REAL-TIME SYSTEM FOR STREAMING WITH LIVE VOLUMETRIC CAPTURE AND ASSOCIATED METADATA

Smart overview of the Invention

A system is described that uses machine learning and 3D volumetric capture to enhance the viewing experience of live events. By capturing depth and color data from various angles, a real-time 3D mesh of the event is created. This allows users to view the event in virtual or augmented reality formats. The system isolates items of interest using background subtraction and can generate textual descriptions or direct predictions of these items through machine learning models.

Background Challenges

Live event viewers often face challenges in identifying items of interest, such as clothing or accessories, due to limited information available during broadcasts. Fast-paced events and brief item appearances complicate this further, making it difficult for viewers to gather sufficient details about these items. The disclosed system addresses these challenges by providing additional context and detailed information about items displayed during live events.

Technical Approach

The system employs depth cameras to capture detailed images of live events. These cameras, positioned strategically around the scene, collect depth information and color data to construct a 3D mesh representation. Technologies such as Time-of-Flight, Structured Light, and Stereo Vision are utilized to obtain accurate depth measurements, allowing for high-quality 3D modeling of subjects and their surroundings.

Data Processing

The captured data is processed through a mesh generation engine that constructs a 3D mesh by fitting polygons to the surface of the scene volume. Background pixels are removed by comparing them to pre-captured background images. The resulting mesh is then analyzed by a machine learning model, which can infer descriptions of items or predict item identities based on their characteristics within the mesh.

Applications and Benefits

This system allows users to gain insights into items featured in live events, such as fashion shows or concerts, enhancing their engagement with the content. It also provides potential for brand promotion by identifying and describing products worn by celebrities or performers in real time. The integration of item metadata further enriches user experience by offering comprehensive information about the items, including purchase options.