Deep Learning Based Causal Image Reprojection for Temporal Supersampling in AR/VR Systems

Smart overview of the Invention

The patent application discusses a method for generating synthesized data in augmented reality (AR) and virtual reality (VR) systems. It involves capturing frames of a scene at a specific frame rate using cameras on a wearable device. The system then determines body position parameters and obtains geometry data for the scene. These inputs are processed by a trained network to predict additional frames, enhancing temporal supersampling in AR/VR environments.

Virtual Data Rendering

For virtual data, the method includes predicting future gaze positions based on current body parameters. A frame's gaze region is rendered at a high resolution according to this future gaze position, while the peripheral region is predicted at a lower resolution. The combination of these regions forms a complete frame, which is then used to drive a display, optimizing both resolution and computational efficiency.

Technical Details

The technique employs deep learning to perform causal temporal supersampling, capturing image frames at an initial frame rate. The system uses various data inputs, such as pose, gaze, and geometry, to synthesize future frames. These inputs are fed into a trained network that generates frames at different resolutions, potentially higher than the original capture, addressing the challenge of producing high-resolution video frames on power-constrained devices.

Power Efficiency

The method offers power savings by reducing the energy required to render high-resolution images. Traditional rendering methods consume power proportional to scene complexity, whereas the neural network-based upsampling technique maintains a stable energy requirement. This approach is particularly advantageous in scenarios with predetermined scene complexity or device power restrictions.

Implementation and Applications

The patent describes various electronic systems that can leverage this technology, including head-mountable systems, projection-based systems, and other display technologies. These systems can adjust content dynamically in response to user movements, providing an immersive experience in XR environments. The described method enhances the efficiency and quality of content rendering in these systems, offering a scalable solution for next-generation AR/VR applications.