NEAR EYE DISPLAY SYSTEM WITH MACHINE LEARNING (ML) BASED STEREO VIEW SYNTHESIS OVER A WIDE FIELD OF VIEW

Smart overview of the Invention

The patent application discusses a head-mounted display (HMD) system designed for stereoscopic view synthesis using machine learning (ML) techniques. The system includes right and left exterior-facing stereo cameras positioned on the front of the HMD, aligned with the user's eyes to capture images for processing. A processor in the HMD executes instructions stored in memory to perform stereo view synthesis, which includes disocclusion filtering to remove hole regions caused by viewpoint differences.

Technical Field

The application pertains to display systems, specifically those employing ML-based stereoscopic view synthesis across a wide field of view. This technology is relevant in artificial environments such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), where immersive content delivery is crucial. The HMDs aim to enhance user experience by offering realistic visual representations of both real and virtual environments.

Background

As interactive content like VR and AR becomes more popular, wearable display systems like HMDs are increasingly used. These devices often employ multiple projectors to create a stereoscopic vision effect. However, traditional passthrough techniques can suffer from poor image quality and reconstruction artifacts due to inaccurate depth information. The described system addresses these issues by leveraging ML techniques for improved image processing.

Stereo View Synthesis

The HMD system performs stereo view synthesis using ML techniques such as depth estimation, image sharpening, forward splatting, and fusion. Disocclusion filtering is a key process that minimizes regions where images are missing due to camera-eye misalignment. The system employs depth-assisted filtering to fill in these regions, enhancing the overall image quality viewed by the user.

Application and Benefits

This innovative approach provides a more immersive experience by improving passthrough imagery, allowing users to perceive their physical environment alongside virtual elements. By addressing limitations of existing systems, such as low resolution and monochromatic images, this technology offers high-resolution, real-time perspective-correct views that meet the demands of VR applications.