CONTROLLABLE DYNAMIC APPEARANCE FOR NEURAL 3D PORTRAITS

Smart overview of the Invention

A method has been developed to create controllable, dynamic appearances for neural 3D portraits. This involves projecting color onto specific points in a digital video portrait, taking into account factors such as location, surface normal, and viewing direction. The technique allows for the projection of dynamic face normals that change according to head poses and facial expressions captured in the video, enabling a realistic representation of movement and emotion.

Importance in Augmented and Virtual Reality

Controllable 3D portraits are essential for augmented reality (AR) and virtual reality (VR) applications, where an immersive experience is crucial. By digitally recording an individual under controlled lighting conditions, a video is created which serves as the basis for training a model that can manipulate head movements and facial expressions accurately. This technology enhances the realism of virtual interactions by allowing for precise control over the animated portraits.

Challenges in Traditional Methods

Creating photo-realistic moving portraits typically requires strict lighting conditions to ensure consistent color across surfaces. However, real-world capture often introduces complications such as shadows, specularities, and interreflections that vary with different head poses and expressions. These challenges have limited the creation of animated portraits to professional setups, making it difficult to produce high-quality outputs in everyday environments.

Innovative Solutions for Capture

The proposed method addresses these limitations by enabling the capture of training videos in any lighting condition, including those that produce shadows or varying skin reflectance. This flexibility allows users to create animated portraits using mobile devices or less sophisticated equipment without the need for a controlled environment. Consequently, it simplifies the process of generating animated content for various applications such as videos or AR presentations.

Technical Implementation and Features

The process utilizes a video processing application that disentangles appearance data from captured videos to create dynamic lighting normals and specularity. Users can provide inputs to control how the portrait is animated over time, specifying details like head position and facial expressions. The resulting neural 3D portrait can be stored or rendered for display, allowing for high-quality animations that are easily produced with accessible technology.