NEURAL HEAD AVATAR CONSTRUCTION FROM AN IMAGE

Smart overview of the Invention

The disclosed system introduces a novel approach for animating a source portrait image by utilizing motion cues such as pose and expression from a target image. Unlike traditional methods, this system constructs an implicit three-dimensional (3D) head avatar from a single-view portrait image. This avatar captures realistic details both within and beyond the face region and is prepared for animation without needing further optimization during inference. The system employs three processing branches to produce tri-planes representing the coarse 3D geometry, detailed appearance, and expression, which are then combined using volumetric rendering to generate an image with the desired identity, expression, and pose.

Background and Challenges

Traditional techniques for animating portrait images often struggle with synthesizing realistic details due to limitations in mesh resolution and texture models, focusing mainly on the facial region and ignoring personal characteristics like hairstyle or accessories. Some recent methods use Generative Adversarial Networks (GANs) to represent motion as a warping field, but these lack a 3D understanding, causing artifacts and distortions when rotating the head beyond limited angles. The need for a system that can generalize to unseen identities and maintain high fidelity without further optimization is evident.

System and Methodology

The system comprises three neural branches that handle different aspects of the portrait image: the canonical branch for 3D geometry, the appearance branch for detailed texture, and the expression branch for target expression. The canonical branch reconstructs a neutral expression and frontal pose in a volumetric format, while the appearance branch maps fine texture details onto this geometry. The expression branch modifies the expression using a 3D morphable model. These branches work together to create a high-fidelity, animated head avatar that generalizes well to new images.

Implementation and Rendering

The process begins by receiving a source image and extracting its 3D shape to produce an identity surface. This surface is combined with an appearance representation to form an identity representation with a neutral expression. By integrating this with an expression representation from a target image, the system uses volumetric rendering to produce a facial image with the source's identity and the target's pose and expression. A super-resolution block may enhance the final output, ensuring high fidelity in the generated images.

System Architecture

The identity representation construction system includes components like a feature encoder, lifting unit, 3D encoder, and integration unit. The 3D encoder processes the source image to predict a coarse 3D reconstruction, ensuring alignment in canonical space and a neutral expression. This setup supports generalization to various camera views and facilitates expression transfer. The described architecture is flexible, allowing for different implementations and combinations of hardware, firmware, and software to achieve the desired outcomes.