Systems and Methods for Efficient Rendering Pipeline for Makeup Including VTO UI Option Configuration/Selection and Looks

Smart overview of the Invention

The patent describes a system and method for efficiently applying virtual effects, such as makeup, to video frames in real-time applications like video chats, conferences, or teleconsultations. It focuses on enhancing the user interface (UI) with features like live and preview modes, predefined look recommendations, and e-commerce integration. The system aims to perform effect rendering and object landmark determination in parallel, ensuring smooth video output with applied effects.

Field of Application

This technology is relevant to image processing applications, including virtual try-on (VTO), augmented reality (AR), and virtual reality (VR) environments. It is particularly useful for video communication platforms where users can apply visual effects like makeup during live interactions. The invention addresses challenges such as face movement, occlusions, and maintaining high frame rates with low latency.

Challenges Addressed

The system tackles common issues in virtual makeup application during video interactions. These include handling dynamic face movements, varying face sizes, and partial occlusions. The goal is to maintain realistic effects without distracting other participants, ensuring that makeup does not appear on occluded parts of the face and that the video remains smooth and responsive.

User Interface Features

The UI offers a preview mode for selecting effects before streaming and a live mode for applying effects during active sessions. Effects are grouped into looks, which users can select through a look selection interface. Each effect is linked to a product, allowing users to refine selections or access e-commerce options directly from the interface. This integration supports personalized recommendations based on user data like existing makeup or skin tone.

Implementation and Benefits

The implementation involves a computing device equipped with a processor and storage for executing the integrated VTO pipeline in video applications. Users can invoke preview mode before or during calls to adjust settings without interrupting the stream. This system enhances user experience by providing seamless effect application and facilitating direct product purchases through an intuitive interface.