Actor-Replacement System for Videos

Smart overview of the Invention

The patent application describes a system for replacing actors in video content efficiently. It utilizes advanced technology to estimate the pose of an original actor in each frame of a video using a skeletal detection model. This model tracks and detects skeletal landmarks over the sequence of frames. The system then obtains images of a replacement actor that correspond to these estimated poses, allowing for seamless integration into the video.

To ensure the replacement actor's speech aligns with the original, the system generates replacement speech in the new actor's voice. This speech matches the original actor's dialogue, maintaining narrative coherence. The synthetic frames generated depict the replacement actor with facial expressions that align temporally with this replacement speech, ensuring realistic lip synchronization.

The process involves generating synthetic frames by using both the estimated poses and images of the replacement actor. These frames are crafted to align with the original video’s timeline and structure, ensuring that the replacement actor appears naturally in place of the original. The final step combines these synthetic frames with the replacement speech to produce a new video where the original actor is replaced.

Additionally, the method can be used to change an actor’s dialogue from one language to another. By generating synthesized speech in a target language using a speech engine, and aligning facial expressions accordingly, the system can produce a version of the video that caters to different linguistic audiences without reshooting scenes.

The architecture includes components like a skeletal detection model, motion capture system, and a speech engine, all working together within a video-generation system. This setup allows for efficient editing and generation of synthetic videos, providing flexibility in post-production processes without extensive reshooting or manual editing.