Invention Title:

POINT CORRESPONDENCE-GUIDED MOTION TRANSFER IN VIDEO GENERATION

Publication number:

US20260330733

Publication date:
Section:

Physics

Class:

G06T13/00

Inventors:

Assignee:

Applicant:

Smart overview of the Invention

Point correspondence-guided motion transfer is a technique in video generation that leverages machine-learning models to transfer motion from one object to another in digital videos. It utilizes a first digital video depicting a reference movement of a first object and a prompt indicating a second object. The machine-learning model tracks point trajectories across frames of the first video to guide the movement of the second object, ensuring consistent motion transfer without appearance drift.

Addressing Conventional Model Limitations

Traditional diffusion-based video generators often face challenges like appearance drift, where visual elements change or degrade over time. The described system addresses these issues by employing an optical flow procedure to establish point trajectories that maintain spatial correspondence across video frames. This ensures that generated videos have consistent and believable motion, overcoming limitations of conventional models in precise motion control.

Machine-Learning Model Process

The process begins with receiving a first digital video and a prompt, which can be text or an image. The machine-learning model, using an optical flow procedure, determines point trajectories that match points across frames. A denoiser, such as a U-Net, uses correspondence loss to update the latent representation of the first object. This updated representation guides the movement of the second object during the inference stage, resulting in a second video with transferred motion.

Training and Minimizing Losses

The machine-learning model is trained to minimize both diffusion loss and correspondence loss. By focusing on these losses, the model is able to generate a second digital video that accurately reflects the reference movement of the first object. The generated video is then presented to the user via a display device, showcasing the successful motion transfer from one object to another.

Application and Environment

The described techniques are implemented in a digital medium environment that includes a service provider system and a computing device connected via a network. The service provider system supports digital services, such as content creation and streaming, using scalable hardware and software resources. Users can access these services through a communication module, enabling the generation and viewing of videos with complex motion transfer.