DIGITAL VIDEO EDITING BASED ON A TARGET DIGITAL IMAGE

Smart overview of the Invention

Digital video editing techniques leverage target digital images to enhance editing accuracy. Inputs include a target text prompt, a target image of a target object, and a source video with frames depicting a source object. Using machine-learning models, regions-of-interest in the source video frames are identified. A generative model then creates frames of a target video featuring the target object, based on these regions, the target image, the source video, and a descriptive source text prompt.

Background

Traditional video editing methods using machine-learning face challenges in maintaining visual and temporal consistency across frames. They often struggle with object replacements of varying shapes and sizes, leading to inaccuracies and inefficiencies. These methods primarily rely on text prompts, which limit their expressive power and accuracy, especially when the edit's nature cannot be fully expressed in text. Consequently, visual inaccuracies and computational inefficiencies arise.

Innovation

New digital video editing techniques incorporate target digital images to guide edits, improving accuracy and expressiveness beyond text-based methods. This approach supports replacing objects of different sizes and shapes, maintaining temporal consistency, and enhancing visual accuracy. By using visual guides, these techniques overcome the limitations of traditional methods, enabling more precise and efficient video editing even when objects differ significantly.

Methodology

The video generation system processes inputs like a target text prompt, target digital image, and source video. For example, replacing a white bus with a red SUV in a video is achieved by identifying regions-of-interest using diffusion models. These models create masks that guide the editing process. The system employs generative AI to transform the source video, using latent noise and denoising operations to ensure the target object is consistently integrated across frames.

Implementation

To maintain temporal consistency, the system uses a latent correction strategy. This involves calculating an inter-frame latent field to map features between consecutive frames. The generative model, utilizing diffusion techniques, processes regions-of-interest, target images, source videos, and text prompts to produce a cohesive target video. This method addresses conventional inaccuracies, ensuring stylistic and spatial coherence across edited frames.