PROMPT-DRIVEN IMAGE EDITING USING MACHINE LEARNING

Smart overview of the Invention

A media application is designed to modify images based on textual prompts while preserving certain aspects of the image, such as the face of a subject. This is achieved through the use of a diffusion model that processes the initial image along with a preserving mask and the text request. The goal is to produce an edited image that aligns with the user's request without altering key facial features.

Process

The application first generates a preserving mask that corresponds to the face of the subject in the initial image. This mask, along with the image and the user's textual request, is fed into a diffusion model. The model performs several operations, including denoising the initial image, applying text conditioning to create a noisy translated image, and finally producing a denoised translated image that meets the request.

Diffusion Model Functionality

The diffusion model involves a sequence of steps to achieve the desired image transformation. Initially, it outputs a denoised version of the original image. Then, it applies forward diffusion to incorporate the textual request, generating a noisy version of the translated image. Using extracted features and self-attention maps, the model outputs a refined, denoised translated image.

Image Blending

The final step involves blending the denoised initial image, the preserving mask, and the denoised translated image to create the output image. The preserving mask ensures that the subject's face remains unchanged, maintaining the integrity of facial features while allowing other requested modifications to be made.

Additional Features

The method supports additional functionalities, such as selecting objects in the image to replace or modify and offering suggestions for background changes. Users can provide textual requests that include predefined options or prompts from a library. This flexibility enhances the utility of the application in various image editing scenarios.