METHOD FOR PERFORMING ZERO-SHOT THREE-DIMENSIONAL-AWARE IMAGE EDITING VIA MULTIMODAL LARGE LANGUAGE MODEL-GUIDED MULTI-VIEW DIFFUSION, AND ASSOCIATED APPARATUS

Smart overview of the Invention

The patent application details a method for zero-shot three-dimensional-aware image editing using a Multimodal Large Language Model (MLLM)-guided approach. This involves a processing circuit executing a Z3DE framework to perform 3D-aware image editing with a trained model. The method transforms user instructions into refined versions, extracts target objects from input images, and generates novel views and output images through 3D-aware integration and synthesis.

Background

Advancements in text-to-image generation using diffusion models have improved image processing, allowing photorealistic images from textual descriptions. However, challenges remain in extending these capabilities to 3D-aware editing of single RGB images, such as inconsistent geometry and incomplete details. This patent addresses these issues by introducing a new method and architecture that minimizes side effects.

Method and Apparatus

The method employs a processing circuit within an electronic device to run the Z3DE framework. It utilizes MLLM-guided scene analysis to refine user instructions and extract target objects. The framework performs meta-information fusion to create novel views of these objects and integrates them into scenes to produce output images. The apparatus includes the processing circuit that executes these operations, potentially comprising the entire electronic device.

Advantages

The proposed method leverages the semantic and reasoning capabilities of MLLMs to enable zero-shot 3D-aware editing without training. It overcomes traditional 3D editing limitations by interpreting complex prompts, generating missing details, and reasoning about spatial layouts. This approach ensures consistent geometry and spatial coherence in edited images, enhancing the natural integration of objects.

Implementation Details

The Z3DE framework incorporates modules for instruction-guided scene analysis, meta-information fusion, and 3D-aware object integration. It refines user instructions, extracts target objects, and generates novel views, leading to coherent output images. This method applies to processing circuits in electronic devices, providing a versatile and comprehensive solution for 3D-aware image editing.