Invention Title:

PERSONALIZED VISION AND LANGUAGE GENERATION

Publication number:

US20260330719

Publication date:
Section:

Physics

Class:

G06T11/60

Inventors:

Applicant:

Smart overview of the Invention

The patent application discusses a method and system for personalized multimodal generation, which involves processing and generating content across various data types, such as images and text, using a generative machine learning model. The system is designed to receive input images depicting an object and a prompt for a multimodal task, and it generates custom tokens that encode information about the object. These tokens include a nonce token for object identification, a learnable image token for visual encoding, and a learnable text token for language processing.

Functionality

The system can perform both image and language generation tasks. When given a prompt describing an image generation task, the system uses the learnable image token to create a synthetic image that maintains the object's identity from the input image. For text-based tasks, such as answering questions or describing the object, the system employs the learnable text token to generate accurate textual responses. This ability to switch between tasks ensures consistency and accuracy in multimodal outputs.

Applications

Multimodal generation is particularly useful in applications requiring the integration of text and visual data, such as visual question answering, image captioning, and content generation. The system's token-based approach enhances personalization, allowing it to generate and describe objects with minimal training data while preserving pre-trained knowledge. This method addresses challenges in conventional systems, such as semantic misalignment and inefficient task switching.

Challenges in Conventional Systems

Traditional systems often rely on full-model fine-tuning, which is computationally expensive and may lead to forgetting previously learned information. Soft-prompt tuning is a more efficient alternative, but it struggles to personalize both image and text generation effectively. Soft prompts may optimize one task but fail to transfer improvements to another, resulting in poor-quality outputs and inaccurate descriptions, especially when data is scarce.

Advantages of the Proposed System

The proposed system overcomes these limitations by using a token-based representation that allows for efficient personalization without extensive retraining. It maintains the core architecture while introducing trainable tokens to adapt to specific tasks, ensuring high-quality multimodal outputs. This approach is particularly advantageous in real-world applications where only a few images are available for personalization, offering a practical solution for generating context-aware content across different modalities.