Invention Title:

Cross-Modal Contrastive Learning for Text-to-Image Generation based on Machine Learning Models

Publication number:

US20260330706

Publication date:
Section:

Physics

Class:

G06T11/00

Inventors:

Applicant:

Smart overview of the Invention

A novel method for text-to-image generation leverages a neural network trained using cross-modal contrastive learning. The approach involves receiving textual descriptions of scenes and generating corresponding image renditions. The neural network is designed to ensure that images generated from the same textual description are similar, while those from different descriptions are distinct. This is achieved by analyzing mutual information between pairs of images and text-image pairs, thereby optimizing the output image rendition of the scene.

Background

Text-to-image synthesis has been enhanced by deep generative models like GANs, which offer high-quality outputs. Existing methods often rely on multiple stages or complex models to generate high-resolution images. These models require detailed object labels and involve multi-step processes, making them less practical for real-world applications. Contrastive learning, which focuses on contrasting positive and negative pairs, presents an alternative approach by maintaining consistency in image representations.

Methodology

The proposed method receives training data consisting of textual descriptions and associated image renditions. The neural network is trained to align image renditions with their corresponding textual descriptions through mutual information analysis. This involves calculating contrastive losses for image-to-image and text-to-image pairs, which guide the network in distinguishing between similar and different descriptions. The trained model is then capable of producing high-fidelity images based on new textual inputs.

Applications

This method can be implemented in various forms, including a computing device with processors and data storage, a computer program with executable instructions, or an article of manufacture with computer-readable media. Each implementation focuses on training the neural network to optimize text-to-image generation, ensuring that the output images accurately reflect the input descriptions. This flexibility allows for integration into different systems and devices.

System Design

The system is designed to efficiently handle text-to-image generation by using contrastive learning techniques. It includes mechanisms for receiving training data, training the neural network, and outputting the final model. By focusing on mutual information and contrastive losses, the system ensures that the generated images maintain a high level of semantic fidelity to their corresponding textual descriptions, making it suitable for complex scenes and diverse datasets.