Invention Title:

DISTRIBUTIONAL PREFERENCE ALIGNMENT OF MACHINE LEARNING MODELS VIA OPTIMAL TRANSPORT

Publication number:

US20260330291

Publication date:
Section:

Physics

Class:

G06N20/00

Inventors:

Assignee:

Applicant:

Smart overview of the Invention

The patent application discusses a method for aligning machine learning models with human preferences using a technique called optimal transport. This approach responds to a prompt by generating a current response and comparing it with a reference response and preference data. The method involves determining positive and negative reward distributions based on these responses and issuing optimization instructions to align the model with the preference data.

Background

The invention addresses the growing need for control over AI systems, particularly large language models (LLMs), which are capable of generating human-like text. These models are trained on extensive datasets and used in various applications, such as chatbots. Aligning these models with human values and goals is crucial to ensure they are helpful, safe, and reliable, reducing the risk of generating harmful or biased content.

Technical Summary

The method involves executing a machine learning model to output a response to a given prompt. A reward component determines positive and negative reward distributions by comparing the current response with a reference response and preference data. An optimizing instruction is issued to align the model according to an optimal transport solution, ensuring that the model's outputs are consistent with human preferences.

Applications and Benefits

This approach is particularly relevant for industries such as healthcare, law, and finance, where the consequences of misleading AI outputs can be severe. By aligning models with human preferences, the method aims to improve the safety and reliability of AI systems. The use of optimal transport allows for efficient processing and resource management, enhancing the model's performance without requiring extensive computation.

Implementation Examples

  • Example 1: A computer-implemented method that aligns a machine learning model with preference data using optimal transport, ensuring positive responses dominate negative ones.
  • Example 2: Enhancements to Example 1, emphasizing the dominance of positive reward distributions over negative ones.
  • Example 3: Further optimizations using convex relaxation techniques, improving computational efficiency and resource utilization.