Invention Title:

MACHINE LEARNING SYSTEM WITH MODALITY SELECTION MODEL

Publication number:

US20260320413

Publication date:
Section:

Physics

Class:

G06N3/10

Inventors:

Assignee:

Applicant:

Smart overview of the Invention

The patent application describes a computing system that utilizes machine learning (ML) to manage input and output modalities efficiently. The system includes input devices, output devices, and processing devices that handle user inputs and interaction context metadata. This metadata includes the input modality and potential output modalities. The system uses a multimodal ML model to select the appropriate output modalities based on the user input and context metadata, ensuring a seamless interaction by computing and outputting the model outputs accordingly.

Background

Recent advancements in ML have led to the development of models capable of handling multiple input and output modalities. Multimodal ML systems can process inputs like text or audio and produce outputs in various forms such as text, images, or audio. These systems often consist of multiple ML models that either handle unimodal or multimodal inputs and outputs. They include scaffolding code to determine the conditions under which different models are executed, which also handles input preprocessing and output postprocessing.

System Process

The system is designed to receive user inputs through a user interface and initiate model calls to different ML models based on the input type. During this process, it can handle additional user inputs and perform subsequent model calls to generate outputs. These outputs are then displayed to the user interface. This approach allows for dynamic interaction without the need for users to specify the input or output modality explicitly, enhancing the user experience.

Challenges Addressed

Existing multimodal ML systems often require users to manually select output modalities, which can disrupt the user experience. They also struggle to adapt to changes in user interaction context, such as switching from audio to text inputs. Additionally, these systems typically follow a static input-output flow, which can introduce delays, especially when handling tasks with long runtimes. The proposed system aims to address these issues by programmatically adapting to changes and optimizing interactions.

System Configuration

The computing system includes processing devices like CPUs, GPUs, or NPUs, and memory devices for executing a user interface that facilitates interaction with the multimodal ML system. Input devices such as touchscreens, keyboards, or microphones, and output devices like displays or speakers, are used to exchange conversational turns between the user and the system. The multimodal ML system comprises multiple ML models specialized in different domains and a central router that manages input and output transmissions. The system can be configured in a server-client setup, with models executed on servers and the user interface and central router on clients.