Invention Title:

STREAMING AUDIO AND VIDEO FOR REAL-TIME VOICE INTERACTIONS GENERATIVE INFERENCE

Publication number:

US20260311694

Publication date:
Section:

Physics

Class:

G10L15/22

Inventors:

Assignee:

Applicant:

Smart overview of the Invention

The patent application describes innovative systems and methods for enhancing real-time interactions using a generative response engine. The technology aims to create more natural and dynamic user experiences by optimizing audio and video streaming. Key features include the reception and transmission of audio and video frames at different rates, which are processed by a generative response engine to produce timely responses. This approach addresses the challenges of latency and real-time processing in speech-based interactions.

Background

Generative response engines, particularly large language models, have transformed artificial intelligence by improving natural language understanding and generation. These engines excel in tasks like text generation and translation but face challenges in real-time speech interactions. Pauses and interruptions in human speech make it difficult to determine complete speech instances, which can disrupt the conversational flow. The application focuses on reducing latency to enhance user experiences in applications requiring quick feedback, such as customer support and learning tools.

Challenges and Solutions

Traditional methods like sliding window-based algorithms introduce significant latency, affecting the natural flow of conversation. These methods require a constant stream of audio to detect speech ends, leading to delays and unnatural interactions. The application proposes dynamic speech instance detection, which identifies complete thoughts based on silence and speech patterns. This approach allows the generative response engine to respond within a natural delay period, significantly improving user experience.

Multi-Modal Integration

While generative response engines typically use text, integrating video frames poses challenges due to high token consumption. The proposed system selectively provides video frames to the generative response engine, enhancing context without overwhelming the processing capabilities. This selective provisioning ensures that the engine receives relevant visual data, improving overall interaction quality without excessive resource use.

Interruption Handling

The application introduces an interruption detection engine to manage conversational interruptions effectively. This engine identifies interruptions and adjusts the generative response engine's state, allowing it to discard unpresented content when necessary. By doing so, the system maintains a coherent and fluid conversation, aligning responses with the user's evolving intent. This feature is crucial for creating a seamless interaction experience, accommodating natural conversation dynamics.