Invention Title:

ACTION ENGINE AND ACTION INDEX FOR TASK PERFORMANCE

Publication number:

US20260339489

Publication date:
Section:

Physics

Class:

G06F3/0484

Inventors:

Applicant:

Smart overview of the Invention

The patent application introduces an action index and an action engine designed to enhance task performance through artificial intelligence (AI) agents. The action index provides detailed information about actionable elements within a graphical user interface (GUI), including the associated actions, preconditions, and postconditions. These elements help determine the likelihood and success of an action performed by an AI agent. The agent utilizes a generative artificial intelligence (GAI) model to devise a plan for task execution, optimizing performance and reducing computational demands.

Background

AI agents are increasingly integrated into applications to perform tasks based on prompts from users, either locally on devices or via remote datacenters. Traditionally, these agents are limited to specific tasks related to the services provided by the entity that developed them. Common applications include purchasing items online or optimizing workflows in enterprise settings. Despite their utility, current AI agents often face performance challenges when interacting with human-centric interfaces like websites and mobile applications.

Technical Challenges and Solutions

Existing methods for AI agents to interact with human interfaces involve complex and resource-intensive processes, such as capturing and analyzing screenshots of web pages. This approach leads to high costs and slow performance. The proposed action index offers a streamlined alternative by providing a predefined set of actions and conditions, reducing reliance on GAI models during execution. This results in faster processing speeds, lower memory and hardware requirements, and decreased network bandwidth usage.

Action Index and API Integration

The action index also accommodates API commands, enhancing flexibility beyond GUI interactions. It includes API definitions, inputs, outputs, and conditions for successful execution. By integrating API calls, the action index can guide AI agents in performing tasks more efficiently, using detailed semantic descriptions and invocation instructions. This dual capability allows agents to operate effectively across various interfaces and systems.

System Architecture

The system architecture includes an AI agent equipped with an action engine that uses plugins and data storage to perform user-requested actions. The agent interfaces with browsers or applications through an automation interface, potentially using frameworks like Selenium. Action crawlers generate the action index by analyzing target services, enabling precise task execution. The system supports both local and remote deployment models, offering flexibility in how the AI agent is accessed and utilized by users.