FUSED MULTIMODAL FRAMEWORK FOR NON-PLAYER CHARACTER GENERATION AND CONFIGURATION

Smart overview of the Invention

The patent application describes a system for generating and animating non-player characters (NPCs) in virtual digital environments using a fused multimodal framework. This system processes multimodal input data, which includes various types of interactions, through neural networks to create animation sequences for NPCs' body and facial features. The framework utilizes a large-language model (LLM) for speech generation and employs reverse diffusion to generate face vertex displacement and joint trajectory data, resulting in realistic NPC animations.

Background

NPCs are essential elements in video games and virtual environments, serving as guides, adversaries, or quest-givers. Traditionally, NPCs follow pre-scripted actions with limited interaction capabilities. The conventional methods also separate facial and body animations, which can result in disjointed character representations. This patent addresses these limitations by creating a unified system that enhances NPC interaction and animation quality.

Technical Approach

The proposed system overcomes the inefficiencies of traditional NPC generation by using generative AI architectures that reduce computational costs and memory usage. It integrates multiple input modes, such as text and audio, to drive interactive conversations and context-aware animations. The fusion of face and body animations is achieved through a unified architecture that resolves the disentangling issues of conventional designs.

Innovation

Key innovations include the use of diffusion-based models for animating 3D faces and bodies, allowing for cross-environment applications. Reverse diffusion refines noisy data into high-fidelity outputs, enabling realistic animations. Additionally, the system incorporates dynamic emotion and motion guidance using textual descriptions to enhance expressiveness in NPC animations.

Applications

An NPC Software Development Kit (SDK) is provided for easy integration into game engines and virtual environments. This SDK leverages a unified AI architecture to adapt NPC models across different platforms. The techniques described have applications in game character control, interactive assistants, video teleconferencing, metaverse environments, and entertainment industries.