SEARCHING AUGMENTED REALITY EXPERIENCES USING VISUAL EMBEDDINGS

Smart overview of the Invention

Overview: The disclosed system utilizes a machine learning model to enhance the search and identification of augmented reality (AR) experiences through visual embeddings. By receiving an image from a client device, the system generates an embedding query vector that maps images and text into a common embedding space. This vector is then used to search a database for AR experiences that align with the generated embeddings, allowing for more accurate and efficient retrieval of relevant AR content.

Technical Field: The innovation pertains to augmented reality systems, which blend virtual objects with real-world environments. Unlike virtual reality, which fully immerses users in a digital world, AR overlays digital content onto the physical world, enhancing user interaction with both virtual and real elements. This technology finds applications in gaming, messaging, and other interactive platforms.

Challenges Addressed: Traditional AR systems often rely on predefined categories to recommend AR elements based on image features. This method can be inefficient and lacks specificity, as it requires manual categorization of AR content. The limited categories can lead to broad matches that do not fully satisfy user needs. The disclosed system overcomes these limitations by using a common visual-semantic embedding space, enabling more precise matches without manual categorization.

Innovative Approach: The system leverages a trained machine learning model to map images directly to AR experiences using visual embeddings. This approach eliminates the need for predefined categories, allowing for dynamic mapping of any image to any AR experience based on user-generated search logs. This not only improves user experience but also reduces resource consumption by automating the labeling process through historical query data.

Networked Environment: The system operates within a networked computing environment that includes client devices such as smartphones, tablets, and AR glasses. These devices communicate with a messaging server system via a network to exchange data and invoke functions. The server system supports various services, including data processing and media augmentation, ensuring seamless integration of virtual content into real-world views.