PhD :"Automatic Generation of Immersive and Interactive Content for Augmented Technicians"

about the role

Your Role is to a PhD thesis on "Automatic Generation of Immersive, Interactive, and Scripted Content for the Augmented Technician"

Global Context and Problem Statement
Augmented Reality and Virtual Reality have proven effective in training intervention technicians and assisting them in the field. These technologies can, for example, display step-by-step guides to technicians during their interventions, describing assembly or maintenance scenarios. A concrete example is the development of an augmented reality application to guide a technician during the installation of equipment in a data center. Such an application can guide the technician to the location of the equipment to be installed, then to the targeted rack, indicate the position of the new equipment, detail the cabling to be carried out, while displaying safety instructions.

The objective of the PhD thesis is to simplify the creation of this type of 3D content to allow users without development skills to define immersive assistance and training scenarios.

Scientific Objective – Results and Challenges to Address
The objective of the PhD thesis is to design a system for the automatic generation of augmented reality training and assistance content for technologies, relying on generative AI models. In particular, this system will need to leverage information from the real environment where the intervention takes place, business knowledge bases (technical documentation, training content), and the tasks to be performed (work orders). The main scientific challenges to address are:

• Extraction and organization of knowledge from a documentary base in a highly technical business context and linking it to the situation encountered by the technician: real environment and task to be performed.
• Selection and filtering of information to achieve the appropriate level of precision according to the needs and skills of the end user.
• Structuring this knowledge in the form of a pedagogical assistance or training scenario.
• Transition from a scenario to an anchored interactive immersive experience (spatial understanding) in the user's real environment: selection and placement of 3D content, scenario progression based on user interactions.
• Establishment of metrics to estimate the relevance of the created content (validation).

The proposed approaches include the use of Retrieval-Augmented Generation (RAG) for knowledge base utilization, large language models (LLMs) for content generation, and immersive interactions (AR, VR) to customize and validate the produced results. The expected outcomes include validated demonstrators in real environments, scientific publications, and patent filings.