Multimodal Interaction for Persons with Autism: The 5A Case Study

Persons with Autism Spectrum Disorder (ASD) experience, in general, cognitive rigidity and behavioral and communication problems. These issues dramatically affect their everyday life abilities, such as interacting with others or being autonomous in most daily tasks. 5A project aims at supporting ASD individuals by exploiting the combination of Virtual Reality (VR), Augmented Reality (AR), and Conversational Agents (CA). Using low-cost headsets and smartphones, 5A offers a unique intervention for ASD people. The VR training allows the user to learn how to perform daily tasks in a safe and controlled environment. The AR experience enables the generalization of the acquired skills in real-world settings. Finally, the customizable CA assists and prompts the user at every step in both experiences. As a first use case for 5A applications, we report its use in urban mobility with trains. A case study evaluation was conducted, measuring the effect of 5A apps on 5 ASD users. Our preliminary results showed that users had improved their autonomy level (perceived and measured by external caregivers). The first lessons learned by this study are also reported as a set of guidelines about technology usage, design, and experimentation for immersive technologies in support of persons with ASD.