PAN-Assistant

Problem

Post-traumatic brain injury, individuals often struggle with adapting to both familiar and unfamiliar environments, including their own residences. The intensive reorientation training necessary to mitigate these issues demands a significant amount of personnel resources. Currently, there is no provision for in-house training.

Fürst Donnersmarck-Stiftung P.A.N. Forschung

Solution

The PAN-Assistant is designed to facilitate independent training for patients within their own environments (route training in the therapy area, orientation training at home). Utilizing software, patients can virtually rehearse their individual routes and visit landmarks.

Gradually, the affected individuals are expected to learn to navigate, for instance, within the therapy area.

Details

Cognitive impairments are a common aftermath of brain damage (e.g., stroke or traumatic brain injury). Some individuals also experience disorientation, which hinders their ability to navigate both familiar and unfamiliar environments, including their own homes. The capacity to acquire new knowledge is often compromised as well.

Currently, disorientation is managed in everyday clinical practice through repeated practice of action steps in individual therapeutic care. This method is labor-intensive and time-consuming, resulting in inadequate treatment coverage, particularly for severely affected patients. The “PAN-Assistant” project is developing software for self-training that enables safe and individually tailored virtual path and orientation training. Technologies from the realms of virtual reality (VR) and augmented reality (AR) are being utilized.

Given the often severe and typically complex physical and/or mental limitations in individuals with severe brain injuries, the development of digital therapy concepts presents both a technical and clinical challenge. The project team comprises industrial partners as well as academic and clinical institutions. Special emphasis is placed on the continuous involvement of the affected individuals to develop a digital training tool that is accepted by people with disabilities and is therapeutically beneficial.

In the future, the “PAN Assistant” aims to provide a technical solution for individually tailored orientation training to enhance the independence and participation of the affected individuals. This introduces a novel therapeutic approach that complements and expands current practice. The fundamental principles of this technology will also be applicable in other areas of neurorehabilitation in the future.

Results

AR-Showcase

In our augmented reality showcase, we aim to raise our patients’ environmental awareness through active engagement with and co-creation of their surroundings. The resident initially chooses the starting area and is then routed to the location with the help of an on-screen map. Once arrived, a specific location is chosen in AR and marked with a personal sculpture object. Throughout successive training sessions, the shape and appearance of the sculpture is designed by the patients choices and the characteristics of its location. Once the rehab residence ends, the patient can take home their creation as a physical 3D print. Additionally our designers have also developed a case for people with hemiparesis that makes it easier to use the iPad with just one hand.

Wearable / Amulet

Lastly, we developed a small, neck-worn wearable called the Amulet that interacts with stationary display devices positioned at various landmarks throughout the rehab center. This device enables patients to practice preset routes at their own pace. The Amulet acts solely as a spatial checklist to navigate sequentially along the chosen route, but when a patient approaches a nearby display device, the display becomes active and indicates the correct direction with a clear visual cue. For example, if a patient passes the entrance and is approaching the garden landmark, the wearable will vibrate to draw the patient’s attention, and a moving arrow will appear on the garden display, indicating the direction to the next waypoint. Progress along the route is updated on the trainer’s user interface.

VR-Showcase

In the first showcase, we use immersive virtual reality with a head mounted display in which residents can virtually navigate through 3D scans of the rehab center. This includes their living area, the outdoor area and the therapy building, but also a route outside the rehab center to the nearest bus stop which includes sequential panorama pictures. They train relevant routes by completing tasks such as finding an object and placing it in a specific location. Depending on the level of difficulty, they can follow arrows or our virtual avatar PANi and consult a map of the surroundings.

Latest update: 06/20/2024

Trainer UI

A trainer interface includes manuals of each system and enables tracking the resident’s progression and adjustment of training difficulty. Each resident has their own profile where relevant info, like their individual needs, is documented and where they and the therapy team can review completed training sessions and plan upcoming ones.