PROCare4Life Sensorial Ecosystem

PROCare4Life offers a personalized, integrated care platform for people with neurodegenerative diseases such as Parkinson’s and Alzheimer’s diseases. Using sensory monitoring, PROCare4Life learns of users’ behavioral habits and detects disease-specific symptoms, e.g., freezing, festination, loss of balance, wandering, confusion, and falls. An essential component of PROCare4Life is the sensorial ecosystem, which provides the necessary tools and modules to track disease progression and facilitate communication between patients with neurodegenerative diseases and the corresponding social and health professionals. The sensorial ecosystem employs state-of-the-art technologies and computational methods to harvest and process sensorial data to identify the symptoms and associated daily activities. Sensory tracking acquires data through sensors from wristbands, mobile phones, binary object sensors, and cameras. In particular, through these sensors, various raw measurements are gathered, such as heart rate, number of sleeping hours, number of steps, accelerometers, gyroscopes, and human-body trajectories. As part of the sensorial ecosystem of PROCare4Life, Sensory Data Tracking (SDT) was developed to enable tracking and gathering of sensorial data from different users. A high-level illustration of the sensorial ecosystem is shown in Figure 1.

Figure 1 An illustration of the sensorial ecosystem of PROCare4Life

In addition, the sensorial ecosystem employs advanced machine learning and AI methodologies to process the sensorial data and identify the disease symptoms. Since neurodegenerative symptoms occur differently, it is important to use specific sensors to identify those symptoms to enhance their identification accuracy. For example, freezing of gait, a symptom associated with Parkinson’s Disease, can be recognized from wristbands’ and cell phones’ sensors. While wandering, a symptom of Alzheimer’s Disease, can be identified from body movement trajectories which are detected using cameras, as shown in Figure 2. Additional daily activities, such as movement within the house, visiting the bathroom, and using the kitchen, can be identified using binary object sensors. Moreover, the sensorial ecosystem contains an experimental tool based on interactive and serious cognitive games. The cognitive games are used to measure parameters related to cognitive skills such as short-term memory, visual recognition, semantic memory, vocabulary, etc. Subsequently, a set of features such as time, the number of errors, the games’ difficulty level, and the accomplished cognitive tasks are employed in predictive machine learning models for cognitive assessments.

Figure 2 Video of Body skeleton detection and its trajectory tracking through cameras

Following sensorial ecosystem measurements and patients’ health-related information, Human Behavior Recognition (HBR), a module within the sensorial ecosystem, conducts a longer-term analysis to define scores and metrics that indicate the following:

The quality of physical activities

The quality of sleep

The severity and frequencies of motor symptoms

The decline or the progress of cognitive and emotional states

The obtained metrics are correlated with a score that defines the patients’ profile, based on their socio-demographic and other health-related information such as comorbidities and stage of the diseases. Afterward, a decision support system detects deviations from the patients’ normal routines. In the decision support system, a multimodal fusion mechanism benefits from domain knowledge provided by clinical partners of ProCare4Life through user interfaces. The multimodal fusion mechanism takes advantage of the clinical partners’ domain knowledge to determine the impact and contribution of each score to either the improvement or the deterioration of the patients’ health conditions. HBR outcomes are utilized in a recommendation engine to provide eHealth recommendations such as: recommending leisure or physical activities, nutrition tips, and reminders to therapy adherence to improve patients’ quality of life. In this manner, the disease progress and the patients’ daily conduct are monitored and communicated with the social and health professionals to provide a better quality of life for patients with neurodegenerative diseases.

This blog was written by Esam Ghaleb, Yusuf Can Semerci, and Stylianos Asteriadis. They are members of the Affective & Visual Computing Lab (AVCL) in the Department of Data Science Knowledge Engineering (DKE) at Maastricht University (UM), the Netherlands. UM leads the Work Package (WP) for the sensorial ecosystem of PROCare4Life. UM also contributes to the user’s behavioral analysis and the realization of the decision support system.