This paper describes an avenue for artificial and computational intelligence techniques applied within games research to be deployed for purposes of physical therapy. We provide an overview of prototypical research focussed on the application of motion sensor input devices and virtual reality equipment for rehabilitation of motor impairment: an issue typical of patients of traumatic brain injuries. We highlight how advances in procedural content generation and player modelling can stimulate development in this area by improving quality of rehabilitation programmes and measuring patient performance.
In many respects the ideas of rehabilitation-driven games share similarities with that of dynamic difficulty adjustment. However, as noted in the likes of @cite_7 , the challenges faced are more nuanced given the physical challenges faced by patients. A notable example can be found in the Intelligence Game Engine for Rehabilitation (IGER) detailed in @cite_11 @cite_15 which is focussed on the recording and management of physiotherapy for stroke patients. This system adopts an adaptive fuzzy-driven approach to catering for the patients rehabilitation process. This is achieved while working with a variety of small mini-games that utilise peripherals such as the Microsoft Kinect and Nintendo Wii Fit board. This system places emphasis on ensuring that the parameters of the current game are such that it should not result in activities that player might find painful.
@cite_15 @cite_7 @cite_11
2785342388 2079022890 2151292517 2019528074 2580770992 2059812677 2533085129 2005110420 2339176273 2147596804 2160770229 2140685838 2145098376 2113096211 2553098005 2104058278 2056358962 2319894388 1985745673 2810102121
Abstract This paper proposes the use of a non-immersive virtual reality rehabilitation system "ReHabgame" developed using Microsoft Kinect™ and the Thalmic™ Labs Myo gesture control armband. The ReHabgame was developed based on two third-person video games that provide a feasible possibility of assessing postural control and functional reach tests. It accurately quantifies specific postural control mechanisms including timed standing balance, functional reach tests using real-time anatomical landmark orientation, joint velocity, and acceleration while end trajectories were calculated using an inverse kinematics algorithm. The game was designed to help patients with neurological impairment to be subjected to physiotherapy activity and practice postures of daily activities. The subjective experience of the ReHabgame was studied through the development of an Engagement Questionnaire (EQ) for qualitative, quantitative and Rasch model. The Monte-Carlo Tree Search (MCTS) and Random object (ROG) generator algorithms were used to adapt the physical and gameplay intensity in the ReHabgame based on the Motor Assessment Scale (MAS) and Hierarchical Scoring System (HSS). Rasch analysis was conducted to assess the psychometric characteristics of the ReHabgame and to identify if these are any misfitting items in the game. Rasch rating scale model (RSM) was used to assess the engagement of players in the ReHabgame and evaluate the effectiveness and attractiveness of the game. The results showed that the scales assessing the rehabilitation process met Rasch expectations of reliability, and unidimensionality. Infit and outfit mean squares values are in the range of (0.68–1.52) for all considered 16 items. The Root Mean Square Residual (RMSR) and the person separation reliability were acceptable. The item person map showed that the persons and items were clustered symmetrically. Computer games are a promising tool to support rehabilitation at home. It is widely recognized that rehabilitation games should (i) be nicely integrated in general-purpose rehabilitation stations, (ii) adhere to the constraints posed by the clinical protocols, (iii) involve movements that are functional to reach the rehabilitation goal, and (iv) adapt to the patients' current status and progress. However, the vast majority of existing rehabilitation games are stand-alone applications (not integrated in a patient station), that rarely adapt to the patients' condition. In this paper, we present the first prototype of the patient rehabilitation station we developed that integrates video games for rehabilitation with methods of computational intelligence both for on-line monitoring the movements' execution during the games and for adapting the gameplay to the patients' status. The station employs a fuzzy system to monitor the exercises execution, on-line, according to the clinical constraints defined by the therapist at configuration time, and to provide direct feedback to the patients. At the same time, it applies real-time adaptation (using the Quest Bayesian adaptive approach) to modify the gameplay according both (i) to the patient current performance and progress and (ii) to the exercise plan specified by the therapist. Finally, we present one of the games available in our patient stations (designed in tight cooperation with therapists) that integrates monitoring functionalities with in-game self-adaptation to provide the best support possible to patients during their routine. Rehabilitation for chronic pain follows a multidisciplinary approach, which despite the effort, often lacks the long term success and patients often fail to translate the skills learned in therapy to every day life. Serious games are hypothesized to support patients to self manage their complaints and keep training their physical functions by themselves, especially, when the game is controlled by the patient's own body performance. In this paper we present the implementation of a system providing multimodal input, including our own full body motion capture system, a low cost motion capture system (Microsoft Kinect) and biosignal acquisition devices to a game engine. In addition, a workflow has been established, that enables the use of the acquired multimodal data for serious games in a medical environment. Finally, a serious game has been implemented, targeting rehabilitation of patients with chronic pain of the lower back and neck. The focus of this work is on the multimodal input and how it is used in a game to support rehabilitation of chronic pain patients. A brief comparison of a marker-based full body MoCap system and Microsoft's Kinect is included. Preliminary results of tests currently underway are provided. Abstract Objective: The aim of this article is to describe a game engine that has all the characteristics needed to support rehabilitation at home. The low-cost tracking devices recently introduced in the entertainment market allow measuring reliably at home, in real time, players' motion with a hands-free approach. Such systems have also become a source of inspiration for researchers working in rehabilitation. Computer games appear suited to guide rehabilitation because of their ability to engage the users. However, commercial videogames and game engines lack the peculiar functionalities required in rehabilitation: Games should be adapted to each patient's functional status, and monitoring the patient's motion is mandatory to avoid maladaptation. Feedback on performance and progression of the exercises should be provided. Lastly, several tracking devices should be considered, according to the patient's pathology and rehabilitation aims. Subjects and Methods: We have analyzed the needs of the clinicians a... Objective: The objective of the proposed research is to develop a methodology for modeling and evaluation of human motions, which will potentially benefit patients undertaking a physical rehabilitation therapy (e.g., following a stroke or due to other medical conditions). The ultimate aim is to allow patients to perform home-based rehabilitation exercises using a sensory system for capturing the motions, where an algorithm will retrieve the trajectories of a patient’s exercises, will perform data analysis by comparing the performed motions to a reference model of prescribed motions, and will send the analysis results to the patient’s physician with recommendations for improvement. Methods: The modeling approach employs an artificial neural network, consisting of layers of recurrent neuron units and layers of neuron units for estimating a mixture density function over the spatio-temporal dependencies within the human motion sequences. Input data are sequences of motions related to a prescribed exercise by a physiotherapist to a patient, and recorded with a motion capture system. An autoencoder subnet is employed for reducing the dimensionality of captured sequences of human motions, complemented with a mixture density subnet for probabilistic modeling of the motion data using a mixture of Gaussian distributions. Results: The proposed neural network architecture produced a model for sets of human motions represented with a mixture of Gaussian density functions. The mean log-likelihood of observed sequences was employed as a performance metric in evaluating the consistency of a subject’s performance relative to the reference dataset of motions. A publically available dataset of human motions captured with Microsoft Kinect was used for validation of the proposed method. Conclusion: The article presents a novel approach for modeling and evaluation of human motions with a potential application in home-based physical therapy and rehabilitation. The described approach employs the recent progress in the field of machine learning and neural networks in developing a parametric model of human motions, by exploiting the representational power of these algorithms to encode nonlinear input-output dependencies over long temporal horizons. Monitoring and interpreting (sub)cortical reorganization after stroke may be useful for selecting therapies and improving rehabilitation outcome. To develop computational models that predict behavioral motor improvement from changing brain activation pattern, we are currently working on the implementation of a clinically feasible experimental set-up, which enables recording high quality electroencephalography (EEG) signals during inpatient rehabilitation of upper and lower limbs. The major drawback of current experimental paradigms is the cue-guided repetitive design and the lack of functional movements. In this paper, we assess the usability of the Kinect device (Microsoft Inc., Redmond, WA, USA) for tracking self-paced hand opening and closing movements. Three able-bodied volunteers performed self-paced right hand open-close movement sequences while EEG was recorded from sensorimotor areas and electromyography (EMG) from the right arm from extensor carpi radialis and flexor carpi radialis muscles. The results of the study suggest that the Kinect device allows generation of trigger information that is comparable to the information that can be obtained from EMG. This paper introduces a human computer interaction system by using hand gesture recognition to apply into game-based rehabilitation application. The basic concept of this project is to assist patient to carry out rehabilitation with hand gesture recognition, meanwhile therapist is able to follow recovery progress of patient through results. In this manner, a rehabilitation system created aimed to increase efficacy in patient treatment with interactive game. The system is applied in real time application to contribute as a function of a virtual mouse. Approach proposed including hand region segmentation, hand palm partition and finger counting. Computer mouse is configured to move according to the location of hand palm region, and perform function of ordinary mouse according to number of fingers detected. In the analysis, experiments is carried out to show how this system collaborate with the simple rehabilitation game in computer. Experiment evaluation as shown has successfully demonstrate the effectiveness and robustness in performing the mouse function and feasible to be implemented into rehabilitation games. Effective stroke rehabilitation must be early, intensive and repetitive, which can lead to problems with patient motivation and engagement. The design of video games, often associated with good user engagement, may offer insights into how more effective systems for stroke rehabilitation can be developed. In this paper we identify game design principles for upper limb stroke rehabilitation and present several games developed using these principles. The games use low-cost video-capture technology which may make them suitable for deployment at home. Results from evaluating the games with both healthy subjects and people with stroke in their home are encouraging. Physical rehabilitation supported by the computer-assisted-interface is gaining popularity among health-care fraternity. In this paper, we have proposed a computer-vision-assisted contactless methodology to facilitate palm and finger rehabilitation. Leap motion controller has been interfaced with a computing device to record parameters describing 3-D movements of the palm of a user undergoing rehabilitation. We have proposed an interface using Unity3D development platform. Our interface is capable of analyzing intermediate steps of rehabilitation without the help of an expert, and it can provide online feedback to the user. Isolated gestures are classified using linear discriminant analysis (DA) and support vector machines (SVM). Finally, a set of discrete hidden Markov models (HMM) have been used to classify gesture sequence performed during rehabilitation. Experimental validation using a large number of samples collected from healthy volunteers reveals that DA and SVM perform similarly while applied on isolated gesture recognition. We have compared the results of HMM-based sequence classification with CRF-based techniques. Our results confirm that both HMM and CRF perform quite similarly when tested on gesture sequences. The proposed system can be used for home-based palm or finger rehabilitation in the absence of experts. In most modern video games, character be- havior is scripted; no matter how many times the player exploits a weakness, that weakness is never repaired. Yet if game characters could learn through interacting with the player, behavior could improve during game- play, keeping it interesting. This paper introduces the real-time NeuroEvolution of Augmenting Topologies (rt- NEAT) method for evolving increasingly complex arti- ficial neural networks in real time, as a game is being played. The rtNEAT method allows agents to change and improve during the game. In fact, rtNEAT makes possible a new genre of video games in which the player teaches a team of agents through a series of customized training exercises. In order to demonstrate this concept in the NeuroEvolving Robotic Operatives (NERO) game, the player trains a team of robots for combat. This paper describes results from this novel application of machine learning, and also demonstrates how multiple agents can evolve and adapt in video games like NERO in real time using rtNEAT. In the future, rtNEAT may allow new kinds of educational and training applications that adapt online as the user gains new skills. In recent years, the field of Human-Computer Interaction (HCI) has been advanced with many technologies, however, most are limited to healthy users. In this paper, we leveraged the technology of free-hand interaction to rehabilitate patients with stroke. We modified the game of Fruit Ninja to use Leap Motion controller's hand tracking data for stroke patients with arm and hand weakness to practice their finger individuation. In a pilot study, we recruited 14 patients with chronic stroke to play the game using natural interaction. Their Fruit Ninja (FN) scores show high correlation with the standard clinical assessment scores such as Fugl-Meyer (FMA) and Box-and-Blocks Test (BBT) scores. This finding suggests that our free-hand Fruit Ninja's score is a good indicator of the patient's hand function and therefore will be informative if used in their rehabilitation. In most modern video games, character behavior is scripted; no matter how many times the player exploits a weakness, that weakness is never repaired. Yet, if game characters could learn through interacting with the player, behavior could improve as the game is played, keeping it interesting. This paper introduces the real-time Neuroevolution of Augmenting Topologies (rtNEAT) method for evolving increasingly complex artificial neural networks in real time, as a game is being played. The rtNEAT method allows agents to change and improve during the game. In fact, rtNEAT makes possible an entirely new genre of video games in which the player trains a team of agents through a series of customized exercises. To demonstrate this concept, the Neuroevolving Robotic Operatives (NERO) game was built based on rtNEAT. In NERO, the player trains a team of virtual robots for combat against other players' teams. This paper describes results from this novel application of machine learning, and demonstrates that rtNEAT makes possible video games like NERO where agents evolve and adapt in real time. In the future, rtNEAT may allow new kinds of educational and training applications through interactive and adapting games. The application of a robot to rehabilitation has become a matter of great concern. This paper deals with functional recovery therapy, one important aspect of physical rehabilitation. Single-joint therapy machines have already been achieved. However, for more efficient therapy, multjoint robots are necessary to achieve more realistic motion patterns. This kind of robot must have a high level of safety for humans. A pneumatic actuator may be available for such a robot, because of the compliance of compressed air. A pneumatic rubber artificial muscle manipulator has been applied to construct a therapy robot with two degrees of freedom (DOF). Also, an impedance control strategy is employed to realize various motion modes for the physical therapy modes. Further, for efficient rehabilitation, it is desirable to comprehend the physical condition of the patient. Thus, the mechanical impedance of the human arm is used as an objective evaluation of recovery, and an estimation method is proposed. Experiments show the suitability of the proposed rehabilitation robot system. This paper presents a real-time vision-based system to assist a person with dementia wash their hands. The system uses only video inputs, and assistance is given as either verbal or visual prompts, or through the enlistment of a human caregiver's help. The system combines a Bayesian sequential estimation framework for tracking hands and towel, with a decision-theoretic framework for computing policies of action. The decision making system is a partially observable Markov decision process, or POMDP. Decision policies dictating system actions are computed in the POMDP using a point-based approximate solution technique. The tracking and decision making systems are coupled using a heuristic method for temporally segmenting the input video stream based on the continuity of the belief state. A key element of the system is the ability to estimate and adapt to user psychological states, such as awareness and responsiveness. We evaluate the system in three ways. First, we evaluate the hand-tracking system by comparing its outputs to manual annotations and to a simple hand-detection method. Second, we test the POMDP solution methods in simulation, and show that our policies have higher expected return than five other heuristic methods. Third, we report results from a ten-week trial with seven persons moderate-to-severe dementia in a long-term care facility in Toronto, Canada. The subjects washed their hands once a day, with assistance given by our automated system, or by a human caregiver, in alternating two-week periods. We give two detailed case study analyses of the system working during trials, and then show agreement between the system and independent human raters of the same trials. The correct execution of well-defined movements plays a crucial role in physical rehabilitation and sports. While there is an extensive number of well-established approaches for human action recognition, the task of assessing the quality of actions and providing feedback for correcting inaccurate movements has remained an open issue in the literature. We present a learning-based method for efficiently providing feedback on a set of training movements captured by a depth sensor. We propose a novel recursive neural network that uses growing self-organization for the efficient learning of body motion sequences. The quality of actions is then computed in terms of how much a performed movement matches the correct continuation of a learned sequence. The proposed system provides visual assistance to the person performing an exercise by displaying real-time feedback, thus enabling the user to correct inaccurate postures and motion intensity. We evaluate our approach with a data set containing 3 powerlifting exercises performed by 17 athletes. Experimental results show that our novel architecture outperforms our previous approach for the correct prediction of routines and the detection of mistakes both in a single- and multiple-subject scenario. This paper proposes a solution to the cross-a-door problem in unknown environments for a robotic wheelchair commanded through a Human-Machine Interface (HMI). The problem is solved by a dynamic path planning algorithm implementation based on successive frontier points determination. An adaptive trajectory tracking control based on the dynamic model of the robotic wheelchair is implemented on the vehicle to direct the wheelchair motion along the path in a smooth movement. An EKF feature-based SLAM is also implemented on the vehicle which gives an estimate of the wheelchair pose inside the environment. The SLAM allows the map reconstruction of the environment for safe navigation purposes. The whole system steers satisfactorily the wheelchair with smooth movements through common doorways which are narrow considering the size of the vehicle. Implementation results validating the proposal are also shown in this work. In this paper, we focus on the problem of motion tracking in unknown environments using visual and inertial sensors. We term this estimation task visual-inertial odometry (VIO), in analogy to the well-known visual-odometry problem. We present a detailed study of extended Kalman filter (EKF)-based VIO algorithms, by comparing both their theoretical properties and empirical performance. We show that an EKF formulation where the state vector comprises a sliding window of poses (the multi-state-constraint Kalman filter (MSCKF)) attains better accuracy, consistency, and computational efficiency than the simultaneous localization and mapping (SLAM) formulation of the EKF, in which the state vector contains the current pose and the features seen by the camera. Moreover, we prove that both types of EKF approaches are inconsistent, due to the way in which Jacobians are computed. Specifically, we show that the observability properties of the EKF's linearized system models do not match those of the underlying system, which causes the filters to underestimate the uncertainty in the state estimates. Based on our analysis, we propose a novel, real-time EKF-based VIO algorithm, which achieves consistent estimation by (i) ensuring the correct observability properties of its linearized system model, and (ii) performing online estimation of the camera-to-inertial measurement unit (IMU) calibration parameters. This algorithm, which we term MSCKF 2.0, is shown to achieve accuracy and consistency higher than even an iterative, sliding-window fixed-lag smoother, in both Monte Carlo simulations and real-world testing. Mobility improvement for patients is one of the primary concerns of physiotherapy rehabilitation. Providing the physiotherapist and the patient with a quantified and objective measure of progress can be beneficial for monitoring the patient's performance. In this paper, two approaches are introduced for quantifying patient performance. Both approaches formulate a distance between patient data and the healthy population as the measure of performance. Distance measures are defined to capture the performance of one repetition of an exercise or multiple repetitions of the same exercise. To capture patient progress across multiple exercises, a quality measure and overall score are defined based on the distance measures and are used to quantify the overall performance for each session. The effectiveness of these measures in detecting patient progress is evaluated on rehabilitation data recorded from patients recovering from knee or hip replacement surgery. The results show that the proposed measures are able to capture the trend of patient improvement over the course of rehabilitation. The trend of improvement is not monotonic and differs between patients. Most formal rehabilitation facilities are situated in a hospital or care center setting, which may not always be conveniently accessible for patients, especially those in geographically isolated areas. Home-based rehabilitation has potential to offer greater accessibility and thus increase consistent uptake. In addition, the exercise performed in conventional rehabilitation contexts may be insufficient to ensure the patient's speedy recovery, with complimentary rehabilitation exercises at home required to make a difference. The goal is to provide effective home-based rehabilitation offering outcomes similar to those obtained through hospital-based rehabilitation under the supervision of an occupational therapist. This paper presents the development of a Kinect-based system for ensuring home-based rehabilitation using a Dynamic Time Warping (DTW) algorithm and fuzzy logic. The ultimate goal is to assist patients in conducting safe and effective home-based rehabilitation without the immediate supervision of a physician. We consider multi-player repeated games involving a large number of players with large strategy spaces and enmeshed utility structures. In these ldquolarge-scalerdquo games, players are inherently faced with limitations in both their observational and computational capabilities. Accordingly, players in large-scale games need to make their decisions using algorithms that accommodate limitations in information gathering and processing. This disqualifies some of the well known decision making models such as ldquoFictitious Playrdquo (FP), in which each player must monitor the individual actions of every other player and must optimize over a high dimensional probability space. We will show that Joint Strategy Fictitious Play (JSFP), a close variant of FP, alleviates both the informational and computational burden of FP. Furthermore, we introduce JSFP with inertia, i.e., a probabilistic reluctance to change strategies, and establish the convergence to a pure Nash equilibrium in all generalized ordinal potential games in both cases of averaged or exponentially discounted historical data. We illustrate JSFP with inertia on the specific class of congestion games, a subset of generalized ordinal potential games. In particular, we illustrate the main results on a distributed traffic routing problem and derive tolling procedures that can lead to optimized total traffic congestion.