An implementation of an imitation game with ASD children to learn nursery rhymes

Previous studies have suggested that being imitated by an adult is an effective intervention with children with autism and developmental delay. The purpose of this study is to investigate if an imitation game with a robot can arise interest from children and constitute an effective tool to be used in clinical activities. In this paper, we describe the design of our nursery rhyme imitation game, its implementation based on RGB image pose recognition and the preliminary tests we performed.


Introduction
Several studies have shown that imitating ASD children can be efficient in enhancing their social behaviour [13]. These studies have been carried with caregivers as imitators of children's movements [15,10]. Upon these results, numerous projects have been using TIC including robots, for ASD care [16,8]. For instance, project Gaming Open Library for Intervention in Autism at Home (GOLIAH) developed a series of 11 serious games on computers and tablets using imitation and joint attention [9]. Other projects have focused on facial expression with robot Keepon [11] or with the expressive robot Face [14]. For instance, Kaspar [5] can be used with the capacity to imitate drum rhythm.  While most imitation studies have focused on facial expressions or sounds, few studies have focused on the imitation of movements. However, impairment in motor imitation abilities have been commonly described in children with ASD. How motricity in interpersonal coordination impacts imitation has little been investigated. Indeed, [17] has shown that both interpersonal synchronization and motor coordination subtly contribute the imitation impairment in ASD children. A human partner was shown to have an impact on the learning abilities of postures by a humanoid robot [7], and when it was interacting with participants, it was able to learn a social signature at the level of individual recognition [2]. These results are interesting, but are only limited to static postures and not yet dynamic movement as required in full motor control. Development of motor control requires forming an internal model of action relying on the coupling between action (motor commands) and perception (sensory feedback). Critical to the development of social, communicative, and motor coordination behaviors, internal model of action accurately predicts the sensory consequences of motor commands [12]. This is why in this study, we propose to examine how ASD children can learn to improve their motor control by a serious game. We propose to use a robot platform as they have been shown to arise their interest.
To allow them to learn motor control and gestures, we examined the program of clinical activities and identified that during their activities, nursing rhymes were sung and mim-icked with therapists. We therefore propose to implement a robot for an imitation game based on a nursery rhyme. We describe our implementation and the test results.

General architecture of our system
We implemented a software for the humanoid robot Pepper to play an imitation game with a nursing rhyme. The robot would thus sing a line of the rhyme while making a gesture and displaying an image on its table, then check if the child imitates the gesture, while encouraging him and displaying an image of the nursery rhyme. This workflow is illustrated by fig. 1.
The system is composed of: (i) pose recognition node; (ii) a display node using Pepper's tablet; (iii) a node to control the Pepper robot movements using Naoqi [1] through the ROS module MoveIt [4]; (iv) the audio player node to play the music line by line (v) the vocalisation node for text-tospeech. These nodes can communicate through ROS as shown in fig. 2. The role of the master node is to synchronise all nodes and to manage the behaviour of the robot.

Pose recognition
Our pose recognition system ( fig. 3), is composed of : • the internal RGB camera of Pepper giving to the system the input video flow.
• the OpenPose library [3] to estimate the pose of the human body using deep learning algorithm, and a classifier to recognise a predefined set of poses. For our application, we used the tensorflow implementation on CPU of openpose, instead of the more vanilla implementation for GPU. The openpose library can output two models of skeleton with either 25 or 18 joints called body-25 and coco, as illustrated in fig. 4. In our case, as the nursery rhyme did not focus on the bottom body part, we opted for the coco model. To avoid false positives, we set the confidence threshold c to 0.5. The openpose library returns at each frame the skeleton features ((x 1 , y 1 ), ..., (x 8 , y 8 )) of the positions (x,y) of the 8 upper body joints (numbered 1 to 8) if its confidences on these joints are above c. These skeleton features are normalised as where (x 0 , y 0 ) is the intersection of joints 1-2 and 3-4.
• a classifier: we used Gaussian Mixture Classifier (GMM) with a dataset of 20 samples for each of the 8 class shown in fig. 5. In the work presented here, the GMM is used to classify poses, but GMM can also be used to classify dynamical movements, as well as evaluate an error in the movement so as to give an advice how to improve the movement, as described in [6] in a project of a robot coach for physical rehabilitation.

Evaluation
We report here the performance of the pose recognition system, and describe our test with an autistic child.

Performance of the pose recognition system
We performed an evaluation on the performance of our pose recognition system on the 8 classes of poses in the rhyme. They are shown in fig 5. The image of the database were captured by a webcam with a resolution of 640x480. We used for openpose the model CMU as it gives the best detects skeletons. Each class had 30 images. The train accuracy is 94.79% while the test accuracy is 94.59 %. Test with a subject We tested the idea of a nursery rhyme game with an autistic children before fully implementing our pose recognition system. For this test, we used a wizard of oz. An engineer was remote-controlling the robot. We tested in 2 configurations: the first when the robot is static, and the second when the robot can move around. In both configurations, the robot can sing the rhymes and make the gestures of the rhymes, and wait for the child to perform. Our subject is a 4-year-old male who shows typical characteristics of infant autism, featuring alterations of social communication.
He also shows limited, repetitive and stereotypical interest and behaviours, conform to the diagnostic criteria DSM V and APA. He shows an important developmental delay in language, can only say a few isolated words, mostly as an echo, and avoids eye contact.
We noticed a clear interest of the subject in the robot and his interest to interact at the beginning of both sessions. However, we noted a clear improvement of the attention in the case the robot can move, and regain attention, whereas in the case of a static robot, the child is easily distracted by other details of the room.

Discussion
These tests show interest of subjects in our proposed imitation game. However, we still need to improve our system. First, we need to extend our algorithm to recognise not only poses, but also dynamic movements such as in [6]. Then, we would need to test our pose recognition system on our subjects, and implement not only a control system of the arms and voice for the nursery rhyme but also a control of the position of the robot to be able to attract the attention of the subject. Moreover, our system needs a good interaction system so as to know when to wait for the child to mimic and sing along the rhyme, or to repeat the line of the rhyme to remind the child of the rhyme, or even to continue to the next line to sustain his interest. The implementation of such an autonomous robot would require a difficult understand-ing of the behaviour of the subject to detect not only if he has repeated the gestures, but also whether the absence of response is due to a lack of time to respond, a bad understanding of the gesture, a lack of attention, or a lack of interest. These first tests support a use of a humanoid robot as a therapeutic tool to improve imitation skills in ASD children. Indeed, imitation constitutes a pivotal skill in children to enable the development of other cognitive and communication skills.