A Fabric-Based Integrated Sensor Glove System Recognizing Hand Gesture


 The research on wearable glove sensor system has been increasing over recent years because of the need for portability and comfort. This study presents a fabric-based integrated sensor glove system with five sensing zones. Five sensors are knitted by silver-plated nylon yarn and embedded into glove directly using intarsia technology. Various parameters including sensor elasticity, sizes of embedded sensor as well as glove are discussed, respectively. Further, selected or chosen integrated sensor glove is manufactured and tested for recognizing gestures. Results show that elasticity affects effective sensing range of knitted sensors, size has significant influence on sensors’ sensitivity, and appropriate glove size helps avoiding nonlinear sensing phenomenon. Finally, the glove system, by extracting feature data, can distinguish Chinese number gestures very well and has also the potential to recognize more hand gestures in the future.


Introduction
Sensing of human hand signal plays an important role in rehabilitation medicine, biomechanics, tele-operation, and human-computer interaction, like robot arm or virtual reality. Consider an example specifi cally-mature sensors that track hand gesture may provide opportunity for a person to communicate with deaf-mutes as well as understand them.
Until now, numerous hand motion sensor systems have been put forward and investigated. Early researches are focused on vision-based methods [1,2], such as camera and refl ective markers on glove, and infrared-based methods. Nevertheless, these methods are limited by hand gestures signifi cantly. For instance, it is hard to recognize one fi nger when it is shielded by another [3]. Moreover, multiple devices fi lming or photographing hand motions are not suitable in some cases.
Moreover, most studies used the method of integrating fl exible or hard sensors with off-the-shelf gloves. Hard sensors used are common transducers that mainly consist of acceleration sensor, angle transducer, inertial measurement unit (IMUs), magnetic sensors [4], etc. However, rigid components need to be small enough so that they are easily wearable, which may lead to diffi culty during integrating [5]. Flexible sensors consist of familiar components that include soft bending sensors [6,7], fi ber optical sensors (FOS) [8][9][10], and various self-assembly stretch sensors. When the factors such as simplicity, portability, and comfort are considered, fl exible glove sensor system is becoming a key system for realizing hand signal recognition.
Except for commercial component, different materials are selected to generate wearable sensor such as nanomaterial [11][12][13] (carbon nanotube, silver nanowires, graphene, etc.), PEDOT:PSS [14], nylon-based silver yarns or other conductive yarns [15,16], etc. However, conductive coating layer may face the problem of generating irreversible cracks during stretch and cohesive materials may have impact on original elasticity and extensibility of fabric. Compared with coating or pasting, fl exible knitted sensors can be embedded into glove directly through the whole-garment knitting process. Regarding wholegarment technology, some underwear or pants [15,17] have had a shot and demonstrated that embedded knitted sensor of these clothing can cling to skin perfectly and monitor human body postures with good sensitivity and stability.
In a previous research [18], an integrated glove with one knitted sensor embedded into it was manufactured for index fi nger and results exhibited that the sensor was able to refl ect fi nger bending condition. In this paper, impact of fabric parameters on sensor properties were investigated further and an integrated glove with fi ve applicable strain sensors was fabricated using a fl at knitted machine. The sensor glove was connected with selfmade fi ve-channel read-out electronics to construct a tracking system. Eventually, results show the system can distinguish numerical hand gestures from 1 to 9 well.

Figuration and mechanism
It is known that lengthwise loops of weft plain knitted fabrics are called wales and crosswise loops are called courses. For fabric-based sensors knitted by conductive yarns, stretching along the lengthwise and crosswise direction generates change in effective contact area between adjacent loops, and as a result, equivalent resistance of the sensor varies. Generally, the extensibility of knitted fabric along the lengthwise direction is larger than that of crosswise direction. For knitted sensors located on chest band or T-shirt, their electrical-mechanical performance is studied based on the crosswise direction, along which fabrics deform. In this study, considering that fi nger bending leads to skin stretch along the lengthwise direction, properties of knitted sensor was tested all along the fabric lengthwise direction.
In the previous study [19], conductive weft plain-knitted fabrics made of silver-plated nylon yarn and elastic nylon/spandex yarns were stretched along the lengthwise direction within certain strain range and found that the sensor resistance increased with good linearity and sensitivity. Hence, this structure is adopted to design new sensors of the glove system here. Figure 1a shows the construction of knitted sensor, where conductive area (i.e., sensor area) was knitted by nylon/ spandex yarns and silver-plated nylon yarn, other area was made of nylon/spandex yarns and elastic polyester yarns, which is non-conductive. Owing to addition of spandex yarns, both the two areas have elasticity.
By designing knitting procedure on computer, an integrated glove with fi ve effective sensors can be obtained from a SHIMA Seiki SWG 061N-15G computer fl at knitting machine directly. For thumb, the embedded sensor cover one interphalangeal joint (IP), but for other four fi ngers, each sensor covers the proximal interphalangeal (PIP), and the distal interphalangeal joint (DIP), as Figure 1b-c shows. Due to glove knitting sequence and design limitation, intarsia technology cannot be used to knit sensors covering all IP, PIP, and MCP joint consequently. Therefore, IP/DIP and PIP are selected and studied here. Being off from knitted machine, two ends of silver yarns of every sensor are connected with read-out electronics so as to obtain resistance value. In addition, knitted elastic glove is washed and ironed to insure stabilization before wearing

Selection of elastic materials
Nylon/spandex yarns that include 20/75 (nylon 8.3 tex, spandex 2.2 tex), 40/75 (nylon 8.3 tex, spandex 4.4 tex), and 70/30 (nylon 3.3 tex, spandex 7.8 tex) are chosen as plating yarn in whole fabric and in addition, silver-plated nylon yarn (44 dtex/12 fx8, resistivity of each filament is 15 Ω/cm) and elastic polyester yarn (33.3 tex) are used as ground yarn of sensor areas and other areas, respectively. Three kinds of knitted sensors are fabricated and elasticity is measured, and Table 1 shows those values. Fixed elongation test method is selected to obtain sensor elasticity, in which the fixed elongation is set as 20%, clamping distance is 100 mm, stretching speed is 100 mm/min, the pre-tensioning force is 1N, the stagnant time is 60 s, the recovery time is 180 s, and number of stretch cycle is 5 times. Three sensors are stretched along the lengthwise direction in one cycle and repeated cycles, and their resistance-strain values are recorded, and Figure 2 depicts these values.
It can be seen from Table 1 and Figure 2 that elasticity of knitted sensor is affected by content of spandex in used yarns, and elasticity obviously influences the electrical-mechanical performance of knitted sensors. Experiments results show that all sensors exhibit increased resistance with good linearity and sensitivity during the first elongation phase, but the corresponding strain range also called as effective working range will be larger when the sensor has better elasticity. Beyond that, it is known that the sensor with better elasticity shows better repeatability during repeated stretch process. In brief, effective working range of sensor S3 can meet the requirement of finger dorsum skin strain (up to 35%), and it has the best repeatability against fabric plastic deformation. Hence, sensor S3 is selected to be used in this glove system.

Size of sensors
To produce an integrated glove with five effective sensors, size parameter of every sensor needs to be considered. Size parameters of embedded sensor include number of rows and wales. In the previous investigation [19], experiment results show that knitted sensor with less wales exhibits better sensitivity but have only smaller effective sensing range (where resistance exhibits growth linearly). On the contrary, knitted and testing, washing and ironing parameters are controlled equally for different glove samples as described later. When experimenter bends fingers, elastic sensors clinging to skin tightly will be stretched and exhibit resistance variation.

Materials and size
In the investigation [19], the strain range where resistance exhibits linear response of elastic knitted sensor is limited (0~10%), however, stretch of finger skin in lengthwise direction can be more than 35% when people do some simple gestures in daily life. As is reported, elastic deformation process of knitted sensors has effect on their electrical-mechanical performance [20]. As the elastic deformation process depends on fabric elasticity and for the sake of obtaining applicable embedded sensor with targeted linear working range in this study, three nylon/spandex yarns of various content of spandex are used to fabricate flexible sensors with different elasticity. Resistance response was recorded by the VICTORY 4105A ground resistance tester (GRT) and YG (B) 026ET fabric mechanical analysis machine (FMA, supplied by Wenzhou Darong) was used to stretch knitted sensor.
As the glove and sensors needs integration, sizes have become important parameters during integration, which includes size of glove (Gs) and size of embedded sensors (ESs). Glove sensor system being too loose or snug for a person may affect sensing process. Here, three sizes of glove are knitted and performance of one single sensor on index finger is compared to determine appropriate Gs. It's worth mentioning that size of finger being monitored and fabric shrinking percentage of post processing should be considered to obtain appropriate Gs. Secondly, ESs also play an important role in affecting sensor performance. In this part, electrical-mechanical performances of different sizes of glove and sensors were recorded by GRT and FMA.

Monitoring finger movement
Preparative gloves are worn by experimenter and connected with a self-made electronic read-out system where five resistance channels are set to obtain resistance signals of one hand, as shown in Figure 1c. Each channel records resistance information of one embedded sensor. In the read-out software used, five resistance response curves were displayed simultaneously. Except for pursuing better linearity and sensitivity, ESs also need to meet the requirements of covering DIP and PIP joints and to save the used materials as much as possible. It is concluded that knitted sensor containing less wales but more rows should be selected based on the fact that it will be able to cover targeted fi nger joints. Considering length of each fi nger is different, various sizes of sensor are determined for 5 wales but with 45, 60, 60, 45, 35 rows, respectively, from thumb to little fi nger.

Size of glove
In addition, three sizes of expected glove are fabricated and the index fi nger sensors' performances are investigated simply.
sensors with more wales show larger sensing range but lower sensitivity. Here, four sensors containing 5 wales are made but under four different rows 10, 20, 30, and 40, respectively and the yarn material is same as used in sensor S3. Photos and longitudinal electrical-mechanical curves of these sensors are shown in Figure 3.
It can be observed from Figure 3 obviously that all four sensors show good linearity over their effective sensing range. Nevertheless, knitted sensor containing 40 rows exhibits better sensitivity, while the sensor having 5 wales and 10 rows exhibits lower sensitivity. But the effective sensing range of four sensors are similar, which is about 0~50%.  http://www.autexrj.com/ exhibits linear increase and decrease when the fi nger bends and returns once in a sequence, and so the resistance signal only generates one summit in one motion cycle. However, both of K 2 and K 3 exhibit two summits in one cycle, while K3 is more remarkable. Two summits appearing in one cycle means resistance response of two phases (bending and recovering) is non-linear, which may be due to the small size of these gloves. After wearing tight gloves, knitted sensor has been pre-stretched before monitoring fi ngers motion, which leads to nonlinear resistance (changing) phenomenon later. In a word, K 1 is selected as a size reference for fabricating following glove system.

Monitoring fi nger movement
Herein, combining chosen elastic materials, size parameters of embedded sensors and glove, a whole fabricating process is determined and an integrated sensing glove is obtained, as shown in Figure 5. Five pairs of electric wires drawn from embedded sensors are connected to the central processing unit part (CPU); in turn CPU gets and sends resistance signals to related computer software. Experimenter performs numbered gestures from one to nine, each gesture is done for about 7 s, and resistance responses are exhibited as shown in Figure 6. It can be seen that each embedded sensor refl ects corresponding fi nger motion successfully. Due to difference of joint angle in various gestures, resistance signal presents diverse characteristics. For instance, all fi ve sensors showed higher resistance value when the hand performed the number 9 First of all, default Gs in design system is chosen and a glove with random size is obtained. After washing and ironing, a shaped glove is generated and its Sp of post processing can be fi gured up. Hand sizes (Hs) of appointed experimenter are also measured, and expected size of sensor gloves (EGs) can be calculated by the formula (1) and (2): where K is the value refl ecting glove's size (larger value demonstrates looser glove), DGs is designed size of sensor gloves, including length and circumference of fi ngers and palms. For a glove to be elastic and to cling to fi nger tightly, K must be a negative number. Here, K is selected to be 0, −0.1, −0.2, and three corresponding sensor gloves are obtained and named K 1 , K 2 and K 3 respectively, which is shown by Figure 4a. Embedded sensors' specifi cation of three gloves is completely identical.
For the appointed experimenter, it's obvious the K 1 is loosest and K3 is tightest among those three sensor gloves. Sensor embedded on the snug gloves gets broadened after being worn, as Figure 4b shows. After wearing the three gloves, the experimenter followed fi nger motion trace, as Figure 4b shows, to observe and compare detected resistance signal of the index fi nger sensor. It can been seen from Figure 4d that K1 than number 5. In summary, the sensor glove with fi ve effective embedded sensors can monitor and recognize simple gestures well now.
However, just obtaining fi ve channels resistance signal is not enough to recognize hand gestures further. To distinguish various number gestures, a mass of raw digits of each gesture are recorded to extract feature data.
In this part, number gesture '5' is selected as a reference position, where fi ve fi ngers are in "straighten" position. Experimenter changes hand from gesture '5' to other number gestures and repeats for 150 times for respective gestures. For example, Figure 7 demonstrates four cycles of fi ve-channels resistance changing from gesture '5' to gesture '6', where the arrow points to resistance of sensor on ring fi nger. When the experimenter performs gesture '5', original resistance (OR) is obtained. When the experimenter performs gesture '6', ambition resistance (AR) is got. Here, the ratio of OR to RR is chosen as a key characteristic value to distinguish different gesture, which is called feature C later. Figure 8 depicts a boxplot of ratios, that is, the value C extracted from 150 cycles for each gesture, where T, I, M, R, L represents the sensors on thumb, index fi nger, middle fi nger, ring fi nger and little fi nger, respectively. The red short line in the middle of the boxplot is the median of the data, which demonstrates average level; the upper and lower line refl ects quartiles of the data, and hence, the box contains 50% of the data. The width of the box refl ect degree of data fl uctuations; the upper and lower short line corresponds to the maximum and minimum value of the data, while red dots represent the abnormal value.
According to Figure 8, the feature C of some gestures shows larger box width, which means there is larger data fl uctuation.
For example, C of the sensor on ring fi nger of gesture '6' varies from 0.3 to 1, which is also affected by diffi culty in repeating accurate and identical gestures completely. In general, from Figure 8, contrast between feature data of different numerical gestures is obvious from Figure 8. So, we can know that the manufactured sensor gloves do have the potential to distinguish various hand gestures with advantages of portability and fl exibility. Further, hand gesture recognition will be investigated using machine learning in future studies and accurate recognition rate will be obtained.

Conclusion
In summary, this work investigates an integrated sensor glove system with fi ve embedded sensors, which is knitted by silverplated nylon yarns and nylon/spandex yarn directly using whole-garment technology. To obtain an applicable sensor   glove system, effect of elastic materials, size of embed sensors, and glove on sensing performance are discussed respectively.
First, elasticity of sensors of various content of spandex is investigated. It is demonstrated that conductive knitted sensor with better elasticity exhibits larger sensing range, better repeatability but lower sensitivity. As a result, 70/30 nylon/ spandex yarn is chosen as elastic plating yarn in whole glove. Second, sensors with various courses but same wales are designed and it is discovered that knitted sensor containing less wales but with more courses has better sensitivity. Considering embedded sensor needs to cover DIP and PIP fi nger joints, sensor with more courses and less wales is designed in later gloves. In addition, according to size of experimenter hand and fabric shrinkage during post-processing, three sizes of gloves are manufactured and results show that glove with largest size can play the role of best sensor. Finally, taking above factors into consideration, an integrated sensor glove is obtained and experiment results demonstrate that the glove system can distinguish nine number gestures well through generating various feature data. For future work, we have planned to build recognizing model by machine learning and obtain accurate recognition rate.