Virtual Training Environment for Gas Operatives: System Usability and Sense of Presence Evaluation

Training of gas operators in real-life settings often has associated risks to health and property. The use of a virtual environment to train gas operators has the potential to offer risk-free training. This study tests the usability of a virtual environment specifically designed to teach new gas operatives in near real-life scenarios. Thirty-two participants tested the virtual environment and performed different tasks required to complete gas safety checks. We used SUS (System Usability Scale) and sense of presence questionnaires to collect data from these participants. The SUS analysis indicated that most participants belonging to a different gender, age, and virtual reality experience groups were comfortable in the VR training environment. The sense of presence data analysis also confirmed similar results as all sense of presence factors scored high regardless of the demographics characteristics of the participants. However, there is still a need to add different scenarios to make the virtual environment into a comprehensive training course.


INTRODUCTION
The recent statistics show that carbon monoxide (CO) poisoning has resulted in thousands of incidents. These incidents have caused more than 697 deaths inside the UK in the last two decades (Safety, 2018). The biggest reason for CO emissions is improper installation of gas ovens, boilers, and cookers. Lack of maintenance of such appliances also contributes to such incidents.
Usually, in gas leak incidents, the gas operatives are called in to fix such situations. Therefore, proper training of gas operatives is imperative to avoid any causalities. However, training new gas operatives in real-world settings has its associated risks. Developing a full-scale gas leak training scenario will not only come at a high cost but will have associated risks to life and property as well. Considering these factors, the researchers have previously advised incorporating virtual reality (VR) as a potential solution to reduce these risks to life and property (Asghar, Egaji, Dando, Griffiths, & Jenkins, 2019). Additional research has also shown that incorporating VR in traditional training can provide a more interactive environment for the trainees and positively impact their skill set and knowledge retention (Fredricks, Blumenfeld, & Paris, 2004).
In literature, we can find many examples of virtual environments to help train people in risk-free environments. Examples include VR training for firefighters in fire evacuation (Cha, Han, Lee, & Choi, 2012). A simulator for tunnel fire evacuation (Ronchi et al., 2015). VR-based emergency rescue training system for railway cranes operators (Xu et al., 2018). VR application for training children to cross railway crossings safely (Dando, Asghar, Egaji, Griffiths, & Gilchrist, 2018). Another VR application to train miners in mine incidents (Kizil & Joy, 2001). These examples motivated us to develop a virtual environment for safety training of the gas operatives explained our previous work (Asghar et al., 2019).
The current paper study aims at analysing the usability of a virtual training environment for the gas operatives. Two data collection instruments called system usability scale (SUS) and sense of presence questionnaires are used to test the usability and level of immersion of the gas operatives while trained through the virtual training environment. In total, 32 participated in the virtual environment testing across multiple sessions. Most of the participants supported the idea of the virtual environment and appreciated the skills and knowledge this system offered to them. 140 The rest of the paper is divided into five sections. A summary of the virtual training environment is presented in the second section. The research process used for this study is summarised in the third section. Research results and discussions are shown in the fourth section. Finally, conclusions from this study and future work recommendations are offered in the fifth section.

VIRTUAL TRAINING ENVIRONMENT
The virtual training environment is based on two scenarios. The first scenario presents a typical residential home in virtual settings. The gas operatives are expected to explore any gas or CO emitting appliances in the virtual home and select these as potential hazardous appliance for gas leaks. The gas operatives can use teleporting devices to enter the house. Teleporting helps them open or close the doors, select appliances, take their readings, and mark any appliance deemed dangerous. The gas operatives can select multiple appliances within one scenario. At the end of this scenario, the gas operatives can see their results in the form of a summary. The gas operatives can repeat this scenario many times. The first scenario is presented in figure 1.

Figure 1. First Scenario with Potential Actions
The second scenario requires the gas operatives to follow a storyboard of a gas leak situation, which can occur in real-home settings. In this scenario, the gas operative will have a time window of 40 seconds to follow the systematic approach from hazards' identification, inspection, and solution. The second scenario showing potential actions that a gas operative can take in a particular situation is presented in figure 2.
The gas operative will read a particular appliance, mark that appliance as hazardous or disconnect that appliance. If the gas operative cannot find the gas leak in 40 seconds or commits any mistakes in this process, there will be an explosion in the virtual training environment. This will add to the realness of dangers involved in such situations in real-life settings. Both scenarios are expected to aid gas operatives and be an excellent addition to their existing training courses.

Figure 2. Second Scenario with Potential Solutions
The current virtual training environment is kept limited to two scenarios only, as previous research has shown that excessive VR exposure can result in dizziness and sickness for some users (Viirre & Ellisman, 2003).

RESEARCH PROCESS
This study followed a three-tier research strategy to accomplish the main research aim as presented in figure 3. Tier one focused on the design of the virtual training environment; tier two concentrate on testing and data collection from the gas operatives, and tier three concentrate on the analysis of the collected data. The ethical approval for the study was obtained from the Faculty of Computing, Engineering and Science, University of South Wales.

RESULTS AND DISCUSSION
In total, 32 participants were trained in the virtual training environment. The training process involved each participant going through both scenarios in the virtual world and filling out both questionnaires at the end of their experience. Table 1 summarises the demographic information for the participants. We have followed the standard SUS data analysis process for data analysis and result interpretation (Harrati, Bouchrika, Tari, & Ladjailia, 2016). For the sense of presence questionnaire data, we used the Shapiro-Wilk and the Mann-Whitney U tests.

SUS Data Analysis
The average historical score for SUS is 68. A SUS score greater than 68 means that the system usability is good, and a lower than average score means there is some problem with the usability of that system (Bangor, Kortum, & Miller, 2008). For the current study, 29 participants have SUS scores of above 68, which means most participants were satisfied with the usability of the virtual training environment. Only three participants with scores of less than 68 were not fully satisfied with the usability of the virtual training environment.
Another dimension in SUS analysis is known as grading levels of SUS scores. In such research, a score equal to or greater than 80.3 means grade A, 68 and above is grade C, and equal to or less than 51 means grade F (Sauro, 2011). The average SUS score of all 32 participants who used the VR training environment is 85.31. This overall survey score corresponds to grade A, which means that the participants did enjoy using the virtual training environment, and they will recommend it to others. Table 2 shows SUS scores based on the demographics of the participants. Interestingly, all groups scored grades A, which confirms the quality of the virtual training environment.

Sense of Presence Data Analysis
For the sense of presence data collection and analysis, we have considered the work of Witmer & Singer, as they are believed to be the pioneers in this domain. Their work revolved around the theories of involvements and immersion. Based on their theoretical and empirical research work Witmer & Singer determined multiple factors, including control, sensory, distraction, and realism, contribute to the sense of presence and immersion (Witmer & Singer, 1998).

Factors Impact on Sense of Presence
The sense of presence was measured on a 7-point Likert scale. The basic descriptive statistics show mean values for control (6.16), sensory (5.97), distraction (4.06), and realism (5.44). The higher values of control, sensory, and realism indicate that sense of presence in the virtual training environment was high amongst the participants. The low value for distraction suggests a minor degree of interference by objects and actions in the virtual training environment, and participants could concentrate on the tasks. In summary, all four factors in this study contribute to a high sense of presence in the virtual environment.

Demographics Impact Sense of Presence
For the second part of the analysis, we wanted to test the impact of demographic characteristics on the presence factors. The Shapiro-Wilk test showed that all demographic characteristics have significant values (p< 0.001), which means the data set in this study was not normally distributed. Therefore, we have to use some non-parametric tests for testing differences among different data groups. As all demographic characteristics, including gender, age group, and previous VR experience, have only two possible values, we used the Mann-Whitney U test for further analysis (Nachar, 2008).
The results of the Mann-Whitney U test are summarised in table 3. These results show no significant values of (p< 0.05) for any demographic characteristics of the participants. This indicates that none of these characteristics has a significant impact on the system's usability under test. Therefore, we can conclude that all participants, regardless of their gender, age, and VR experience, enjoyed the virtual training environment as the factor mean scores indicated in the last subsection.
From the data and results for both questionnaires, we can conclude that system usability and sense of presence are highly appreciated by the gas operatives who got the chance to have hands-on experience in the virtual training environment. Furthermore, the usability results are slightly better than the previous study (Asghar et al., 2019). This indicates that involving end-users in the testing process and modifying the system based on their feedback results in improved system usability and sense of presence. The study limitation includes that this system is still undergoing further development based on user feedback. Furthermore, as only 32 people participated in the testing, the results cannot be generalised and need further testing.

CONCLUSION AND FUTURE WORK
A virtual training environment for the gas operatives is evaluated in this paper. This virtual training environment can be integrated into existing modules of gas operatives training that can help in improving the skill set and decision-making capabilities among the gas operatives. The potential advantages of using a virtual training environment are its ability to construct close to reallife situations at less cost, risk-free training of new gas operatives, and execution of the same problem multiple times.
The virtual training environment is tested with 32 participants with the help of the SUS and sense of presence questionnaires. The average SUS score for all participants was 85.31, equivalent to a Grade A in the SUS analysis. There are no significant differences between different age groups, gender, and participants with or without previous VR experience. The sense of presence data analysis supports the results from the SUS scores reflecting high system usability and close to a real-life virtual training environment.
Future work will address some of the limitations of the current paper by testing the virtual training environment with a much larger sample size to generalise the results. In addition, more VR scenarios can add extensive details and interaction opportunities between the gas operatives and their trainers by having trainee and instructor views (multiplayer).