s2fieldsofstudy:
Computer Science
Medicine

text:
Usability Evaluation of Dwell-free Eye Typing Techniques

Dwelling is an essential task to be performed to select keys from an on-screen keyboard present in the eye typing interface. This selection task can be performed by fixing eye gaze on a key for a prolonged time. Spending sufficient amount of time on each key effectively decreases the overall eye typing rate. To address the problem, researchers proposed mechanisms, which diminish the dwell time. We conducted a within-subject usability evaluation of four dwell-free eye typing techniques. The results of first-time usability study, longitudinal study and subjective evaluation conducted with 15 participants confirm the superiority of controlled eye movement based advanced eye typing method (Adv-EyeK) than the other three techniques.

INTRODUCTION
In recent times, gaze-based interaction has evolved as an alternate user interaction modality. It has significant impact in the field of text entry in the last few decades [14]. The probable cause may be its similarity with traditional text entry methods, only by changing the controlling organ from finger to the eye. An advantage of the eye modality which may attract the developers is, with the same setup supporting ablebodied, we can develop systems for disabled users who are capable of visual interaction. These days, many applications support text entry through eye gaze [13,16,26], also in the mobile environment [5].
Unlike other input modalities, eye gaze supports few commands (as eye is always moving and always activated, it is unnatural to hold the gaze for long time etc.) like eye movement, fixation, blinking and winking of an eye. In an eye-tracking Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Permissions@acm.org. November 25 -26 2014, Kochi, Japan setup, eyes can be attached with the mouse pointer with the help of an eye tracking device (i.e. camera) and gaze-tracking software. Eye gaze-based text composition can be performed in three ways namely a) direct gaze pointing or Eye typing, b) eye gesturing and c) continuous writing [2,11].

IDHF 2014,
In Eye typing method, users require to perform text entry through an on-screen keyboard. There, selecting a key from the keyboard can be performed by placing eye pointer for a slightly prolonged duration, called as dwell time [11]. The other eye behavior used for selection is through eye blink performed on the desired key button. The second method, i.e. Eye gesturing, supports eye movements to draw a specific pattern (gesture) for selecting a character if that pattern matches. In Continuous eye writing method, commands are activated based on the natural movement of gaze (like positioning the gaze into an overlay button area selects a character, etc.). It is supporting the users' natural gaze movements as the eyes are always activated and roamed constantly. Methods where switching off the gaze is not required (like continuous drawing with no lifting of pen) can be chosen as a suitable platform to implement the method (e.g. using Dasher interface for gaze-based text entry, where always it is needed to select the characters from the character stream). Gaze-controlled continuous writing method is different from Eye typing as it does not require the eye pointer to be continuously active. In Eye typing method, rather after selecting a character, the eyes can stay which is not possible in other methods.
In this research, we aim to develop Eye typing methods as those can support dwell-freeness. Besides gaining popularity among the alternate text entry mechanisms [11], eye typing creates a number of design issues which make it a unique technique with the own set of research problems.
One design principle of keyboard suitable for eye typing is decreasing the number of keys as well as the space between keys in the layout to save screen space [16]. On the other hand, larger size keys help the user to select them easily, even in a setup having less spatial resolution [6]. In some cases, the large size of the interface is observed with the fewer number of keys accommodated in it, which inevitably takes larger screen space for a button. Thus, to optimize between eye movement, screen space and user comfort, an optimal size of the keys and space between keys needs to be decided. Before selecting the key, searching of it in the interface is an essential task which is significantly affected by visual stimuli presented in the interface such as color, orientation, shape, size, spatial frequency etc. [31]. Another important objective of develop-ing effective eye typing keyboard is to optimize the screen area utilization, especially for small display devices.Špakov and Majaranta [23] (see Fig. 1a), Panwar et al. [19] developed eye typing keyboards which save screen space.
Minimizing speed-accuracy trade-off at different levels of cognitive complexity is one of the major concerns in dwellbased eye pointing [32]. Large dwell time prevents users from false selections most of the time, as well as brings tiredness in their eyes [15]. On the other hand, shorter dwell time increases the chance of Midas Touch problem (the classic eye tracking problem referring in our topic as wrong character selection from the keyboard through gaze pointing [8]). As a result, it is difficult to conclude that shorter dwell time always produces better text entry rate with accuracy. The fixed dwell time also sets the maximum typing speed limit as the user has to wait for the stipulated time on each character button before selecting it. Majaranta and Räihä [15] stated that most gaze typing evaluations were conducted with novices using a constant, fairly long dwell time (450 − 1000 ms). Wobbrock et al. [30] used a short dwell time (330 ms) and achieved fair text entry rate (7 wpm).Špakov and Miniotas [24], Majaranta and Räihä [15] and Panwar et al. [19] studied dynamic adjustment of dwell time. Although the typing result of those systems were better, they reported delay (participants committed that it was hard for them to change typing speed quickly as the system responded with a delay [25]) and involuntary variation (after selecting a key with less dwell time, users cannot move their eyes off the target. In turn, as the dwell time decreases, this adaptive adjustment becomes less convenient to the user [25]) as critical problems. So, a trade-off still remains among dwell time, text entry rate and accuracy of the interface.
An effective way to increase the eye typing rate is to minimize dwell time. It has been observed by researchers that depending on the flexible cognitive complexity of users during the eye typing experiment, dwell time can be altered instead of fixing at a particular value [12]. Majaranta et al. [12] developed an interface which dynamically adjusts the dwell time. Their results revealed the effectiveness of the method in achieving faster eye typing rate. Further, researchers attempted to develop gaze-based interaction methods which can diminish dwelling task for selecting a character key. Effective execution of dwell-free methods can produce a moderate improvement of text typing rate. Urbina and Huckoff [26], Morimoto and Amir [17], Bee and Andrě [2], Kristensson and Vertanen [9] (see Fig. 1b), Sarcar et al. [21] and Chakraborty et al. [3] proposed dwell-free eye typing mechanisms.
According to eminent interaction designer Jacob Nielsen [18], usability is defined as a quality attribute that assesses how easy and learnable a user interface is. The word "usability" also points to methods for improving ease-of-use during the design process. Our work seeks to examine the effectiveness of existing dwell-free eye typing techniques through two usability studies (a first-time usability study followed by longitudinal study) with respect to quantitative evaluation (i.e., eye typing rate and error rate) and subjective measures. The first-time usability study 9(a) Scrollable keyboard (Špakov and Majaranta, 2008) 9(b) QWERTY keyboard (Kristensson and Vertanen, 2012) Figure 1: Keyboards in eye typing interfaces is aimed to analyze the usability of popular eye typing methods/interfaces for short-time accessing. On the other hand, the objective behind performing longitudinal study is to examine the combined effect of eye typing methods and on-screen keyboard layouts in accessing different eye typing interfaces of users over a long period of time. A subjective evaluation was performed using questionnaire responses collected during the experiment. From the results of usability between different dwell-free techniques examined in this comparative study, we find out the most effective technique which can further be used in developing efficient eye typing interfaces.
The organization of the paper is as follows. Section 2 surveys the state of the arts in dwell-free eye typing research area. First-time usability study and longitudinal study details and result analysis are provided in Section 3 and 4, respectively. Finally, Section 5 concludes the paper.

DWELL-FREE EYE TYPING MECHANISMS
The detailed methodology descriptions of dwell time diminishing mechanisms considering for the usability evaluation are given below.

Iwrite
Iwrite [26] is a square-shaped interface for gaze-based text entry placing character buttons at the outer side and text area in the middle (Fig. 2). The characters are selected by gazing toward the outer frame of the application. The text window is placed in the middle of the screen for comfortable and safe text review. The order of the characters, parallel to the display borders, reduces errors like the unintentional selection of items placed on the way to the screen button (e.g., [28]). The interface is very simple to use taking full advantage of the short saccade selection. on Key-focus and Key-selection task oriented eye movements. The KKBoard interface replicates same keyboard layout placed in two separate screen regions, called as contexts (see Fig. 3) where the user needs to switch alternately for selecting a key. In the eye typing process, key-focus task, activated through short dwell time, is followed by key selection, which is made by switching contexts (a saccade to the other context). The key which is last in focus in the previous context is taken at the selection phase. Context switching task diminishes the effect of the Midas Touch problem. The CSbased text input replaces the traditional long dwell time with a short dwell followed by a context-switching saccade. Sarcar et al. [21] proposed the EyeK eye typing interface where they introduced a dwell-free typing mechanism. According to the method, a key button in the keyboard is selected automatically through specific interaction, which actually activates while a user moves the eye pointer through the button areas in inside-outside-inside fashion (see Fig. 4). As eye movement is faster than mouse or finger movement, this interaction takes minimum effort and time, which is negligible with respect to the traditional dwell time. According to the example shown in Fig. 4, suppose a user wants to select character 'C' While hovering on the character, key and its overlay area are visible. The user starts moving the eye pointer within the key area, goes to outside overlay area and again comes back inside to the key area to complete the interaction phase (Fig. 4a, enlarged portion). After placing the eye pointer inside the key area, character selection gets activated (Fig. 4b). After selecting a character, visual feedback is given by changing its font color to red, which remains up to next character selection. If users need to enter same characters twice, they have to get out of the character initially and enter it again in the similar manner. This proposed mechanism provides a facility that the "going out" and "coming back" sides of a button may not be same and fixed.

An effective dwell-free eye typing technique (Adv-EyeK)
A slight improvement over Sarcar et al.'s method [21] for dwell-free eye typing was proposed by Chakraborty et al. [3] (throughout the article, we will call it as Adv-EyeK, which is an advanced method of EyeK). The interaction pattern for this advanced method is different from the previous method. It supports more controlled eye movement for key selection (Fig. 5). In the overlay area which is activated after hovering, a black point is placed at a fixed position (upper side of the key, preferably at the center of the upper portion of the outer key area). After users hover on the intended key, they require to "go out" from the inner key, reach to that prominent point and after looking, "come back" inside the inner key area (preferably from same side) (Fig. 5a). A feedback system, same as the previous method, is applied to provide selection confirmation to the user (Fig. 5b). As an example, suppose, a user needs to select character 'C'. Then, on hovering 'C', the outer layer becomes visible and user "goes out" from the upper side, sees the point and "comes back" into the inner key from the same side. For the double selection of a single key, the same procedure stated in the previous method is to be followed.
Kriestensson and Vertanen [9] proposed a novel work in the related field, but they did not provide the working methodology of the system. So, we could not replicate the method. Bee and Andrě [2] mimicked the design methodology of Quickwriting text entry system for eye typing. We were unable to consider the system for our usability experiment also as it supports Eye Writing instead of Eye typing [2].
We conducted two usability studies namely first-time usability test and longitudinal user experiment and analyzed the results. Our aim of conducting two esperiments was to fulfill two objectives: a) to observe the efficacy of popular dwellfree eye typing techniques with respect to user response in

FIRST-TIME USABILITY TEST
The purpose of this usability test was to study the speed, errors and perception of usability in the first encounter with the eye typing interfaces. Conducting the short term study, we not only observed the initial reaction of users toward different eye typing interface, but also judged the efficacy of the interfaces with respect to immediate usability. We, in this study, wanted to observe the efficacy of different eye typing methods for short-term usability of users in different conditions (like accessing eye typing interface without and with + without help etc.). The other side of conducting this short study with every user is to provide scope to get acquainted with different eye typing interfaces which would further help them while performing long-term studies.

Apparatus
Experiments were conducted in a low-cost eye tracking setup using 2.2GHz Intel Core2Duo processor with 15" screen LCD color monitor having 1440 × 900 resolution. Modified Sony PlayStation Eye webcam, original lens was replaced by manual focus and Infrared (IR) filter removed lens, IR Lamp, consisting a matrix of 10 IR LED, along with open source ITU GazeTracker software [1], developed by IT University of Copenhagen, were used for experiments (see Fig. 6). The key press events and gaze positions were recorded automatically and stores into log files using separate event hooking programs. All experiments were performed in Windows 7 environment. Controlled light conditions and positioning of the setup were maintained.
Three applications were run during testing the efficacy of 4 dwell-free typing methods namely Iwrite [26], KKBoard [17], EyeK [21] and Adv-EyeK method [3]. In each case, real-time eye movement was obtained with a tailored version of the application namely ITU GazeTracker. Apart from this, one C# application was run to display eye typing interfaces. Another application was used for presenting phrases during the typing session running. This program randomly presented a target phrase from a corpus of 500 to the user while simultaneously recording various text entry metrics for further analysis. Real-time (x, y) eye movements were converted to cursor coordinates by a C# program that simply masqueraded the gaze point as the mouse position to other Windows applications.
The program was based on ITU Gaze tracker developed at IT University of Copenhagen [1].

Figure 6: Experimental setup
Usability was measured by speed of text entry as words per minute (wpm) and total errors. In this study, we wanted to explore the effect of dwell-free eye typing mehods on users in short duration (in both with and with+without help situation).
To judge user performance of traditional text entry systems, speed and accuracy were the basic and effective quantitative evaluation metrics. Thus, we collected user results on the basis of these two metrics.

Participants
Eye typing experiments were performed by 15 participants (11 male, 4 female) recruited from the local university area. Participants ranged from 25 to 34 years (mean = 28.5). All were regular computer users, accessing on an average 4 hours per day and have prior experience in composing text using eye typing techniques. All participants have normal or correctedto-normal visual acuity. 14 participants were right-eye dominant and 1 was left-eye dominant, as determined from eye dominance test [4].

Designs
Three popular keyboards namely, a) compact screen space optimized full Scrollable Keyboard layout proposed by Spakov et al. [23] (Fig. 1a), b) Panwar et al.'s key size and space optimized EyeBoard layout (Fig. 4b) [19], incorporated in EyeK interface [21] and c) popular QWERTY layout were considered for the evaluation. The size and space between key buttons were maintained same as specified in first two keyboards whereas for QWERTY, these were modified as 1.6 cm for both height and width and 0.6 cm for distance between two keys, respectively [21]. During the first-time usability experiments, sentences to be typed were taken from MacKenzie and Soukoreff's phrase set [10]. The experiment was a within-subjects design. Participants were randomly assigned to keyboard and eye typing method.

Procedure
Every eye typing session was preceded by synchronization of the eye movement with the gaze tracker through Calibration. During the typing session, users were instructed to type as fast as possible allowing few errors and not to move their eyes beyond the visibility range of the screen. To tackle this situation, users wrote the to be typed phrase using pen and paper before the session began or they listened to word by word from the instructor and composed them at runtime. Participants could only correct errors occurred due to wrong key selection by erasing them using backspace and then retyping them.
Users spent first few sessions before the experiment as training sessions where they were briefed about the nature of the experiment and then they completed a short demographic questionnaire. After that, users made themselves familiarized with eye tracking setup (fixing up camera and Infrared lamp positions) and eye typing methods applied on keyboard interfaces ( Fig. 7) by repeatedly accessing them. The total time for the interaction for each participant was about 10−15 minutes. Before typing, participants first practiced some phrases on pen and paper, taken from standard phrase set [10] (this phrase set contains 500 phrases, complete set is available in the web (link:http://www.yorku.ca/mack/PhraseSets.zip) to memorize them and then composed these phrases for each of the 3 keyboards (to enhance the familiarity with keyboard based eye typing and typing texts). We collected feedbacks from the participants after the training session. If, after these practice sessions, participants felt the strong attachment with the keyboards and eye typing methods, then we started the testing sessions keeping a log for each. In the testing phase of the first-time usability test, the initial speed of eye typing, errors, and perception of usability were studied. The first session of the initial usability test, on an average, took more than one hour, and data were not considered for analysis. Before starting of each session in short and long term test, users assured the instructors about their memorability of the practice set. On average, each testing session took about 45 minutes. Among 12 texts selected in each experimental procedure for a user, 2 were taken from the in-domain Mackenzie and Soukreff's phrase set [10] and other 10 were taken from out-of-domain texts such as novels, stories etc. for judging the design efficacy.
This test was performed to prepare the participants for longitudinal eye typing tasks with on-screen keyboards following specific typing methods within the constrained hardware scenario. At starting, an initial assessment of eye typing familiarity of the users was done. For this, in any of the keyboards chosen randomly, users were required to search some character keys given by instructor through eye movement and then select by fixing their eyes on the key for some time. This task was termed as first time test. The eye typing experiment was planned such that participants were required to listen to the text and then typed it. Each user typed 20 words in a session.
In this context, we divided the session into two subgroups where the order became random for each participant. One was to type 5 words, where we trained participants with single word typing after listening with no external help provided (without help phase). Here, key selections were performed through dwelling. When the task began, instructor uttered one word, and the user was expected to type it with his eye as fast as possible. Once finished, he gave the signal to the instructor for supplying the next word. If a word was typed correctly, the system played a "correct" sound. If the word was typed wrong on the first attempt, the system gave a "wrong" beep rang once, and the instructor might either mark the error or repeat (predetermined) the help instructions. For the other method, known as with + without help, each user was given to type 3 phrase each containing 5 words. Here, the instructor read the phrases once, user listened to that and typed as their own. Again, if users committed mistakes, instructors helped them to rectify those (this with + without help mode was allowed only for two attempts per word or phrase. If the user got the second attempt wrong as well, the current word was skipped and he had taken to the next word). The aforementioned eye typing methods were conducted at least two times each for a user and ordering of the keyboard selection was counterbalanced across participants. Although taken from the same corpus, the testing words or phrases were different from those chosen in the training session. We allowed a maximum of two days gap between two consecutive sessions of a user.

Dependent measures
The dependent measures used in this experiment were words per minute (WPM) and the total error rate [22,29] which is an addition of corrected and uncorrected error rate.

Results
In the first-time usability test (15 participants ×20 words), total attempts can be categorized into a) successes without help, b) the proportion of successes without help, c) successes with and without help, and d) the proportion of successes with and without help for each keyboard. We averaged individual category results with respect to an user. Out of 300 attempts against each dwell-free methods, EyeK and Adv-EyeK method proposed by Chakraborty et al. [3] earned best results in success with + without help as well as in success without help category. Using advanced EyeK method, users achieved 215 correct attempts (results ranging from 180 to 230, SD = 1.15) on an average which did not seek any help. The EyeK interface also performed better as it achieved on an average 205 attempts correct (range from 189 to 223, SD = 1.2) without seeking help out of 300 attempts. The results achieved in this category by Iwrite and KKBoard interface augmented dwell-free mechanisms were 189 attempts (range from 169 to 203, SD = 1.21) and 175 (range from 156 to 189, SD = 1.28), respectively. It was observed that the impact of advanced EyeK method was significantly better than other methods in terms of average success rate (F-test result, F (3, 292) = 5.92, p < 0.05). In success with + without help category also, users preferred controlled eye movement which was supported by EyeK augmented dwell-free method and advanced EyeK based efficient method. Users got on an average 290 correct attempts (results ranging from 285 to 294, SD = 1.08) and 296 (range from 187 to 205, SD = 1.05) toward entering text through these methods. In contrast, the other two methods i.e. Iwrite and KKBoard interface associated dwell-free methodologies achieved only 239 number of attempts (results ranging from 230 to 249, SD = 1.07) and 252 (values from 238 to 266, SD = 1.14) with + without help correct within 300 attempts. In this category also, Statistical analysis revealed the significant difference of the newly proposed efficient dwell-free mechanism from other 3 methods (F (3, 293) = 6.47, p < 0.05). In this study, we observed no effect on different keyboard layouts while accessing dwellfree methodologies.
During the user performing the usability task, we observed users' eye typing speed along with corrected (number of backspaces pressed) and uncorrected (error remaining in the typed text) error rate for all the methods. According to the scenario, all users typed in each keyboard augmented with every typing methodologies at least once. We gathered all user and keyboard based results of eye typing rate and error rate for a dwell-free eye typing methods and averaged them. The average eye typing rate of users in the first-time study ranged from 3 to 6 wpm. For without as well as joint with and without help based interaction, controlled eye movement based eye typing methods performed better. We observed total error rate (based on majorly uncorrected error rate because very few persons corrected the mistakes during composition with any keyboard), on an average over all the cases, of Adv-EyeK method as 28.21% whereas 33.52%, 33.69% and 31.89% for methods augmented with Iwrite, KKBoard and EyeK keyboard interfaces, respectively. We observed from the results that using the Adv-EyeK method, users left fewer errors uncorrected. Further study on analysis of variance revealed that there was no significant difference in error rates between the methods (F(3, 295) = 1.03, n.s.).

Discussion
Results indicated that eye-typing with the Adv-EyeK interface keyboard was significantly faster and yielded less error rate than three other eye typing methods namely Iwrite, KKBoard and EyeK. The achieved high typing speed with the Adv-EyeK method and low error rate can become more prominent with the number of sessions exercised by the users. Further, to explore the joint effect of eye typing method and augmented onscreen keyboard layouts, a longitudinal study was conducted.

LONGITUDINAL STUDY
To analyze the effects on user performance on different dwellfree eye-typing schemes along with keyboards, we conducted a controlled experiment spanning 8 sessions attaching the methods with different eye typing keyboards including QW-ERTY. Using this many sessions allowed us to assess the performance of the 4 methods over time, as we expected the learning rates of each method to be different. Individuals performed no more than 2 sessions per day. maximum 48 hours gap was kept between two consecutive sessions for a user. If two sessions were performed on the same day, at least two hours were kept between sessions.

Apparatus
The apparatus matches that of the first-time usability test.

Participants
All the participants participated to first time usability tests were invited to participate in a longitudinal test.

Procedure
In the longitudinal test, it was decided that evaluating the performance of the participants could be judged by typing sufficient number of words through each of the three on-screen keyboard augmented with different eye typing methods, because it was observed that a typical user would not put much effort in small trials to learn typing. We divided the task into 8 sessions for each user, each keyboard with each method. While most sessions ran on consecutive days, we ensured that 2 sessions could be completed for each user in a day. There could be a maximum gap of 2 days between any two sessions for a user. No training was provided before the task. Users were asked to input approximately 8 phrases (40 words) in a session as fast as possible. When the task began, the instructor spoke a phrase to the user which they listened and tried to type. No feedback, help or a second attempt was provided at the middle of typing a phrase. After completion of the phrase, participants pressed the "save" button, a sound was generated which alerted instructors for either declaring the session end or supplied the next phrases. Once the user had finished typing all the 8 phrases, it was shown how well he has done (what he was supposed to type, what he typed on the first attempt, what he typed on the second attempt if any, the errors if any, and the speed). In this way, a user who wanted to complete all the sessions, he required to type at least 3 keyboards ×4 methods ×8 sessions ×8 phrases = 768 unique phrases taken from the standard phrase set. It was assured that instead of keyboard and method were selected randomly for a session, each user needed to type same 768 phrases overall. During the study, keyboard and method order were random and counterbalanced across participants. Each session lasted, on an average, approximately 45 to 60 minutes. A user, who evaluated all keyboards with different typing methods, approximately took 2 to 4 months to finish. Such a long duration is required to evaluate minimum 192 (2 trials × 3 keyboards × 4 methods × 8 sessions) trials per user. If the instructor, after a session, realized that user performance was not satisfactory, he conducted the session again. After a session completion, the user was asked to rate the task on a scale of 1 to 5 for difficulty on the basis of some demographic questionnaire. Then, the user was reminded to come for the next session before leaving.

Dependent measure
Aligned with the pilot study, the dependent measures used in this experiment were words per minute, overhead time, total error rate and subjective evaluaqtion parameters likeease-ofuse, distracting, fatigue etc.

Results
Within-subject experiments were performed with 3 keyboards each having 4 dwell-free methods measuring eye typing rate, total error rate and overhead time [9]. Data for each participant were averaged in each session to form single measure per participant per session on a variety of metrics. Participants completed a total of 2 trials ×3 keyboards ×4 methods ×8 sessions = 192 trials. With 15 participants, the entire study comprised of 2880 trials. Also, for testing sessions, keyboard order was also kept counterbalanced across participants. 3 sessions were performed per day by each participant. The whole study lasted for approximately 4 months. Each trial was made of 8 phrases taken randomly from new phrase set [27] (this phrase set, developed by Vertanen and Kristensson [27], contains total of 2239 sentences and sentence fragments generally used in E-mails; it can be accessed online (link:http://aactext.org/comm2/comm2.zip).

Eye typing rate
The overall average eye typing rate achieved by participants with 4 dwell-free methods applied on 3 different keyboards was ranged from 6 to 9 wpm (Fig. 8) (see Table 1). Using method applied in Iwrite [26], keyboards Scrollable keyboard, QWERTY and EyeBoard earned the eye typing rate ranging from 5.9 wpm to 7.1 wpm (mean = 6.6, SD = 1.05), 6.2 to 7.9 wpm (mean = 7.2, SD = 1.15) and 6.4 to 8.1 wpm (mean = 7.4, SD = 1.18), respectively (Table 1). Implementing KKBoard method, irrespective of users, 3 keyboards achieved 6.4 wpm to 7.9 wpm (mean = 7.1, SD = 1.02), 6.5 wpm to 8.2 (mean = 7.4, SD = 1.06) and 6.55 wpm to 7.8 wpm (mean = 7.1, SD = 1.17), respectively. In case of EyeK method, 3 keyboards achieved 6.8 wpm to 8.1 wpm (mean = 7.5, SD = 1.09), 6.7 wpm to 7.8 (mean = 7.3, SD = 1.06) and 7.1 wpm to 8.7 wpm (mean = 7.9, SD = 1.08), respectively. Finally, participants on an average achieved eye typing rate of 6.7 wpm to 7.8 wpm (mean = 7.3, SD = 1.07), 7.4 wpm to 8.3 (mean = 7.8, SD = 1.06) and 7.3 wpm to 8.9 wpm (mean = 8.1, SD = 1.08), respectively through 3 keyboards by using the Adv-EyeK method. For all the mechanisms, it was observed that participants' eye typing rates got improved in the first few sessions and then reached to saturation. The analysis of variance (ANOVA) on text entry speeds showed that there was a significant difference between the means of user's performance on different eye typing mechanisms (F (3, 716) = 5.42, p < 0.05). Further, The Post-hoc using Tukey HSD test reveals significant difference between performance of Adv-EyeK and other methods (p < 0.05) (ANOVA or F-test can only tell about whether significant difference between the groups exist or not, but not which group significantly differs from the others. For that, we need  (Fig. 9). However, total error rates were dropped significantly over sessions (F (7, 350) = 4.29, p < 0.05). The results concluded the observation that using the Adv-EyeK method, users left fewer errors uncorrected in different keyboard designs, i.e., the number of corrected errors is more in case of associated interfaces. We also analyzed a number of errors left in the typed text for all the 4 methods applied on 3 keyboard designs. An analysis of variance revealed that there was no significant difference in error rates between the keyboard designs (F (2, 946) = 1.01, n.s.)

Subjective Evaluation
We collected the subjective ratings from the participants with the nonparametric Wilcoxon Matched Pairs Signed Ranks Test [20] as Likert-scale data did not often conform to the assumptions required for ANOVA procedures. We talked with the participants before and after each session asking them about their eye strain and tiredness in a scale of 1 to 5. The level of tiredness was calculated by subtracting the first value from the later value. Analyzing the experimental results, we observed no significant difference between the average level of the tiredness, which was 0.52 in the first and 0.71 in the last session. We also calculated the text entry speed, ease of use, and general fatigue after each session using a questionnaire with a scale from 1 to 5. An increment of text entry rate of every user was observed (average from 3.1 to 4.3). By analyzing the participant preferences given in the Likert scale, Adv-EyeK method performed significantly better than other eye typing techniques in terms of easier to use (z = 42.00, p < .001), faster (z = 40.00, p < .01), less distracting (z = −47.00, p < .01) and less fatiguing (z = −55.00, p < .001) (detail demographic questions and user feedbacks are shown in Table 2).
Finally, participants were again interviewed after completion of the series of sessions. Participants felt that the concept of key area increment and allowing user's eye to move through those areas in a pattern to select an object (here, key button in an on-screen keyboard) was much more intuitive than other methods. They also admitted that typing by gaze was fairly easy, easier than their expectations and less boring than dwellbased eye typing methodologies, but clearly slower than using a conventional, hand operated hardware/virtual keyboard.
Participants said that during first few sessions, the Adv-EyeK and EyeK interfaces were felt to be more difficult for them in terms of selecting an intended key than other dwell-free eye typing methods. Soon, they started to control their eyes easily and after some time, usually took hold of the methods and gradually geared up the speed with them (as well as felt comfortable). Although the Adv-EyeK method was not known to participants previously, it can be pointed out as quickly learnable and perceived as easier to use than others. This might be for their simple going to a point and then coming back gesture of performing the selection task. Earlier, it might create a problem which soon got overcome.
Regarding Iwrite eye typing interface [26], participants expressed positive feedback about the simplicity and intuitive- ness of it. But many of them said that they had difficulty to perform the interaction for selecting a key. The ambiguity between interaction and user's natural eye movement could hamper the eye typing process.
Participants, after performing eye typing task with KKBoard interface, admitted that in spite of the presence of popular QWERTY keyboard familiarity, the interface implemented two keyboard layouts which took double space than a normal keyboard size used in eye typing interface. Users suggested that the wastage of space should be minimized, but not that much where the keys become so small and inappropriate for use with eye trackers.
People might think that speed would also lead to greater perceived fatigue, but, in fact, from the observed results it was prominent that the Adv-EyeK method was significantly less fatiguing than other methods executed through on-screen keyboards.

Learning curve
To understand the learning of the selected dwell-free eye typing methods augmented with on-screen keyboards we performed a longitudinal study with those 4 methods augmenting QWERTY layout. 5 new participants (who did not perform any of the previous experiments) having familiarity with QWERTY based text entry but unfamiliarity with eye tracking methods performed the eye typing sessions with testing phrases selected from Vertanen's phrase set [27]. For each session, 5 phrases had been typed by each user with each method. The average user result was depicted in Fig. 10. It indicates that the Adv-EyeK method needed more initial effort to learn compared to other methods. However, after 20 sessions, Adv-EyeK outperformed other methods. We derived standard regression models in the form of the power curve fitting as it followed Power law of learning. The longitudinal study lasted for 60 sessions. The learning curve inevitably reflected the increasing efficiency of users after performing several sessions. The highest eye typing rate achieved through Iwrite, KKBoard, EyeK and Advanced EeyK methods were 6.55, 5.79, 7.31 and 8.01 wpm, respectively. Figure 10: Learning curve Through user experiment, 4 dwell-free eye typing methods were compared in terms of performance measures and subjective usability criteria. The first time usability and longitudinal study results reflected that Adv-EyeK eye typing mechanism performed better in eye typing rate, error rate, and usability than other 3 methods. The interface with on-screen keyboard implementing the method was learned quickly because of a fixed pattern based selection for every key. In view of users' subjective impressions, it is observed that they performed small eye pattern movement [3] well within a small region, after sufficient training. This scenario saves the screenspace offering an advantage over off-screen targets in limiting saccade distance to the dimensions of the attached keyboard's layout area. Further, subjective evaluations using standard task assessment tool NASA Task Load Index (NASA-TLX) [7] is underway and the preliminary observations assure the superiority of Adv-EyeK design over others with respect to physical, mental and temporal workloads along with performance, effort and frustration measures.

CONCLUSION AND FUTURE WORK
In this work, we present a usability study on 4 dwell-free eye typing mechanisms comparing them with respect to eye typing rate, error rate, ease-of-use, eye fatigue etc. and collect results from short-term as well as longitudinal user experiments. The obtained user results reveal the suitability of Adv-EyeK dwell-free eye typing method according to user's eye typing behavior. The learning rate and text entry rate of the Adv-EyeK method augmented with on-screen keyboards are moderately higher than other eye typing techniques. This result undoubtedly support the superiority of the method even while attached with any of the popular on-screen keyboards applied for eye typing. Also, due to the controlled nature, the underlying interaction can accurately be performed by users, once become familiarized.
In our study, we used our self-developed easy replicable lowcost eye tracking set-up, as one of the initial objective of this project was to develop low-cost eye typing task supporting hardware. Thus, we used low-cost apparatus and open-source software ITU gaze-tracker. However, the accuracy of the setup still is not up to the mark and thus, the validity of results confines within performing the experiments in controlled environments. To collect more accurate data from this lowcost setup, we introduced calibration before each session and also instructed participants not to see outside the screen area during the experiment. Irrespective of all these precautions, sometimes when we found that participants could not finish the experiment due to many errors committed, we had to conduct the same experiment again for each of them. Extending the current work, we can further improve the setup quality by many ways like fixing the infrared (IR) filters within visible range, placing the camera as close to eye for more accurately detecting eye gaze during calibration phase etc.


SimpleNets: Quality Estimation with Resource-Light Neural Networks

We introduce SimpleNets: a resource-light solution to the sentence-level Quality Estimation task of WMT16 that combines Recurrent Neural Networks, word embedding models, and the principle of compositionality. The SimpleNets systems explore the idea that the quality of a translation can be derived from the quality of its n-grams. This approach has been successfully employed in Text Simplification quality assessment in the past. Our experiments show that, surprisingly, our models can learn more about a translation’s quality by focusing on the original sentence, rather than on the translation itself.

Introduction
The task of Machine Translation Quality Estimation (QE) has gained noticeable popularity in the last few years. The goal of QE is to predict the quality of translations produced by a certain Machine Translation (MT) system in the absence of reference translations. Reliable solutions for QE can be useful in various tasks, such as improving post-editing efficiency (Specia, 2011), selecting high quality translations (Soricut and Echihabi, 2010), translation re-ranking (Shah and Specia, 2014), and visual assistance for manual translation revision (Bach et al., 2011).
QE can be performed in various ways in order to suit different purposes. The most widely addressed form of this task is sentence-level QE. Most existing work addresses this task as a supervised learning problem, in which a set of training examples is used to learn a model that predicts the quality of unseen translations. As quality labels, previous work uses either real valued scores estimated by humans, which require for a given QE system to address the task as a regression problem, or likert scale discrete values, which allow for the task to be addressed as either a regression or a classification problem.
Sentence-level QE has been covered by shared tasks organised by WMT since 2012, with subsequent years covering also word and documentlevel tasks. Recent advances in Distributional Semantics have been showing promising results in the context of QE strategies for different prediction levels. An example of that are modern word embedding architectures, such as the CBOW and Skip-Gram models introduced by (Mikolov et al., 2013b), which have been used as features in some of the best ranking systems in the sentence and word-level QE shared tasks of WMT15 (Bojar et al., 2015). Word embeddings are not only versatile, but also cheap to produce, making for both reliable and cost-effective QE solutions.
Neural Networks have also been successfully employed in QE. The FBK-UPV-UEdin (Bojar et al., 2014) and HDCL (Bojar et al., 2015) systems are good examples of that. They achieved 1st and 2nd places in the word-level QE tasks of WMT14 and WMT15, respectively, outperforming strategies that resort to much more resource-heavy features. Another successful example are neural Language Models for sentence-level QE (Shah et al., 2015).
We were not able to find, however, any examples of sentence-level QE systems that combine word embedding models and Neural Networks. In this paper, we present our efforts in doing so. We introduce SimpleNets: the resource-light and language agnostic sentence-level QE systems submitted to WMT16 that exploit the principle of compositionality for QE. In the Sections that follow, we describe the sentence-level QE task of WMT16, introduce the approach used by the SimpleNets systems, and present the results obtained.

Task, Datasets and Evaluation
SimpleNets are two systems submitted to the sentence-level QE task of WMT16. In this task, participants were challenged to predict real-valued quality scores in 0,100 of sentences translated from English into German. The translations were produced by an in-house phrase-based Statistical Machine Translation system, and were then postedited by professional translators. The real-valued quality scores are HTER (Snover et al., 2006) values that represent the post-editing effort spent on each given translation.
The task organisers provided three datasets: • Training: Contains 12,000 translation instances accompanied by their respective postedits and HTER values.
• Development: Contains 1,000 translation instances accompanied by their respective postedits and HTER values.
• Test: Contains 2,000 translation instances only, without their respective post-edits or HTER values.
Each instance is composed by the original sentence in English along with its translation in German. HTER scores were capped to 100. The organisers also provided 17 baseline feature values extracted using QuEst++  for each dataset.

The SimpleNets Approach
SimpleNets aim to provide a resource-light and language agnostic approach for sentence-level QE. Our main goal in conceiving SimpleNets was to create a reliable enough solution that could be cheaply and easily adapted to other language pairs, moving away from the use of extensive feature engineering.
The SimpleNets approach was first introduced by Paetzold and Specia (2016a) as a solution to the shared task on Quality Assessment for Text Simplification of QATS 2016 1 , in which participants were asked to create systems that predict discrete quality labels for a set of automatically produced text simplifications. Labels could take three values: "Good", "Ok" and "Bad". Text Simplification differs from Machine Translation in the sense 4. Structuring: Using a trained word embeddings model, it transforms each n-gram into a training instance described by a matrix M×N , where M is the previously mentioned maximum n-gram size, and N the size of the word embeddings used. Each of the M rows in matrix M×N represent a given word in the n-gram, and each of the N columns, its embedding values. If an n-gram is smaller than N , the matrix is padded with embedding values composed strictly of zeroes.

5.
Learning: Training instances are then fed into a deep Long Short-Term Memory (LSTM) Recurrent Neural Network in minibatches so that a quality prediction model can be learned.
Notice that this process yields a model that predicts the quality of individual n-grams rather than the quality of a simplification in their entirety, which is not what was required for the task. To address this, each simplification in the test set is first processed through Decomposition, Union and Structuring, but the complete quality prediction process has two complementary steps: 1. Prediction: The trained model is used to predict the quality of each n-gram of the simplification in question.

2.
Merging: The quality of all n-grams in the simplification are merged using a certain policy, such as averaging.
After merging, a quality estimate for the simplification as a whole is produced. Although this approach has certain limitations, it addresses a very important challenge in using Recurrent Neural Networks for Text Simplification Quality Assessment: the small amount of training data available. With only 505 simplifications for training, it becomes very unlikely that a Recurrent Neural Network would be able to reliably learn a quality prediction model if it was presented with sentences in their entirety, such as how it has been done in Neural Translation and Text Generation (Schmidhuber, 2015). By splitting the sentences in the simplification in n-grams, the number of training instances available grows considerably, allowing for a better informed learning step. Addition-ally, the length of the sequences used in the Recurrent Neural Network becomes shorter, which can help the network to generalise the knowledge available in the training set.
The results of the Quality Assessment for Text Simplification of QATS 2016 serve as evidence of the potential of this approach: SimpleNets ranked 1st in predicting the overall quality of simplifications. Nonetheless, the inherent differences between Machine Translation and Text Simplification make it impossible for the strategy described above to be directly applied to sentence-level QE without any adaptation. In the next Section, we describe how we adapt the SimpleNets approach for sentence-level QE.

SimpleNets for Machine Translation
In order to use the SimpleNets strategy for sentence-level QE, we must address the biggest difference between Machine Translation and Text Simplification: while Text Simplification encompasses transformations within the constraints of a single language, Machine Translation has to handle two languages, which often have distinct vocabularies, grammar, etc. This difference prevents the application of the Union step from the process described in Section 3, since source (original) and target (translated) sentences are not in the same language, and hence cannot share the same word embeddings model during Structuring.
Another challenge in adapting SimpleNets for Machine Translation lies in the often found disparity in quality between the source sentence and its translation. Inspecting the datasets provided by the WMT16 organisers, we found that, unlike the source sentences, the majority of translations contain at least one noticeable error with respect to either grammar or coherence. This means that even if we used techniques such as the one employed by bivec (Luong et al., 2015), which allows for the training of bilingual word embeddings, the contrast between the quality of source and target sentences could confuse the SimpleNets approach, and hence compromise its capability of learning a reliable quality prediction model.
To overcome these challenges, we explore the hypothesis that SimpleNets can learn a better model for sentence-level QE by looking strictly at one of the sides of translations, rather than by trying to somehow combine the information from both the source and translated sentences. We train two variants of SimpleNets: • SimpleNets-TGT: Explores the idea that the quality of a translation can be reliably determined based solely on the characteristics of the machine translated sentence itself, without the need to assess its relationship with the original sentence. This variant of SimpleNets aims to learn a model that is capable of quantifying the differences in quality of translated sentences.
• SimpleNets-SRC: Explores the idea that a translation's quality can be determined based solely on the original sentence itself, without any need to assess the intricacies of its translated version. This variant assumes that, by focusing on the original sentences and the quality scores of their translations, Sim-pleNets can learn how to quantify just how likely the MT system in question will be of making a mistake while attempting to translate an unseen sentence. This is in line with work on QE that explores source features to measure the complexity of the source sentence (Specia et al., 2010).
Finally, we must also address the fact that, while the quality scores provided for the QATS 2016 shared task are discrete labels, the scores for the WMT16 task are real-valued. We solve this problem by simply replacing the multiple softmax activation nodes used in the QATS 2016 SimpleNets with a single dense node, and also by replacing the cross-entropy loss function with Mean Average Error.
The workflow followed by SimpleNets-TGT and SimpleNets-SRC is illustrated in Figure 2. In the Section that follow, we describe our experiments with these approaches.

Experimental Setup
To assess the efficacy of our SimpleNets, we train them over the training set provided by the organizers, which contain 12,000 instances. In order to select the architecture to be used by the LSTM networks of our SimpleNets, we resort to the technique used in (Paetzold and Specia, 2016a), in which each aspect of a Neural Network is determined through parameter optimisation over the development set. The optimisation metric used is Pearson correlation, since it is the main evaluation metric adopted by the WMT16 task. The aspects of the architecture considered and the values tested for each one of them are: 1. Number of hidden layers: 1 to 5 in steps of 1.

Embeddings model: CBOW or Skip-Gram.
Even though SimpleNets-TGT and SimpleNets-SRC were optimised individually, the resulting architectures of the two approaches are surprisingly the same: three hidden layers with 200 nodes each, with CBOW embeddings.
The word embedding models used were trained with word2vec (Mikolov et al., 2013a). We use 300 word vector dimensions and train the  (Paetzold and Specia, 2016b), UMBC webbase 2 , News Crawl 3 , SUB-TLEX (Brysbaert and New, 2009), Wikipedia and Simple Wikipedia (Kauchak, 2013). For evaluation we use the task's official metrics, which are Pearson correlation (r), Mean Average Error and Root Mean Squared Error. We compare our SimpleNets with the baseline provided by the task organisers, as well as all other systems submitted. The baseline uses SVM regression with an RBF kernel and grid search for parameter optimisation.

Results
The task results illustrated in Table 1 reveal that SimpleNets are not as effective in sentence-level QE as they were for Text Simplification Quality Assessment. Although they outperform a few systems in terms of MAE and RMSE, when it comes to Pearson correlation, SimpleNets-SRC and SimpleNets-TGT feature at the bottom of the ranking.
What is even more surprising, however, is the difference between the performance of our Sim-pleNets systems. Intuitively, one would think that the n-grams of the translated sentence itself would be a more reliable indicator of a translation's quality, given that it only becomes possible for one to assess the grammaticality and meaning errors in a translation after inspecting the translated sentence itself. Interestingly, the performance scores suggest that the model employed by SimpleNets is more proficient in learning how difficult it will be for the source sentence to be translated.
The difference in performance between the SimpleNets variants became much more clear once we inspected the individual n-gram quality predictions made by them. Tables 2 and 3 show the n-grams in the development set with the highest and lowest HTER scores, as predicted by SimpleNets-TGT and SimpleNets-SRC, respectively, along the average gold HTER of the sentences in the development set which contain them. It can be noticed that the correlation between the highest scoring n-grams of SimpleNets-SRC and their average gold HTER seem to be much more pronounced than the one observed for the highest scoring n-grams of SimpleNets-TGT. The same phenomenon can be observed between the lowest scoring n-grams of the SimpleNets variants.
The Pearson correlation scores between predicted and average gold n-gram scores provide further insight on the limitations of Sim-pleNets in the context of sentence-level QE. While SimpleNets-TGT achieves a correlation score of 0.127, SimpleNets-SRC achieves a score of 0.151. Although SimpleNets-SRC does obtain a slightly higher Pearson score, both of them are low in comparison to other approaches, which ultimately suggests either that n-grams alone do not provide with enough information on the quality of a translation in order for a reliable Quality Estimation model to be learned, or that our method of assigning   Table 3: N-grams with highest and lowest HTER scores, as predicted by SimpleNets-SRC the translation's quality score to all n-grams during training prevents our models from learning to effectively differentiate between good and bad ngrams.

Final Remarks
In this paper we have described the Sim-pleNets systems for the sentence-level QE task of WMT16. SimpleNets aims to offer a resourcelight solution to the task by exploiting Recurrent Neural Networks, word embedding models, and the principle of compositionality. Two SimpleNets variants were described, SimpleNets-SRC and SimpleNets-TGT, which attempt to predict the quality of a translation based solely on the quality of the n-grams present in its source or target (translated) sides, respectively.
Although interesting and efficient, the Sim-pleNets systems have been shown not to perform well for the task at hand, featuring at the bottom of the task's final ranking. Nonetheless, our experiments have still provided with valuable insight on the impact of the source segment of a translation on the quality of its translation.


extfieldsofstudy: Biology Medicine

s2fieldsofstudy:
Computer Science
Biology

text:
eRFSVM: a hybrid classifier to predict enhancers-integrating random forests with support vector machines

Background Enhancers are tissue specific distal regulation elements, playing vital roles in gene regulation and expression. The prediction and identification of enhancers are important but challenging issues for bioinformatics studies. Existing computational methods, mostly single classifiers, can only predict the transcriptional coactivator EP300 based enhancers and show low generalization performance. Results We built a hybrid classifier called eRFSVM in this study, using random forests as a base classifier, and support vector machines as a main classifier. eRFSVM integrated two components as eRFSVM-ENCODE and eRFSVM-FANTOM5 with diverse features and labels. The base classifier trained datasets from a single tissue or cell with random forests. The main classifier made the final decision by support vector machines algorithm, with the predicting results of base classifiers as inputs. For eRFSVM-ENCODE, we trained datasets from cell lines including Gm12878, Hep, H1-hesc and Huvec, using ChIP-Seq datasets as features and EP300 based enhancers as labels. We tested eRFSVM-ENCODE on K562 dataset, and resulted in a predicting precision of 83.69 %, which was much better than existing classifiers. For eRFSVM-FANTOM5, with enhancers identified by RNA in FANTOM5 project as labels, the precision, recall, F-score and accuracy were 86.17 %, 36.06 %, 50.84 % and 93.38 % using eRFSVM, increasing 23.24 % (69.92 %), 97.05 % (18.30 %), 76.90 % (28.74 %), 4.69 % (89.20 %) than the existing algorithm, respectively. Conclusions All these results demonstrated that eRFSVM was a better classifier in predicting both EP300 based and FAMTOM5 RNAs based enhancers. Electronic supplementary material The online version of this article (doi:10.1186/s41065-016-0012-2) contains supplementary material, which is available to authorized users.

Background
Transcription regulation in human genes is a complex process. Systematic and precise identification of these regulatory DNA elements, especially enhancers [17] is a prerequisite to understand gene expression in both healthy and diseased cells [9]. More and more studies indicate that mutations in enhancers are associated with human diseases, such as cancers [7,15], cardiovascular diseases [21] and immunological diseases [27].
Enhancers increase the transcriptional output in cells manifesting distinct properties, which are summarized as follows: (a) enhancers are distal regulatory elements, which may locate 20kb or further away from transcription start sites, (b) they can activate gene transcription by recruiting transcription factors (TFs) and their complexes, (c) they may be enriched with chromatin modifications, such as monomethylation of histone H3 lysine 4 (H3K4me1) and the acetylation of histone H3 lysine 27 (H3K27ac), (d) they can initiate RNA polymerase II transcription, producing a new class of non-coding RNAs called enhancer RNAs (eRNAs) [1], (e) they are tissue specific and merely conservative functioning in different spaces and stages. Thus, single experimental validation of them seems to be a time-consuming and costly task. Predicting enhancers based on conservation analysis of genomic sequences also doesn't work well [23]. With the development of high-throughput sequencing technologies, the advanced computational tools make this task possible in the big data era.
Machine learning algorithms [31] were used to predict enhancers with chromatin immune precipitation sequencing (ChIP-Seq) datasets [10,30], such as the chromatin modification loci and the TF binding sites (TFBs) [24,31]. Single classifiers used supervised learning algorithms, e.g., CSI-ANN [13] introduced an artificial neural network approach; RFECS [25] identified enhancers with RF; ChromaGenSVM [12] applied SVMs with a Genetic Algorithm (GA) to optimize the parameters of SVMs; EnhancerFinder [11] used multi-kernel SVMs to predict enhancers in the eukaryotic genome; DEEP [18] used an algorithm combined SVMs with ANN including the components of DEEP-ENCODE and DEEP-FANTOM5. Some classifiers mentioned above, such as CSI-ANN, ChromaGenSVM, REFCS and EnhancerFinder used ChIP-Seq datasets as features, and were strongly relied on EP300, which was a transcriptional coactivator and could activate gene transcription by combining with TFs, considering EP300 binding sites as enhancers. However, EP300 binding sites are very possible enhancer sites, but not the 100 % real ones [19]. DEEP used the same datasets in predicting EP300 based enhancers as the classifiers mentioned above. It firstly used FANTOM5 datasets as training enhancers, which were reconstructed from enhancer RNAs (eRNAs) datasets using Cap Analysis of Gene Expression (CAGE) [14]. However, it used DNA sequence features as features not ChIP-Seq datasets in DEEP-FANTOM5. It used SVMs as base classifiers to train datasets from single tissues or cell lines and it used ANN as a main classifier to make the final decision combining the results of the base classifiers. It has been proved that getting the global optimum of ANN is a NP-hard problem. The simple implementing algorithm, back-propagation algorithm, is a heuristic algorithm, which is easy to trap in local optimal solution [26]. Thus, the weakness of DEEP was obvious. In the layer of algorithm, the predicting result was not the global optimum; in the layer of datasets, it didn't use effective features in predicting eRNAs based enhancers.
In this study, we built a hybrid classifier eRFSVM including eRFSVM-ENCODE and eRFSVM-FANTOM5. We used RF as base classifiers [3,4], which was a fast and easy-paralleled algorithm good at dealing with unbalanced datasets. For eRFSVM, we could get the global optimum when making the final decision for both RF and SVMs algorithms were P problems [29].
In the process of data pre-processing, to reduce unbalanced ratio, we used a sub-sampling algorithm, and combined it with the k-means method, comprehensively considering the running time for the program and the loss of information in samples. The base classifiers trained datasets from single tissues or cell lines with the RF algorithm, which used 60 % of the datasets for training and the remaining 40 % for testing. With the testing results of base classifiers, we built the main classifier with SVMs. For eRFSVM-ENCODE, we trained datasets of cell lines like Gm12878, Hep, H1-hesc and Huvec, and then tested datasets of Hela and K562 cell lines, with enhancers identified by transcriptional coactivator EP300 binding sets as labels. For eRFSVM-FANTOM5, we trained on datasets of blood, lung, liver, kidney, and then tested on the datasets of adipose, with eRNAs based enhancers as labels.

The ENCODE datasets
The National Human Genome Research Institute (NHGRI) launched a public research consortium named ENCODE, the Encyclopedia of DNA Elements [6], in September 2003, to carry out a project to identify all functional elements in the human genome sequence. The datasets generated by next-generation sequencing technologies or software prediction in the project were available at https://genome.ucsc.edu/ENCODE/, including raw data, alignment data and peak calling data in different kinds of formats. The datasets were used for the studies, such as TF binding sites, histone modifications. In this study, we used the datasets for training and testing in the classifier of eRFSVM-ENCODE from ChIP-Seq experiments of different cell lines. The peak calling datasets of ChIP-Seq experiments from Gm12878, Hep, H1-hesc, Huvec, Hela and K562 in the form of broadpeak were used as features including eleven histone modifications, H2AFZ (Variant of H2A), H3K27ac (Detects of acetylation), H3K27me3 (Detects of trimethylation in Lysine 27), H3K36me3 (Marks of actively transcribed regions), H3K4me1 (Associated with enhancers), H3K4me2 (Marks of promoters and enhancers), H3K4me3 (Associated with active promoters), H3K79me2 (Marks of transcriptional transition regions), H3K9ac (Marks of promoters in chromatin regions), H3K9me3 (Associated with silenced chromatin) and H4K20me1 (Associated with active and accessible regions). Gm12878, Hep, H1-hesc, Huvec, Hela and K562 were cell lines from blood, liver, embryonic stem cell, blood vessel, cervical cancer and blood, respectively. The EP300 datasets were used as labels for the classifiers. The human genome sequences were downloaded from Ensembl (Homo_sapiens.GRCh38.dna).

The FANTOM5 datasets
The sra or bed format of ChIP-Seq datasets of lung, blood, liver, kidney and adipose including H3K4me1, H3K4me3, H3K9ac, H3K9me3, H3K27ac, H3K27me3 and H3K36me3 from NIH Epigenome Roadmap [2] were used as features for training. The ChIP-Seq of the same tissue were from different kinds of cells and different phases of life. Datasets of kidney were from fetal and adult; datasets of adipose were from mesenchyme and gland; datasets of lung were from left lung and right lung in the fetal; datasets of blood were from K562 and CD4+; datasets of liver were from adult. The FANTOM5 project [1] examines how our genome encodes the fantastic diversity of cell types that make up a human. Using Cap Analysis of Gene Expression (CAGE), studies in FANTOM5 project had mapped the sets of transcripts, transcription factors, promoters and enhancers active in the majority of mammalian primary cell types and a series of cancer cell lines and tissues. Enhancer labels were obtained from FANTOM5 consortium publicly available at http://enhancer.binf.ku.dk/Pre-definedtracks.html.

Data normalization for histone modifications
Firstly, to process datasets from NIH Epigenome Roadmap, we transformed data in the form of sra [16] to fastq by a software from NCBI. With bwa, we transformed fastq data to sam data. With samtools, we transformed sam data to bed data. Secondly, we used MACS [32] for sharp peak calling and SICER for broad peak calling [22,28]. ChIP-Seq peaks counted in a 200, 400, 600, 800, 1000-bp window were output.
The resolution of preprocessed histone modification data was 200bp. We used the signal value and normalized it in the formula below to train the datasets.

Reduction of imbalanced data ratio
Unbalanced datasets will lead to machine learning bias that classification will tend to put a sample prediction for a majority class sample, which will lead to a lower recognition rate for minority class samples. According to the priori data, enhancers accounts for about 2 % of the genome data as a minority class [19]. So the recognition of minority class samples is the key point of this issue. We used sub-sampling combined with the k-means to get negative samples from the whole genomes and made the ratio between positive and negative datasets as 1:10 [12] to reduce the imbalanced ratio.

Base classifier algorithm (The random forests algorithm)
RF [3,4] are ensemble of decision trees, which are based on information gain, the computation formula are presented as: The step of RF can be represented as: (a) use bootstrap to extract k samples from the original training sets with N samples for k times, (b)establish k decision trees, (c) vote according to the classification results of all decision trees, the voting results called confidencescore can be described as: RF overcomes the limitation of over-fitting of decision trees, with advantage of fast and simple, tolerance to noise and abnormal value. In this study, we used the signal values of histone modifications in each sample as features and computed confidence-score of all decision trees. The confidence-score was used as a feature for the main classifier.

Main classifier algorithm (The support vector machines algorithm)
Recently, the support vector machines (SVMs) [8] is an algorithm for classification or regression, which aims to separate the positive data and negative data through a hyperplane with datasets mapped to high dimensions with maximal margin. The algorithm tries to minimize structural risk and is less vulnerable to the over-fitting problem. Given training vectors x i ∈ R n , i = 1,…, n, in two classes, and a vector of label y ∈ R m such that y i ∈(−1,1) [20], SVMs solves a quadratic optimization problem by means of a kernel function like polynomial and radial basis function (RBF). The principle of SVMs can be described as follows: For any testing instance x, the decision function (predictor) is described as follows: , the kernel function, to train with SVMs. The RBF kernel is used in our experiments: With the RBF kernel, there are two parameters to be determined in the SVMs: C and γ. To get the good generalization ability, we conduct a validation process to decide parameters. The procedure is described as the following: (a) consider a grid space of (C, γ) with C ∈ {0,2,4,…,50} and γ∈{0,2,4,…,200}, (b)for each parameter pair (C, γ) in the search space, conduct 10-fold cross validation (CV) on the training sets, (c) choose the parameter pair (C, γ) that leads to the lowest CV balanced error rate, (d)use the best parameter pair to create a model as the predictor.

Implementing eRFSVM-ENCODE
Firstly, with the normalized features from Gm12878, Hep, H1-hesc, Huvec, we built base classifiers with RF algorithm separately, using 60 % of each datasets for training and the rest 40 % for testing. RF was not sensitive to input parameters, thus, we just used the default parameters for each classifier. We calculated the precision, F-score of each base classifiers. With the results of base classifiers, we used SVMs to merge the testing results of the four single classifiers. With the confidence-score multiplying the F-score of each classifier as features of main classifier, we used 10-k fold cross-validation. SVMs was sensitive to the parameters, so we used the Gaussian kernel as default and searched for the best C and γ with the grid search algorithm. With base classifiers training on datasets from Gm12878, Hep, H1-hesc, Huvec, we

Implementing eRFSVM-FANTOM5
We trained the datasets from blood, lung, kidney, liver and tested them on the adipose as the same framework ( Fig. 1) that eRFSVM-ENCODE used. The computational time for model training on a specific cell line was 10 min and training on the four datasets merged with SVMs for 1 h and testing on adipose for 5 min.

Performance evaluation of classifiers
The trained classifiers return confidence scores between 0 and 1 for a combined histone modification profiles. These scores are then transformed to a binary state indicating 'enhancer' or 'not enhancer' by choosing a cutoff. For each combination of profiles, the existence of regulatory element is considered positive (P) or negative (N) otherwise. True (T) means that the predicted functional states are enhancers, and false (F) implies otherwise. The notations of TP, FP, TN and FN combined these labels to return the number of each class. The performance evaluation of classifiers is made according to the following formulas: The predicted confidence scores are transformed into binary predictions by using different cut-offs yielding sensitivity and specificity over the entire score range. ROC plots can well evaluate the performance of classifiers, which display the FP (1-specificity) values on the x-axis, and the TP (sensitivity) values on the y-axis. ROC plots show the direct relationship between the FP and TP rates. The total AUC (area under the curve) for the ROC plot is used to measure the prediction performance of this method.

Performance of eRFSVM-ENCODE
With the histone modification datasets and EP300 datasets of cultured cell lines in broadpeak format downloaded from ENCODE, we discretized the positive datasets with 200bp as a unit and used sub-sampling [5] and k-means algorithms to get the negative datasets (Additional file 1: Table S1).
For the training steps, the best performed base classifier was hesc, with precision, recall and F-score of 84.53 %, 83.03 % and 83.78 %, respectively. For eRFSVM-ENCODE, we found that the precision, recall and F-score were 92.16 %, 90.70 % and 91.43 %, respectively, which meant that the hybrid classifier fitted better than the base classifiers (Additional file 1: Table S2).
When using the classifiers to test on K562 datasets (Table 1), among the base classifiers, GM12878 classifier showed the highest precision (84.39 %); huvec classifier showed the highest recall (6.34 %), F-score (11.76 %) and accuracy (69.79 %). When using classifiers to test on hela datasets, among the base classifiers, hep classifier showed the highest precision (30.24 %) and F-score (6.05 %); GM12878 showed the highest recall (5.47 %) and accuracy (99.33 %). For the hybrid classifier eRFSVM, when testing on K562 datasets, the precision, recall, F-score and accuracy were 83.69 %, 4.92 %, 9.29 % and 69.50 %, respectively, with precision higher than hep and huvec, recall higher than GM12878 and hesc, F-score higher than GM12878 and hesc, accuracy higher than GM12878 and hesc. When testing on hela datasets, the precision, recall and F-score of eRFSVM-ENCODE were 15.35 %, 0.38 % Fig. 1 The overview of eRFSVM (Different RF classifiers are made as base classifiers and SVMs classifier is made as main classifier) and 0.75 %, with precision higher than GM12878, hesc and huvec, recall higher than hep and huvec, F-score higher than hep and huvec. The ROC curves could comprehensively evaluate the performance of classifiers (Figs. 2 and 3), with higher the AUC value, the better performance of the classifier. The AUC values of GM12878, hep, hesc, huvec and eRFSVM-ENCODE are 0.6049, 0.5369, 0.5385, 0.5444 and 0.6357 for K562 datasets, respectively; 0.4650, 0.6106, 0.5415, 0.5474 and 0.6165 for hela datasets, respectively. eRFSVM-ENCODE showed the highest AUC value in both K562 and hela datasets, which meant the hybrid classifier was of better performance.

Performance of eRFSVM-FANTOM5
The datasets from NIH Epigenome Roadmap were of different formats. With softwares (see Materials and Methods), we made the peak datasets of different tissues (Additional file 1: Table S3), positive and negative datasets for training of these classifiers (Additional file 1: Table S4). For the training steps, the best performed base classifier was liver. The mean values of precision, recall and F-score of these base classifiers were 75.42 %, 53.90 % and 61.79 %, respectively. For eRFSVM, we found that the precision, recall and F-score were 90.59 %, 48.16 % and 62.22 %, respectively (Additional file 1: Table S5). The F-score of eRFSVM was higher than that of blood, lung and kidney, which meant eRFSVM fitted better than these base classifiers.
When using the classifiers to test on adipose datasets (Table 2), among the base classifiers, lung classifier showed the highest precision (83.72 %); kidney classifier showed the highest recall (47.31 %), F-score (57.41 %) and accuracy (93.55 %). For the hybrid classifier eRFSVM, when testing on adipose datasets, the precision, recall, F-score and accuracy were 86.17 %, 36.06 %, 50.84 % and 93.38 %, respectively, with precision higher than all the base classifiers, recall higher than blood, liver and lung, F-score higher than blood, liver and lung, accuracy higher than blood, liver and lung.
The AUC value of blood, liver, lung, kidney and eRFSVM-FANTOM5 are 0.6026, 0.5808, 0.5707, 0.5411 and 0.6344 for adipose datasets, respectively. eRFSVM-FANTOM5 showed the highest AUC value with better performance than base classifiers (Fig. 4).   (89.20 %) using DEEP-FANTOM5, respectively. To compare the features and algorithms in the function of classifiers with eRFSVM-FANTOM5 separately, we built two classifiers in the following. To compare the effects of different features for classifiers, we used the same algorithm (RF-SVMs) for the classifiers (Tables 2 and 4). DNA sequence features, such as the frequencies of di-nucleotide, tri-nucleotide, tetranucleotide, single base were used to make the feature matrix (Additional file 1: Table S6). For the base classifiers, the mean values of precision, recall, F-score were 73.02 %, 29

Discussion
Previous studies [29] demonstrated that hybrid algorithms could well improve the performance of classifiers, which could be well interpreted in the layer of statistics. A learning algorithm could be considered to search the best performed parameters in the hypothesis space (H), which was near to the real whole space (f). However, the training datasets were not sufficient to the whole real space. Thus, the optimum found in the hypothesis space was very likely not the best in the whole space. The algorithms were easy to trap in local optimum. By combining these hypothesis using hybrid algorithms, classifiers could average these results, leading to much more close to the optimum in the whole real space. In our study, with AUC values to evaluate the performance of classifiers, when testing on the K562, hela and adipsoe datasets, the values of hybrid classifier were 0.6357, 0.6165 and 0.6344, higher than all the base classifiers, verifying that hybrid classifiers were performing better than single classifiers.
The previous methods used EP300 datasets as criteria of enhancers. For eRFSVM-ENCODE, the performance in predicting K562 was better than that in hela. Hela was generated from cervical cancers, much more different from Gm12878 (blood), Hep (liver), H1-hesc (embryonic stem cell) and Huvec (blood vessel), but K562 was from blood, of the same tissue of Gm12878. With more similar cell lines, it would have the better predicting performance. Thus, combining cell lines from much more kinds of tissues would strengthen the predicting performance. We found that F-score of these classifiers was nearly 80 %. However, when testing on other cells, the average F-score of these algorithms was just about 10 %, indicating that the generalization performance of the classifiers were poor. EP300 and enhancers were associated but the relevance was not the same in different tissues, which was one of the reasons leading to the previous classifiers with low generalization performance.
Interpreting enhancers in a variety of angles with biological meaning was an important clue and basis for predicting enhancers with compute methods [19]. Therefore, from a biological perspective, looking for labels with stronger relevance with enhancers was important to build the effective classifiers. FANTOM5 consortium identified enhancers in different tissues and organs in transcription levels. DEEP-FANTOM5 used FANTOM5 enhancers as labels to predict new enhancers. However, DEEP-FANTON5 only used gene sequence features, such as CpG islands, G + C content to create feature matrixes. The average precision of the classifiers was only about 30 % (Additional file 1: Table S8). In this study, we used the ChIP-Seq datasets of different tissues and organs from NIH Epigenome Roadmap as feature matrixes, the average precision rate was about 70 % (Additional file 1: Table S5), which was 2.1 times that of DEEP-FANTOM5. When using eRFSVM-FANTOM5 to predict the adipose datasets, the F-score was 50.84 %, which was close to the F-scores training in blood, kidney and lung. These classifiers performed better compared with classifiers using EP300 datasets as labels.
There are some shortcomings of eRFSVM. Firstly, in the FANTOM5 datasets, only part of the tissues and organs had ChIP-Seq datasets in Roadmap database, therefore, eRFSVM-FANTOM can only predict enhancers in part of tissues and organs. Classifiers suitable for more tissues and cell lines with better generalization performance still need to be developed. Secondly, our knowledge of eRNAs was limited and not all enhancers generate eRNA, thus, enhancers not generating eRNAs may be missed in eRFSVM-FANTOM5. Thirdly, a web based user-friendly tool should be developed in the future. ChIP-Seq datasets were better features of classifiers in predicting FANTOM5 enhancers. When using the the same algorithm (RF-SVMs), the precision, F-score and accuracy were 86.17 %, 50.84 % and 93.38 % using ChIP-Seq features, increasing 2.7 fold (23.61 %), 57.35 % (32.31 %), 6.68 % (87.53 %) comparing with using DNA sequence features, respectively.
In conclusion, we provided a better classifier eRFSVM with higher generalization performance both in ENCODE datasets and FANTOM5 datasets comparing with DEEP and other state-of-art classifiers. However, in the FANTOM5 datasets, only part of the tissues and organs had ChIP-Seq datasets in Roadmap, therefore, eRFSVM-FANTOM can only predict enhancers in part of tissues and organs. Classifiers suitable for more tissues and cell lines with better generalization performance still need to be developed.

Additional file
Additional file 1: Table S1. The datasets for training of cell lines from ENCODE. Table S2. Results training on four cell lines.