A review on the public benchmark databases for static keystroke dynamics <s> Datasets Characterization <s> Abstract The test is based on the maximum difference between an empirical and a hypothetical cumulative distribution. Percentage points are tabled, and a lower bound to the power function is charted. Confidence limits for a cumulative distribution are described. Examples are given. Indications that the test is superior to the chi-square test are cited. <s> BIB001 </s> A review on the public benchmark databases for static keystroke dynamics <s> Datasets Characterization <s> In this paper, we propose a method to realize a classification of keystroke dynamics users before performing user authentication. The objective is to set automatically the individual parameters of the classification method for each class of users. Features are extracted from each user learning set, and then a clustering algorithm divides the user set in clusters. A set of parameters is estimated for each cluster. Authentication is then realized in a two steps process. First the users are associated to a cluster and second, the parameters of this cluster are used during the authentication step. This two steps process provides better results than system using global settings. <s> BIB002 </s> A review on the public benchmark databases for static keystroke dynamics <s> Datasets Characterization <s> Our daily interactions are based on recognizing people. As information technology advances, our interactions will be accomplished through newly designed interfaces, with a vital element being the automated process of recognition of a persons real identity: Biometrics. In recent years, biometrics has moved from using fingerprints, to many different methods of assessing human physical and behavioural traits. This useful guide introduces a new performance evaluation framework designed to provide complete coverage of performance evaluation of major biometric systems. Two key areas are presented: The first gives a unique snapshot of the performance evaluation of various biometrics within a common evaluation framework (databases and protocols). The second presents a state-of-the-art benchmarking evaluation framework for use in future performance evaluations comprised of open-source reference systems. Features and benefits: Includes a Foreword by renowned biometric expert Prof. Anil K. Jain, Michigan State University, USA Introduces the concept of reproducible results and comparative evaluations Presents an extensive benchmarking methodology Describes reference algorithms in detail, with a link to source code Biometrics Reference and Evaluation Toolkit provided via http://share.int-evry.fr/svnview-eph/ Implements this methodology throughout all chapters Offers readers a tool for a better evaluation of their algorithms by giving them the instruments to fully reproduce the benchmarking (reference) experiments (software, database, experimental protocols) Provides a global perspective with a common evaluation scheme and methodology Examines current research results with this framework Consolidates results from publicly available databases, using well defined protocols This unique text/reference combines state-of-the-art research with practical tools for researchers, practitioners and graduates in the field of Biometrics and Pattern Recognition, and with its comprehensive coverage, it will prove an indispensable resource and frequently used tool. Key topics BioSecure Benchmarking Methodology Biometric Performance Evaluation Iris Recognition Fingerprint Recognition Hand Recognition Online Handwritten Signature Verification Text-independent Speaker Verification 2D and 3D Face Recognition Talking Face Verification <s> BIB003
After having described the meta-data present in the datasets, it is interesting to analyze their content. Table 4 and Figure 1 present the information regarding the template ageing, while Figure 2 summarises the correlation between the various criteria. We observe the following points: • The Pearson correlation factor between entropy and text complexity is 0.89. So, it would be better to only use the entropy which is well defined in comparison to the text complexity. • Passwords are totally different between datasets, as well as their complexity and difficulty. This is a good point, which can allow testing various factors depending on these values. There is almost no correlation between the entropy or the text complexity with the typing difficulty (resp. 0.28 and 0.15). So the "strong" passwords for standard password authentication are not necessarily "strong" passwords for KD. • There is a huge impact on the number of sessions used in the dataset and the degradation of the EER. This clearly shows that keystroke dynamics authentication systems are subject to template ageing and this point should be systematically treated in any industrialised system. Table 5 and Figure 1 give the information related to the performance of each dataset. The baseline classifier we use is trained using user's gallery samples (no need of impostors' samples). It computes a distance score as explained in BIB002 . The reference of user i of dataset j consists of θ j i = (µ j i , σ j i ) with µ j i and σ j i , the mean and standard deviation of the gallery's samples. When there are several sessions, we use the first session for training, otherwise we use half of 14 the samples. The distance computation method between the biometric reference θ j i and a query sample q j k is the following one: BIB003 with N the size of the feature vector (it depends on the size of the password). We can observe the following points: • EER with the baseline classifier is really different between the datasets. Its value varies from 10% to 42%, so we can observe a relative difference of 76% between the best (BA) and the worst (EB) dataset with the same authentication method. This emphasizes the importance of using various datasets in order to validate algorithms. We could also test several algorithms because one algorithm can be more efficient on a specific dataset which corresponds to specific constraints well managed by the algorithm. • The decreasing of performance over time is linked to the number of sessions (the Pearson correlation coefficient between the number of session and the degradation of EER is 0.73). This emphasizes two different aspects: -The typing changes over time are significants. -Once again, it is mandatory to evaluate KD methods with datasets captured under several sessions. • The Pearson correlation coefficient between the inconsistency property and the EER degradation is about 0.901 except on WEBGREYCB (the dataset with one password per user), and is about 0.55 with WEBGREYCB. This asserts the interest of this measure in order to control the performance of user recognition (remember this measure has been created in order to ask users to create stable patterns). However, it is less true when each user has its own password. Figures 3, 4 and 5 respectively present the distribution of the uniqueness, discriminability and inconsistency values of users for each dataset. They have been computed after normalising each dataset of score using the zscore method (zscore(s j k ) = s j k −µ j σ j , with s j k the sample to normalize, and µ j and σ j the mean samples and standard deviation sample estimated on the whole dataset). It would be interesting to compare the obtained values with the ones provided by the creators of these indices, but sadly, the dataset they used is not public. The confusion matrix of the p-value of a Kolmogorov-Smirnof test BIB001 between the different distribution of the datasets is also presented in order to analyse if the distribution of values is different between the datasets (values close to 0 mean the distributions are different). We can note a correlation for datasets EA, EB and several subdatasets of G for the uniqueness, D0, D1, D2 and several subdatasets of G for the discriminability and D0, D1, D2 and EA, EB, EC and the subdatasets of G for the inconsistency. Thus acquisition protocol, or the involvement of the user influence these metrics. Figure 6 presents the ROC curve of each system. The biometric reference of each user is computed using some of its samples (the number is shown in Table 3 ). The other samples are used to make verification comparisons. Analysing the global performance on the ROC curves, we see in figure 6 that a large part of the area available for the ROC curves (from the worst at the middle line to the best at the top left corner) is used to display the complete set of ROC curves of each dataset. This figure is sufficient by itself to prove that it is mandatory to use several datasets to test new authentication methods. Otherwise researchers could develop methods which work well only on their own dataset and not on other ones. Figures 7a and 7b respectively present the AUC and EER evolution over sessions of each dataset providing several sessions. The first session is used to build the references of each user. The samples of the other sessions are used to compute the performance of their session against the biometric reference of the first session. So, three sessions are needed before being able to compute evolution over time. We make the following observations: • The performance on all datasets, except D2 (Pressure with the password "pr7q1z"), decreases over time, even when the number of samples is quite small. It shows the necessity to take into account biometric reference ageing or instability over time. An additional analysis (probably with another dataset) would be necessary in order to understand why we observe a different behaviour on one dataset. • The decreasing speed of performance is not the same for all the datasets. As a conclusion, individuals and session delays are important factors in this decrease (each dataset has been captured with sessions separated by various amount of time).