A review on the public benchmark databases for static keystroke dynamics <s> State of the Art <s> Gait recognition has become a popular new biometric in the last decade. Good recognition results have been achieved using different gait techniques on several databases. However, not much attention has been paid to get major questions: how good are biometrics data; how many subjects are needed to cover diversity of population (hypothetical or actual) in gait and how many samples per subject will give good representation of similarities and differences in the gait of the same subject. In this paper we try to answer these questions from the point of view of statistical analysis not only for gait recognition but for other biometrics as well. Though we do not think that we have a whole answer, we content this is the start of the answer. <s> BIB001 </s> A review on the public benchmark databases for static keystroke dynamics <s> State of the Art <s> In order to advance face recognition research, algorithm performance has to be measured and compared using a range of metrics and operating characteristics. While public challenges such as the NIST-sponsored FERET, FRVT, FRGC, and MBGC are helpful to gauge comparative performance and improvement for a particular scenario, they typically are not sufficient to fully characterise the strengths and weaknesses of the face recognition algorithm, thus researchers need to do additionally benchmarking independently. This paper provides: (1) a detailed review and categorisation of publicly available face biometrics benchmarks; (2) a discussion of metrics and performance factors to consider; (3) a proposal for a meta-face biometric benchmarking regime which suggests guidelines for benchmarking across multiple datasets to more fully characterise and quantify face recognition performance across various operating characteristics; and (4) a sample demonstration which compare the performance of a face recognition algorithm before and after inclusion of a face quality metric. <s> BIB002 </s> A review on the public benchmark databases for static keystroke dynamics <s> State of the Art <s> Understanding the difficulty of a dataset is of primary importance when it comes to testing and evaluating fingerprint recognition systems or algorithms because the evaluation result is dependent on the dataset. Proposed in this paper is a general framework of assessing the level of difficulty of fingerprint datasets based on quantitative measurements of not only the sample quality of individual fingerprints but also relative differences between genuine pairs, such as common area and deformation. The experimental results over multi-year FVC datasets demonstrate that the proposed method can predict the relative difficulty levels of the fingerprint datasets which coincide with the equal error rates produced by two matching algorithms. The proposed framework is independent of matching algorithms and can be performed automatically. <s> BIB003
Dataset characterization does not seem to be a research field deeply studied. It is a real issue for the biometric community, as the results of biometric authentication methods deeply depend on the chosen dataset. Mau et al. analyzed the characterization of face recognition systems BIB002 . They reviewed large public datasets for still images and videos and listed various points of differences between datasets or studies: (i) Does the evaluation uses a single image as reference, image sets or videos? (ii) Is it an open-set or a closed set evaluation? (iii) Is the capture controlled or uncontrolled? (iv) Which kind of protocol has been used (identification, verification, screening)? (v) Which evaluation metrics are used? All these parameters must be fixed in order to compare various algorithms. They argued that several evaluation measures must be used, because none of them can be adapted to all scenarios. It is also important to analyze the relationships between dataset size and accuracy, dataset size and processing speed, and memory usage of the face recognition algorithms. Li et al. have presented a method to characterize fingerprint datasets BIB003 . They compute a level of difficulty of a fingerprint dataset by doing a linear combination of various metrics: the samples quality, the relative rotation and deformation, and the common area between the reference and probe to compare. Three difficulty levels can be obtained (i) easy, (ii) normal, (iii) and difficult. They only used genuine mated pairs for computing the level of difficulty. Although their method is correlated to the Equal Error Rate (EER), we think that it is biased as it only analyses the facility of matching genuine samples, but not the difficulty for impostors to be accepted as genuine (which may explain why in two cases, EER and difficulty are not correlated). We think that EER is well correlated to this difficulty factor, most of the time because fingerprints of different users are rather different (which is less true for KD). Veres el al. presented a method for computing the minimal number of samples, users, and samples per user to create a dataset allowing to obtain representative results for a specified error rate BIB001 . The optimal values are obtained from a first dataset used to compute the selected error rate. It is then necessary to acquire another dataset, under the same conditions, in order to obtain representative results. Their work is illustrated on gait recognition, but can be generalized to other modalities. The humanID gait challenge presents a good overview of gait recognition performances  . HumanID proposed one gait dataset of 122 users and one baseline verification method, which is tested among 12 different experiments. The experiments were of various difficulties corresponding to different configurations (walking surface, camera angle, time difference between captures, shoes, carrying or not a bag). The baseline system performed quite well in the simplest scenarios. The variable factors had a different impact on the recognition performance. They also showed that normalizing the scores allows to improve the verification performance. The aim of the study is to allow researchers to test their own algorithms in the same conditions in order to improve them by comparison to the baseline system. Note that the dataset is larger in number of user than previous ones. Based on this state of the art, we propose a specific method to characterize keystroke dynamics datasets and its associated criteria.