RaspiReader: An Open Source Fingerprint Reader Facilitating Spoof Detection

We present the design and prototype of an open source, optical fingerprint reader, called RaspiReader, using ubiquitous components. RaspiReader, a low-cost and easy to assemble reader, provides the fingerprint research community a seamless and simple method for gaining more control over the sensing component of fingerprint recognition systems. In particular, we posit that this versatile fingerprint reader will encourage researchers to explore novel spoof detection methods that integrate both hardware and software. RaspiReader's hardware is customized with two cameras for fingerprint acquisition with one camera providing high contrast, frustrated total internal reflection (FTIR) images, and the other camera outputting direct images. Using both of these image streams, we extract complementary information which, when fused together, results in highly discriminative features for fingerprint spoof (presentation attack) detection. Our experimental results demonstrate a marked improvement over previous spoof detection methods which rely only on FTIR images provided by COTS optical readers. Finally, fingerprint matching experiments between images acquired from the FTIR output of the RaspiReader and images acquired from a COTS fingerprint reader verify the interoperability of the RaspiReader with existing COTS optical readers.

I. INTRODUCTION
O NE of the major challenges facing biometric technology is the growing threat of presentation attacks, commonly referred to as spoofing 1 [1]. In this paper, we are interested in fingerprint presentation attacks in which a hacker intentionally assumes the identity of unsuspecting individuals, called victims here, through stealing their fingerprints, fabricating spoofs with the stolen fingerprints, and maliciously attacking fingerprint recognition systems with the spoofs into identifying the hacker as the victim 2 [2], [3], [4], [5].
The need to prevent presentation attacks is paramount due to the monumental costs and loss of user privacy associated with spoofed systems. Consider for example India's Aadhaar program which (i) provides benefits and services to an ever growing population of over 1.17 billion residents through fingerprint and/or iris recognition [6], [7] and (ii) facilitates electronic financial transactions through the United Payments Interface (UPI) [8]. Failure to detect presentation attacks on Fig. 1: Prototype of RaspiReader: two images (b, (i)) and (b, (ii)) of the input fingerprint (a) are captured. The raw direct image (b, (i)) and the raw, high contrast FTIR image (b, (ii)) both contain useful information for spoof detection. Following the use of (b, (ii)) for spoof detection, image calibration and processing are performed on the raw FTIR image to output a high quality, 500 ppi fingerprint for matching (b, (iii)).
the Aadhaar system could cause the disruption of a commerce system affecting untold numbers of people. Also consider the United States Office of Biometric Identity Management (US OBIM) which is responsible for supporting the Department of Homeland Security (DHS) with biometric identification services specifically aimed at preventing people who pose a risk to the United States from entering the country [9]. Failure to detect spoofs on systems deployed by OBIM could result in a deadly terrorist attack 3 . Finally, almost all of us are actively carrying fingerprint recognition systems embedded within our personal smart devices. Failure to detect spoof attacks on smartphones [11] could compromise emails, banking information, social media content, personal photos and a plethora of other confidential information.
In an effort to mitigate the costs associated with presentation attacks, a number of presentation attack detection techniques involving both hardware and software have been proposed 3 Border control fingerprint recognition systems have already been shown to be vulnerable to presentation attacks as far back as 2012 when a journalist successfully spoofed the fingerprint recognition system at the Hong Kong-China border [10]. Fig. 2: Fingerprint images acquired using the RaspiReader. Images in (a) were collected from a live finger during a single acquisition. Images in (b) were collected from a spoof finger during a single acquisition. Using features extracted from both raw image outputs ((i), direct) and ((ii), high-contrast FTIR) of the RaspiReader, our spoof detection algorithms are able to discriminate between live fingers and spoof fingers. In particular, because the images in (i) and (ii) contain color information, discriminative color local binary patterns (CLBP) can be extracted for spoof detection. The raw FTIR image output of the RaspiReader (ii) can be post processed (after spoof detection) to output images suitable for fingerprint matching. Images in (c) were acquired from the same live finger (a) and spoof finger (b) on a COTS 500 ppi optical reader. The close similarity between the two images in (c) qualitatively illustrates why current spoof detectors are limited by the low information content, processed fingerprint images output by COTS readers.
While hardware and software spoof detection schemes provide a reasonable starting point for solving the spoof detection problem, current solutions have a plethora of shortcomings. As noted in [18], [19], [20] most hardware based approaches can be easily bypassed by developing very thin spoofs ( Fig. 3 (a)), since heartbeat, thermal output, and blood flow can still be read from the live human skin behind the thin spoof. Additionally, some of the characteristics (such as odor and heartbeat) acquired by the hardware vary tremendously amongst different 4 Several fingerprint vendors have developed hardware spoof detection solutions by employing multispectral imaging, infrared imaging (useful for sub-dermal finger analysis), and pulse capture to distinguish live fingers from spoof fingers [12], [13]. 5 Raw fingerprint images are "processed" (such as RGB to grayscale conversion, contrast enhancement, and scaling) by COTS readers to boost matching performance. However, useful spoof detection information (such as color and/or minute textural abberations) is lost during this processing. human subjects, making it very difficult to build an adequate model representative of all live subjects [18], [19].
Spoof detection software solutions have their own limitations. Although the LiveDet 2015 competition reported state-of-the-art spoof detection software to have an average accuracy of 95.51% [32], the evaluations were limited to spoofs fabricated with only six material types. Furthermore, the spoof detection performance at desired operating points such as False Detect Rate (FDR) of 0.1% was not reported, and very limited evaluation was performed to determine the effects of testing spoof detectors with spoofs fabricated from materials not seen during training (cross-material evaluation). In the limited cross material evaluation that was performed, the rate of spoofs correctly classified as spoofs was shown to drop from 96.57% to 94.20% [32]. While this slight drop in accuracy seems promising, without knowing the performance at field conditions, namely False Detect Rate (FDR) of 0.1% on a larger collection of unknown materials, the reported levels of total accuracy should be accepted with caution. Chugh et al. [31] pushed state-of-the-art fingerprint spoof detection performance on the LiveDet 2015 dataset from 95.51% average accuracy to 98.61% average accuracy using a CNN, but they also demonstrated that performance at strict operating points dropped significantly in some experiments 6 . Additionally, many other studies have reported up to a threefold increase in error when testing spoof detectors on unknown material types [33], [34], [35].
Because of the less than desired performance of spoof detection software to adapt to spoofs fabricated from unseen materials, studies in [36], [37], and [38] developed openset recognition classifiers to better detect spoofs fabricated with novel material types. However, while these classifiers are able to generalize to spoofs made with new materials better than closed-set recognition algorithms, their overall accuracy (approx. 85% -90%) still does not meet the desired performance for field deployments.
Given the limitations of state-of-the-art fingerprint presentation attack detection (both in hardware and software), it is evident that much work remains to be done in developing robust and generalizable presentation attack detection solutions. We posit that one of the biggest limitations facing the most successful spoof detection solutions to date (such as use of textural features [35] and CNNs [29], [30], [31]), is the processed COTS fingerprint reader images used to train spoof detectors. In particular, because COTS fingerprint readers output fingerprint images which have undergone a number of image processing operations (in an effort to achieve high matching performance), they are not optimal for fingerprint spoof detection, since valuable information such as color and textural aberrations is lost during the image processing operations. By removing color and minute textural details from the raw fingerprint images, spoof fingerprint impressions and live fingerprint impressions (acquired on COTS optical readers) become very similar (Fig. 2 (c)), even when the physical live/spoof fingers used to collect the respective fingerprint impressions appear very different (Fig. 3).
This limitation inherent to many existing spoof detection solutions motivated us to develop a custom, optical fingerprint reader, called RaspiReader, with the capability to output 2 raw images (from 2 different cameras) for spoof detection. By mounting two cameras at appropriate angles to a glass prism (Fig. 4), one camera is able to capture high contrast FTIR fingerprint images (useful for both fingerprint spoof detection and fingerprint matching) ( Fig. 2 (ii)), while the other camera captures direct images of the finger skin in contact with the platen (useful for fingerprint spoof detection) ( Fig. 2 (i)). Both images of the RaspiReader visually differentiate between live fingers and spoof fingers much more than the processed fingerprint images output by COTS fingerprint readers ( Fig. 2 (c)). RaspiReader's two camera approach is similar to that which was prescribed by Rowe et al. in [12], [16] where both an FTIR image and a direct view image were acquired using different wavelength LEDs, however, the commercial products developed around the ideas in [12], [16] act as a proprietary black box outputting only a single processed composite image of a collection of raw image frames captured under various wavelengths. As such, fingerprint researchers cannot implement new spoof detection schemes on the individual raw frames captured by the reader. Furthermore, unlike the patented ideas in [12], RaspiReader is built with ubiquitous components and open source software packages, enabling fingerprint researchers to very easily prototype their own RaspiReader, and further customize it with new spoof detection hardware.
By utilizing the novel hardware of the RaspiReader, we are able to develop robust fingerprint presentation attack detection algorithms. In particular, because both image outputs of the RaspiReader are raw and contain useful color information, we can extract discriminative color local binary patterns  [39], [40] from each of the image outputs. Our experimental results demonstrate that the color local binary patterns from each image contain complementary information such that when the features are fused together and passed to a binary SVM classifier, state-of-the-art spoof detection performance can be achieved. Additionally, by calibrating and processing the FTIR image output of the RaspiReader (post spoof detection), we demonstrate that RaspiReader is not only interoperable with existing COTS optical readers but is also capable of achieving state-of-the-art fingerprint matching accuracy. Finally, RaspiReader is cost effective (approx. $165 in total component costs), built with off-the-shelf components (Table 1), and can be easily assembled and operated by those with little expertise in hardware, enabling easy integration and extension of this research (such as adding additional hardware specifically for spoof detection).
More concisely, the contributions of this research are: • An open source, easy to assemble, cost effective fingerprint reader, called RaspiReader, capable of producing fingerprint images useful for spoof detection and that have sufficient quality for fingerprint matching. The custom RaspiReader can be easily modified to facilitate presentation attack detection studies. • A customized fingerprint reader with two cameras for image acquisition rather than a single camera. Use of two cameras enables robust fingerprint spoof detection, since we can extract features from two complementary, information rich images instead of processed grayscale images output by traditional COTS optical fingerprint readers. • The implementation of a new fingerprint spoof detection schema through the extraction of highly discriminative fingerprint spoof detection features (CLBP) using the raw images of the RaspiReader. Our algorithm delivers stateof-the-art spoof detection accuracy.

II. RASPIREADER CONSTRUCTION AND CALIBRATION
In this section, the construction of the RaspiReader using ubiquitous, off-the-shelf components is explained. Additionally, the steps for calibrating and processing the raw FTIR fingerprint images of the RaspiReader for fingerprint matching are prescribed.

A. Construction of RaspiReader
The construction of RaspiReader adheres to the following process. First, an outer casing is electronically modeled using Meshlab (Fig. 5) [41] and subsequently fabricated using a high resolution 3D printer [42]. Next, the components enumerated in Table 1 are assembled together according the the specifications of the schematic diagram shown in Figure 4. The exact dimensions of assembly are not critical, however, depending on the dimensions used, the focal lengths of the two cameras need to be adjusted to focus on the finger surface as it makes contact with the acrylic prism.
Because the Raspberry Pi only has a single camera connection port, a camera port multiplexer is used to enable the use of multiple cameras on a single Pi [43]. Using the Raspberry Pi GPIO pins, the code available in [43], and the camera multiplexer, one can easily extend the Raspberry Pi to use multiple cameras. Note that an alternative approach that has been suggested for enabling multiple cameras on the Raspberry Pi is to attach one camera to the Pi's camera connection port, and a separate USB web-cam to the Raspberry Pi USB port. This method was experimented with, however, the frame Fig. 4: Schematic illustrating RaspiReader functionality. Incoming white light from three LEDs enters the prism. Camera 2 receives light rays reflected from the fingerprint ridges only (light rays are not reflected back from the valleys due to FTIR). This image from Camera 2, with high contrast between ridges and valleys can be used for both spoof detection and fingerprint matching. Camera 1 receives light rays reflected from both the ridges and valleys. This image from Camera 1 provides complementary information for spoof detection. rate of the USB camera is significantly reduced on the Pi due to the latency in loading images from the USB port to the Pi's graphics card. As such, using the camera multiplexer is recommended.
Once the components of Table 1 have been assembled, and the camera port multiplexed for two cameras, open source python libraries [43] can be used to acquire two images from the fingerprint reader (one raw FTIR fingerprint image and another raw direct fingerprint image).

B. Fingerprint Image Processing
In order for the RaspiReader to be used for spoof detection, it must also demonstrate the ability to output high quality fingerprint images suitable for fingerprint matching. As previously mentioned, the RaspiReader performs spoof detection on non-processed, raw fingerprint images. While these raw images are shown to provide discriminatory information for spoof detection, they need to be made compatible with processed images output by other commercial fingerprint readers.  In this image, the checkerboard pattern is not frontalized to the image plane. Therefore, destination coordinate pairs are defined on a frontalized checkerboard (b) such that perspective transformation parameters can be estimated to map (a) into (b). These transformation parameters are subsequently used to frontalize fingerprint images acquired by RaspiReader for the purpose of fingerprint matching. Therefore, after spoof detection, the RaspiReader performs (1) image enhancement operations and (2) image transformations on the raw high contrast, FTIR image frames in order to output high fidelity images compatible with COTS optical fingerprint readers (Fig. 7).
1) Fingerprint Image Enhancement: Let a raw (unprocessed) FTIR fingerprint image from the RaspiReader be denoted as F T IR raw . This raw image F T IR raw is first converted from the RGB color space to grayscale (F T IR gray ) Fig. 7: Processing a RaspiReader raw FTIR fingerprint image into a 500 ppi fingerprint image compatible for matching with existing COTS fingerprint readers. (a) The RGB raw FTIR image is first converted to grayscale. (b) Histogram equalization is performed on the grayscale FTIR image to enhance the contrast between the fingerprint ridges and valleys. (c) The fingerprint is negated so that the ridges appear dark, and they valleys appear white. (d),(f) Calibration (estimated using the checkerboard calibration pattern in (e)) is applied to frontalize the fingerprint image to the image plane and down sample (by averaging neighborhood pixels) to 500 ppi in both the x and y directions.
( Fig. 7 (a)). Then, in order to further contrast the ridges from the valleys of the fingerprint, histogram equalization is performed on F T IR gray (Fig. 7 (b)). Finally, F T IR gray is negated so that the ridges of the fingerprint image are dark, and the background of the image is white (as are fingerprint images acquired from COTS readers) (Fig. 7 (c)).
2) Fingerprint Transformation: Following the aforementioned image processing techniques, the RaspiReader FTIR fingerprint images are further processed by performing a perspective transformation (to frontalize the fingerprint to the image plane) and scaling to 500 ppi (the native resolution of the RaspiReader images, approx. 900 ppi, was obtained using a calibration target.) (Fig. 7 (d),(f)).
A perspective transformation is performed using Equation 1, where x and y are the source coordinates, x and y are the transformed coordinates, (a, b, c, d, e, f, g, h) is the set of transformation parameters, and λ = gx + hy + 1 is a scale parameter. In this work, we image a 3D printed checkerboard pattern ( Fig. 7 (e)) to define source and destination coordinate pairs (Fig. 6) such that the transformation parameters could be estimated. Once the perspective transformation has been completed, the image is downsampled (by averaging neighborhood pixels) to 500 ppi ( Fig. 7 (f)). Upon completion of this entire fingerprint reader assembly and image processing procedure, the RaspiReader is fully functional and ready for use in both presentation attack detection and subsequent fingerprint matching.

III. LIVE AND SPOOF FINGERPRINT DATABASE
CONSTRUCTION To test the utility of the RaspiReader for presentation attack detection and its interoperability for fingerprint matching, a database of live and spoof fingerprint impressions was collected for performing experiments. This database is constructed as follows.
Using 7 different materials ( Fig. 3 (a)), 66 spoofs were fabricated. Then, for each of these spoofs, 10 impressions were captured at varying orientations and pressure on both the RaspiReader (Rpi) and a COTS 500 ppi optical FTIR fingerprint reader (COT S A ). The summary of this data collection is enumerated in Table 2.
To collect a sufficient variety of live finger data, we enlisted 15 human subjects with different skin colors ( Fig. 3 (b)). Each of these subjects gave 5 finger impressions (at different orientations and pressures) from all 10 of their fingers on both the RaspiReader and COT S A 7 . A summary of this data collection is enumerated in Table 3.
In addition to the images of live finger impressions and spoof finger impressions we collected for conducting spoof detection experiments, we also verified that for spoofs with optical properties and/or 3D structural properties too far from that of live finger skin (Fig. 8), images would not be captured by the RaspiReader. These "failure to capture" spoofs are therefore filtered out as attacks before any software based spoof detection methods need to be performed.

IV. EXPERIMENTAL PROTOCOL AND RESULTS
Given the database of live and spoof fingerprint images collected on both COT S A , and the prototype RaspiReader, a number of spoof detection experiments are conducted to demonstrate the value of the raw images from the RaspiReader for training spoof detectors. In addition, experiments are 7 Acquiring a fingerprint on RaspiReader involves the same user interactions that a COTS optical reader does. A user simply places their finger on an acrylic prism. Then, LEDs illuminate the finger surface and images are captured from both cameras over a time period of 1 second (Fig. 1). The only difference in the acquisition process between a COTS reader and RaspiReader is that RaspiReader acquires two complementary images of the finger in contact with the acrylic platen from two separately mounted cameras. The spoof materials used to fabricate these spoofs were in accordance with the approved materials by the IARPA Odin project [1]. 2 The spoofs are all of unique fingerprint patterns.

A. Spoof Detection Experiments
In all of our spoof detection experiments, we use a binary linear SVM for classification 8 , namely to classify images into live vs. spoof fingerprints. A binary linear SVM informally is the hyperplane with the "largest" margin separating the positive and negative samples (here spoof and live impressions). More formally, this optimal decision boundary can be found by solving the following unconstrained optimization 8 Other classifiers were experimented with such as K-Nearest Neighbor and Support Vector Machines with various kernels (RBF and Polynomial), however, linear SVM provided the best results in our experiments. equation (Eq. 2) given a set of instance label pairs (x i , y i ), i = 1, ..., n, x i ∈ R d , y i ∈ {−1, +1} and the quadratic hinge loss function ξ 2 (w; x i ; y i ) defined in Equation 3 [44].
In practice, finding a "large margin" (hyperplane) which perfectly separates the positive and negative samples is infeasible due to noise in the training data. As such, linear SVMs are relaxed from a "hard margin" SVM (no training sample misclassifications allowed by the margin) to a "soft margin" SVM (some training sample misclassifications allowed by the margin). The number of misclassifications allowed is dependent upon the user defined parameter C in Equation 2. If C is set very large, then a large penalty incurs for allowing any training sample misclassifications. Consequently, a very strict decision boundary is chosen which may not generalize well to the testing data. In an effort to better generalize the SVM to the testing data, the value of parameter C can be reduced. However, if C is reduced too much, the selected decision boundary may begin to deviate from the optimal decision boundary for the testing data. Thus, selecting the appropriate value of C is a tradeoff that needs to be empirically determined for the dataset on hand. In our case, we use five-fold cross validation to select the value of C (from the list of 10 −5 10 −4 ... 10 4 10 5 ) such that the best performance is achieved in different folds. In our experiments, the best classification results were achieved with C = 10 2 .
Experiment 1: LBP Features From COTS Acquired Images. With a classifier selected, we begin our experiments using grayscale processed fingerprint images acquired from COT S A (Fig. 9) for training and testing. The purpose of this initial experiment is to set a baseline performance for spoof detection when classifiers are trained on grayscale images from COTS fingerprint readers. In the subsequent experiments, the images from the RaspiReader are used for training classifiers. This first experiment is then used as a reference point in demonstrating the improvements in classification error when RaspiReader images are used for extracting features rather than images from a 500 ppi COTS optical fingerprint reader.
Given the reported success of local textural features for spoof detection in comparison to anatomical or physiological features [35], we use the very prevalent grayscale and rotation invariant local binary patterns (LBP) as features for this baseline experiment [45]. LBP features are extracted by constructing a histogram of bit string values determined by thresholding pixels in the local neighborhoods around each pixel in the image. Since textures can be present at different spatial resolutions, parameters R and P are specified in LBP construction to indicate the length (in pixels) of the neighborhood radius used for selecting pixels and also the number of neighbors to consider in a local neighborhood. Previous studies have shown that more than 90% of fundamental textures in an image can belong to a small subset of binary patterns called "uniform" textures (local binary patterns containing two or fewer 0/1 bit transitions) [45]. Therefore, in line with previous studies using local binary patterns for fingerprint spoof detection, we also employ the use of uniform local binary patterns. More formally, extracting a uniform local binary pattern for a pixel g c and a set of P neighborhood pixels g 0 g 1 ... g P −1 g P selected at radius R from g c (Fig. 10) as defined in [45] is performed by: where, and Finally, let H P,R be the uniform local binary pattern histogram constructed by binning the local binary patterns for each pixel g c in an image according to equations 4, 5, and 6 using P and R as parameters to the LBP feature extractor. In our experiments, we extract H 8,1 , H 16,2 , and H 24,3 in order to capture textures at different spatial resolutions. These histograms (each having P + 2 bins) are individually normalized and concatenated into a single feature vector x of dimension 54.
Using the live and spoof finger images from COT S A as described in Tables 2 and 3, five-fold cross validation is used to partition training and testing splits in the data. When partitioning each fold of the live fingers, the finger impressions from 12 subjects are used for training, and the finger impressions from the remaining 3 subjects are used for testing. When partitioning spoof finger impressions, 80% of the impressions for each spoof material are used for training, and the other 20% for testing.
Next, a feature vector x is extracted from every image in the training and testing splits and used to train and test a binary linear SVM according to the parameters previously described. Using the proposed classification scheme and training/testing splits, we report a mean (over five folds) True Detection Rate (TDR) of 58.33% at a False Detection Rate (FDR) of 0.1%. Additionally, we report the standard deviation for the TDR (over five folds) to be 37.69%.
Not only is the performance of this baseline classifier unacceptably low, it is also not robust, as evidenced by the extremely high standard deviation of the TDR over the five folds. Furthermore, it demonstrates that 500 ppi processed grayscale fingerprint images from COTS FTIR fingerprint readers have relatively low information content for discriminating spoof fingers from live fingers. This serves as motivation for developing the RaspiReader to output two raw RGB images for spoof detection. In the subsequent sections, we perform experiments on the RaspiReader images and demonstrate that the performance of classifiers trained with RaspiReader images far surpass the performance of classifiers trained on processed grayscale images. Experiment 2: Color LBP Features Extracted from RaspiReader Raw FTIR Images. In this experiment, we make use of the information rich, raw FTIR images from the RaspiReader (Fig. 11) for presentation attack detection. As with the experiment 1, we again pursue the use of LBP textural features given their superior performance in comparison with other types of features that have been tried for fingerprint spoof detection [35]. However, since the raw FTIR images from the RaspiReader contain color information, rather than using the traditional grayscale LBP features, we employ the use of color local binary patterns (CLBP). Previous works have shown the efficacy of CLBP for both face recognition and face spoof detection [39], [40]. However, because fingerprint images from COTS fingerprint readers are grayscale, CLBP features have, to our knowledge, not been investigated for use in fingerprint spoof detection until now.
Unlike traditional grayscale LBP patterns, color local binary patterns (CLBP) encode discriminative spatiochromatic textures from across multiple spectral channels [39]. In other words, CLBP extracts textures across all the different image bands (HSV used here) in a given input image. More formally, given an input image I with K spectral channels, let the set of all spectral channels for I be defined as S = {S 1 , ..., S K }. Then, the CLBP feature x can be extracted using algorithm 1. Note that H P,R (g c , g N ) returns a normalized histogram Algorithm 1 Extraction of Color Local Binary Patterns x ← x H 8,1 (Si, Sj) H 16,2 (Si, Sj) H 24,3 (Si, Sj) end for end for return x of local binary patterns using g c as the image channel that the center (thresholding) pixels are selected from, and g N as the image channel from which the neighborhood pixels are selected from in the computation of LBP as defined in equations 4, 5, and 6. Also, indicates vector concatenation. This extraction of CLBP features is also visually presented in Figure 12.
Using the live and spoof finger images from the RaspiReader's FTIR camera as described in Tables 2 and 3, five-fold cross validation is again used to partition training and testing splits. Then, we (i) convert each image to the HSV 9 color space and (ii) extract 486 dimensional CLBP features from each image to train/test a binary linear SVM. With this proposed classification scheme and training/testing data, we report a mean (over five folds) True Detection Rate (TDR) of 90.75% at a False Detection Rate (FDR) of 0.1%. Additionally, we report the standard deviation for the TDR (over five folds) to be 8.88%.
To better understand which presentation attacks the spoof detector has a difficult time detecting using the proposed CLBP features and also how well the detector classifies live fingers, we report the mean (over five folds) correct detection rate for live fingers and spoofs (of every material) individually (Table 4). From these results, we can see that that most difficult 100.0% ± 0.0 Silver Coated Ecoflex 100.0% ± 0.0 Crayola Model Magic 100.0% ± 0.0 1 The mean and standard deviation is reported for the correct detection rates over 5-folds, using 0.5 as a threshold (where scores range from 0-1). spoofs to detect using the CLBP features extracted from the RaspiReader raw FTIR images are ecoflex spoofs. This is to be expected since the ecoflex spoofs are quite thin. As such, much of the human skin color texture passes through the spoof material, making the spoof impression of similar color to live finger impressions (Fig. 15 (a)). We can also see from these results that the detector correctly classifies spoof fingers at a higher rate than live fingers. This is likely due to a small dataset of live fingers (only 15 human subjects provided live fingerprints) used during the training of the spoof detector. Given a larger number of live finger impressions to train on, the detection rates of live fingers will likely improve.
Overall, in comparison to the baseline classifier (trained on grayscale LBP features extracted from processed images from a COTS fingerprint reader), this classifier (trained on CLBP features extracted from raw FTIR images from the RaspiReader) achieves significant improvement (90.75% mean TDR compared to 58.33% mean TDR both at FDR of 0.1%). This huge jump in TDR from the baseline classifier at the same fixed FDR (0.1%) demonstrates the usefulness of the raw FTIR images from the RaspiReader for fingerprint spoof detection. In particular, because the raw FTIR images from the RaspiReader contain color information, we are able to extract color textural (CLBP) features which are much more discriminative than traditional grayscale textural features. Fig. 12: Illustration of the extraction of the color LBP features from a spoof fingerprint impression captured by the RaspiReader. First, the fingerprint images from the RaspiReader are converted from the RGB color space to the HSV color space. Then, LBP as defined in equations 4, 5, and 6 is applied to (i) each individual channel of an image (a, b, c), and (ii) across opponent channels (e.g. a center pixel is taken from one channel and the neighborhood pixels are taken from another channel)(d, e, f, g, h, i). Finally, each of these LBP histograms are normalized and concatenated into a single feature vector. Note, the green squares illustrate the location of a center pixel and the red circles illustrate the location of neighboring pixels during each LBP extraction.
However, a TDR of 90.75% still leaves much room for improvement, especially if the proposed spoof detection scheme were to be installed in the field where a 9.25% miss rate could prove very costly. Therefore, in the following experiment, we attempt to achieve even higher performance by using the direct image output of the RaspiReader for spoof detection. Experiment 3: Color LBP Extracted From RaspiReader Raw Direct Images. To perform this experiment, we essentially repeat the entire structure of Experiment 2. However, instead of extracting CLBP features from the raw FTIR image output of the RaspiReader, 486 dimensional CLBP features are instead extracted from the direct image output of the RaspiReader (Fig. 13).
Using the SVM classifier with the same parameters and 5fold cross validation training/testing data, we report a mean (over five folds) TDR of 92.88% at a FDR of 0.1% (a slight improvement over the results of Experiment 2). Additionally, we report the standard deviation for the TDR to be 10.97%. As in Experiment 2, we report the correct detection rates for live fingers and each individual spoof material separately to better determine how well the classifier correctly detects live fingers and each respective spoof type (Table 5).
From these results we can see that ecoflex spoofs are again the most difficult spoof type for RaspiReader to defend against since very thin clear spoofs allow much of the human skin The mean and standard deviation is reported for the correct detection rates over 5-folds, using 0.5 as a threshold (where scores range from 0-1).
color behind the spoof to pass through ( Fig. 15 (b)). Also, in these direct images, if the surface area of the spoof is smaller than the surface area of the finger presenting the spoof (Fig. 15  (b)), much of the peripheral surface area of the live finger will be included in the spoof images used to make a classification decision, confusing the classifier. In the future, we plan to solve this problem by collecting more spoof impressions from thin clear spoofs, enabling us to better learn the more subtle differences between thin clear spoofs and live fingers. We can also see from these results that using the direct images of the RaspiReader enables higher correct detection rates of live fingers than the raw FTIR images do. This is likely because the raw direct images contain less variability amongst the various fingers than the raw FTIR images do. As such, the small amount of live finger training data does not affect performance as much in Experiment 3 as it does in Experiment 2.
Finally, even at a TDR of 92.88% in this experiment, there is still room for improvement. Therefore, in the final spoof detection experiment, we extract features from both of the RaspiReader images from camera-1 and camera-2 and concatenate them together in an attempt to achieve even higher and more robust spoof detection performance. Experiment 4: Fusion of Color LBP Features. In this penultimate spoof detection experiment, we extract CLBP features from both image outputs of the RaspiReader (raw FTIR output and raw direct output). These two CLBP features are then fused into a single 972 dimensional feature vector and used to train a binary linear SVM.
Using this classification scheme, we report a mean (over five folds) TDR of 95.3% at a FDR of 0.1%. The standard deviation over the five folds was 6.05%. By fusing CLBP features extracted from both the raw FTIR output of the RaspiReader and the direct output of the RaspiReader, we boost the classification performance beyond the performance when only a single image output from RaspiReader is used for spoof detection. This indicates that using two cameras for fingerprint image acquisition and subsequent spoof detection is advantageous, since they both contain complementary and discriminative spoof detection information.
In summary, we have demonstrated in these spoof detection experiments that both raw images generated by our custom RaspiReader contain far more discriminative information for spoof detection than grayscale images output by COTS optical fingerprint readers. In addition, we have shown that these images contain complementary information such that when simple CLBP features extracted from each image are fused together, state-of-the-art spoof detection performance on known materials can be achieved. By showing that the RaspiReader images are far more useful for spoof detection than grayscale COTS optical images and by open sourcing the design of RaspiReader, we have removed what we posit to be one of the biggest limitations facing current state-of-the-art presentation attack detection schemes (namely low information processed images on which these detectors are trained). A full summary of these spoof detection experiment results is listed in Table  6. 1 These results are reported over 5-folds. 2 The LBP and CLBP feature extractions and SVM classifications were performed with a single CPU core on a Macbook Pro running a 2.9 GHz Intel Core i5 processor with 8 GB of RAM. In the future, we will parallelize the feature extractions and directly run the spoof detection algorithm on the Raspberry Pi controlling the RaspiReader.

B. Interoperability of RaspiReader
In addition to demonstrating the usefulness of the RaspiReader images for fingerprint spoof detection, we also demonstrate that by processing the RaspiReader FTIR images, we can output images which are compatible for matching with images from COTS fingerprint readers. Previously, we discussed how to process and transform a RaspiReader raw FTIR image into an image suitable for matching. In this experiment, we evaluate the matching performance (of 11,175 imposter pairs and 6750 genuine pairs) when using (i) the RaspiReader processed images as both the enrollment and probe images, (ii) the COT S A images as both the enrollment and probe images, and (iii) the COT S A images as the enrollment images and the RaspiReader processed images as the probe images. The results for these matching experiments are listed in Table 7. From these results, we make two observations. First, the best performance is achieved for native comparisons, where the enrolled and search (probe) image are produced by the same capture device. RaspiReader's native performance is slightly better than that of COT S A . This indicates that the RaspiReader is capable of outputting images which are compatible with state of the art fingerprint matchers 10 . Second, we note that the performance does drop slightly when conducting the interoperability experiment (COT S A is used for enrollment images and RaspiReader is used for probe images). However, the matching performance is still quite high considering the very 10 We use the Innovatrics fingerprint SDK since we recently acquired this matcher, and it is shown to have high accuracy in the NIST FpVTE evaluation [46]. stringent operating point (FAR = 0.1%). Furthermore, studies have shown that when different fingerprint readers are used for enrollment and subsequent verification or identification, the matching performance indeed drops [47], [48], [49]. Finally, we are currently investigating other approaches for processing and downsampling RaspiReader images to reduce some of the drop in cross-reader performance.

V. CONCLUSIONS
We have designed and prototyped a custom fingerprint reader, called RaspiReader, with ubiquitous components. This fingerprint reader is both low cost and easy to assemble, enabling other researchers to easily and seamlessly develop their own novel fingerprint presentation attack detection solutions which use both hardware and software. By customizing RaspiReader with two cameras for fingerprint image acquisition rather than one, we were able to extract discriminative color local binary patterns (CLBP) from both raw images which, when fused together, enabled us to match the performance of state of the art spoof detection methods (CNN). Finally, by processing the raw FTIR images of the RaspiReader, we were able to output fingerprint images compatible for matching with COTS optical fingerprint readers.
In our ongoing work, we plan to integrate specialized hardware into RaspiReader such as Optical Coherence Tomography (OCT) for sub-dermal imagery, IR cameras for vein detection, or microscopes for extremely high resolution images of the fingerprint. Because the RaspiReader uses ubiquitous components running open source software, RaspiReader enables future integration of these additional hardware components. In addition to the integration of specialized hardware, we also plan to use the raw, information rich images from the RaspiReader to pursue one-class classification schemes for fingerprint spoof detection. In particular, we posit that the RaspiReader images will assist us in modeling the class of live fingerprint images, such that spoofs of all material types can be easily rejected. Anil K. Jain is a University distinguished professor in the Department of Computer Science and Engineering at Michigan State University. His research interests include pattern recognition and biometric authentication. He served as the editor-in-chief of the IEEE Transactions on Pattern Analysis and Machine Intelligence and was a member of the United States Defense Science Board. He has received Fulbright, Guggenheim, Alexander von Humboldt, and IAPR King Sun Fu awards. He is a member of the National Academy of Engineering and foreign fellow of the Indian National Academy of Engineering.


Similarity Based Genre Identification for POS Tagging Experts & Dependency Parsing

POS tagging and dependency parsing achieve good results for homogeneous datasets. However, these tasks are much more difficult on heterogeneous datasets. In (Mukherjee et al. 2016, 2017), we address this issue by creating genre experts for both POS tagging and parsing. We use topic modeling to automatically separate training and test data into genres and to create annotation experts per genre by training separate models for each topic. However, this approach assumes that topic modeling is performed jointly on training and test sentences each time a new test sentence is encountered. We extend this work by assigning new test sentences to their genre expert by using similarity metrics. We investigate three different types of methods: 1) based on words highly associated with a genre by the topic modeler, 2) using a k-nearest neighbor classification approach, and 3) using perplexity to determine the closest topic. The results show that the choice of similarity metric has an effect on results and that we can reach comparable accuracies to the joint topic modeling in POS tagging and dependency parsing, thus providing a viable and efficient approach to POS tagging and parsing a sentence by its genre expert.

Introduction
POS tagging and dependency parsing can be performed reliably on homogeneous datasets such as the Penn Treebank. However, both POS taggers and parsers are often used on out-of-domain data, which generally results in considerably lower accuracies. Domain adaptation provides a poten-tial for improving out-of-domain accuracy if annotations in the target domain are available. In our work, we assume a more flexible approach in which POS tagging and parsing of an individual sentence is performed by the closest genre expert where the genres are identified automatically prior to training.
In (Mukherjee et al., 2016(Mukherjee et al., , 2017, we approach this problem by creating genre/domain experts using topic modeling: We use Latent Dirichlet Allocation (LDA) (Blei et al., 2003;Blei, 2012) for an unsupervised clustering of sentences into topics. We assume that these topics correspond to genres; previous experiments (Mukherjee et al., 2016) have shown that the topic modeler models the split into genres in a very similar way to the original split, with error rates around 2%. We then train one expert per topic. I.e., we train the expert on all the training sentences that were assigned to the corresponding topic. During testing, we assign test sentences to topics, which means that they are POS tagged and parsed by the corresponding expert. We tested the approach on an artificial, heterogeneous corpus, consisting of a balanced mix of sentences from the WSJ portion of the Penn Treebank (financial news) (Marcus et al., 1994) and from the GENIA corpus (biomedical abstracts) (Tateisi and Tsujii, 2004). For POS tagging, we show a moderate increase in performance over a competitive baseline of training on the full training set, and a considerable increase for dependency parsing.
However, in our previous approach, assigning test sentences to the relevant training topic experts is handled in the simplest possible way: We perform topic modeling on the combination of training and test data. Since topic modeling clusters the data but does not create a predictive model, we cannot assign sentences to topics after the initial clustering. This means that each time a new test sentence is encountered, the topic modeler is rerun to consistently determine the appropriate genres across training and test sentences. In order to avoid retraining, we propose to use similarity estimation techniques to determine which genre the test sentence belongs to. In this setup, we only create the experts once, and then assign new sentences to genres in an asynchronous fashion. For a new test sentence, we evaluate the similarity of the test sentence to the sentences of a genre and then assign it to the expert with the highest similarity score. We investigate a range of different techniques, based on 1) words closely associated with a topic by the LDA, 2) k-nearest neighbors, or 3) perplexity models to estimate the similarity.
Our results indicate that for word based similarity metrics, we reach results very similar to the joint topic modeling approach, thus proving the feasibility of an asynchronous approach. In the case of unigram-based perplexity, parsing accuracy even surpasses the joint modeling approach.
The remainder of the paper is structured as follows: Section 2 outlines our system architecture in greater detail, and section 3 discusses related work. In section 4, we describe the similarity methods, section 5 describes the setup for the experiments, and section 6 shows the results of our experiments. In section 7, we draw conclusions and discuss future steps.

Architecture
Following our previous work, we use LDA (Blei et al., 2003;Blei, 2012) to generate genres and train genre experts. But then, instead of having the test sentences clustered along with the training sentences, we assign test sentences to the genres via similarity metrics. The test sentences are consequently annotated by the corresponding genre expert. The complete architecture of our approach is shown in Figure 1.
For the current experiments, we use 2 topics, parallel to the 2 domains, and assign each training sentence via hard clustering to the genre for which LDA showed the highest probability 1 . We then test the different similarity metrics: We compute the similarity of a test sentence to the training sentences of the individual experts and then assign the sentence to the expert for tagging/parsing for which it has the highest similarity. We use the following similarity metrics: • Topic words from LDA: LDA does not only cluster sentences into genres, it also determines which words are highly correlated with each genre. Thus, we can utilize these words along with their probabilities for a specific genre. We sum over all genre words that we find in the sentence, weighted by their probability, and then assign the sentence to the topic that has the highest score.
• k-nearest neighbors: There exist a wide range of metrics to calculate similarity between two feature vectors. However, since we compare a sentence to a set of genre sentences, we decided to use memory-based classification using the k-nearest neighbors to classify each test sentence into the relevant class (or genre, in our case). This has the advantage over a pure similarity metric that we have a principled way of handling the comparison to a set of sentences. Additionally, we do not consider the whole search space, as we would if we used a centroid.
• Perplexity: Another obvious choice for determining the similarity of a sentence to a set of sentences is language modeling. We calculate the perplexity of a test sentence with regard to set of sentences of an expert. We then assign a sentence to the expert for which it has the lowest perplexity.

Related Work
Our work cannot exactly be called domain adaptation since we automatically determine a range of genres and then train experts per genre while domain adaptation starts from a general model and adapts it to a specific genre. However, the two research questions are closely related and generally face the same problems. Our work is closest to the work by Plank and van Noord (2011)  In their setting, a parser learns the domain differences and various statistics from being trained on datasets from multiple domains. Our approach is comparable to the extent that both approaches profit from a range of domains. Plank and van Noord (2011) adopt an approach where they build a highly specialized training set that is most similar to an out-of-domain document. They create a specialized expert each time the parser encounters a new document. Our approach is more general than Plank and van Noord's in that we do not create an expert for every new document, but rather create experts per genre. In contrast, our approach is more fine grained in that we assign individual sentences to genres rather than complete documents. Plank and van Noord (2011) use the topic distribution from LDA as features for determining the most similar training set. This is comparable to our approach of assigning test sentences to the proper genre (but not to the creation of genres). Domain adaptation has been studied more extensively in parsing than in POS tagging. For POS tagging, Blitzer et al. (2006) have a similar setup to ours, where they train on WSJ data and test on MEDLINE abstracts. They use structural correspondence learning by identifying "frequently occurring" pivot features which can appropriately represent source as well as target do-main. The problem of adapting to a new domain is compounded in cases where no adequate data from the target domain is available. Differences in annotation scheme between the source and target domain can pose additional challenges (Dredze et al., 2007). In our case, GENIA follows a similar annotation scheme as WSJ with a few differences in assigning POS tags to names.
Agreement based approaches and co-training have been employed for domain adaptation of POS tagging. In the agreement-based method adopted by Clark et al. (2003), a Markov model tagger and a maximum entropy tagger are used. For a sentence to be included in the training set, both the taggers have to reach a unanimous decision. Sagae and Tsujii (2007) apply a similar approach but using MaxEnt and SVM to simulate an iteration of co-training. Kübler and Baucom (2011) extended this further by demonstrating that an agreement in terms of word sequences rather than complete sentences is more robust and achieves better results.
In the CoNLL 2007 shared task on domain adaptation for dependency parsing, Attardi et al. (2007) adapt an error correction approach to revise mistakes caused by the base parser in the target domain. Kawahara and Uchimoto (2008) employ a single parser approach using a second order MST parser and combining labeled data from the known domain with unlabeled data of the new domain by simple concatenation and judging the efficacy of the resulting most reliable parses. Finkel and Manning (2009) devise a model for dependency parsing by using a hierarchical Bayesian prior based on the notion that different domains may have different features specific to each domain. Instead of applying a constant prior over all the parameters, a hierarchical Bayesian global is used.

Similarity Estimation
There are different ways of determining the similarity of a sentence to the sentences in a genre. We investigate methods based on the topic words from LDA, k-nearest neighbor approaches, and perplexity.

Topic Words from LDA
LDA provides a list of the words most closely associated with a topic and assigns a weight to each word. Thus, we can use the words that are highly correlated with each topic as good indicators for a sentence belonging to the genre represented by the topic. Additionally, we utilize the probabilities provided by LDA as weights to determine a word's contribution to the similarity. For this experiment, we select the top 50/100/200 words from each topic. I.e., we assume that these words can be considered to be the most representative words in their respective domain. Then, for each sentence, we check how many of those words occur in the current sentence and add up their weights. We then assign the sentence to the topic with the higher value. Since we only look at a small number of words, we have to consider the cases where a sentence does not contain any of the topic words. We resolve these cases by extending beyond the top words and considering all the words in the training set for each genre. If there is a tie in values, we assign the sentence randomly to one of the experts.

k-Nearest Neighbors
In this method, we perform k-nearest neighbor classification to assign test sentences to topic experts. We create a feature vector by using the top 50/100 words from each genre, as determined by LDA, and assigning the weights as values. We tune the classification parameters on the validation set. For the setting using 50 words, the highest accuracy corresponds to a setting with 7 nearest neighbors, Dice coefficient as the distance metric, and gain ratio for feature weighting. When we increase the number of words per genre to 100, we use Dice coefficient, gain ratio, and 3 nearest neighbors.

Perplexity-Based Similarity
As the third set of methods, we turn to language modeling and use perplexity as a measure to determine the similarity, i.e., we assign test sentence to the expert which has a lower perplexity. Perplexity is calculated based on unigrams, bigrams, or trigrams.

Dataset
We create our corpus manually by combining the Wall Street Journal (WSJ) (Marcus et al., 1994) section of the Penn Treebank and the GENIA corpus (Tateisi and Tsujii, 2004). This gives us an artificial balanced corpus for which we know to which genre a sentence belongs. While WSJ consists of newspaper reports, GENIA consists of biomedical abstracts from Medline.
For the WSJ corpus, we use the POS annotation and syntactic annotations from the treebank. The GENIA Corpus is annotated on different linguistic levels, including POS tags, syntax, coreference, and events, among others. We use GENIA 1.0 trees (Ohta et al., 2002) created in the Penn Treebank format 2 . Both treebanks are converted to dependencies using pennconverter (Johansson and Nugues, 2007).
Following our data split in Mukherjee et al. (2016Mukherjee et al. ( , 2017, we create a balanced dataset comprising 17 181 sentences from each corpus for the training set and 850 sentences for the test set. Since GENIA is rather small and since there is no standard data split for GENIA, we decided to extract the last 850 sentences for the test set, and the 850 sentences before that for the validation set. The remaining 17 181 sentences are used for training. For WSJ, we chose the same number of sentences for the training, validation, and test set, the training sentences are selected randomly from sections 02-21 and the validation and test sentences from section 22 and 23 respectively.

Baselines
In Mukherjee et al. (2016Mukherjee et al. ( , 2017, we have used two baseline cases: The first baseline considers the entire training set and does not employ any topic modeling. Since the topic experts have access to only a fraction of the data, a second and more comparable baseline consists of randomly distributing training and test sentences into sets that correspond in size to the genres. We use these baselines and add a third, which is more relevant to our current setting. In this case, we use the experts trained on the genres but then randomly assign test sentences to the experts. This allows us to gauge how important a correct assignment to the corresponding expert is.

Topic Modeling
Probabilistic topic modeling is a class of unsupervised algorithms which detects the thematic structure in volumes of documents (Blei, 2012). We use Latent Dirichlet Allocation (LDA), a generative probabilistic model that approximates the underlying hidden topical structure of a collection of texts based on the distribution of words in the documents (Blei et al., 2003). We use the topic modeling toolkit MALLET (McCallum, 2002). The topic modeler in MAL-LET implements Latent Dirichlet Allocation clustering documents into a predefined number of topics.

POS Tagging and Parsing
For part of speech tagging, we use the Markov model POS tagger TnT (Trigrams'n'Tags) (Brants, 2000). We use TnT mainly because of its speed and because it allows the manual inspection of the trained models (emission and transition frequencies). For the parsing experiments, we use the dependency parser of the MATE Tools 3 , a Java implementation of a graph-based parser (Bohnet, 2010). For the parsing experiments, we use gold POS tags.

Similarity Estimation
For the k-nearest neighbor estimation, we use the Tilburg Memory-Based Learner (TiMBL) (Daelemans et al., 2010). For the perplexity models, we derive n-grams of the training experts using Laplace smoothing, in NLTK (Bird et al., 2009), and compute the perplexity of the training experts to a test sentence.

Evaluation
We use the script tnt-diff that is part of TnT to evaluate the POS tagging results and the CoNLL shared task evaluation script 4 for evaluating the parsing results.

Experimental Results
Genre Assignment. We first investigate how well the different similarity metrics can assign the 3 code.google.com/p/mate-tools 4 http://ilk.uvt.nl/conll/software/eval.pl test sentences to the correct genre. I.e., we calculate accuracy in terms of whether a GENIA sentence is assigned to the GENIA genre, and a WSJ sentence to the WSJ genre. Table 1 shows the results of classification accuracy of using different similarity estimation techniques. The reference here is the joint LDA for training and test data, with an accuracy of 98.94%.
Perplexity based on unigrams reaches the highest accuracy, reaching an accuracy of 99.76%, thus surpassing the joint LDA. Surprisingly, using bior trigrams instead decreases accuracy by 15-20 points absolute. We assume that this is due to data sparsity since the language model is trained on a relatively small dataset. The second highest accuracy is reached by the methods based on topic words. Here, the number of words considered does not seem to make a significant difference. The k-NN approach performs at around 91%. These results indicate that using single words without context (i.e., the context in bi-and trigrams) provides the most reliable information. The language model has an additional advantage, potentially because it can smooth over unseen words.
POS Tagging. Table 2 shows the accuracies of the POS tagging experiments for different similarity metrics. We notice that assigning the test sentences randomly to genres has a detrimental effect, and we reach an accuracy of 91.36%, which is more than 5 points absolute lower than the random split baseline. This difference shows how important it is that sentences are assigned to the correct genre.
Perplexity based on unigrams and the method using topic words reach comparable accuracies to the original topic expert results based on the joint LDA clustering. These results follow the same   trends as the genre assignment accuracies, but the differences between the methods are smaller. The perplexity setting using unigrams does not only surpass the joint LDA scores but also all the baselines: by nearly 0.3 percent points for the full training set, by 0.5 percent points for the random split, and by 5 percent points for the random test assignment.
Dependency Parsing. Table 3 shows labeled and unlabeled attachment scores of the dependency parses. These results mirror the trends in the POS tagging experiments: Randomly assigning sentences to genres results in the lowest LAS score of 82.17%, and using perplexity based on unigrams reaches the highest LAS of 90.54%. This LAS is considerably higher than the full training baseline of 88.67%. Note that the differences between the accuracies based on different similarity metrics are considerably more pronounced than in the POS tagging experiments. This mirrors the trend that we have previously seen for the joint LDA assignment. It is also interesting to see that the results for all the topic word settings are the same. This is due to 5 sentences that did not contain any of the topic words and thus had to be randomly assigned to one genre. We now have a closer look at the two best settings, i.e., the perplexity experiment using unigrams and 50 topic words: We separate the sentences that were assigned to the wrong genre from the correctly assigned ones and evaluate them separately. For the unigram setting, 4 sentences were assigned to the wrong genre, for the 50 topic words, 41 sentences. The results are shown in Table 4. These results show that the sentences that were assigned to the wrong genre receive POS and dependency analyses with significantly lower accuracies, the difference to the correct ones ranging between 5 points absolute for POS tag-  Table 4: Results for POS tagging and dependency parsing when we separate incorrectly assigned sentences from correct ones.
ging and 10-14 points for parsing. This corroborates our findings that the correct assignment to a genre is of utmost importance, which also corroborates our conclusion that the genre experts model genre-specific information. If they did not, misassigning sentences would not have any impact.

Conclusion
Using topic modeling to create experts can be very beneficial, but this approach is only viable if we can assign new sentences asynchronously to genres without having to retrain the LDA to determine genres that include the new sentences. We have investigated similarity based methods for assigning the new sentences to genres. More specifically, we have investigated the following methods: 1) using topic words that LDA associates with a genre, 2) k-nearest neighbor models, and 3) perplexity in language models. A baseline that assigns test sentences randomly to genres performs poorly, thus showing that the correct assignment to genres is indispensable.
Our results show that the perplexity model based on unigrams surpasses the accuracy of a joint LDA model that assigns the sentences synchronously. For POS tagging, the accuracy of the unigram perplexity model is very close to that of the joint LDA. For parsing, the unigram perplexity model outperforms the joint LDA model. Using the 50 topic words to assign sentences to their genre reaches accuracies close to the best performing model. This shows that word-based methods are more robust in comparison to bigram and trigram methods, which should be able to profit from more context but also face data sparsity issues.
For the future, we plan to investigate models with a more dynamic mix of genres during the POS tagging and parsing process. I.e. rather than creating independent experts, we will investigate methods to create a POS tagger and parser that have access to expert views during each decision about the next POS tag or parsing step. We will also investigate whether we can integrate gold POS tags or dependency information into the topic modeling process, so that the topic modeler has access not only to the specialized lexical information but also to the linguistic information that it is ultimately tasked to distinguish.


extfieldsofstudy:
Biology
Computer Science
Mathematics

text:
Insulin Regimen ML-based control for T2DM patients

\begin{abstract} We model individual T2DM patient blood glucose level (BGL) by stochastic process with discrete number of states mainly but not solely governed by medication regimen (e.g. insulin injections). BGL states change otherwise according to various physiological triggers which render a stochastic, statistically unknown, yet assumed to be quasi-stationary, nature of the process. In order to express incentive for being in desired healthy BGL we heuristically define a reward function which returns positive values for desirable BG levels and negative values for undesirable BG levels. The state space consists of sufficient number of states in order to allow for memoryless assumption. This, in turn, allows to formulate Markov Decision Process (MDP), with an objective to maximize the total reward, summarized over a long run. The probability law is found by model-based reinforcement learning (RL) and the optimal insulin treatment policy is retrieved from MDP solution.

Introduction
Diabetes mellitus type 2 (T2DM) makes up for about 90% of all Diabetes cases [15], with increasing rates since as early as 1960 [9]. The goal of therapy for T2DM is to bring the average blood glucose (BG) as close to the normal range as possible, which can be done by various oral medications or/and insulin injections [12]. The exact dosage and frequency of the medication intakes are coordinated by blood glucose measurements normally being performed by designated glucose meters, see , e.g., [14]. The recent recommendations are such that T2DM patients are advised to measure their blood glucose level (BGL) at least 3 times a day [8].
In this work, we address individually adopted insulin regimen control. For this purpose, we harness tools from the area of Machine Learning (ML) and stochastic control theory. In particular, we build Markov Decision Process of blood glucose level. The solution to the defined MDP is expressed by a policy which assigns an action according to the measured BGL.
Note that we pose no contradiction to other known methods employed by medical care. The ML theory is agnostic to physiological processes, e.g., biochemical insulin impacts e.g. renal, hepatic effects of insulin resistance, effect of other possibly malfunctioning processes inside beta-cells, impairments of insulin pathways and so on. We merely taking a different approach, modeling the insulin treatment as a controllable stochastic process. In contrary to other known, including ML-based works, we exploit no mathematical or biological models which express chemical dependencies.
One of the well-known difficulties related to insulin regimen control is associated with short and ultradian insulin secretion peaks, which, at the recent time, are not well understood, [5]. The BGL is known to increase after meal intake and to decrease during fasting periods and after physical activity. However, the peaks render the BGL to vary according to the pattern which is hard to mathematically characterize.
In particular, patients with insulin resistance and especially those who had progressed to T2DM, have abnormal insulin oscillations, no first phase and diminished/scattered 2nd phase of insulin secretion out of 1 arXiv:1710.07855v1 [q-bio.QM] 21 Oct 2017 beta-cells. Even more important, these oscillations may not exactly follow glucose oscillations, like it is known to be in healthy people [5].
In order to characterize the BGL state of a patient, we assume continuous and bounded BG axes and perform discretization by dividing the entire region into finite BG states. For example, between 70 and 300 mg/dL (see, e.g., [6]). Each state refers to continuous region, such that any BG in this region is assigned to the corresponding state. The state can be augmented by additional individually measured parameters. In particular, we may also account for glucose absorption rate (GAT) and HbA 1c (glycosylated haemoglobin) values.
The patient's BGL, hence, is viewed as a stochastic process which varies according to physiological dynamics and according to the patient activities. This process is the controllable subject such that the control is applied by means of medications, e.g. insulin injections (IJ). The finesse of the discretization is set in a way to assure that the next state will solely depend on the previous state and the action, i.e., IJ, if any, taken in that state. While the BGL process is clearly continuously fluctuating through the time, the measurements are performed in discrete time slots. Hence, we view the state of the patient at certain time marks. We are interested in transition law, i.e. transition probabilities from one state, at some time mark t, to the next state which the patient sees (i.e, feels) right after the next measurement which occurs at time mark t + 1. This law is expressed by the transition probability from a state, associated with the corresponding BGL, given the set of actions taken by the patient in that state, to the next state.
The aforementioned state transition probabilities are learned by a model-based reinforcement learning (RL). The input data for the RL is taken from existing samples of measurements individually performed under effect of medication applied by physician administration. We assume that such samples exist for each individual patient. Alternatively, the individual can be assigned to a category of patients for which the initial insulin policy is already defined. This policy will be improved henceforth and individually adjusted by applying reinforcement learning on the individual's measurements. In this document, we show results from database open for purposes of academic usage [7].
The methods of ML were already addressed in [2] for the treatment of Type I diabetes Mellitis (T1DM). The paper [2] proposes linearizion of MDP solution. The model is based on high dependence of insulin and glucose levels. However, for T2DM this assumption is not necessarily true. T1DM treatment is facilitated by very frequent (e.g., every several minutes) BG measurements and actions, while T2DM measurements are normally performed several times a day. This makes T2DM control an appropriate candidate for the stochastic modeling by discrete MDP with discrete actions control. The prediction of BGL was addressed in [3] and references therein, giving a stronger emphasis to T1DM. Neural Network (NN) based model is presented in [16], aimed to predict BGL in T2DM patients. The training was performed on other patients for the purpose of the NN convergence. In this work, the crucial learning part is performed using individual data, with objective to individually adjust to each patient. Authors of this model suggest that convergence of NN by sample data from other patients may cause a bias and dependency on that specific data. (Note that we do use data samples from various patients for research phases of the algorithm presented in this paper, while adopting to the specific patient is fulfilled in the phase of on-line control.) Block-oriented Wiener modeling for BG prediction is done in [13].
Previous works which apply ML in the context of Diabetes employ a great deal of system parameters, which is especially helpful for the prediction. Our approach is control-oriented and is designed to completely hide the complex impact of multiple parameters behind the stochastic process of the BGL, which we consider as the main indication for the insulin control purpose. That is, we allow the assumption that BGLs' stochastic process embodies in itself the impact of all relevant physical parameters (including, in most extremely simplified cases, carbohydrates intakes and physical activities), hence we rely on learning the statistical properties of the process.
Using the states, actions and transition laws, we formulate Markov Decision Process (MDP), which aims to maximize the average reward over time. For this purpose we heuristically define a mapping which transfers the BGL state into reward, such that a positive reward is obtained for the healthiest states and negative rewards are obtained at undesired BG levels. We also introduce non-linear components which aim to fine for being in dangerous states, e.g., overly high BGL or hypoglycemia.
We formally define the MDP and describe the RL algorithm in the next section.

The probabilistic model
We start by defining the state space, next we define the reward function, the reward functional and the action space.

The measurement space
We consider blood glucose interval of gl min to gl max , where the lower bound means an extremely low BGL (possibly corresponding to hypoglycemia) while the upper bound means extremely high level of glucose in blood. The BGL finesse coefficient (BFC) defines the size of the interval which is mapped to the single BG state. For example for BF C = 1 and gl min = 70mg/dL, the lowest interval of [70, 71) corresponds to the lowest state. The set of all possible BGL forms the space: Denote the BGL-state by gl, where gl ∈ L, We allow to augment state space by additional measured parameters, e.g., glucose absorption rate (GAT), Similarly to BGL, we perform discretization for GAT by defining the bounds of the GAT axes ga min to ga max . The GAT finesse coefficient (GFC) is used in order to define the GAT-state: The measurement space M defines the set of all possible measurements of the patient. In the case both BGL and GAT can be measured at all times we have M = L × G. In the simplest scenario where the only component of the measurement space is BGL, we merely have M = L.

The daytime space
We differentiate the BGL process instantiations during the day at different measurement points. These points are expressed by daytime space, denoted by where T is the number of measurements per day. For simplicity we assume this number is constant. The detailed time is denoted by a couple {t, τ }. In this sense t ∈ {0, 1, · · · } represents the date, while τ ∈ T represents the time during the day, corresponding to one of the measurements. The absolute measurement points can be counted by denoting ι = t · T + τ . At any ι we consider ms(ι) ∈ M. 3

Carbohydrates intake history space
The carbohydrates finesse coefficient (CFC) is used in order to define the quantity of carbohydrates consumed at some time mark by a patient: In order to wisely incorporate the Carbohydrates intake history (CIH) we will assume that at each ι last H Such a structure allows for impact of the past carbohydrate intakes on one hand and preserves memoryless property on th other hand. That is, CH(ι) depends only on CH(ι − 1) and ch(ι).

Physical activity history space
We assume physical activities have a prolonged effect, similarly to CIH. The measurements of energy spent for a given activity can be done in metabolic equivalents, or METs, see, e.g., the table of translation of activity to METs in [4]. The physical activity finesse coefficient (PFC) is used in order to define the number of METs corresponding to the energy spent by an individual at some measurement (or in a period between two consecutive measurements): PH = {1, · · · , P H}, C = ph max − ph min P F C .
To define the Physical activity history (PAH), denote at time ι the set P H(ι) = {ph ι }, where ph i ∈ PH, ∀i ∈ [ι, ι − Y ], and Y is the size of the relevant history.

Patient activity space
We define the patient activity (PA) space by At any ι we consider pa(ι) ∈ PA. We reason we differ between PA space and the measurement space is summarized as follows • ph is directly known by a patient and can be foreseen by conducted activities. In contrary, ms is assumed to be purely stochastic and can be only measured.
• The insulin medication is applied in order to control the ms, i.e., the BGL, while pa may only effect the dosage.
• We aim to apply optimal control for given ms and ,optionally, for a given pa. The pa can be seen as the secondary means of control, however the maintaining certain level of activities is not a part of the insulin policy (see definition below).

Overall state space definition
We now ready to define the state space and the state. Denote Denote by s ι ∈ S, a state at absolute measurement point ι. That is, the state consists of BGL and other measurements according to definition of M, patient activities history according to definition of PA and daytime of ι, according to definition of T.
Note that in the simplest scenario where the only component of the measurement space is BGL and no patient activities are accounted for, we merely have S = L × T, and s = {gl, τ }, where gl ∈ L and τ ∈ T.
Also note, that even though ι already carries the information about τ , we will still sometimes explicitly specify τ , for the clarity.

Reward function
Denote the optimal (the most healthy) BGL by gl h . Define reward function r(s (t,τ ) ) which maps the BGL at s at some given time {t, τ } to the value which quantifies the quality of the healthiness of a patient with BGL at s (t,τ ) . The linear reward component, normalized to 1, is given by Define a subset of dangerous states L hyp ∈ L of being close to hypoglycemia by L hyp = {1, · · · , L c }, L c < L. Normally, L c is small. It defines the number of states with critically low BGL. Define the non-linear reward component which fines for being close to hypoglycemia where I is the standard indicator function. The joint instantaneous reward is given by r(g) = r l (g) − r n (g) .

Action Space
The action space consists of set of possible insulin related medical actions. We allow for finite number of treatments, in particular, the most known of them -short-acting (Actrapid-like) insulin preparations, intermediate-acting (NPH-like) and long-acting (Ultratard-like) insulin preparations. Denote the sets of possible dosages corresponding the short-acting, intermediate-acting and long-acting insulin medications. Note that the first values of the spaces a s 1 , a i 1 , a l 1 are equal to 0, meaning no mediation of that type is given. As long as the medications can be administered concurrently we assume that the action space is given by At each state s ∈ S, an action a ∈ A is chosen. At time mark ι, a (ι) is a triplet {a s (ι) , a i (ι) , a l (ι) }. 5

Transition Probabilities
The transitions between states occurs at every absolute measurement point according to transition probabilities given by p(s (ι+1) |s (ι) , a (ι) ). That is, the probability of getting to the next state s , given that the previous state was s, and action a was taken. Clearly, s p(s (ι+1) |s (ι) , a (ι) ) = 1, where the summation is over all possible states at ι + 1.
Clearly, these probabilities are unknown a priori. Hence, the objective is to statistically learn them using a patient's measurements, under previously applied insulin regime. This regime could be applied according to doctor's purely medical (i.e., non-mathematical) considerations. Once these probabilities are learned, the new, optimal insulin policy is derived.

Total reward functional
Define the total discounted reward, with positive discount factor γ < 1.
The discount factor γ has two important interpretations. The first one follows from the known analytical property which allows the sum in (1) to converge. The second one advocates the reasoning of "feeling better now" is more important than "feeling better in a distant future". Therefore, BGL rewards for the distant time marks are discounted with geometrically decreasing discount multiplicative. The alternative approach, which values equally the feeling throughout the day employs the following definition r(s (t,τ ) ). (2)

MDP and RL solution 3.1 The Bellman equation
Write the equation (1), for the given initial state s (0) as follows Denote by π a policy which selects action at each state. State s(1) is set at time mark t = 1, according to the transition probability from state s (0) , given action a π (s (0) ) = {a π,s (s (0) ), a π,i (s (0) ), a π,l (s (0) )}, which is chosen according to π.
Define the value function given an initial state: V (s (0) ) indicates the optimal total reward obtained for the BGL throughout time, when starting BGL was as in s (0) . We use next the dynamic programming (DP) principle. Expand (3) as follows Applying DP on the value function gives the Bellman equation (see e.g., Denote V the vector of all V (s). The Bellman equation (6) is solved by the value function, a result of a mapping from the state space to the space of the total discounted reward, which is complete metric space, denote it by V: equipped with a suitable metric V a − V b . Observe that (6) can be written with operator T , that is, V = T V . By a well-known result (see, e.g., [1]), operator T is strict contraction (i.e., α V a − V b < V a − V b , for some 0 < α < 1). Therefore, T has a fixed point (see, e.g. [11,Theorem V.18]). This fixed point is V and it constitutes the unique solution to the Bellman equation (6). The reasoning above means V is found by a well-known method of value iteration (see, again, [1]). That is, T is iteratively applied, starting with initial values of some U ∈ V, till the arbitrarily close convergence to the fixed point V .

Algorithm formulation
Observe that (6) can only be solved by using transition probabilities, as were defined in previous section. These probabilities constitute the dynamics of the stochastic process of BGL and the response to the actions, i.e. the insulin preparations. The straightforward approach to calculate them is by mere statistical learning over the existing vector of samples of measurements of BGL paired with the corresponding actions.
We formulate the complete Insulin Optimization Policy Algorithm next. See Figure 3.2 below for the complete formulation. The phase A is the initialization of the vector of statistics of ν(s , a, s) which counts all occurrences of being in state s, acting by applying a and passing to the next state s . The phase B comprises the learning phase where acting is done by initial policy π 0 , which normally taken from the same samples, and stems from doctor's heuristic consideration treating the individual patient. The output of this phase is the transition probabilities. These probabilities are exploited in the phase C to perform the value iteration and to retrieve the optimal policy π * . The number of iterations depends on the size of the vector V and is heuristically set, in order to achieve the desired level of convergence. Finally, phase D is the exploitation part, which exploits the latest found optimal policy π. The phase keeps tracking the transition occurrences and updates the policy every M transitions.
The final phase reflects possible changes in the BGL process properties through the time.

Optimal policy result
In this section we use data from [7], in order to demonstrate the activation of the algorithm described in the previous section. Both examples presented in this section are solely based on short vectors of several hundreds of measurements of BGL. Anything else, e.g., carbohydrates intakes and physical activities are presumed to be embodied in the BGL process. Clearly, the obtained policy is only a presumable outcome, as the learning was done over short-sized vectors. While the BGL (simply denoted as Glucose in the graphs below) scale is presented starting from minimal measurement and ending by maximal BGL measurement, not at all points in the BGL scale the policy any policy is indicated. The reason for this stems from the limited data size. In particular, the action outcome cannot be learned in the case no occurrence of any action for that specific BGL was ever recorded. For this reason the preliminary policy π 0 should be carefully selected. The best anticipated practice would be having an assessment of a particular individual and assigning them to the specific category of policies. Once the successful assignment is done, the correction is expected to be effective even if only limited period of individualized measurements is offered. We further elaborate in the Future Work section.

Future Work
The future work is aimed at two directions.
• Producing a set of categories of individuals. The individuals would be assigned into the finite number of policy sets. Each set will contain patients with resembling (but clearly not similar) insulin regime.  The patients may also have other resembling parameters, e.g., body weight, age, etc. The categorizing process will be done by both heuristic assignment and correlation study.
• The second direction relates tot the pure ML approach and is directed towards coping with the size of of the state-space. For this purpose, approach from approximate MDP (AMDP) will be introduced. See, e.g., methods described in [10,Chapter 7].