text:
Classifying Emotions in Customer Support Dialogues in Social Media

Providing customer support through social media channels is gaining increasing popularity. In such a context, automatic detection and analysis of the emotions expressed by customers is important, as is identification of the emotional techniques (e.g., apology, empathy, etc.) in the responses of customer service agents. Result of such an analysis can help assess the quality of such a service, help and inform agents about desirable responses, and help develop automated service agents for social media interactions. In this paper , we show that, in addition to text based turn features, dialogue features can significantly improve detection of emotions in social media customer service dialogues and help predict emotional techniques used by customer service agents.

Introduction
An interesting use case for social media is customer support that can now take place over public social media channels. Using this medium has its advantages as described, for example, in (De-Mers, 2014): Customers appreciate the simplicity and immediacy of social media conversations, the ability to reach real human beings, the transparency, and the feeling that someone listens to them. Businesses also benefit from the publicity of giving good services almost in real-time, online, building an online community of customers and encouraging more brand mentions in social media. A recent study shows that one in five (23%) customers in the U.S. say they have used social media for customer service in 2014, up from 17% in 2012 1 . Obviously, companies hope that such uses are associated with a positive experience. Yet there are limited tools for assessing this. In this paper, we analyze customer support dialogues using the Twitter platform and show the utility of such analyses.
The particular aspect of such dialogues that we concentrate on is emotions. Emotions are a cardinal aspect of inter-personal communication: they are an implicit or explicit part of essentially any communication, and of particular importance in the setting of customer service, as they relate directly to customer satisfaction and experience (Oliver, 2014). Typical emotions expressed by customers in the context of social media service dialogues include anger and frustration, as well as gratitude and more (Gelbrich, 2010). On the other hand, customer service agents also express emotions in service conversations, for example apology or empathy. However, it is important to note that emotions expressed by service agents are typically governed by company policies that specify which emotions should be expressed in which situation (Rafaeli and Sutton, 1987). This is why we talk in this paper about agent emotional techniques rather than agent emotions.
Consider, for example, the real (anonymized) Twitter dialogue depicted in Figure 1. In this dialogue, customer disappointment is expressed in the first turn ('Bummer. =/'), followed by customer support empathy ('Uh oh!'). Then in the last two turns both customer and support express gratitude.
The analysis of emotions being expressed in customer support conversations can take two applications: (1) to discern and compute quality of service indicators and (2) to provide real-time clues to customer service agents regarding the cus-Service-Barometer-US.pdf tomer emotion expressed in a conversation. A possible application here is recommending to customer service agents what should be their emotional response (for example, in each situation, should they apologize, should they thank the customer, etc.) Another interesting trend in customer service, in addition to the use of social media described above, is the automation of various functions of customer interaction. Several companies are developing text-based chat agents, typically accessible through corporate web sites, and partially automatized: In these platforms, a computer program handles simple conversations with customers, and more complicated dialogues are transferred to a human agent. Such partially automated systems are also in use for social media dialogues. The automation in such systems helps save human resources and, with further development based on Artificial Intelligence, more automation in customer service chats is likely to appear. Given the importance of emotions in service dialogues, such systems will benefit from the ability to detect (customer) emotions and will need to guide employees (and machines) regarding the right emotional technique in various situations (e.g., apologizing at the right point).
Thus, our goal, in this paper, is to show that the functionality of guiding employees regarding appropriate responses can be developed based on the analysis of textual dialogue data. We show first that it is possible to automatically detect emotions being expressed and, second that it is possible to predict the emotional technique that is likely to be used by a human agent in a given situation. This analysis reflects our ultimate goal: To enable a computer system to discern the emotions expressed by human customers, and to develop computerized tools that mimic the emotional technique used by a human customer service agent in a particular situation. We see the main contributions of this paper as follows: (1) To our knowledge, this is the first research focusing on automatic analysis of emotions expressed in customer service provided through social media. (2) This is the first research using unique dialogue features (e.g., emotions expressed in previous dialogue turns by the agent and customer, time between dialogue turns) to improve emotion detection. (3) This is the first research studying the prediction of the agent emotional techniques to be used in the response to customer turns.
The rest of this paper is organized as follows. We start by reviewing the related work and a description of the data that we collected. Then we formally define the methodology for detection and prediction of emotion expression in dialogues. Finally, we describe our experiments, evaluate the various models, conclude and suggest future directions.

Emotion Detection
Approaches to categorical emotion classification often employ machine learning classifiers, and SVM has typically outperformed other classifiers. In (Mohammad, 2012;Roberts et al., 2012;Qadir and Riloff, 2014) a series of binary SVM classifiers (one for each emotion) were trained over datasets from different domains (news headlines, social media). These works utilize unigrams and bigrams among other lexical based features (e.g., utilizing the NRC emotion lexicon (Mohammad and Turney, 2013)) and punctuation based features. In our work, we also used an SVM classifier, however, while these works aim at classifying single posts (i.e., sentence, tweet, etc.) without context, our work utilizes the context while con-sidering dialogues. The work in (Hasegawa et al., 2013) showed how to predict and elicit emotions in online dialogues. Their approach for emotion classification is different from ours, for example they only considered the last turn as informative (we consider the full context of the dialogue), and focused on eliciting emotions, while we focus on predicting the agent emotional technique.

Emotion Expression Prediction
The works in (Skowron, 2010) and (D'Mello et al., 2009) presented dialogue systems that sense the user emotions, such that the system further optimizes its affect response. Both systems use rulebased approaches to generate responses, however, the authors do not discuss how they developed the rules.
It is worth mentioning the works in (Ritter et al., 2011;Sordoni et al., 2015) that are focused on data-driven response generation in the context of dialogues in social media. These works generated general responses, while we focused on predicting the appropriate emotional response.

Emotions in Written Customer Service Interactions
In the domain of customer support, several papers studied emotions as part of written interactions. The work in (Gupta et al., 2013), analyzed emotions in textual email communications and the authors focused on prioritizing customer support emails based on detected emotions. In the setting of online customer service (chats), in (Zhang et al., 2011) the authors studied the impact of emotional text on the customer's perception of the service agent. To extract the emotions, the authors used relatively basic features such as emoticons, exclamation marks, all caps, and some internet acronyms (such as 'lol' or 'imho'). Emotion detection is also applied to the domain of call centers (Vidrascu and Devillers, 2005;Morrison et al., 2007) and this differs from our focus since call center data are voice, and, thus, emotion detection is mainly based on paralinguistic aspects rather than on the text. In addition, if the textual part is considered, then the texts are transcripts of calls that are very different from written text (Wallace Chafe, 1987), and even more different from the social media setting where the dialogue is fully public.

Data
In this section we describe the data collection process and provide some statistics about the Twitter dialogue dataset we have collected.

Data Collection
Companies that utilize the Twitter platform as a channel for customer service use a dedicated Twitter account which provides real-time support by monitoring tweets that customers address to it. At the same time corporate support agents reply to these tweets also through the Twitter platform. A customer and an agent, can use the Twitter reply mechanism to discuss until the issue is solved (e.g., a solution is provided, or the customer is directed to another channel), or until the customer is no longer active.
In the present work, we define a dialogue to be a sequence of turns between a specific customer and an agent, where the customer initiates the first turn. Consecutive posts of the same party (customer or agent) uninterrupted by the other party, are considered as a single turn (even if there are several tweets). Given the nature of customer support services, we assume the last turn in the dialogue is an agent turn (e.g., "You're very welcome. :) Hit us back any time you need support"). Thus, we expect an even number of turns in the dialogue. We filtered out dialogues in which more than one customer or one agent are involved. Formally, we define a dialogue to be an ordered list of turns [t 1 , t 2 , · · · , t n ] where odd turns are customer turns, and even turns are agent turns, and n is even.
Each turn t i is a tuple consisting of {turn number, timestamp, content} where turn number represents the sequential position of the turn in the dialogue, timestamp captures the time the message was published on Twitter, and content is the textual message.

Data Statistics
We gathered data for two North America based customer support services Twitter accounts that provide support for customers from North America (so tweets are in English). One service is for general customer care (denoted as Gen), and the other is for technical customer support (denoted as Tech). We extracted this data from December 2014 until June 2015. Specifically, for each customer that posted a tweet to the customer support accounts, we searched for the previous, if any, turn  to which it replied. Given this method we traced back previous turns and reconstructed entire dialogues. Table 1 summarizes some statistics about the collected data, and Figure 2 depicts the frequencies of dialogue lengths which follow a power-law relationship. Table 1 shows differences between the two services; the dialogues in Tech tend to be longer (i.e., typically include more turns), with an average of 6.81 turns vs. average of 4.83 turns for Gen.
As most of the dialogues include at most 8 turns (88% and 76% for Gen and Tech, respectively), we removed dialogues longer than 8 turns. In addition, we removed dialogues that contained only 2 turns as these are too short to be meaningful as the customer never replied or provided more details about the issue. After applying these preprocessing steps, we had 1189 dialogues of Gen support, and 1224 dialogues of Tech support.

Methodology
The first objective of our work is to detect emotions expressed in customer turns and the second is to predict the emotional technique in agent turns. We treated these two objectives as two classification tasks. We generated a classifier for each task, where the classification output of one classifier can be part of the input to the other classifier. While both classifiers work at the level of turns, i.e., classify the current turn to emotions ex-pressed in it, they are inherently different. When detecting emotions in a customer turn, the turn's content is available at classification time (as well as the history of the dialogue) -meaning, the customer has already provided her input and the system must now understand what is the emotion being expressed. Whereas, when predicting the emotional technique for an agent turn, the turn's content is not available during classification time, but only the agent action and the history of the dialogue since the agent did not respond yet. This difference stems from the fact that in order to train an automated service agent to respond based on customer input, the agent's emotional technique needs to be computed before the agent generates its response sentence.
We defined a different set of relevant emotion classes for each party in the dialogue (customer or agent), based on our above survey of research on customer service (e.g., (Gelbrich, 2010)). Relevant customer emotions to be detected are: Confusion, Frustration, Anger, Sadness, Happiness, Hopefulness, Disappointment, Gratitude, and Politeness. Relevant agent emotional techniques to be predicted are: Empathy, Gratitude, Apology, and Cheerfulness.
We utilized the context of the dialogue to extract informative features that we refer to as dialogue features. Using these features for emotion classification in written dialogues is novel, and as our experimental results show, it improves performance compared to a model based only on features extracted from the turn's text.

Features
We used the following features in our models.

Dialogue Features
Comprises three contextual feature families: integral, emotional, and temporal. A feature can be global, namely its value is constant across an entire dialogue or it can be a local, meaning that its value may change at each turn. In addition, a feature can be historical (as will be discussed below).
The integral family of features includes three sets of features: 1. Dialogue topic: a set of global binary features representing the intent of the customer who initiated the support inquiry. Multiple intents can be assigned to a dialogue from a taxonomy of popular topics, which are adapted to the specific service. Examples of topics include ac-count issues, payments, technical problem and more 2 . This feature set captures the notion that customer emotions are influenced by the event that led the customer to contact the customer service (Steunebrink et al., 2009). 2. Agent essence: a set of local binary features that represent the action used by the agent to address the last customer turn, independently of any emotional technique expressed. We refer to these actions as the essence of the agent turn. Multiple essences can be assigned to an agent turn from a predefined taxonomy. For instance, "asking for more information" and "offering a solution" are possible essences 3 . This feature set captures the notion that customer emotions are influenced by actions of agents (Little et al., 2013). 3. Turn number: a local categorical feature representing the number of the turn.
The emotional family of features includes Agent emotion and Customer emotion: these two sets of local binary features represent emotions predicted for previous turns. Our model generates predictions of emotions for each customer and agent turn, and uses these predictions as features to classify a later customer or agent turn with emotion expression.
The temporal family of features includes the following features extracted from the timeline of the dialogue: 1. Customer/agent response time: two local features that indicate the time elapsed between the timestamp of the last customer/agent turn and the timestamp of the subsequent turn. This is a categorical feature with values low, medium or high (using categorical values yielded better results than using a continuous value). 2. Median customer/agent response time: two local categorical features defined as the median of the customer/agent response times preceding the current turn. The categories are the same as the previous temporal features.
2 Currently this feature is not supported in social media. In other channels, for example, customer support on the phone, the customer is requested to provide a topic before she is connected to a support agent (usually using an IVR system). As this feature is inherent in other customer support channels, we assume that in the future it will also be supported in social media. 3 We assume that if the agent is human, then this input is known to her e.g., based on company policies. For the automated service agent case, we assume that the dialogue system will manage and provide this input. When representing a turn, t i , as a feature vector, we added some features originating in previous turns j < i to t i . These features, that are historical, include the emotional features family and local integral features (namely agent emotions, customer emotions and agent essence). We do not include the turn number of previous turns, as this is dependent on the turn number of t i . We denote these features as historical features. The value of history, that is a parameter of our models, defines the number of sequential turns that precede t i which propagate historical features to t i . Figure 3 shows an example of the historical features in relation to the classification of customer turn t i , for history size between 1 and 3.

Textual Features
These features are extracted from the text of a customer turn, without considering the context of the dialogue. We use various state-of-the-art text based features that have been shown to be effective for the social media domain (Mohammad, 2012;Roberts et al., 2012). These features include various n-grams, punctuation and social media features. Namely, unigrams, bigrams, NRC lexicon features (number of terms in a post associated with each affect label in NRC lexicon), and presence of exclamation marks, question marks, usernames, links, happy emoticons, and sad emoticons. We note that these are the features we used in our baseline model detailed below, in the description of our experiments.

Turn Classification System
For both of the agent and customer turn classification tasks, we implemented two different models which incorporate all of the feature sets we have detailed above. We considered these tasks as multi-label classification tasks. This captures the notion that a party can express multiple emotions (e.g., confusion and anger) in a turn. We chose to use a problem transformation approach which maps the multi-label classification task into several binary classification tasks, one for each emotion class which participates in the multi-label problem (Tsoumakas and Katakis, 2006). For each emotion e, a binary classifier is created using the one-vs.-all approach which classifies a turn as expressing e or not. A test sample is fully classified by aggregating the classification results from all independent binary classifiers. We next define our two modeling approaches.

SVM Dialogue Model
In our first approach we trained an SVM classifier for each emotion class as explained above. The feature vector we used to represent a turn incorporates dialogue and textual features. The history size is also a parameter of this model. Feature extraction for a training/testing feature vector representing a turn t i , works as follows. Textual features are extracted for t i if it is a customer turn, or for t i−1 if it is an agent turn (recall that the system does not have the content of agent turn t i at classification time). The temporal features are also extracted using time lapse values between previous turns as explained above. As discussed above, agent essence is assumed to be an input to our module, while agent emotion and customer emotion features are propagated from classification results of previous turns during testing (or from ground truth labels during training), where the number of previous turns is determined according to the value of history. These historical features are also appended to the feature vector of t i , similarly to (Kim et al., 2010) where this method was used for classifying dialogue acts.

SVM-HMM Dialogue Model
Our second approach to classifying dialogue turns is to use a sequence classification method (SVM-HMM), which classifies a sample sequence into its most probable tag sequence. For instance (Kim et al., 2010;Tavafi et al., 2013) used SVM-HMM and Conditional Random Fields for dialogue act classification. Since emotions expressed in customer and agent turns are different, we treated them as different classification tasks (like in our previous approach) and trained a separate classifier for each emotion. We made the following changes when using SVM-HMM: (1) We treated the emotion classification problem of turn t i as a sequence classification problem of the sequence t 1 , t 3 , ..., t i (i.e., only customer turns) if t i is a customer turn and t 2 , t 4 , ..., t i (i.e., only agent turns) if it is an agent turn. (2) The SVM-HMM classifier generates models that are isomorphic to a k th -order hidden Markov model. Under this model, dependency in past classification results is captured internally by modeling transition probabilities between emotion states. Thus, we removed historical customer emotion (resp. agent emotion) feature sets when representing a feature vector for a customer (resp. agent) turn.
(3) We note that in our setting we provide classifications in real-time during the progress of the dialogue, so at classification time we have access only to previous turns and global information, and we cannot change classification decisions for past turns. Thus, we tagged a test turn, t i , by classifying the sequence which ends in t i . Then, t i was tagged with its sequence classification result.

Experimental Setup
A first step in building a classification model is to obtain ground truth data. For this, we sampled dialogues from our dataset, as detailed in Table 2  judge performed the following tagging tasks given the full dialogue: 1. Emotion tagging: indicate the intensity of emotion expressed in each turn (customer or agent) for each emotion, on a scale of ([0...5]), such that 0 defines no emotion, 1 a low emotion intensity and 5 a high emotion intensity. The intraclass correlation (ICC) among the judges was 0.53 which indicates a moderate agreement which is common in this setting (LeBreton and Senter, 2007 We generated true binary labels from the emotion tagging. For turn t i , we considered it to express emotion e if tag(e, t i ) ≥ 2 where tag(e, t) is the average judges' tag value of e in t. This process generated the class sizes detailed in Table 3. Dialogue topic tagging was converted to binary features representing the top-2 selected topics. Agent essence feature set representation for each turn was defined analogously. The temporal response time values were translated to low/medium/high categorical values according to their relation to the 33-th and 66-th percentiles.
We evaluated our methods by using leave-onedialogue-out cross-validation (as in (Kim et al., 2010)), over the whole dataset (for the two cus-Customer Agent Emotion # of instances Emotion  # of instances  Happiness  66  Apology  146  Sadness  31  Gratitude  81  Anger  160  Empathy  163  Confusion  68  Cheerfulness  177  Frustration  342  Disappointment  257  Gratitude  119  Hopefulness  30  Politeness  180   Table 3: Class size per classification task tomer service data sources together). Each test dialogue was classified by its order of turns, where each turn type (customer or agent) is classified by its corresponding classifier.
Our baseline in all experiments is an SVM classifier that uses only the textual features described above, which do not utilize the dialogue context. This was used as a state-of-the-art single sentence emotion detection approach in many cases, e.g., (Mohammad, 2012;Roberts et al., 2012;Qadir and Riloff, 2014) and more. As described above, agent turn emotion prediction is performed before its content is known. Thus, the baseline representation of an agent turn consisted of textual features extracted from its preceding customer turn. We evaluated each emotion's classification performance by using precision (P ), recall (R) and F1score (F ). We evaluated the total performance for all emotion classes using micro and macro averages. We used Liblinear 5 as an SVM implementation and SVM-HMM 6 for sequence classification. Additionally, we used ClearNLP 7 for textual features extraction.

History Size Impact
Since history size is a parameter of our models, we first tested the classification results for all possible history sizes (given that that maximum dialogue size in our dataset is 8). For each task and for each possible history size, we generated SVM Dialogue and SVM-HMM Dialogue models and evaluated them as detailed above. We compared the macro and micro average F1-score of our classifiers against the baseline classifier performance. As depicted in Figure 4  for all history ranges and for both tasks. Examining the customer turns emotion detection performance, we can see in Figure 4(a) that it increases until history = 3, and then remains relatively stable for larger history sizes. This means that information about the behavior of the customer and agent in past turns is beneficial for detecting customer emotions in a current turn. For assessing the performance of our predictions of agent turns emotion techniques, we first note that we tested with history > 0 range, since we assume that the minimal information needed for agent turn classification is the information extracted from the last customer turn. Figure 4(b) shows that overall, performance is highest when history = 1, and does not decline much for higher history values. This indicates that for agent emotion technique prediction the last customer turn is the most informative one. In all of our experiments, we used the Wilcoxon signed-rank test to validate the statistical significance of our models' micro and macro average F1-score comparing to baseline performance. Additionally, we used McNemar's test on the contingency tables aggregated over all emotions. These tests showed that both of our models were significantly different from the baseline model, under a value of 0.001, for both classification tasks and all history sizes. Table 4 depicts the detailed classification results for optimal history values that obtained maximal macro F1-score, namely for customer emotion detection history = 4 and for agent emotion technique prediction history = 1. The table presents performance for each emotion, for macro and micro average results over all dialogues, and for each data source (Gen or Tech) separately. For both classification tasks, both of our models outperformed baseline results for almost all emotions, where average macro and micro results are statistically significant compared to the baseline, as described above.

Detailed Classification Results
For customer turn emotion detection, the SVM-HMM Dialogue model performed better than the SVM Dialogue model, and reached a macro and micro average F1-score improvements over all dialogues of 17.8% and 11.7%, respectively. Furthermore, the macro and micro average F1-score results of the SVM-HMM Dialogue model (0.519 and 0.6, respectively) are satisfying given the moderate ICC score between the judges (0.53). For predicting the agent emotional technique, the SVM Dialogue model obtained slightly better results than SVM-HMM Dialogue model, and reached a macro and micro average F1-score improvements over all dialogues of 53.9% and 43.5%, respectively. These results emphasize the differences between the SVM Dialogue and SVM-HMM Dialogue models. Specifically, when history size is large, as in customer emotion prediction, SVM-HMM Dialogue model, which internally captures dependencies in past classifications, outperforms the simplistic SVM Dialogue model. We note that an improvement is also obtained when calculating macro and micro average performance for each data source separately. This highlights our models' superiority as well as their general applicability and robustness for different data sources.

Feature Set Contribution Analysis
We examined the contribution of different feature sets in an incremental fashion, using the optimal history value detailed above. Based on the families of feature sets that we defined in the Methodology section, we tested the performance of different feature set combinations in our models, added in the following order: baseline (textual features), emotional, temporal and integral. Figure 5 Table 4: Detailed performance results for customer and agent classification tasks given optimal history size. For brevity, the table presents improvement relative to baseline in percentages only for F1-score.
the results for both classification tasks. The x-axis represents specific combination of features sets, and the y-axis represents the macro or micro average F1-score value obtained. Figure 5 shows that adding each feature set improved performance for all models, for both tasks, which indicates the informative value of each feature set. Additionally, the figure suggests that the most informative dialogue feature sets are the integral and emotional.

Conclusions
In this work we studied emotions being expressed in customer service dialogues in the social media. Specifically, we described two classification tasks, one for detecting customer emotions and the other for predicting the emotional technique used by support service agent. We have proposed two different models (SVM Dialogue and SVM-HMM Dialogue models) for these tasks. We studied the impact of dialogue features and dialogue history on the quality of the classification and showed improvement in performance for both models and both classification tasks. We also showed the robustness of our models across different data sources. As for future work we plan to work on several aspects: (1) In this work, we showed that it is possible to predict the emotional technique. In the future, we plan to run experiments in which the predicted emotional technique is actually applied in the context of new dialogues to measure the effect of such predictions on real support dialogues.
(2) Distinguish between dialogues that have positive outcomes (e.g., high customer satisfaction) and others.


Quantized neural network design under weight capacity constraint

The complexity of deep neural network algorithms for hardware implementation can be lowered either by scaling the number of units or reducing the word-length of weights. Both approaches, however, can accompany the performance degradation although many types of research are conducted to relieve this problem. Thus, it is an important question which one, between the network size scaling and the weight quantization, is more effective for hardware optimization. For this study, the performances of fully-connected deep neural networks (FCDNNs) and convolutional neural networks (CNNs) are evaluated while changing the network complexity and the word-length of weights. Based on these experiments, we present the effective compression ratio (ECR) to guide the trade-off between the network size and the precision of weights when the hardware resource is limited.

Introduction
Deep neural networks (DNNs) begin to find many real-time applications, such as speech recognition, autonomous driving, gesture recognition, and robotic control (Sak et al., 2015;Chen et al., 2015;Corradini et al., 2015). Recent works show that the precision required for implementing fully-connected deep neural networks (FCDNNs), convolutional neural networks (CNNs) or recurrent neural networks (RNNs) needs not be very high, especially when the quantized networks are trained again to learn the effects of lowered precision. In the fixed-point optimization examples shown in , neural networks with ternary weights showed quite good performance which was close to that of floating-point arithmetic. However, the performance of DNNs usually degrades when the weights are represented using a very low precision. Thus, we have a question whether it might be a better option to reduce the network size, instead of severely quantizing the weights, for efficient implementations.
In this work, we compare the performance of FCDNNs and CNNs under two constraints for hardware implementation, one is reducing the network size and the other is lowering the precision of the weights. We conduct the experiments with FCDNNs for phoneme recognition and CNNs for image classification. To control the network complexity, the number of units in each hidden layer is varied in the FCDNN. For the CNN, the number of feature maps for each layer is changed. The retraining based quantization algorithm is used for fixed-point optimization of weights .
Based on the experiments, we propose a metric called the effective compression ratio (ECR) that compares the complexity of floating-point and fixed-point networks showing the same performance. This analysis intends to provide a guideline to network size and word-length determination for efficient hardware implementation of deep neural networks (DNN).

Related Work
Fixed-point design of DNNs with ternary weights show quite good performances that are very close to the floating-point results Anwar et al., 2015a;. The ternary weight based FCDNN is used for VLSI implementations, by which the algorithms can operate with only on-chip memory consuming very low power (Kim et al., 2014). The CNN is implemented by XNOR-bitcounting operations (Rastegari et al., 2016). Binary weight based deep neural network design is also studied (Courbariaux et al., 2015). Pruned floating-point weights are utilized for efficient GPU-based implementations, where small valued or less important weights are forced to zero to reduce the number of arithmetic operations and the memory space for weight storage (Yu et al., 2012;Han et al., 2015;Anwar et al., 2015b).
Most of the above works are experimented using large size neural networks. However, mobile or embedded portable devices have limited resources, and thus small size fixed-point networks showing good performances are very needed.

Fixed-Point FCDNN and CNN Design
This section explains the design of FCDNN and CNN with varying network complexity and weight precision.

FCDNN and CNN Design
In this work, we examine an FCDNN for phoneme recognition and a CNN for image classification. The reference DNN has four hidden layers. Each of the hidden layers has N h units; the value of N h is altered to control the complexity of the network. We conduct experiments with N h value of 32, 64, 128, 256, 512, and 1024. The input layer of the network has 1,353 units to accept 11 frames of a Fourier-transform-based filter-bank with 40 coefficients (+energy) distributed on a mel-scale, together with their first and second temporal derivatives. The output layer consists of 61 softmax units which correspond to 61 target phoneme labels (Mohamed et al., 2012). Phoneme recognition experiments were performed on the TIMIT corpus.
The CNN used is for CIFAR-10 dataset (Krizhevsky & Hinton, 2009). It contains a training set of 50,000 and a test set of 10,000 images. We divided the training set to 40,000 images for training and 10,000 images for validation. The reference CNN has 3 convolution and max-pooling layers, a fully connected hidden layer with 64 units, and the output with 10 softmax units. We control the number of feature maps in each convolution layer. The reference size has 32-32-64 feature maps with a 5 by 5 kernel size as used in Krizhevskey (2014). To know the effects of network size variation, the number of feature maps is reduced or increased. The configurations of the feature maps used for the experiments are 8-8-16, 16-16-32, 32-32-64, 64-64-128, 96-96-192, and 128-128-256. Note that the fully connected layer in the CNN is not changed.

Fixed-Point Optimization of DNNs
Reducing the word-length of weights brings several advantages in hardware based implementation of neural networks. First, it lowers the arithmetic precision, and thereby reduces the number of gates needed for multipliers. Second, the size of memory for storing weights is minimized, which would be a big advantage when keeping them on a chip, instead of external DRAM or NAND flash memory. Note that FCDNNs demand a very large number of weights. Third, reduced arithmetic precision or minimization of off-chip memory accesses leads to low power consumption.
The fixed-point DNN algorithm design consists of three steps: floating-point training, direct quantization, and retraining of weights. Refer to  for the details.

Analysis of Quantization Effects
The fixed-point performance of the FCDNN is shown in Figure 1a, where the number of hidden units in each layer varies. The performance of direct 2 bits (ternary levels), direct 3 bits (7-levels), retrain-based 2 bits, and retrain-based 3 bits are compared with the floating-point results. We can find that the performance gap between the floating-point and the retrain-based fixed-point networks converges very fast as the network size grows. Direct quantization does not show good results at any network size. In this figure, the performance of the floating-point network almost saturates when the network size is about 1024. Note that the TIMIT corpus that is used for training has only 3 hours of data. Thus, the network with 1024 hidden units can be considered in the 'training-data limited region'. Here, the gap between the floating-point and the fixed-point networks almost vanishes when the network is in the 'training-data limited region'. However, when the network size is limited, such as 32, 64, 128, or 256, there exists some performance gap between the floating-point and highly quantized networks even if retraining on the quantized networks is performed. The similar  Figure 1: Comparison of retrain-based and direct quantization. All weights are quantized with ternary and 7-level weights. Note that each FCDNN has four hidden layers. In the figure (b), x-axis label " 16, 32, 64, 128, 192, 256" represents the number of feature map is " 8-8-16, 16-16-32, 32-32-64, 64-64-128, 96-96-192, 128-128-256". experiments are conducted for the CNN with varying feature map sizes, and the results are shown in Figure 1b. The configurations of the feature maps used for the experiments are 8-8-16, 16-16-32, 32-32-64, 64-64-128, 96-96-192, and 128-128-256. The size of the fully connected layer is not changed. In this figure, the floating-point and the fixed-point performances also converge very fast as the number of feature maps increases. The floating-point performance saturates when the feature map size is 128-128-256, and the gap is less than 1% when comparing the floating-point and the retrain-based 2-bit networks. This suggests that a fairly high-performance feature extraction can be designed even using very low-precision weights if the number of feature maps can be increased.

Efficient DNN Design with Hardware Constraints
As the number of quantization levels decreases, the memory space needed is reduced at the cost of sacrificing the accuracy. Therefore, there can be a trade-off between the network size reduction and aggressive quantization. Figure 2a shows the framewise phoneme error rate on TIMIT corpus while varying the layer size of FCDNNs with a various number of quantization bits from 2 to 8 bits. Note that the network has four hidden layers containing the same number of units.
In this section, we propose a guideline for finding the optimal bit-widths when the desired accuracy or the network size is given. Note that we assume 2 n − 1 quantization levels are represented by n bits (i.e. 2 bits are required for representing a ternary weight). For simplicity, all layers are quantized with the same number of quantization levels. Based on this observation, we introduce a metric called the effective compression ratio (ECR), which is defined as follows: quantization is employed. Thus, the compression obtained by 3-bit quantization can be roughly figured as (32 × 256 2 )/(3 × N 1 2 ). If N 1 is very close to 256, there can be about 10 (= 32/3) times compression. But, if N 1 is 512, the compression drops to only about 2.5 because the number of parameters of FCDNN is proportional to the square of the network size.
The ECRs for various network sizes and quantization bits are shown in Figure 3. The figure illustrates that the 2-bit or (maybe 1-bit) quantization can lead to the best compression ratio when the target performance is low, which means a high phoneme error rate in this example. However, for designing a fairly high-performance network, increasing the network size with severe quantization does not yield hardware efficient networks. The optimum number of bits for obtaining the performance corresponding to that of 512 sizes floating-point DNNs is 4 or 5 bits. Further reducing the wordlength demands very increased size networks, and as a result, the total number of bits increases.

Conclusion
Hardware efficient deep neural networks can be designed either by lowering the number of units in each layer or reducing the number of bits for weight quantization. We evaluate the performance of fixed-point deep neural networks and analyze the trade-off between the complexity and the precision of the weights. This study shows that low-performance hardware efficient DNNs can be designed with severely quantized weights. In the low-performance region, the DNN performance increases very rapidly as the network size grows. Thus, it is possible to compensate for the quantization effects by slightly increasing the network size. However, for a high-performance DNN design, compensation of quantization effects by increasing the network size is difficult, and thus severe quantization does not lead to efficient hardware design. The effective compression ratio is given for a DNN design when the network size and the precision vary.


Network anomaly detection for railway critical infrastructure based on autoregressive fractional integrated moving average

The article proposes a novel two-stage network traffic anomaly detection method for the railway transportation critical infrastructure monitored using wireless sensor networks (WSN). The first step of the proposed solution is to find and eliminate any outlying observations in the analyzed parameters of the WSN traffic using a simple and fast one-dimensional quartile criterion. In the second step, the remaining data is used to estimate autoregressive fractional integrated moving average (ARFIMA) statistical models describing variability of the tested WSN parameters. The paper also introduces an effective method for the ARFIMA model parameters estimation and identification using Haslett and Raftery estimator and Hyndman and Khandakar technique. The choice of the “economically” parameterized form of the model was based on the compromise between the conciseness of representation and the estimation of the error size. To detect anomalous behavior, i.e., a potential network attack, the proposed detection method uses statistical relations between the estimated traffic model and its actual variability. The obtained experimental results prove the effectiveness of the presented approach and aptness of selection of the statistical models.

Introduction
Intelligent transportation systems (ITS) are currently a key technology that is identified as an answer to the growing need for mobility of goods and people. Owing to the use of ITS, it is possible to establish a fully functioning, accurate, real-time, and efficient transportation management system. It can be achieved by combining information systems and technologies like wireless networks and sensors, computing/networking devices, Global Positioning System (GPS), mobile telephony, and camera recognition systems. Thanks to ITS, it is possible to improve the level of services and capacity of transportation systems. In particular, ITS can help to enhance the transportation infrastructures, its overall safety, and security of critical information for different transportation means. It must be noted that currently, the main focus of academics and industry is on vehicular networks and more precisely on developing inter-vehicle and vehicle-to-infrastructure networks [1][2][3]. However, ITS are not limited only to manage vehicular traffic but they can also provide services and they can be successfully implemented in air, water transport, and rail systems [4,5] as well.
An important aspect of any ITS is to correctly address potential security and privacy issues. Aijaz et al. [4] define the following vital attack aspects depending on what the target is. Authors identify attacks on the following: (i) wireless interface; (ii) sensor inputs to different processing units; (iii) software and hardware parts of the systems; and (iv) security infrastructure behind wireless access networks (e.g., certification and traffic authorities, transportation vehicle manufacturers). On the other hand, various security solutions have been proposed to tackle these problems and they can be classified [6] as proactive (e.g., tamper resistant hardware, proprietary system design, and digitally signed messages) or reactive (e.g., anomaly-based, context-based, and signature-based approaches). Especially anomaly-based detection systems constitute an important part of every ITS-based management system. They allow to assess the imminent emergence of any incidents, i.e., to detect deviations from normal patterns (events, situations). Therefore, identifying anomalous events is essential as they can lead to critical conditions where immediate actions must be taken.
In the existing literature, several anomaly detection approaches for ITS have been proposed. However, they have been mostly proposed for the vehicular networks, see e.g., [7][8][9]. However, very few solutions so far have been proposed for railway systems and they are mentioned below.
Rabatel et al. [10] focused on the field of train maintenance. Monitoring of trains is provided using sensors positioned on the main train components, e.g., motors wheels, to transmit information regarding, e.g., the temperature, acceleration, and velocity. Then an automatic detection system is introduced to identify anomalies in order to predict potential failures in advance. The proposed approach considers also the contextual criteria associated to railway data like weather conditions and itinerary.
Holst et al. [11] developed a statistical anomaly detection method which has been deployed in a tool which aim is to monitor train fleets and that allows inspecting and visualizing the occurrence of event messages generated on the trains. The designed anomaly detection component is based on the Bayesian Principal Anomaly [12] and aids operators to quickly find significant deviations from normal behavior and to detect early indications of potential problems.
Anomaly detection system that is able to indicate degraded condition of track and rolling stock has been proposed by Goodman et al. [13]. Authors utilized a sensor system installed on one of the 110 boxcars on a train on a high-tonnage loop test track. The data from the sensors was sent to a specialized collection gateway hub which was mounted inside the boxcar. The main goal was to discover abnormalities in railroad tracks, rolling stock, bearings, rotating shafts, and gears. The obtained results confirmed that such a detection system is efficient enough to identify, locate, and characterize such types of anomalies.
Considering the above, in this article we introduce another type of a novel anomaly detection system dedicated for wireless sensor networks (WSN) traffic that is based on clustering and statistical model with long memory, i.e., autoregressive fractional integrated moving average (ARFIMA). The main idea of the proposed approach is to analyze the deviations between parameters of the real network traffic and the estimated statistical models of that traffic. We develop a two-stage anomaly detection method. The first step is to find and eliminate possible observations outlying from the analyzed WSN traffic parameters. This step is performed by means of a simple yet effective one-dimensional quartile criterion. In the second step, the remaining data are used to estimate the ARFIMA statistical models describing variability of the tested WSN parameters.
The proposed anomaly detection method is used as a security measure for railroad gates tracking system that is based on WSN sensors. It is a part of a more comprehensive system responsible for supervising and visualization of the critical infrastructure, i.e., railroad crossings. The feasibility and effectiveness of the introduced method is proved based on the abovementioned real-life railroad crossings monitoring system; however, it must be noted that it can be conveniently ported to any other transportation system as well.
The rest of this paper is structured as follows. The next section focuses on the main security issues related to WSN. Then in Section 3, the assumed scenario as well as details of the proposed anomaly detection approach is outlined. Section 4 presents the real-life setup as well as obtained experimental results. Finally, Section 5 concludes our work.

Overview of security in WSN
Ensuring security of the WSNs is an important factor for their correct operation due to the fact that they are distinctly sensitive to hazards emerging from human intentional actions that include illegal use or incapacitation (e.g., impersonating or eavesdropping the user, terrorist attacks) [14], or from environmental influence, for instance fire, electromagnetic signal, etc. [15]. When comparing cable networks to WSNs, the latter offer restricted computational abilities and limited energy resources [16].Taking this into account, maintaining safe operation of WSN is difficult, nevertheless necessary. It is because the networks must be able to uphold their basic functioning which consists in collecting data from the sensors and transmitting it to both the monitoring unit and the WSN infrastructure management centers.
Wireless sensor networks might be subjected to different types of attacks (either passive or active). Passive attacks happen when an intruder does not utilize signal emissions aimed at disturbing the proper operation of WSN in order to, e.g., access its data, infrastructure, or modify transmitted messages. On the other hand, active attacks rely on utilization of such emissions of signals or actions that may be detected [15,17], while trying to obtain an unauthorized access or a possibility to alter the messages.
During passive attacks on WSN, the intruder aims at passive interception of the exchanged network traffic to implicitly acquire the transmitted data. An example of such action is eavesdropping of the data that is transmitted between the nodes. The WSN radio medium, because of its specific features, is relatively vulnerable to such attacks. Another instance may be the network traffic analysis which intends to examine and disclose the WSN topology. A characteristic aspect of wireless sensor networks is a great load of information transmitted through a part of their nodes. If this data transmission is increased on these nodes, the neighboring nodes counteract by retransmitting the information to the base station. Due to the network traffic analysis, the attacker may obtain knowledge of how much workload the sensor network critical nodes are burdened with [18,19].
Contrary to the passive methods, active attacks enable the intruder to directly or indirectly influence data content transmitted in the WSN. Moreover, the active attacks can be easily detected because their impact on WSN performance is direct, i.e., they may degrade the WSN quality, or even deny access to some services or completely disenable control over the network. For the network critical infrastructure, direct attacks on WSN hardware are especially dangerous. Such attacks can cause diminishing of the monitoring area of the sensor network, or entire disposal of the WSN [19,20].
The aim of manipulating WSN nodes is to distract the operator of the sensor network from the main origin of the threat. e.g.. from spoofing or distributed denial of service ((D)DoS) attacks. Moreover, if the attacker uses a short-term high-energy electromagnetic pulse then an annihilation of either the given sensor network or any electronic device within the EMP destruction field [21] is possible.
The attacks aimed at the data's confidentiality or integrity constitute an immense threat because they let the intruder enter the network without authorization to transmit data. The Sybil Attack is an example of a masking technique which consists in spoofing by transmitting numerous identifiers through a harmful confluence, or framing a legal confluence and taking over its specification to obtain access to the WSN infrastructure [22].
The (D)DoS attacks in WSN aim at excessive charging of the attacked sensor networks features in order to disenable data gathering from the attacked nodes or to restrain the efficiency of tools provided by the victim WSN. The (D)DoS invasions can be directed onto every level of the network model, i.e., ISO/OSI [21].
Because the WSN are vulnerable to a large number of dangers, limiting the possibility of a successful attack requires the use of advanced methods and algorithms. These methods are spread spectrum techniques (hindering the successful interfering with radio transmission); methods using cryptographic algorithms (securing the confidentiality and integrity of the transferred information); a proper nodes' device construction to deny access to their internal systems (for instance, information about cryptographic algorithms, or keeping secret keys); utilizing related protocols (documents guarding the transfer of information); and ways of monitoring and identifying abnormalities in the WSN data transfer [23][24][25].
In this paper, we suggest the two stages of analysis of abnormal behavior detection for WSN traffic. The first step prepares proper data by removing the outlier values. In the second step, the parameters of ARFIMA statistical model are used for detecting anomalies. In the course of stages, performed according to particular scenarios, the satisfying results were obtained. The following scenarios of anomaly attacks are analyzed and calculated for the sake of efficient protection of railroad crossings: (i) electromagnetic distortion; (ii) intentional damage of selected infrastructure; and (iii) attacks performed by means of the important WSN component, i.e., the WSN IP gateway.

Network anomaly detection: the proposed approach
In the rest of the paper, we assume the scenario depicted in Fig. 1 in which there is a management system that is utilized for visualizing and controlling railroad crossings critical infrastructure. The required infrastructure for monitoring a single railroad crossing consists of WSN sensors used for analyzing the state and position of the railroad crossing separate gates, WSN IP gateway which aggregates the traffic from sensors, firewall, and multi-WAN router for providing different links to the WAN network.
As mentioned in the introduction, an anomaly detection system is a vital component of any ITS management solution. That is why the Intrusion Detection/Prevention Systems (IDS/IPS) for detecting attacks and/or intrusions are currently utilized as one of the main components to provide security of the critical infrastructure. Their main function is to accurately identify, detect, and respond to an unauthorized activity directed against the protected network resources [26].
Generally, we may classify IDS/IPS based on the utilized threat identification technique to signature-based or anomaly detection systems. The first consists in the detection of intrusions using the signature of previously known attacks [27]. Comparatively, the latter relies on monitoring the defended system and to detect any abnormalities. Thus, any deviation from a defined model or profile of legitimate activities reflected in the WSN network traffic parameters is treated as a symptom of the attack. Such a deviation from normal reference is called an anomaly [14,28].
The outstanding profit of abnormal behavior noticing solutions is the fact that they are able to detect unknown intrusions interrupting correct network traffic parameters. Therefore, ID/PS relying on anomaly detection are (if properly configured) more effective than signature-based ones [29].
Considering the above, in this article the recognition of abnormal behavior is applied. This approach is based on the idea of analyzing the deviations of parameters of the real network traffic from the estimated statistical models of that traffic (see Fig. 2). We suggest a two-stage anomaly detection method. In the first step, ARFIMA model base for the analyzed WSN network traffic parameters is built. This is realized on the formerly selected and calculated features of the network traffic. In the following steps, outlier observations are eliminated and estimation of ARFIMA models parameters of the analyzed WSN network traffic features is performed. In result, statistical models base is created and serves as a basis for an anomaly detection system. The second step is a normal operation of an anomaly detection system (ADS), i.e., selection and calculation of the relevant network traffic features, and assessment of the difference between the actually transmitted data (i.e., network traffic) and the calculated ARFIMA representation of the traffic for the chosen WSN network parameters.
The motivation for choosing the ARFIMA statistical model was based on the results of previous authors' research, i.e., on the use of autoregressive models and heteroscedastic and regression models with variable sampling resolution of the dataset for anomaly detection in the LAN/WAN networks. Findings included in [30][31][32] clearly indicate the superiority of the ARFIMA model for modeling network traffic parameters' variability for the purpose of anomaly detection.
It must be emphasized that the first stage is always initiated after performing any change in WSN network infrastructure or topology. It can also be performed periodically to update the statistical models base, which is a basis for the anomaly detection system. However, the elimination procedure of outliers' observations (realized at this stage) disables degradation of ARFIMA models by rejecting non-standard parameters of the analyzed network traffic.
Below we present and discuss main components of the proposed approach in details.

Detection of outlying observations-one-dimensional quartile criterion
Due to the nature of the transportation critical infrastructure and its monitoring using WSN, there is a real hazard of fluctuation of the analyzed network traffic parameters, i.e., possibility of emerging outlying observations (outliers). These fluctuations may have diverse sources, for instance (i) environmental-connected with interference of radio wave propagation, (ii) technical-related to changes in the infrastructure, (iii) devices' damage; or (iv) they can be a consequence of network attack. In our approach, identification of the outliers of the analyzed WSN traffic parameters is performed by means of one-dimensional quartile criterion introduced by Tukey [33], which is used for the construction of box plots. For every parameter, we calculate the first (Q1) and third (Q3) quartile and interquartile range IRQ = Q3 − Q1. Quartiles divide all our observations into four equal-number groups (Fig. 3, left).
The first quartile (Q1) divides observations in respect of 25-75 %, which means that 25 % are lower or equal to Q1, and 75 % of observations are equal or greater that Q1. The second quartile (Q2), otherwise known as the median, divides observations into 50-50 % proportion. The third quartile (Q3) divides the observations in respect of 75-25 %, which means that 75 % of observations are lower or equal to Q3, and 25 % are equal or greater that Q1. Observations which can be considered as outliers are those whose values exceed the range (Q1 − 1.5IRQ, Q3 + 1.5IRQ). In contrast, observations of extreme outliers (see Fig. 3, right) are identified as those for which the attributes are outside the range (Q1 − 3IRQ, Q3 + 3IRQ).

The ARFIMA model-estimation of the WSN traffic features variability
Grange, Joyeux [34], and Hosking [35] introduced a model called the autoregressive fractional integrated moving average (ARFIMA) which is composed of the two different processes, i.e., fractional differenced noise and auto regressive moving average. ARFIMA's aim is to examine the attribute of long memory, and for data presented as time series y t n È É , it is: where t n e 0; σ 2 ð Þ is the statistic process (white noise process) with zero mean and variance σ 2 , The gamma function is marked by Γ(*) and the number of differences necessary to present the stationary series is marked with d. The d th power of the differencing operator, included in Eq. (1) is marked with (1 − B) d .
When the value of the differencing parameter is in range (−0.5, 0.5), the ARFIMA model can be described as stationary, and if the value of the differencing parameter belongs to (0, 0.5), the process is characterized as a longmemory behavior. If there are suitable k differences, it is possible to transform many non-stationary processes into stationary ones by fulfilling condition (1). Consequently, the non-stationary processes obtain the long-memory attribute [36]. It is possible to predict the ARFIMA processes by means of an infinite autoregressive representation of formula (1), recorded as Π B ð Þy t n ¼ t n , also where From the perspective of numerical realization, the above equation requires truncation after k lags; nevertheless, it is not easy to obtain. The difficulty in truncation will influence the forecast horizon included in predictions (see [36]). Formula (2) explains that the predicting rule absorbs the impact of the remote lags, by which it captures their persistent impact. However, if shifts appear in the process, the pre-shift lags will also influence the prediction, and in consequence, the postshift horizons may have some biases [37,38].

Estimation and selection of parameters of the ARFIMA model
To find a proper prognostic model, contrary to using the highest number of precise parameters that describe the variability of the analyzed data presented as time series, it is of crucial importance to understand that too large adjustment of series may provide either the description of the signal itself or the random noise (that may show accidental regularity in a definite number of attempts). Therefore, the main aim is to find a model which, with the use of a limited number of statistically important parameters, will be able to describe the essential features of the analyzed time series.
There are two relatively simple and effective methods for calculation of the autoregressive models' parameters: maximum likelihood estimation (MLE) and quasimaximum likelihood estimation (QMLE) [39,40]. For the MLE, the basic computational problem is finding the solution of the following equation: where θ is the calculated data set, L T (ϑ) is the likelihood function, and T is the quantity of modeled parameters controls. For many cases, it is impossible to find analytical solution to Eq. (3) for the defined form of the model; thus, the numerical estimation is used. Using the maximum likelihood method requires establishing the complete model, hence the formed estimator's sensitivity to possible mistakes in procedure of the auto regressive (AR) and moving average (MA) polynomials that define the dynamics of the process. A universal criterion for selecting model's form does not exist. The common practice is that mapping the model onto the data is most optimal when the model's likelihood function and level of complexity increase concurrently. Nevertheless, there is a bigger possibility of an error occurrence when a greater number of parameters is being estimated. Therefore, one should seek an optimization of the quantity of parameters that appears i.e., they suggest selecting the form that possesses the minimal value of information criterion [41]. Below we present results of parameter estimation obtained by means of MLE and QMLE methods, and the ARFIMA form of the model. The differentiation parameter d value is calculated with the use of the mentioned techniques and the HR estimator, which is developed by Haslett and Raftery [42]. Furthermore, we calculated the selection of the row of the analyzed model using exponential smoothing in the state space and such information factors as Akaike (AIC) (see Hyndman and Khandakar [43]). Owing to the above approach, we were able to obtain satisfactory computational efficiency and automatic realization of the used algorithms.

MLE method in estimation of parameter d
The analysis of an ARFIMA process Z t from the perspective of the Gaussian Log-likelihood presented by formula (1) that refers to with z = (z 1 , z 2 , …, z n ) t being the vector described by parameter, and ϱ = (σ 2 , H e ), Σ describes the n x n covariance matrix of Z relying on z and ϱ, H e denotes the Hurst exponent and where the determinant of Σ is described by |Σ|. The MLE ofρ may be calculated by the maximum value of log G L (z; ϱ) respectively to ϱ. The calculation of first partial derivative of formula (4) has been described by The maximum likelihood estimation ϱ̂creates the result of the G 0 L z; ϱ ð Þ ¼ 0 . Provided that the parameters present high dimension, or there is a long time series, it is difficult to compute the exact MLE due to its numerical instability, for the formula (5) stimulates the estimation of the determinant and the elements of matrix Σ are inversed [36,44,45].
Out of numerous analogous MLE methods that can be conveyed by calculation of approximation of the likelihood function, we decided to use the HR estimator based on a quick and precise Haslett and Raftery's method [42], whose heuristic idea is to achieve autoregressive approximations. Such autoregressive, infinite order process may represent a Gaussian ARFIMA.
However, since the quantity of samples is definite, the truncated model is obtained in accordance with m < t ≤ n, with ϱ being the coefficients of the formula Φ(B)Θ(B)(1 − B) d . Since approximating as well as refining are performed, a QMLE ofρ is brought about by the operation of maximization where C is a constant, A more extensive study on this approximation method can be found in [42].

The calculation and selection of model features
The ways of exponential smoothing of the models of state space are obtained as follows: where {ϵ t } is a Gaussian white noise process with zero mean and variance σ 2 , and μ t = W(a t − 1 ). The sample containing additional mistakes has R(a t − 1 ) = 1, and consequently, b t = μ t + ϵ t . The analyzed model with multiplicative mistakes has R(a t − 1 ) = μ t , and consequently, b t = μ t (1 + ϵ t ). Hence, ϵ t = (b t − μ t )/μ t creates the mistake related to the multiplicative model. The models created as a result of this action are not distinctive. Apparently, each value of R(a t − 1 ) results in creating identical prediction points for b t . The values of a 0 and the parameter ϑ are necessary for these models to be useful in terms of forecasting. Hence, calculating the likelihood of the improvements of models of state space (8) does not create difficulties, and so is achieving the maximum likelihood estimates: It is easy to calculate (9) with the recursive equations in [43,46]. As far as multiple sources of mistakes of the state space models are concerned, it is necessary to apply the Kalman filter to estimate likelihood; our calculations are free of that requirement. The sets of the parameters ϑ and the initial states a 0 are realized by operation of computing of minimization of G Ã L . The method for choosing the present model is based on the Akaike criterium where V AIC is the value of AIC, n p creates the quantity of parameters in ϑ along with the quantity of free states in a 0 , andθ and â 0 define the calculations of ϑ together with a 0 . From the models applicable for the data, we selected the one that minimizes the AIC. On the basis of the mentioned ideas we achieve an efficient and commonly appropriate algorithm for automatic predicting. To summarize, the stages of the undertaken performances are as follows [43]: Stage 1: optimize the parameters, i.e., smooth them and the initial state variable, for every series use all the matching models Stage 2: choose the most effective model in terms of AIC Stage 3: create point forecasts with the use of most applicable model (with parameters that are created in the optimization process) for any stages in advance as necessary A thorough explanation of the proposed method is described in Hyndman and Khandakar work [43].
In this section, we describe our experimental setup which has been implemented on a real-life railroad transportation system. Using this installation, a set of experiments has been performed to prove that the approach proposed in this paper is feasible and effective.

Experimental setup
As mentioned above, experimental results presented in this paper have been obtained using real-world installation placed on the active railroad crossings. Railroad gates tracking component that is based on WSN sensors is a part of the more comprehensive system for supervising and visualization railroad crossings. The presented system is an original solution for supervising critical infrastructure of railroad crossings.
In Fig. 4, main components of the control and visualization system for railroad crossings is illustrated. Telecommunication infrastructure for one railroad crossing consists of WSN sensors used for analyzing the state and position of the railroad crossing separate gates, industrial SCADA computer, classic Intrusion Detection System (IDS) using previously known attacks signatures database, firewall, and multi-WAN router for providing different links to the WAN network. In our solution, we propose an anomaly detection system (ADS) for the WSN part that is complementary to the implemented classic IDS. ADS system obtains the WSN traffic from Ethernet link provided by the WSN IP gateway. As already mentioned proposed in this paper, a novel detection approach is based on the statistical model with long memory-ARFIMA. Parameters of the ARFIMA models obtained for different traffic features are stored in ADS database. For every railroad crossing, a separate ADS instance exists and the same telecommunication infrastructure. WAN routers are used for communication with control and visualization management application. Railroad crossings were situated on the same rail link and connected to the control and visualization system by fiber or radio WAN connection, e.g., by means of long-term evolution (LTE). Practical realization of WSN IP gateway is depicted in Fig. 5 where a printed circuit board and the gateway installed on one of the railroad crossing column are presented.
WSN sensors are installed on the top of railroad gates. Installed sensors on the railroad crossing and a sensor printed circuit board are presented in Fig. 6.The sensor is powered by battery banks and additionally supported by a small solar panel. Static position/tilt of the railroad gate is measured by a Microelectromechanical Systems (MEMS) sensor which provides position in three dimensions-x, y, and z. WSN sensors transmit packets in an idle state (gates are not moving) in approximately constant periods of time. In the idle state, we control physical presence of railroad gates and battery health, signal strength (RSSI), ambient temperature, and gates' three dimensional position. In the idle state, insignificant railroad gates' movements resulting from, e.g., wind or vibrations caused by heavy vehicles are not taken into account. For the sake of reliability, packets from sensors are received by two redundant IP gateways. Every sensor transmits packets to the WSN IP gateway when triggered by railroad gates' movements.
In Fig. 7, part of the railroad crossings management application with railroad gates visualization is depicted. One   Fig. 7). The list below button ribbons provides information, for example, about the state and the position of separate railroad gates (e.g., gate open, closed, changing position, broken gate). As mentioned, presented in Fig. 7 screenshot is a part of the more comprehensive system for analyzing railroad crossings critical infrastructure which considers also, for example, video images.

Experimental results
As mentioned before, the proposed anomaly detection system comprises two main steps. In the first step, we remove outlier values (see Section 3.1) for every observed traffic feature (see Table 1). This step prepares data for the next step where parameters of the statistical model with long memory dependence are calculated. The second step is based on the calculation of parameters of ARFIMA statistical model (see Section 3.2). We selected seven features which are related to the most important functionalities of railroad crossing critical infrastructure (see Table 1). For every WSN traffic feature, we achieve forecasting interval (30 samples forecasting horizon) based on ARFIMA model (see Figs WSN traffic features presented in Table 1 are captured from the Ethernet link of the WSN IP gateway. WSN sensors transmit packets to WSN IP gateways placed on both sides of the railroad crossing. WSN gateway converts received packets to IP packets. Then, packets converted by WSN IP gateway are captured in the next step by software sensor installed on railroad industrial computer. Every WSN traffic feature presented in Table 1 is extracted from packets captured by the IP network sensor.
In a subsequent step, WSN traffic features are in real time processed by the proposed anomaly detection solution that indicates possible anomaly/attack when the value of the online calculated traffic feature is outside an interval determined by two prediction intervals. When values for a given traffic feature are inside 80 % prediction intervals, we assume that there is no anomaly/attack for a given traffic feature. When WSN traffic features lie inside interval described by 80 to 95 % of prediction intervals, we treated this traffic as suspicious where an anomaly or attack can be present. Traffic features with values outside 95 % prediction intervals triggers anomaly/attack by anomaly detection algorithm. The proposed anomaly detection method for WSN traffic has been tested with different anomaly/attack scenarios assumed. Because railroad crossing is a critical railway infrastructure anomaly/attacks had to be simulated and carefully controlled in order to preserve safety on active testing railroad crossings. In this paper, we evaluated the following anomaly/attack scenarios:  Table 2: Scenario 2 can be understood as a situation where, for example, railroad gates will be hit or bent by a vehicle (but sensors are still able to communicate with the IP gateway). In this scenario, the most noticeable impact will be seen for features responsible for measurement of three dimensional positions of railroad gates-F2, F3, and F4. Results for this scenario are presented in Table 3.
A different variant of scenario 2 covers situations where railroad gates will be moved outside the railroad crossing area, but the sensor will not be damaged or sensors and railroad gates will be completely damaged. In these cases, we can observe an impact on features F2, F3, F4, F5, F6, and F7. Results can be observed in Table 4.
In scenario 3, WSN IP gateway was used to perform an attack. The aim of this attack was to drain batteries or delay packets from sensors. This attack requires the knowledge of the specific communication protocol between sensors and the IP gateway. F1, F6, and F7 are features influenced by this attack scenario (see Table 5).
Experimental results presented in Table 6 contains aggregated detection rate together with false positive for all seven WSN traffic features (see Table 1). Based on the obtained results, an overall performance of the proposed ADS solution can be observed taking into account all anomaly/attack scenarios described earlier. Most of the anomaly/attacks have been successfully identified. Detection rates varied between 93 and 98 %, while false positive rates were below 9 %. The best results have been achieved for WSN features: F2, F3, and F4 calculated based on readings from MEMS sensors and feature F1 (but only for scenario 3).

Conclusions
Ensuring a proper level of security for resources and systems of critical infrastructure, particularly transportation ones, realized as sensor radio networks is currently an intensively explored research topic. It is apparent that WSN, due to their nature, are vulnerable to a substantial number of threats originating both from the outside and inside of their own infrastructure. Therefore, these networks require ensuring the integrity and confidentiality   of the transmission, as well as protection of nodes and data transferred with their use. While developing mechanisms, algorithms, or protocols that increase transmission security in WSN, one also needs to consider the restrictions imposed by the unique characteristics of WSN, such as self-organization, dislocation, equipment limitations, and ease of fiasco of nodes and protocols. The increasing number of novel attacks, their global scope, and complexity level enforce dynamic development of network security systems. The most often implemented solution aiming at ensuring security are detection and classification methods that allow to identify abnormal behaviors reflected in the analyzed network traffic.
The advantage of such an approach is the protection against so far unknown attacks, developed specially (targeted attacks) in order to realize attacks onto particular resources of network infrastructures or simply constituting so called zero-day exploits. Anomaly detection systems may play a crucial role in those environments. Their purpose is to detect (for auto-response) unusual traffic behavior representing symptoms of unauthorized actions directed against protected critical infrastructure resources, implemented as WSN networks.
That is why in this paper we introduced a novel network traffic anomaly detection method for a critical railway transportation infrastructure which is utilizing sensor radio network. In order to detect anomalies, differences between the actual network traffic and the estimated ARFIMA model of that traffic for the analyzed WSN network parameters were used. For the purpose of suitable preparation of data for statistical modeling, observations outlying in the analyzed WSN network parameters with the use of a simple and fast onedimensional quartile criterion were found and eliminated. Parameter estimation and identification of the row of the ARFIMA statistical models were realized as a compromise between the model's coherence and the size of its estimation error. The obtained experimental results performed on the real-life railway crossings infrastructure confirm efficacy and accuracy of the presented anomaly detection method. We achieved overall detection rates varied between 93 and 98 %, while false positive rates were below 9 %. Most valuable WSN features for anomaly detection purposes were F1, F2, and F3 and they were based on reading from MEMS sensor. In case of sensor failure, we took into account other undamaged sensors readings in order to ensure system successful operation. To conclude, we consider utilization of an efficient IDS system as a must in every railway critical infrastructure management system. Future work will be related to further exploring the most efficient set of parameters used for the proposed network traffic anomaly detection method. Moreover, we also plan experiments on a greater scale and during a longer time period which will allow to further tune the proposed solution and to model the abnormal behaviors even better.