On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> Probabilistic clustering techniques use the concept of memberships to describe the degree by which a vector belongs to a cluster. The use of memberships provides probabilistic methods with more realistic clustering than "hard" techniques. However, fuzzy schemes (like the fuzzy c means algorithm, FCM) are often sensitive to outliers. We review four existing algorithms, devised to reduce this sensitivity. These are: the noise cluster (NC) algorithm of Dave (1991), the possibilistic c means (PCM) scheme of Krishnapuram and Keller (1996), the least biased fuzzy clustering (LBFC) method of Beni and Liu (1994), and the fuzzy possibilistic c means algorithm of Pal et al. (1997). We then propose the new credibilistic fuzzy c means (CFCM) algorithm to improve on these methods. It uses a new variable, credibility of a vector, which measures the typicality of the vector to the whole data set. By taking credibility into account CFCM generates centroids which are less sensitive to outliers than other techniques, and closer to the centroids generated when the outliers are artificially removed. <s> BIB001 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> Essentially all data mining algorithms assume that the data-generating process is independent of the data miner's activities. However, in many domains, including spam detection, intrusion detection, fraud detection, surveillance and counter-terrorism, this is far from the case: the data is actively manipulated by an adversary seeking to make the classifier produce false negatives. In these domains, the performance of a classifier can degrade rapidly after it is deployed, as the adversary learns to defeat it. Currently the only solution to this is repeated, manual, ad hoc reconstruction of the classifier. In this paper we develop a formal framework and algorithms for this problem. We view classification as a game between the classifier and the adversary, and produce a classifier that is optimal given the adversary's optimal strategy. Experiments in a spam detection domain show that this approach can greatly outperform a classifier learned in the standard way, and (within the parameters of the problem) automatically adapt the classifier to the adversary's evolving manipulations. <s> BIB002 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> Assuming an ellipsoidal model of uncertainty a robust formulation for classifying noisy data is presented. The formulation is a convex optimization problem, in particular it is a instance of second order cone programming problem. The formulation is derived from a worst case consideration and the robustness properties hold for a large class of distributions. The equivalence of ellipsoidal uncertainty and Gaussian noise models is also discussed. The generalized optimal hyperplane is recovered as a special case of the robust formulation. Experiments on real world datasets illustrates the efficacy of the formulation. <s> BIB003 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> Unsupervised or unlabeled learning approaches for network anomaly detection have been recently proposed. In particular, recent work on unlabeled anomaly detection focused on high speed classification based on simple payload statistics. For example, PAYL, an anomaly IDS, measures the occurrence frequency in the payload of n-grams. A simple model of normal traffic is then constructed according to this description of the packets' content. It has been demonstrated that anomaly detectors based on payload statistics can be "evaded" by mimicry attacks using byte substitution and padding techniques. In this paper we propose a new approach to construct high speed payload-based anomaly IDS intended to be accurate and hard to evade. We propose a new technique to extract the features from the payload. We use a feature clustering algorithm originally proposed for text classification problems to reduce the dimensionality of the feature space. Accuracy and hardness of evasion are obtained by constructing our anomaly-based IDS using an ensemble of one-class SVM classifiers that work on different feature spaces. <s> BIB004 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> A very effective means to evade signature-based intrusion detection systems (IDS) is to employ polymorphic techniques to generate attack instances that do not share a fixed signature. Anomaly-based intrusion detection systems provide good defense because existing polymorphic techniques can make the attack instances look different from each other, but cannot make them look like normal. In this paper we introduce a new class of polymorphic attacks, called polymorphic blending attacks, that can effectively evade byte frequency-based network anomaly IDS by carefully matching the statistics of the mutated attack instances to the normal profiles. The proposed polymorphic blending attacks can be viewed as a subclass of the mimicry attacks. We take a systematic approach to the problem and formally describe the algorithms and steps required to carry out such attacks. We not only show that such attacks are feasible but also analyze the hardness of evasion under different circumstances. We present detailed techniques using PAYL, a byte frequency-based anomaly IDS, as a case study and demonstrate that these attacks are indeed feasible. We also provide some insight into possible countermeasures that can be used as defense. <s> BIB005 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> How do we find a natural clustering of a real world point set, which contains an unknown number of clusters with different shapes, and which may be contaminated by noise? Most clustering algorithms were designed with certain assumptions (Gaussianity), they often require the user to give input parameters, and they are sensitive to noise. In this paper, we propose a robust framework for determining a natural clustering of a given data set, based on the minimum description length (MDL) principle. The proposed framework, Robust Information-theoretic Clustering (RIC), is orthogonal to any known clustering algorithm: given a preliminary clustering, RIC purifies these clusters from noise, and adjusts the clusterings such that it simultaneously determines the most natural amount and shape (subspace) of the clusters. Our RIC method can be combined with any clustering technique ranging from K-means and K-medoids to advanced methods such as spectral clustering. In fact, RIC is even able to purify and improve an initial coarse clustering, even if we start with very simple methods such as grid-based space partitioning. Moreover, RIC scales well with the data set size. Extensive experiments on synthetic and real world data sets validate the proposed RIC framework. <s> BIB006 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> This paper aims to introduce the robustness against noise into the spectral clustering algorithm. First, we propose a warping model to map the data into a new space on the basis of regularization. During the warping, each point spreads smoothly its spatial information to other points. After the warping, empirical studies show that the clusters become relatively compact and well separated, including the noise cluster that is formed by the noise points. In this new space, the number of clusters can be estimated by eigenvalue analysis. We further apply the spectral mapping to the data to obtain a low-dimensional data representation. Finally, the K-means algorithm is used to perform clustering. The proposed method is superior to previous spectral clustering methods in that (i) it is robust against noise because the noise points are grouped into one new cluster; (ii) the number of clusters and the parameters of the algorithm are determined automatically. Experimental results on synthetic and real data have demonstrated this superiority. <s> BIB007 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> Incorporating invariances into a learning algorithm is a common problem in machine learning. We provide a convex formulation which can deal with arbitrary loss functions and arbitrary losses. In addition, it is a drop-in replacement for most optimization algorithms for kernels, including solvers of the SVMStruct family. The advantage of our setting is that it relies on column generation instead of modifying the underlying optimization problem directly. <s> BIB008 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> The standard assumption of identically distributed training and test data is violated when an adversary can exercise some control over the generation of the test data. In a prediction game, a learner produces a predictive model while an adversary may alter the distribution of input data. We study single-shot prediction games in which the cost functions of learner and adversary are not necessarily antagonistic. We identify conditions under which the prediction game has a unique Nash equilibrium, and derive algorithms that will find the equilibrial prediction models. In a case study, we explore properties of Nash-equilibrial prediction models for email spam filtering empirically. <s> BIB009 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> Pattern recognition systems are increasingly being used in adversarial environments like network intrusion detection, spam filtering and biometric authentication and verification systems, in which an adversary may adaptively manipulate data to make a classifier ineffective. Current theory and design methods of pattern recognition systems do not take into account the adversarial nature of such kind of applications. Their extension to adversarial settings is thus mandatory, to safeguard the security and reliability of pattern recognition systems in adversarial environments. In this paper we focus on a strategy recently proposed in the literature to improve the robustness of linear classifiers to adversarial data manipulation, and experimentally investigate whether it can be implemented using two well known techniques for the construction of multiple classifier systems, namely, bagging and the random subspace method. Our results provide some hints on the potential usefulness of classifier ensembles in adversarial classification tasks, which is different from the motivations suggested so far in the literature. <s> BIB010 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> The standard assumption of identically distributed training and test data is violated when test data are generated in response to a predictive model. This becomes apparent, for example, in the context of email spam filtering, where an email service provider employs a spam filter and the spam sender can take this filter into account when generating new emails. We model the interaction between learner and data generator as a Stackelberg competition in which the learner plays the role of the leader and the data generator may react on the leader's move. We derive an optimization problem to determine the solution of this game and present several instances of the Stackelberg prediction game. We show that the Stackelberg prediction game generalizes existing prediction models. Finally, we explore properties of the discussed models empirically in the context of email spam filtering. <s> BIB011 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> In adversarial classification tasks like spam filtering, intrusion detection in computer networks, and biometric identity verification, malicious adversaries can design attacks which exploit vulnerabilities of machine learning algorithms to evade detection, or to force a classification system to generate many false alarms, making it useless. Several works have addressed the problem of designing robust classifiers against these threats, although mainly focusing on specific applications and kinds of attacks. In this work, we propose a model of data distribution for adversarial classification tasks, and exploit it to devise a general method for designing robust classifiers, focusing on generative classifiers. Our method is then evaluated on two case studies concerning biometric identity verification and spam filtering. <s> BIB012 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> In adversarial classication tasks like spam ltering and intrusion detection, malicious adversaries may manipulate data to thwart the outcome of an automatic analysis. Thus, besides achieving good classication performances, machine learning algorithms have to be robust against adversarial data manipulation to successfully operate in these tasks. While support vector machines (SVMs) have shown to be a very successful approach in classication problems, their eectiveness in adversarial classication tasks has not been extensively investigated yet. In this paper we present a preliminary investigation of the robustness of SVMs against adversarial data manipulation. In particular, we assume that the adversary has control over some training data, and aims to subvert the SVM learning process. Within this assumption, we show that this is indeed possible, and propose a strategy to improve the robustness of SVMs to training data manipulation based on a simple kernel matrix correction. <s> BIB013 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> The standard assumption of identically distributed training and test data is violated when the test data are generated in response to the presence of a predictive model. This becomes apparent, for example, in the context of email spam filtering. Here, email service providers employ spam filters, and spam senders engineer campaign templates to achieve a high rate of successful deliveries despite the filters. We model the interaction between the learner and the data generator as a static game in which the cost functions of the learner and the data generator are not necessarily antagonistic. We identify conditions under which this prediction game has a unique Nash equilibrium and derive algorithms that find the equilibrial prediction model. We derive two instances, the Nash logistic regression and the Nash support vector machine, and empirically explore their properties in a case study on email spam filtering. <s> BIB014 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> We propose a non-convex training objective for robust binary classification of data sets in which label noise is present. The design is guided by the intention of solving the resulting problem by adiabatic quantum optimization. Two requirements are imposed by the engineering constraints of existing quantum hardware: training problems are formulated as quadratic unconstrained binary optimization; and model parameters are represented as binary expansions of low bit-depth. In the present work we validate this approach by using a heuristic classical solver as a stand-in for quantum hardware. Testing on several popular data sets and comparing with a number of existing losses we find substantial advantages in robustness as measured by test error under increasing label noise. Robustness is enabled by the non-convexity of our hardware-compatible loss function, q-loss. <s> BIB015 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> We analyze the problem of designing pattern recognition systems in adversarial settings, under an engineering viewpoint, motivated by their increasing exploitation in security-sensitive applications like spam and malware detection, despite their vulnerability to potential attacks has not yet been deeply understood. We first review previous work and report examples of how a complex system may be evaded either by leveraging on trivial vulnerabilities of its untrained components, e.g. parsing errors in the pre-processing steps, or by exploiting more subtle vulnerabilities of learning algorithms. We then discuss the need of exploiting both reactive and proactive security paradigms complementarily to improve the security by design. Our ultimate goal is to provide some useful guidelines for improving the security of pattern recognition in adversarial settings, and to suggest related open issues to foster research in this area. <s> BIB016 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> Today's systems produce a rapidly exploding amount of data, and the data further derives more data, forming a complex data propagation network that we call the data's lineage. There are many reasons that users want systems to forget certain data including its lineage. From a privacy perspective, users who become concerned with new privacy risks of a system often want the system to forget their data and lineage. From a security perspective, if an attacker pollutes an anomaly detector by injecting manually crafted data into the training data set, the detector must forget the injected data to regain security. From a usability perspective, a user can remove noise and incorrect entries so that a recommendation engine gives useful recommendations. Therefore, we envision forgetting systems, capable of forgetting certain data and their lineages, completely and quickly. This paper focuses on making learning systems forget, the process of which we call machine unlearning, or simply unlearning. We present a general, efficient unlearning approach by transforming learning algorithms used by a system into a summation form. To forget a training data sample, our approach simply updates a small number of summations -- asymptotically faster than retraining from scratch. Our approach is general, because the summation form is from the statistical query learning in which many machine learning algorithms can be implemented. Our approach also applies to all stages of machine learning, including feature selection and modeling. Our evaluation, on four diverse learning systems and real-world workloads, shows that our approach is general, effective, fast, and easy to use. <s> BIB017 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> Pattern recognition and machine learning techniques have been increasingly adopted in adversarial settings such as spam, intrusion, and malware detection, although their security against well-crafted attacks that aim to evade detection by manipulating data at test time has not yet been thoroughly assessed. While previous work has been mainly focused on devising adversary-aware classification algorithms to counter evasion attempts, only few authors have considered the impact of using reduced feature sets on classifier security against the same attacks. An interesting, preliminary result is that classifier security to evasion may be even worsened by the application of feature selection. In this paper, we provide a more detailed investigation of this aspect, shedding some light on the security properties of feature selection against evasion attacks. Inspired by previous work on adversary-aware classifiers, we propose a novel adversary-aware feature selection model that can improve classifier security against evasion attacks, by incorporating specific assumptions on the adversary's data manipulation strategy. We focus on an efficient, wrapper-based implementation of our approach, and experimentally validate its soundness on different application examples, including spam and malware detection. <s> BIB018 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Attack Defenses <s> In spam and malware detection, attackers exploit randomization to obfuscate malicious data and increase their chances of evading detection at test time, e.g., malware code is typically obfuscated using random strings or byte sequences to hide known exploits. Interestingly, randomization has also been proposed to improve security of learning algorithms against evasion attacks, as it results in hiding information about the classifier to the attacker. Recent work has proposed game-theoretical formulations to learn secure classifiers, by simulating different evasion attacks and modifying the classification function accordingly. However, both the classification function and the simulated data manipulations have been modeled in a deterministic manner, without accounting for any form of randomization. In this paper, we overcome this limitation by proposing a randomized prediction game, namely, a noncooperative game-theoretic formulation in which the classifier and the attacker make randomized strategy selections according to some probability distribution defined over the respective strategy set. We show that our approach allows one to improve the tradeoff between attack detection and false alarms with respect to the state-of-the-art secure classifiers, even against attacks that are different from those hypothesized during design, on application examples including handwritten digit recognition, spam, and malware detection. <s> BIB019
We will now cover a few approaches to defending against the attacks discussed in Section 6. Note that a complete survey is outside the scope of this work, so we only focus on defenses relevant to C&C detection systems. For a more detailed discussion on defenses against attacks, we refer the reader to BIB016 . Defending against ML attacks is a difficult task. Defenders can take both reactive and proactive approaches BIB016 . In a reactive approach, the defender observes an attack and then incorporates countermeasures into the existing system. In a proactive approach, the defender anticipates the adversary's strategy and develops countermeasures before deployment. The majority of defenses are proactive in nature, attempting to limit the capability of the adversary before deployment. In particular, the two common approaches are the use of multiclassifier systems (MCSs) and ideas from game theory to predict and model the attack at training without having access to attacker data. One of the most common approaches to defend against evasion attacks on classifiers is to make use of MCSs  BIB018 BIB010 , in which the combined output of several classifiers is used to make a decision. The different classifiers vary by either using different subsets of the feature set or different subsets of the training data. One variation of MCSs is to make use of one-class classifiers, which can be more effective in identifying attacks. One-class classifiers produce a tight decision surface around the target objects (e.g., the benign objects) and output how much a data point matches that target rather than outputting a label from a set. These work particularly well when combined with two-class classifiers, as described by . They find that whereas two-class classifiers achieve better accuracy when not under attack, one-class classifiers achieve better accuracy while under attack. The classifiers are all based on SVMs. By combining two one-class classifiers and one two-class classifier, a more robust MCS is produced. Performance of the MCS is less affected when under attack, particularly good word insertion (GWI)/bad word obfuscation (BWO) attacks against spam filters and adding features in PDF files, compared to a single SVM classifier. BIB004 make use of a set of one-class SVM classifiers to provide resistance against mimicry attacks in payload-based IDSs (e.g., PAYL). When tested against polymorphic blending attacks BIB005 ], the SVM-based multiclassifier IDS is able to detect the occurrence of all attacks with a 0.5% FP rate (correctly identifying 99.2% of all attack packets). BIB010 evaluate the robustness of two approaches to MCSs: bagging (in which each classifier is trained with a subset of the training data) and the random subspace method (RSM, where each classifier is trained with all of the data but a subset of the feature set). The two approaches are tested using linear SVM and LR in the spam detection scenario under the GWI/BWO evasion attack (with both PK and LK). When not under attack, the bagging and RSM MCSs almost always outperform the single classifiers. Under the PK attack, the bagging method reduces the impact of the attack for both the SVM and LR classifiers, whereas the RSM method only improves the robustness of LR. In the LK case, LR is improved slightly by both the bagging and RSM methods, whereas SVM is improved by bagging only. BIB018 present a method for performing feature selection while providing robustness to evasion attacks. Feature selection, where only a subset of the possible features are used for training, is used to reduce the time and computational complexity of the algorithm, and provides better learning on smaller training sets. However, this can lead to further evasion if the adversary is aware of the features that are selected due to the fact that there are fewer features to attack. The proposed method, wrapper-based adversarial feature selection, is based on the popular forward feature selection and backward feature elimination algorithms. The algorithm is tested in the spam detection scenario using a linear SVM classifier and in the PDF malware detection scenario using SVM with the RBF kernel. The algorithm produces relatively robust classifiers when subjected to gradient descent-based evasion attacks compared to the forward feature selection, backward feature selection, and adversarial backward feature selection algorithms.  use an approach to MCS that involves a feature reweighting step as an alternative to feature selection. In the reweighting step, the classifier is trained in two passes. In the first pass, the classifier (SVM or LR) is trained outputting feature weights. In the second pass, the classifier is retrained using feature weights inverse to the feature importance, derived from the weights output in the previous pass. When tested on spam email detection under GWI/BWO evasion attacks, an MCS built using 10 LR classifiers, differing by the included features (each has 50% of the total feature set), is shown to be able to withstand the attack to a higher degree than standard LR, a single reweighted LR, or fscale LR (a variant of a worst-case feature noise injection algorithm). Game theory has been used to incorporate knowledge of attacker capability into learning algorithms. Broadly, the goal is to find an equilibrium representing the optimal strategy against the opponent (where no player benefits from deviating from the set of actions represented by the equilibrium). One such approach is to incorporate Stackleberg games, which are sequential games in which the follower (adversary) can observe and react to the leader's (learner) action. BIB011 apply a Stackleberg prediction game to the field of spam email detection. The Stackleberg equilibrium is found using an optimization function. When tested on four email datasets, the approach is shown to outperform SVM, LR, and Nash logistic regression (NLR) in accurately identifying spam emails. In a static prediction game, the learner and adversary both act simultaneously, without prior information on the opponent's move. BIB009 provide a method for finding the unique Nash equilibrium in a static game, focused on the email spam detection example. The equilibrium is found using a convex loss function. BIB014 revise the earlier work of BIB009 to repair the theorem dictating under which conditions a unique Nash equilibrium for a game exists and develop NLR and Nash SVM (NSVM). When tested on email spam detection, the NLR and NSVM variants outperform the non-Nash variants. BIB019 extend this approach to incorporate randomization on behalf of the attacker. In this approach, player strategies are replaced with probability distributions defined over the strategy sets. As with NSVM, a Nash equilibrium is found, although the requirement for a unique equilibrium is relaxed. A randomized SVM classifier (RNashSVM) is produced with both a linear and RBF kernel. The approach is found to produce decision boundaries tight to the legitimate points, resulting in a more difficult task for the attacker, but increasing the amount of FPs. The linear approach is compared against standard SVM, NSVM, and InvarSVM BIB008 . When tested against handwritten digits, a spam detection dataset, and a PDF malware dataset, the approach is shown to perform well when not under attack in all cases (except for the PDF malware test) and is shown to be more robust to attack than traditional SVM and the two previous secure variants, requiring greater levels of modification on behalf of the attacker to provide similar levels of performance degradation. A different approach is to incorporate attack data into the training set to increase classifier robustness. BIB012 propose a system that addresses a lack of attack data for use in training by making use of a generative classifier and incorporating a model of the expected attack into the training phase. The technique is evaluated in two settings. First, it is evaluated in the biometric identity verification scenario, where the LLR fusion rule is extended with the model. The extended LLR is shown to be slightly less effective when compared to standard LLR when not under attack but achieves significantly better performance when under attack. The technique is also tested in the spam filtering case using a modified naive Bayes text classifier (as used in SpamBayes) against the GWI/BWO attack. The modified naive Bayes is shown to outperform the standard naive Bayes. In both cases, the defesce reduces the effect of the attack rather than provides complete protection. Machine unlearning BIB017 ] is a reactive approach that enables the defender to remove compromised (poisoned) data from a training set without having to retrain the classifier. This approach is effective as long as the defender knows what to unlearn. Therefore, the defender needs some form of attack detection in place. Another common approach is to take an existing ML algorithm, and modify it to provide resilience to a particular adversary BIB002 , such as through retraining frequently. This approach is limited by the fact that it is an arms race between the defenders keeping the system up to date and attackers producing new attack variants. Noise robustness techniques in ML have been around for several years. In the case of clustering, these techniques are generally focused on the handling of outliers or Gaussian noise BIB001  BIB007 BIB006 , whereas in the case of classification, the goal is to handle label noise BIB013 BIB003 BIB015 . However, these techniques are focused on "naturally occurring" noise rather than adversarial noise, and thus the techniques are likely to be circumventable by the well-motivated attacker. Of course, there are limitations to the mentioned defenses. In almost all cases, the defenses do not provide immunity to the discussed attacks. Although total immunity is impossible without already knowing the attack samples, the level of protection is often not enough. The usual result is that the effect of the attack is reduced, but the attacks can still be successfully performed, just with more effort on behalf of the attacker. In almost all cases, the defenses are evaluated against a single attack with one or two limited datasets. Spam email detection is the motivating example in the majority of the defense papers mentioned earlier, with GWI/BWO attacks being common. The emails are usually converted into a bag-of-words feature vector made up of binary values. This evaluation methodology leaves two points to consider. What is the effectiveness of the defenses against further attacks? And how well will the defenses work in situations with far more complex, non-binary features such as in the case of malware C&C detection? There is a lack of evaluation in other security applications, such as C&C detection, where ML use is common. Part of this issue is left to the authors of detection systems themselves, as where they may evaluate with a few different ML algorithms, the set of algorithms will consist of nonsecure variants. Whereas spam detection and PDF malware detection (one of the other common use cases in the defense papers) both share similarities to some C&C detection systems, such as those performing analysis on packet payloads, others such as those focused on complex features like packet timings and malware behavior represent far more complex problems. Some defenses, namely those based on game theory, rely on the attacker performing as expected by the attacker model. However, in reality, the attacker may not behave as expected (e.g., using the feedback of the learning algorithm to formulate future attacks) BIB016 . Probabilistic models such as those described in BIB012 rely on an accurate prediction of the behavior of the attacker. However, as in any security problem, the attacker may orchestrate an attack that has not been considered.