-5 mm  many problems in biology , chemistry and material science can be formulated as the study of the energy or free energy landscape of the underlying system .well - known examples of such problems include the conformational changes of macromolecules , chemical reactions and nucleation in condensed systems .very often the dimension of the state space is very large , and the energy landscape exhibits a hierarchy of structures and scales .these problems are becoming a major challenge in their respective scientific disciplines and are beginning to receive attention from the mathematics community . in this article , we report recent work in this direction . for a detailed account , we refer to .we begin with a simple example .plotted in figure 1 is the solution of the stochastic differential equation where the potential and is gaussian white noise , . without the random perturbation ,the solution would be .indeed the deterministic part of dynamics in does nothing but taking the system to local equilibrium states . with the random perturbation ,the solution , over long time , exhibits completely different behavior .it fluctuates around the two local minima of and 1 , with sudden transitions between these two states .the time scale of the transition , is much larger than the time scale of the fluctuation around the local minima , . for this reason, we refer to and 1 as the metastable states .time series of the solution to the stochastic differential equation ( 1.1 ) , with .obviously the transition between the metastable states is of more interest than the local fluctuation around them .the transition time is much larger since it requires the system to overcome the energy barrier between the two states .this is only possible because of the noise .when is small , a huge noise term is required to accomplish this .for this reason , such events are very rare , and this is the origin of the disparity between the time scales and .this simple example illustrates one of the major difficulties in modeling such systems , namely the disparity of the time scales .it does not , however , illustrate the other major difficulty , namely , the large dimension of the state space and the complexity of the energy landscapes .indeed for typical systems of interests the energy landscape can be very complex .there can be a huge number of local minima in the state space .the usual concept of hopping over barriers via saddle points may not apply ( see ) .  in applications ,the noise comes typically from thermal noise . in this case, we should note that even though the potential energy landscapes might be rough and contain small scale features , the system itself experiences a much smoother landscape , the free energy landscape , since some of the small scale features on the potential energy landscape are smoothed out by the thermal noise .our objective in modeling such systems are the following :  1 .  find the transition mechanism between the metastable states .2 .  find the transition rates .3 .  reduce the original dynamics to the dynamics of a markov chain on the metastable states .our discussion will be centered around the following model problems : or  is related to the temperature of the system by where is the boltzmann constant .we refer to as type - i gradient flow and as type - ii gradient flow .  before proceeding further ,let us remark that there is a very well - developed theory , the large - deviation theory , or the wentzell - freidlin theory , that deals precisely with questions of the type that we discussed above .however as was explained in , this theory is not best suited for numerical purpose .therefore we will seek an alternative theoretical framework that is more useful for numerical computations .-5 mm  transition state theory ( tst ) has been the classical framework for addressing the questions we are interested in .it assumes the existence and explicit knowledge of a reaction coordinate , denoted by , that connects the two metastable states .in addition it assumes that along the reaction coordinate there exists a well - defined transition state , which is typically the saddle point configuration , say at , and the two regions and defines the two metastable regions and . for these reasons ,transition state theory is restricted to cases when the system is simple and the energy landscape is smooth , i.e. the energy barriers are larger than the thermal energy .  knowing the transition state, tst calculates the transition rates by placing particles at the transition state , and measuring the flux that goes into the two regions .for example , the transition rate from to is given approximately by where here is the surface delta function at , is the heaviside function , is the lebesgue measure restricted to . for a system with a single particle of mass and potential , this gives  where is the energy barrier at the transition state , , denotes the location of the local minimum inside .formulas such as are the origin of the arrhenius law for chemical reaction rates and boltzmann factor for hopping rates in kinetic monte carlo models .-5 mm  for simplicity , we will discuss mainly type - i gradient flows .the fokker - planck equation can be expressed as where is the equilibrium distribution  is the normalization constant .the states of the markov chain consist of the sets , where satisfies  1 .the s are mutually disjoint 2 .  where .an illustration of the collection the is given in figure 2 . depends on .as decreases , the s exhibits a hierarchical structure .illustration of the collection of metastable sets at different energies .having defined the states of the markov chain , we next compute the transition rates between neighboring states .denote by and two such neighboring states .we would like to compute the transition rate from to . without loss of generality, we may assume .let be the metastable region containing and respectively , and let , , . applying laplace s method to ( 3.1 )we get  where  is a so - called minimal energy path , to be defined below , is the family of hyperplanes normal to .the minimal energy path ( mep ) is defined as follows . if is smooth, then is a mep if for all $ ] , i.e. restricted to is parallel to . in generalthere is not a unique that is particularly significant , but rather a collection ( a tube or several tubes ) of paths contribute to the transition rates .however , one can define a mep self - consistently via the equation where . in the case when has two scales : , , and is smooth , then can be defined as the mep of .  for type - i gradient systems , if the two metastable sets are separated by a single saddle point , then the mep is the unstable manifold associated with the saddle point .mep for type - ii gradient systems is less trivial . in this case( 3.3)-(3.6 ) have to be modified .consider the simple example where .it has two local equilibrium states and .the mep that connects and is plotted in figure 3 .it is not a smooth curve .the velocity is reversed at the saddle point . to verify that the mep does reflect the true behavior of the transition path, we also plot the transition path obtained from direct simulation of the stochastic differential equation .mep for type - ii gradient systems .the red line is the mep in phase space , the green line is the transition path computed from solving the stochastic differential equation .mep is a very important concept since it defines the `` most probable '' transition path from which transition rates can be computed via equation .however we should emphasize that from a numerical point of view our task is not that of a conventional optimization or control problem , since there is not an objective function that we can easily work with .instead our aim is to perform importance sampling in path space to sample the paths that contribute significantly to the switching .finally , if there exists a mep that connects two metastable sets without going through a third one , then we connect these two metastable sets by a link . in this way, we form a graph .the original dynamics is then reduced to a markov chain on this graph .-5 mm  a variety of numerical techniques have been developed , most prominently in chemistry , but also in biology and material science , for computing meps and sometimes transition rates . among the most well - known techniques in the chemistry literatureare the nudged elastic band method and the transition path sampling technique .the nudged elastic band method ( neb ) aims at computing the mep defined by ( 3.5 ) .it represents the mep by a discrete chain of states .these states evolve according to the potential forces of the system .to prevent the states from all falling to the two local equilibrium states , a spring force is applied to neighboring states to penalize the non - uniformity in the distribution of the states along the chain .this by itself may cause convergence to a path which is not a mep .hence a nudging technique is used , namely only the normal component of the potential force and the tangential component of the spring force is applied .neb is a very effective method for small systems with relatively smooth energy landscapes .it has two main drawbacks .one is that it is highly inefficient and may not even be applicable to systems with rough energy landscapes .the other is the choice of the elastic constant .a large elastic constant requires a small time step in the evolution of the states .a small elastic constant will not achieve the desired uniformity of the states and hence will not give the required accuracy for the energy barrier .a second important technique is transition path sampling ( tps ) .this method aims at complex systems with rough energy landscapes by developing a monte carlo technique that samples the path space .its efficiency hinges on the ability to produce new accepted paths from old ones .other techniques include the ridge method , blue mooth sampling , etc . .often these methods require knowing beforehand the reaction coordinate .al propose to minimize the onsager - machlup action as a way of finding the most probable path for macromolecular systems .the onsager - machlup action is the same as the wentzell - freidlin action . from a numerical point of view, there are certain difficulties associated with minimizing this action functional .these issues are discussed in .-5 mm  the basic idea in the string method , developed in , is to represent transition paths by their intrinsic parameterization in order to efficiently evolve and sample paths in path space .it has two versions .the zero temperature version is designed for smooth energy landscapes .the finite temperature version is designed for rough energy landscapes in which case thermal noise acts to smooth out the small scale features .the simplest example of a zero - temperature string method is to evolve curves in path space by the gradient flow here is the tangent vector of the curve , denotes the component of normal to , is the lagrange multiplier that enforces certain specific parameterization of the curves .for example if we require equal arclength parameterization , then we need , i.e. we call such curves with intrinsic parameterization strings .  in practicethe strings are discretized into a collection of points .these points move according to the normal component of the potential force .a reparameterization step is applied once in a while to enforce the proper parameterization of the strings .the finite temperature string method is designed for systems with rough energy landscapes , particularly the case when the potential can be expressed in the form where is smooth and . in this casewe would like to compute the mep of without first computing explicitly .this is achieved by creating an ensemble of a special type .our computational object will be a string connecting the two metastable sets , together with a family of probability measures on the hyperplanes normal to the string .consider the stochastic equation where is gaussian noise with mean 0 and correlation the projection operator is defined by projecting to the hyperplane normal to the string , where  is the tangent vector of at , is the lagrange multiplier that enforces proper parameterization of .  *theorem 1.1 . *_ _  1 .  the statistical steady state of satisfies : where is the hyperplane normal to at , .the stationary distribution of is given by the family of distributions   knowing and , the transition rates and free energy landscapes can be computed , see .the finite temperature string method is applied to the perturbed mueller potential where is the so - called mueller potential ( see ) , is a random perturbation .the results of the ( finite temperature ) string method is shown in figure 4 .also plotted is the mep of ( since is explicitly known for this particular example ) as well as the fluctuations around .effective mep and local fluctuations for the perturbed mueller potential .the red curve is the mep for .the black curve is the mep computed from the finite temperature string method .the green curves show the size of the fluctuations .-5 mm  there are several important topics that we did not cover in this brief report .these include the effect of dynamics , non - gradient systems , and acceleration techniques .these are discussed in . also found in these referencesare applications of the ideas discussed here to thermal activated reversal of magnetic thin films , models of martensitic transformations , and the formation of from 60 carbon atoms .the last example is a case when the barrier is entropic . even though the potential energy is mainly going downhill , the free energy has barriers because of entropic effects .such examples are found frequently in biopolymers .  from a numerical point of view , our main idea for overcoming the difficulty caused by the disparity of the times scales is to reformulate the problem as a boundary value problem instead of initial value problem , since we have some knowledge of the initial and final state of the system .compared with other existing methods that assume explicit knowledge of a reaction coordinate , our method finds the reaction coordinate self - consistently during the computation .the topic discussed here is relatively new in applied mathematics , but it is of paramount importance in science and particularly computational science .progress in this area will likely have a fundamental impact in many areas of applications .jnsson , h. ; mills , g. ; jacobsen , k. w. nudged elastic band method for finding minimum energy paths of transitions . in : classical and quantum dynamics in condensed phase simulations .edited by : berne , b. j. ; cicotti , g. ; coker , d. f. world scientific , 1998 .

Abstract: many problems in physics , material sciences , chemistry and biology can be abstractly formulated as a system that navigates over a complex energy landscape of high or infinite dimensions . well - known examples include phase transitions of condensed matter , conformational changes of biopolymers , and chemical reactions . the energy landscape typically exhibits multiscale features , giving rise to the multiscale nature of the dynamics . this is one of the main challenges that we face in computational science . in this report , we will review the recent work done by scientists from several disciplines on probing such energy landscapes . of particular interest is the analysis and computation of transition pathways and transition rates between metastable states . we will then present the string method that has proven to be very effective for some truly complex systems in material science and chemistry .   4.5 mm  * 2000 mathematics subject classification : * 60 - 08 , 60f10 , 65c .  * keywords and phrases : * energy landscapes , stochastic effects , rare events , transition pathways , transition rates , string method .


today , the internet connects millions of users , networks , and network collections worldwide , where the internet s security and stability are quite important to the global economy and well - being of the human race .however , challenged by various forms of cyber - attacks , ensuring the security of the internet and combatting the various attacks requires proper reconnaissance that prelude countermeasure actions .the information security and threat landscape has grown significantly , making it difficult for a single defender to defend against all of these attacks alone . as such information sharing for threat intelligence ,a paradigm in which threat indicators are shared in a community of trust to facilitate defenses , has been on the rise  .  in threat intelligence sharing ,participants exchange patterns of threats with each others , in a form of threat indicators or signals .participants are defined over a community of trust , and collaborate towards a common goal or mission ; to understand and respond to emerging threats  . for such intelligence sharing to happen ,standards for representation , exchange , and consumption of indicators are proposed in the literature  .communities of trust are established , and systems and initiatives for sharing are built . for such initiatives to work ,participants need to contribute information in those systems to be consumed by other community members .however , sharing threat - related information have posed various risks to organization , which pertain to security , privacy , and competitiveness . given the coinciding benefits and risks of threat information sharing ,some community members have adopted an elusive behavior of `` free - riding ''  so that they can achieve utility of the sharing paradigms without contributing much to the community .so far , understanding the effectiveness of sharing has been viewed from the point of view of whether participants contribute or not using volume - based notions of contributions .thus , a community member who does not contribute a volume of data ( indicators ) is a free - riding community member  .the state - of - the art on the problem did not include other metrics beyond simple measures of volume - based contribution , particularly metrics that capture and assess the quality of indicators ( qoi ) as a mean of understanding contribution in the information sharing paradigms .we believe that the nature of the information sharing as a concept and its application to threat intelligence both make quality a very relevant notion , and call for further investigation into the notion s definition and quantification in various settings .a well - defined measure of qoicould provide a better way of capturing contribution in general , and distinguishing community members who contribute useful data .furthermore , threat intelligence systems present distinct challenges and opportunities to counter the problem of free - riding and other abusive behaviors once quality is defined . on the one hand, these systems generally lack the enforcement of a central authority , and therefore entities in these systems share information at their own will in a ( somewhat ) peer - to - peer fashion . as a result , it is necessary to envision a set of notions and mechanisms that characterize contribution in general , and are capable of capturing free - riding , while being implemented in a distributed manner and used by each community member .an ideal measure of qoishould be robust to distinguish between the various members based on their contribution , rather than a predefined notion of trust . with the possible speciality of community members , and the varying usefulness of indicators shared based on the context in which they are used , a major challenge is to assign context - dependent quality markers for indicators .   to the best of our knowledge, the problem of free - riding in information sharing for threat intelligence , while sparsely mentioned in other work  , is not treated properly in the literature .thus , this work is the first of its type to be dedicated to the problem by identifying qoias a new metric of contribution to capture free - riding in information sharing for threat intelligence .  given the aforementioned challenges , one simple measure of contribution in information sharing systems weigh the volume of indicators contributed by various community members .however , for many reasons , such measure is insufficient , as described earlier .therefore , it is important to understand the quality of shared information as a form of participation . without a high quality of shared information , we can not achieve actionable intelligence that is effective in combating cyber threats .unfortunately , this issue is not well understood in the literature , and requires further exploration by identifying the meaning of quality , and basic methods and tools for assessing them are lacking .   in  ,et al._explored the potential correlation between qoiand privacy . however , privacy is not the only factor that affects qoi .one feature of quality is the _ correctness _ of an indicator ; a meaningful annotation and label of the indicator that is true and accurate .a second possible feature of quality is the _ relevance _ of the indicator to the community members ; because of the targeted nature of modern cyberthreats , information that is shared has to be contextual to the domain .a third plausible feature of quality of an indicator is its _ utility _ ; informally , some indicators are more indicative than others about cyber - attacks , and therefore it is critical that participants in the threat intelligence community share information that capture prominent features of cyber - threats . finally , the _ uniqueness _ of an indicator is another assessor of quality , which is defined as a measure of ( dis)similarity with previously seen indicators .this property ensures that participants deliver indicators that are not duplicates or redundant , and provide additional threat information to other community members .besides these features , indicators are often time - sensitive , making temporal features very important when evaluating qoi . a timely indicator such as a source of an attackcould be used to defend against an emerging attack , unlike a stale indicator that could be ( potentially ) used for postmortem analysis . as mentioned above, there is also potential correlation between qoiand privacy .privacy can affect qoi(although perhaps negatively when privacy of indicator is ensured ) .we elaborate on this quality metric in this work , and show its quantification through data - driven analysis .while each of the aforementioned measures can be used as a separate feature of quality , we envision that a single indicator could have multiple of those features . as such, we also assess qoiwith respect to these metrics in the form of a weighted ( continuous ) score .our method for evaluating qoiis based on exploiting a fine - grained historical records as benchmark for assessing the contributions of community members .we illustrate the concept through a concrete evaluation of a real dataset from various security vendors of antivirus scans and their results of labeling malware samples as seen in the virustotal service ( https://virustotal.com/ ) .the main contributions of this paper are multifold .first , we identify the need for qoito capture contributions by community members in information sharing paradigm .qoicaptures a wide spectrum of behaviors , from altruistic behavior , where a community member contributes a lot of ( high quality ) indicators to free - riding , where a community member contributes less , or contribute a lot of low quality indicators .second , we develop and formulate various metrics that are robust to capture the notion of quality .third , we experimentally demonstrate those measures and metrics , and show their robustness , and how they differ in identifying contributor s behavior ( particularly free - riding ) from the simple volume - based measure of contribution .the organization of this paper is as follows . in section [ sec :overview ] , we provide an overview on cyber - threat intelligence and the risks of information sharing . in section [ sec : methodology ] , we provide an overview of our quality of indicator(qoi ) assessment methodology . in section [ sec : qoi ] , we describe the processes involved in our qoi - based assessment . in section [ sec : results ] , we present the results of our benchmark experiment , afterwards we discuss related work in section [ sec : related ] and finally we conclude in section [ sec : conclusion ] .we first provide an overview of cyber - threat intelligence systems , then introduce to unique problem with information sharing in these systems which demand quality measures .the internet today connects hundreds of millions of users worldwide , and is operated by service providers who connect businesses , education institutes , and government agencies , collectively forming a global village . in the recent years , the internet has been challenged by various forms of cyber attacks , ranging from endpoint malware attacks  to massive network disruptions and instabilities  .  at the endpoint side , malware is capable of penetrating a perimeter s security in many enterprise systems , exfiltrating sensitive data from such systems , and causing great damage to both private and public sector networks  . at the larger scale , multiple endpoint infections by malwareare more powerful , and pose a greater risk , seen often in systemized large - scale botnets  .botnets , defined as collections of networks of infected hosts are the basic fabric for the operation of many cybercriminal activities .botnets rely on principled designs , where bots ( infected hosts in a botnet ) execute commands on behalf of their herder ( botmaster ) , utilizing command and control ( c&c ) infrastructure  .botnets today are used for a variety of cybercriminal activities , including spam , massive denial - of - service ( ddos ) attacks , and data exfiltration , among many others .botnets represent a major component of the cybercrime ecosystem , with the rise of botnet - as - a - service . today, hackers utilize network reconnaissance to probe targets for vulnerabilities and craft custom payloads to gain control over their infrastructure by spreading malware in propagation efforts .defending against the threat vectors of malware and botnets is a challenging task , which resulted in a rich body of literature .the literature on defending against malware and botnets looks into identifying `` signals '' , `` indicators '' , or simply `` features '' that could be useful in identifying endpoint systems , malware , and botnets . for malware , for example , such features could include static strings in that piece of malware , dynamic artifacts that the malware generates when executed in the wild , or external context information associated with the binary binary of the malware ( such as the author s information , operating system , etc . ) . for botnets , the c&c infrastructure may include domain names and internet protocol ( ip ) addresses , and knowing such information can be very helpful in identifying a botnet .for example , botnets tend to use domain generation algorithms ( dgas )  , which result in random domain strings with high entropy , and being able to identify those domain names is key to detection of botnets .furthermore , being able to distinguish between various dgas is key to attribution of threat to a certain botnet family .an effective cyber defense would rely on a good visibility into many of those features .combatting cyber threats and attacks requires intelligence gathering that prelude countermeasure actions , as seen in the above examples . to this end, cyberthreat intelligence has become a growing concept . today , organizations in the public and private sector , government and industry , have established tools seeking first - hand knowledge about new cyber - attacks and malware threats .this includes the ability to recognize and act upon indicators of attack and compromise scenarios , essentially putting the pieces together for analysis about attack methods and processes using static and dynamic analysis and profiling techniques , open source , social media , and dark web intelligence .the need for information sharing for threat intelligence is necessitated by both economical and technical realities .being able to identify all the types of indicators and features useful for characterizing , identifying , and defending against all types of threats , while desirable , is infeasible from both technical and economical standpoints . with new technologies such as cloud , mobile computing , social networks , and the internet of things ( iots ) , and the the persistence of adversaries through cybercrime and advanced persistent threats ( apts )have also brought several challenges .therefore , it is reasonable to say that no single player in this ecosystem is capable of addressing all security issues alone .for this reason , sharing information of threat intelligence among vendors and government entities has emerged as a plausible technique for efficiently and effectively defending against new and emerging threats . with threat intelligence sharing ,operational experience is communicated to other parties in an ecosystem to enable them to effectively defend against current attacks , and to improve their defense posture by preventing such attacks from happening utilizing such actionable intelligence .  to enable information sharing , organizations need to agree on standardizing threat information .this requires defining the content fields , encoding , and exchange format of the information relevant to a particular threat or incident , along with a pre - defined protocol to communicate the criticality of such information .various standards for information sharing have been proposed  to automate and structure the exchange of threat information with a community of trust .today , standards are used in the exchange of indicators of software , hardware , and network artifacts , and are intended for operationalizing those indicators in many applications , including security operations related to malware characterization , vulnerability analysis , remediation , platform hardening , and incident - response  .while threat information sharing brings many benefits to the sharing community members , it may incur security risks about participants , their operational contexts , and security posture .not only that , the same information , once exposed to an adversary , may be used to test their applicability on other target systems , who may lag behind in security updates or miss out on patching vulnerabilities . therefore , the adversary will be able to utilize such information for attacking other unpatched systems .the risk of sharing may go beyond fingerprinting systems to leaking personal identifiable information about individuals .various types of sharing standards are proven to leak personal identifiable information ( pii ) that may contain names , email addresses , and other types of sensitive data  .for example , privacy violations in sharing standards may occur in the form of a document which contains contact information for the constituent responsible for an incident report .this type of information may become personally identifiable in the case when the contact information of a particular individual are used .participants in a threat intelligence sharing community may interact with one another with various degrees of collaboration and competition , which may affect the way they share  .because of that , many companies and organizations today are reluctant in sharing firsthand intelligence , and mostly gather and ingest information from neighboring sources that are less significant .   given the triad of security , privacy , and risks associated with threat intelligence sharing , some members might be joining communities of sharing for the purpose of benefiting from the platform without offering valuable information themselves , hence , the term , `` free - riding '' is coined to refer to the behavior of such users who act to maximize their own utility at the expense of the welfare of the community .this problem is not new , and is manifested in other distributed settings , most notably peer - to - peer ( p2p ) systems . in p2p systems , cooperation is required for the operation the system .however , cooperation may incur significant communication and computational overhead , thus users may refuse to contribute their fair share of resources . at the same time they may utilize the system by consuming the resources of other peers .researchers demonstrated the impact of free - riders in p2p systems , such as bittorent  , and observed a significant increase in download times for high contributing nodes in the presence of few low contributing ones .so far , and to the best of our knowledge , understanding the effectiveness of sharing has been viewed from the point of view of whether participants contribute or not ( thus the literal meaning of free - riding ) .this form of contribution is perceived as a volume - based contribution , since the level of contribution by any participant is evaluated directly by the amount of information communicated to the community regardless of its nature , whether it is used by community members or not .given the large amount of unprocessed threat - related events , which are generated by automation tools , such as security information and event management ( siem ) technologies , and the fact that in many sharing systems today , the contribution level amounts to the volume of data , actors may find it more convenient to submit raw , unprocessed , or unused events as indicators of threat to avoid the investment on resources for cleaning , contextualizing , operationalizing , and filtering such information .for this reason , it is important to consider the quality of shared information as the basis for evaluating the level of participation , because a simple and coarse measure of participation is insufficient . in order to overcome these obstacles , in this paperwe propose qoi as a quantifiable and measurable metrics and provide a methods for quality assessment .assessing qoiis a nontrivial task . however , as other quality notions ( e.g. , quality of services , quality of experience , etc . ) , qoirequires defining metrics , methodology for assessing such metrics , and methods for validation of the proposed metrics based on sound assumptions , to find out what capabilities they provide , including addressing free - riding .our approach in developing qoimetrics is intuitive , and uses several sounds assumptions driven from the context in which indicators are used . in particular , qoimetrics include correctness of an indicator with respect to a label feature , the relevance of the indicator to a consuming community member , the utility of the indicator , and its uniqueness .more ( informal ) details are provided in section [ sec : qoimetrics ] .our approach for assessment of qoiuses a reference `` golden '' dataset as ground truth .based on whether the indicator provided by a community member is in the reference dataset or not , the assessor proceeds by either matching the metric attribute of the indicator to that of the golden dataset . if the indicator is not in the golden set , and assuming that the golden dataset has `` similar indicators '' , a machine learning algorithm is used for predicting the attribute of the qoimetric , and compare it to the one provided by community member . for an arbitrary number of indicators provided by the community member , a scoreis then established for that community member based on the normalized weighted sum of qoivalues across all indicators .  in this section ,we describe the quality metrics and the details of our methodology including a system architecture for assessment , processes followed and the data - flow .   in the following , we identified four metrics to be used for assessment of quality : correctness , relevance , utility , and uniqueness , as described below .   for a given reference dataset , the correctness metric of qoicaptures whether attributes of an indicator ( e.g. , label used for attribution , severity score used for risk assessment , etc . ) are consistent with the assessor s reference . for that , and using the labeling of a malware sample as an example of an attribute for an indicator , we compute the correctness score as the fraction of samples that match the anticipated labels . specifically , this is computed as the aggregate binary score of the correct samples normalized by the size of the sample set of indicators .informally , the relevance metric of a qoimeasures the extent to which an indicator submitted by a community member to the community is contextual and of interest to the rest of the community . as such , in defining and assessing the relevance of an indicator , we use a reference weight assignment to the class labels giving higher weight to labels of greater interest to a particularly community member ( assessor ) and lower weights to less desirable labels .the relevance score is then computed as the average weighted sum for all sample indicators in the set .the utility is similar to the relevance of an indicator , although at a finer - granularity than an indicator . as such, we view the utility of an indicator as the average weighted sum of all of its feature components .this is , we assign a different weight to each feature of the indicator to leverage features that are a better candidate input to prediction of threats . while the weighting of the features of an indicator could be realized using one of many ways , we suggest the information gain as a measure of weighting features .for example , using a similar notion , the weights of the feature components can be computed using various statistical models for sensitivity analysis , including the principal component analysis ( pca ) technique  ) .the uniqueness of an indicator is a measure of the ( dis)similarity of the indicator in comparison with other submitted indicators by contributors in the community .a vector distance ( e.g. , using the mahalanobis distance  , which captures the difference between a point and a distribution of points with a mean ) is computed to determine the degree of uniqueness .we also define a threshold on the minimum distance between the feature vector of an indicator to other indicators , and use that threshold to tell whether an indicator is unique or not .having elaborated on the informal definition of qoimetrics , we now move to discuss the qoisystem architecture , first as a strawman highlighting the main concept of assessing qoi , and then as a fully functioning system that addresses various issues in the strawman design .  our system for assessing qoioperates for a set of distributed nodes in a community of trust .those nodes are are logically connected with each other in a p2p fashion , as shown in figure [ fig : arch ] . as such , each of these nodes would participate in the sharing and consumption of threat indicators provided by other peers , which is achieved within a community of trust that is separated from other communities in the sharing ecosystem . before nodes can accept and operationalize ( process ) these threat indicators , they need to evaluate their quality by asking a special node , an assessor , which has sufficient information to perform such function , for a rating ( scoring ) of the indicator . in response , the assessor assigns a quality score for the indicator based on a ground - truth available to the assessor , and using a reference dataset the assessor has access to .    in this architecture, we assume that the messages sent between the peers in the system are authentic and tamper - evident , using existing threat exchange protocols that provide end - to - end security guarantees .one issue that the above architecture may suffer from is the amount of trust each community member has to put in the assessor , and the validity of his scoring of indicators .we address this issue by assuming that the architecture can support more than one assessor node , and these nodes may coordinate among each each other .we leave exploring the spectrum of the number of assessors and the rationale for various numbers as a feature work .however , as a feature , a consumer of a score by the assessors ( where the assessors number is greater than 2 ) could perform a majority voting to improve the robustness of the scoring to address intentional bias ( dishonesty of assessors , if any ) , and unintentional bias ( due to issues with the underlying ground truth dataset ) . on the other hand , any and every node in the system could act as an assessor , if the golden reference dataset is available to them .  while the latter assumption of the availability of the data to each node in the system is very implausible, a milder assumption for the operation of the strawman design above is the coverage of data : the system assumes the reference dataset has sufficient information about every possible indicators presented by the various community members .however , based on a prior work assessing coverage of indicators  , no single community member ( antivirus scanner ) in the case of malware detection and labeling has a 100% coverage or accuracy .based on the same study , and for a malware family such as zeus  , it takes 6 and 18 community members to provide close to perfect coverage of detection and correctness of labeling , respectively .such numbers are close to 10% and 30% , respectively , of the entire set of community members with antivirus scans in the virustotal dataset .this in particular calls for a more `` intelligent '' process for the assessment of qoi , using not only explicitly provided labels , but also using learned labels from features of indicators utilizing advanced machine learning techniques .   at a high - level , our ideal system for assessment has the following specific procedures for system setup :  1 .  _ defining quality metrics and scoring procedures ._ quality metrics are used as a measurement criteria to ensure that community members who participate in information sharing provide threat indicators that are valuable to other members , while scoring procedures are methods that specify how these metrics are used to generate a quality score . 2 .  _ defining annotations for threat and quality labeling ._ annotations can either be labels that indicate the type of threat or they can be labels for identifying the quality ( severity , timeliness , etc . ) level or quality type of an indicator . utilizing these annotations ,a weight value is assigned to each quality label , and a scoring method is utilized to convert the quality labels to a numeric aggregate score for the indicator .3 .  _ building the reference dataset . _the reference dataset will be used to evaluate qoifor a sample of indicators submitted by a sample provider . to build the initial reference dataset , data that is collected through security operations ( e.g. , monitoring , profiling , analyses , etc . )is vetted for their validity and applicability to the domain , perhaps using often expensive by necessary manual vetting  .4 .  _ defining extrapolation procedures and training the classifier . _extrapolation procedures enable a quality assessor to predict the label of an indicator using its feature set and classifier model .the classifier is trained using a supervised learning process extracted from the reference dataset .this reference dataset is collected for the purpose of initializing the system .after the initial setup of the system , the sample indicator is assessed for its quality and a quality score is computed .the following describes the steps of the assessment .1 ) obtain a set of sample indicators where each sample is composed of a tuple ( label , vector ) that consists of a label and a vector of features .2 ) for each sample , extract the feature vector and feed the data as test input to the trained classifier which predicts its label .3 ) compare between the predicted label and the label provided by the sample .indicate whether the two labels match , and record the comparison result as a quality annotation .4 ) compute the confidence level and include other quality annotations for the indicator using the labeling rubric .5 ) use scoring procedures and quality labels to compute a quality score for the indicator .the illustration for the complete process and dataflow of the qoiassessment method embraced in our design is depicted in figure [ fig : method ] .as can be seen in the figure , the setup of the system is achieved through the use of a supervised learning process over the reference dataset , rather than the direct matching of explicit labels of indicators in the dataset .    in this assessment system , we assume a reference labeled ( training ) dataset that contains a comprehensive library of artifacts , such as malware samples , incident reports , and logs , and that has been collected through typical operational intelligence gathering procedures .ways for obtaining such labels falls out of the scope of this work , and industry s best practices , as described in  could be used . upon ingesting those artifacts in our system ,they are converted into a set of training samples with their corresponding features .each sample is a pair that consists of a feature vector as an input object for the machine learning algorithm and the corresponding threat label as the desired output value for the object class . to reach such end goal of predicting a label correctly ,the build of our trained model encompass multiple components , namely a feature selection procedure , a machine learning algorithm selection procedure ( e.g. , svm , logistic regression , random forest , etc . ) and the corresponding parameters ( e.g. , procedure for regularization and linearization in case of svm and lr , respectively ) , and cross validation procedures ( e.g. , fold size , validation strategy , etc . ) .  upon building the machine learning model , establishing a confidence in its performance through typical evaluation metrics ( e.g. , low false positive and false negative , and high true positive and true negative ) , we then use the built model as a predictor for previously unseen threat indicators . in such operational setting , given a sample indicator provided by a community member , and before ingesting it the community member would pass it to the assessor for further evaluation and scoring .the assessor then extracts a feature set corresponding to the indicator using a standard form , and converts it into a feature vector .the assessor then uses the previously built model as a predictor , and assigns a label ( e.g. , using a multiclass svm , the assessor can indicate the label closest in the training set to that of the newly observed indicator ) .the assessor then decides the quality of the indicator by taking both the predicted and the self - provided label by the community member into account .the quality scoring engine then uses the individual scores of the various indicators provided by each community member to assess their actual contribution , and detect free - riders .we note that the `` intelligent '' system above addresses various issues in the strawman system .first , rather than requiring the actual indicators to be present in the ground - truth dataset , this technique requires only the availability of a sufficient number of indicators of the same label .second , with such flexibility in defining the ground - truth through a learning and model - building process , the number of community members that can act as assessors greatly increase .finally , the even when the labels of indicators are not provided by the community member , e.g. , artifacts provided to the community are not operationalized , this technique makes use of those indicators through other measures of qoi , such as relevance , or utility , or uniqueness , which do not require a label to be provided by the contributing community member .as discussed in section [ sec : procedure ] , the qoiassessment process is composed of a series of steps in order to initialize and operate our system for both assessment of individual indicators and scoring of community members as a whole . specifically , these steps begin with collecting the reference dataset and building the prediction model , then extrapolating , benchmarking , and computing a quality score for a given indicator . in the following ,we outline in more details each of those processes and procedures .   after identifying metrics for defining quality , as exemplified in section [ sec : qoimetrics ] , we demonstrated the use of qoifor the assessment of the contribution level of participants . as mentioned before , our methods for computing the qoiinvolve multiple processes . in order to initialize the system the reference datasetis used to build prediction model through supervised techniques of machine learning .specifically , this involves submission of sample artifacts from multiple sources , and the premise is to utilize the notion of quality as opposed to a simple view of contribution based on the volume of data ( i.e. the number of samples ) . in order to evaluate the qoiprovided by community members, a reference dataset is used as a resource of ground truth .while proposing methods for obtaining ground truth falls out of the scope of this work , we only use an example in this work to bootstrap the evaluation .in particular , in this work we demonstrate evaluating the quality of malware labels by av vendor using virustotal as a reference dataset with samples that are manually vetted by one community member  ( in such setting virustotal could be loosely defined as the community of trust ) . in short , virustotal is a multi - engine av scanner that accepts submissions by users and scans the samples with those engines .the results from virustotal provide many useful artifacts and annotations , including the labeling of a sample by the various engines of av scanners , as well as other behavioral and static features of malware samples .though there might be some inconsistencies in the final labeling between the results of virustotal and across vendors , the premise is that the tool can be trusted for samples that have been submitted multiple times over sufficient periods of time , particularly since av vendors update their results with virustotal whenever acquiring a new signature for a previously unknown sample .features provided in the ground truth are particularly useful in a learning a model for their label prediction . to identify the family to which a malware sample belongs , security vendors ( av scanners , community members ) usually gather various characteristics and features of the sample using static analysis , dynamic analysis , and memory forensics . for static analysis , artifacts like file name , size , hashes , magic literals , compression artifacts , date , source , author , file type and portal executable ( pe ) header ,sections , imports , import hash , among others , are used . for dynamic analysis , file system , user memory , registry , and network artifacts and featuresare collected . for memory forensics ,memory byte patterns are captured to create a signature .after building the prediction function by training classifier with the reference dataset , the next step is to assess qoithrough extrapolation from the prediction function results .the remaining question becomes how the reference set are used to assess and extrapolate the values and quality of indicators . in order to answer this question , we elaborate on a particular machine learning techniques , the semi - supervised learning and its application to the problem at hand .  model .while our system described in the previous section uses multiple off - the - shelf algorithms , we highlight the operation of qoiusing a classifier model based on the nearest centroid classifier ( ncc ) , specifically we adopt a variant called the linear discriminant analysis  to map threat indicators to their respective labels . in this model ,each label is characterized by its vector of average feature values ( i.e. class centroid ) .a new sample indicator is evaluated by computing the scaled distance between the features of the sample and each class centroid , and then the sample is assigned to the class to which it is nearest .  to build the classifier, we obtain samples for training from the reference dataset .this dataset is built such that there are training samples per class , with features per sample .for each training sample , we observe a label and a sample vector .for simplicity we refer to the classes labels by their indices .note that each is a vector of length .we assume that samples labeled by are distributed as , the multivariate normal distribution with mean vector and standard deviation matrix .we denote by , the corresponding probability density function .finally , let be the prior probability that an unknown sample comes from class labeled by .bayes theorem states that the probability that an observed sample comes from class is proportional to the product of the class density and prior probability : where is the posterior probability that sample comes from class .the classifier assigns the sample to the class with the largest posterior probability to minimize the misclassification error .this can be written as a rule : therefore , a sample is assigned to the nearest class and the distance is computed using the lda metric : , where is the square of the mahalanobis distance between and .  rate .a misclassfication occurs when an indicator is assigned to an incorrect label .the probability of making a classification error is : .\ ] ] the misclassification rate using the lda rule can be derived from ( [ eq:2 ] ) . in particular , we can calculate the misclassification rate of the nearest - centroid using : \times \pi_{j},\ ] ] where is the cumulative distribution function ( cdf ) of the standard normal distribution .note that this assumes that the sample data are normally distributed as stated by the model .the equation above can be interpreted as a measure of the collective distance between all of the class centroids taking into account class prior probabilities . in general , the misclassification rate is small when the centroids are far apart and increases otherwise .we used the ncc to predict labels for observed indicators and compared the results with the sample labels .this enables us to compute a score on the correctness and quality of the feature set for the indicators . in the followingwe formalize the steps to compute a score for the samples based on the quality metrics described earlier .denote by the number of users in the system .each user provides a set of samples  with feature vector and sample label for and .as described earlier , the reference dataset is used as the benchmark for determining the correct label for an arbitrary sample .each sample consists of a feature vector and an associated label . in procedure [ alg : correct ] we outline the algorithm used for computing the correctness as a qoimetric .obtain reference dataset  where for .evaluate by applying the ncc method as follows :  1 ._ training _ : compute reference indicators for class labels in , as per - class centroids , where is a subset of belonging to the class label .prediction _ : for every sample , apply the classifier function to compute the label ,   for every sample , compute the sample score as :   compute the correctness score ( c ) of by taking the average sum :   as shown above , we first build a classifier by utilizing the reference dataset as the training set and forming a prediction on the label of to obtain .then , the assigned label of is compared against the predicted label and a positive score is given if labels match .the correctness is computed as the average sum of scores for all samples in .the steps for computing the relevance variable of a set of indicators are shown in scoring procedure [ alg : relevance ] .as can be seen , the weight values are arbitrarily chosen , and a mapping function is defined to assign weights labels such that higher weight values are assigned to labels of greater interest to the community members .define weight values : define a weight function to assign elements in the label set as compute the relevance of as average weighted sum :   for each sample , the corresponding label is evaluated using the mapping function to obtain the weight value as the sample score .the relevance score of , denoted by , is calculated as the average weight sum of the scores .next , we provide the sequence of steps required for calculating the utility variable of a set of indicators , in procedure [ alg : utility ] .in this procedure we note that the utility of an indicator is determined by the sum of the utility weights of the samples .the weights and weight function are defined by the application .define utility types define weight values , where each weight value corresponds to a utility type . define a weight function s.t . maps to a utility weight , i.e. where .compute a weight of , using compute the utility score of as the average sum of the sample weights : where is the corresponding label type of sample    another metric of qoiis their uniqueness , where highly unique indicators are considered more valuable than common indicators . in procedure [ alg : uniqueness ]we outline the steps used for calculating the uniqueness of a set of indicators .consider the set which is initially empty , i.e. build the set by considering unique samples from the sets  add to . compute the uniqueness score for as follows compute the uniqueness score ( n ) of by accumulating the scores of the elements :   in this procedure , we assume that samples can be uniquely identified ( e.g. using hashes ) . in set notation, we can say that an element is unique if it is not an element of other sample sets , i.e. .   qoiis a comprehensive measure of the various notions of quality defined earlier .in particular , qoifor is the average weighted sum of the four components : correctness ( c ) , relevance ( r ) , utility ( u ) and uniqueness ( n ) , as shown in procedure [ alg : qoi ] .the weights assigned for individual metrics are application- and community member - specific .define normalized weights for the components : , and .calculate the quality of indicator ( qoi ) as the weighted sum of the components : this section , we evaluate the scoring method for contribution based on quality of indicators , and highlight how it addresses the free - riding problem in information sharing in a unique way .we start by analyzing the dataset that we obtained from av vendors about their sample labeling , then we utilize this dataset and apply qoi - based and volume - based scoring methods to compare between the vendors .to highlight qoias a new notion of evaluating contribution in information sharing for threat intelligence , we compare the difference between quality - based and volume - based scoring methods for the contribution of av vendors . to this end , our dataset enumerates av vendors who submitted their artifacts of malware samples , including labels , to virustotal during the period of our data collection from mid 2011 to mid 2013  .a key goal of the evaluation is to demonstrate the deficiency in the use of volume - based scores , since one vendor can achieve a high rating by submitting a large number of artifacts about malware samples of low quality . as discussed previously, this could happen because of several reasons : the submitted artifacts about some malware sample are incorrect , the sample family is uninteresting , or that the kind of information submitted about the samples are not helpful in identifying or detecting them .table [ tab : malfam ] depicts the malware families used in this study , their sample size , and the corresponding brief description of each family type .all scans are carried out on those malware samples around may 2013 timeframe .the dataset provides a diverse representation of families , which nicely facilitate our study .as can be seen , avzhan and darkness are the most popular ddos malware being submitted . on the other hand , shadyrat is the most popular targeted malware with the largest sample size in its category ( represents about 43% of the targeted samples and 24% of the total samples ) , while zeus has the largest sample size for trojan malware , roughly about 77% of the trojan samples and 16% of the total samples .furthermore , we observe that more than half of the samples are ddos ( 54% ) , 21% are trojans , and 25% are targeted malware.this breakdown provides an insight about the threat landscape and the frequency in which these types of malware appear in the wild .oftentimes av vendors harvest malware samples by utilizing deployment of internet sensors for packet capturing , or using isolated environments such as honeypots and virtualization tools for behavioral analysis .the increased number of samples for ddos is justified by the need for vast deployment for scaling up the number of infected hosts to launching attacks . on the other hand , targeted malware are less common in the wild because they are deployed in limited number of hosts , and are typically designed with covertness in mind . in the rest of this analysis ,the identity of the vendors is anonymized ..malware families used in the study .ddos stands for distributed denial of service .also , ddoser is known as blackenergy while darkness is known as optima .dataset and description are from .[ cols="<,<,<",options="header " , ]   in this section , we introduce the results and finding by performing an evaluation of the various qoimetrics over our evaluation dataset ( we evaluate all but uniqueness , since it is trivial to assess ) .first , we note that while there are more samples gathered for ddos - type malware in comparison with others , the threat - intelligence community often gives more weight to identify malware or incidents that are less observable , which present a level of sophistication .thus , for our evaluation , we consider trojan and targeted malware more relevant than ddos , from the point of view of community members consuming the shared information .  .figure [ fig : relevance ] presents the normalized scores for the relevance of indicators for each vendor . in assessing relevance , we give more weight to targeted malware and trojans over ddos samples of each vendor ( community member ) .specifically , the weights are , , and m and `` 0 '' otherwise .as shown , in the relevance - based assessment a higher score is given to vendors who contribute more targeted and trojan samples , and vendors who only contribute ddos samples are greatly de - emphasized .we observe two distinct behaviors .first certain contributors who have high volume - based score tend to have a very low ( close to `` 0 '' ) score when using the qoimetric of relevance for their evaluation .in particular , with the two relevant and one less relevant family types of interest identified , such community members tend have more unidentified ( irrelevant ; i.e. , individual score of `` 0 '' ) malware samples and families ( e.g. , vendor 7 , vendor 21 , vendor 59 , etc . ) . on the other hand , certain contributors ( although smaller in numbers ) , and despite having a small volume - based contribution , tend to have higher relevance score , thanks to having the very relevant family identified in their shared indicator labels ( e.g. , vendor 10 , vendor 16 , vendor 27 , etc . ) .  .an assessment of the correctness of the av indicators is depicted in figure [ fig : correct ] . as can be seen , vendor 4 , vendor 27 , and vendor 30 , outperformed other vendors in this metric with a score in the 80s up to top 90s percentile range , highlighting a quality of the labels provided by those vendors corresponding to the actually learned labels of the samples provided by them . on the other hand , the majority of the rest of vendors tend to have a gap in the score between the volume - based and the correctness - based contribution measures , where the correctness - based measure is significantly lower .there are various reasons for why some vendors tend to score low for correctness despite their large ( volume - based ) contribution .this could be caused by them labeling samples under unknown names , mislabeling to other family types due to similarities between families , or assigning generic labels like `` trojan '' , `` virus '' , `` unclassified '' among other misleading labels .examining the correctness of av indicators also leads to a more subtle discussion about their utility .looking closer into the label generated by some vendors , we find out that some labels are too generic in the sense that they only describe the behavior rather than name of a known malware family type , e.g. , trojan.win32.servstart vs. avzhan .  . to evaluate the utility of av indicators , we give weights for three classes of malware labels : complete labels ( ) are based on industrially popular name , generic labels ( ) are based on placeholders commonly used for labeling the family such as `` generic '' , `` worm '' , `` trojan '' , `` start '' and `` run '' , and incomplete labels , ( ) , including `` suspicious '' , `` malware '' , and `` unclassified '' , which do not hold any meaning of a classsimilar to the strategy with relevance , we assign weights of , , and .we plot the results of evaluating the utility of indicators in figure [ fig : utility ] .we notice that vendors such as vendor 51 , vendor 53 , and vendor 59 are rated as high utility indicator providers that surpass their volume - based scores .nevertheless , these vendors high utility indicator is offset in figure [ fig : metrics ] that includes two more metrics : correctness and relevance .these additional metrics show that these vendors achieve insignificant correctness and relevance .figure [ fig : metrics ] captures the importance of displaying the three scores in a single plot to allow direct comparison of the various quality - based metrics .we notice that there is a clear correlation between the correctness and relevance scores .this is because we only compute relevance scores for correctly submitted samples , where incorrect labels are zeroed in the relevance score .as described earlier , we aggregate a single qoiscore for each vendor based on the weighted sum of the various qoimetrics . in figure[ fig : qoi ] , we present a comparative bar - plot between qoi - based and volume - based scoring for assessment of contribution by the av vendors . as can be seen , many vendors , such as vendor 14 , vendor 57 , and vendor 58 which received high qoiscores are rated with lower scores in their volume - based rating . in particular, they received from 15 - 75 percent lower rating in their volume - based scores . on the other hand ,vendors like vendor 11 and vendor 18 , and vendor 20 , which tended to provide a very high volume - based indicators , have very small ( close to zero ) qoiscore , highlighting their potential as free - riding candidates .the free - riding problem in threat intelligence sharing is not addressed before , nor measures of quality broadly defined , or closely identified for assessing contribution by community members . however , the problem of free - riding in general is not new and has been a topic of discussion in the peer - to - peer ( p2p ) systems community .going early back in the literature , adar and huberman  were first to spot the problem in p2p systems as they noticed the existence of a large fraction of users who do not share useful content in the file sharing network gnutella .a few years later , feldman et al . characterized the problem of free - riding in peer - to - peer systems and proposed potential directions for research . in response ,a stream of papers were published on the topic , most notably , the work of locher et al . , who developed a free - riding client as a proof - of - concept and demonstrated how entire files can be downloaded in the bittorrent network without providing any content to the peers . while locher et al . have concentrated on analyzing the feasibility of free - riding attacks , other papers  are more focused on analyzing their root - cause and impact on the overall system utility .  in light of operationalizing the functions of threat intelligence ,various information sharing standards were proposed including those developed by ietf ( https://www.ietf.org/ ) , mitre ( https://www.mitre.org/ ) , and nist ( https://www.nist.gov/ ) .industry leaders have picked up on these standards and developed application program interfaces ( apis ) to facilitate delivery and retrieval of raw , processed , and structure and intelligence data , such as threatexchange  by facebook and intelgraph by verisign .however , sharing standards have shown to exhibit privacy violations including leaking pii fields , as demonstrated by  , potentially encouraging the act of free - riding .of relevance to the notion of quality of indicators in threat intelligent systems is malware attribution .malware attribution have been widely employed in the literature for training algorithms and techniques of malware classification and labeling  , and understanding the utility of attributes as detector patterns of malware samples has been an important subject matter .bailey et al . were one of the early folks to characterize malware in terms of system state changes ( e.g. registry changes , files created ) and investigated the problem of behavior - based clustering as a method for classifying and analyzing internet malware .more focused on the labeling problem , canto et al . analyzed the quality of labeling of malware samples for a couple of vendors and pointed out their labeling inconsistencies .in the same vein , perdisci  analyzed the shortcomings of malware labeling of various av vendors by constructing a graph from the labels and measuring the distance between them .on the other hand , mohaisen and alrawi  quantified the inconsistencies in labeling against a reference dataset collected from thousands of samples of various types which were manually vetted by analysts . in their study , the authors evaluated the detection rate , correctness , and consistency of labeling of av scanners .in this paper , we have the first look at the notion of the quality of indicators ( qoi ) for understanding the contribution of community members in information sharing paradigms . unlike other peer - to - peer systems in which the volume of contribution ( bandwidth , size of files , etc . )is a good indicator of contribution , we argue that the special nature of security applications calls for more elaborate notion of contribution . as such , we define multiple metrics for assessing contribution , including correctness , utility , and relevance of indicators .as compared to volume - based measures for contribution , and thus free - riding , our metrics are more robust , contextual , and reasonably quantify the actual contribution of individuals . by verifying our metrics on a real - world data of antivirus scans we unveil that contribution measured by volume is not always consistent with those quality measures , and that qoias notion is capable of capturing forms of contribution beyond free - riding .d. k. tosh , s. sengupta , c. a. kamhoua , k. a. kwiat , and a. p. martin , `` an evolutionary game - theoretic framework for cyber - threat information sharing , '' in _ 2015 ieee international conference on communications , icc 2015 , london , united kingdom , june 8 - 12 , 2015 _ , pp . 73417346 , 2015 .h. tanaka , k. matsuura , and o. sudoh , `` vulnerability and information security investment : an empirical analysis of e - local government in japan , '' _ journal of accounting and public policy _ , vol .24 , no . 1 ,pp . 3759 , 2005 .d. k. tosh , s. sengupta , s. mukhopadhyay , c. a. kamhoua , and k. a. kwiat , `` game theoretic modeling to enforce security information sharing among firms , '' in _ieee 2nd international conference on cyber security and cloud computing , cscloud 2015 , new york , ny , usa , november 3 - 5 , 2015 _ , pp . 712 , 2015 .s. yadav , a. k. k. reddy , a. reddy , and s. ranjan , `` detecting algorithmically generated malicious domain names , '' in _ proceedings of the 10th acm sigcomm conference on internet measurement _ , pp .4861 , acm , 2010 .m. friedewald , e. vildjiounaite , y. punie , and d. wright , `` privacy , identity and security in ambient intelligence : a scenario analysis , '' _ telematics and informatics _ , vol .24 , no . 1 ,pp . 1529 , 2007 .a. mohaisen and o. alrawi , _ detection of intrusions and malware , and vulnerability assessment : 11th international conference , dimva 2014 , egham , uk , july 10 - 11 , 2014 .proceedings _ , ch .av - meter : an evaluation of antivirus scans and labels , pp .cham : springer international publishing , 2014 .m. feldman , c. papadimitriou , j. chuang , and i. stoica , `` free - riding and whitewashing in peer - to - peer systems , '' in _ proceedings of the acm sigcomm workshop on practice and theory of incentives in networked systems _ , pp . 228236 , acm , 2004 .c. rossow , c. j. dietrich , c. grier , c. kreibich , v. paxson , n. pohlmann , h. bos , and m. van steen , `` prudent practices for designing malware experiments : status quo and outlook , '' in _ 2012 ieee symposium on security and privacy _ , pp . 6579 , ieee , 2012 .m. bailey , j. oberheide , j. andersen , z. m. mao , f. jahanian , and j. nazario , `` automated classification and analysis of internet malware , '' in _ international workshop on recent advances in intrusion detection _ , pp . 178197 , springer , 2007 .j. canto , m. dacier , e. kirda , and c. leita , `` large scale malware collection : lessons learned , '' in _ ieee srds workshop on sharing field data and experiment measurements on resilience of distributed computing systems _ , citeseer , 2008 .

Abstract: threat intelligence sharing has become a growing concept , whereby entities can exchange patterns of threats with each other , in the form of indicators , to a community of trust for threat analysis and incident response . however , sharing threat - related information have posed various risks to an organization that pertains to its security , privacy , and competitiveness . given the coinciding benefits and risks of threat information sharing , some entities have adopted an elusive behavior of `` free - riding '' so that they can acquire the benefits of sharing without contributing much to the community . so far , understanding the effectiveness of sharing has been viewed from the perspective of the amount of information exchanged as opposed to its quality . in this paper , we introduce the notion of quality of indicators ( qoi ) for the assessment of the level of contribution by participants in information sharing for threat intelligence . we exemplify this notion through various metrics , including correctness , relevance , utility , and uniqueness of indicators . in order to realize the notion of qoi , we conducted an empirical study and taken a benchmark approach to define quality metrics , then we obtained a reference dataset and utilized tools from the machine learning literature for quality assessment . we compared these results against a model that only considers the volume of information as a metric for contribution , and unveiled various interesting observations , including the ability to spot low quality contributions that are synonym to free riding in threat information sharing .