The Role of Diversity in Cybersecurity Risk Analysis: An Experimental Plan

Cybersecurity threat and risk analysis (RA) approaches are used to identify and mitigate security risks early-on in the software development life-cycle. Existing approaches automate only parts of the analysis procedure, leaving key decisions in identification, feasibility and risk analysis, and quality assessment to be determined by expert judgement. Therefore, in practice teams of experts manually analyze the system design by holding brainstorming workshops. Such decisions are made in face of uncertainties, leaving room for biased judgement (e.g., preferential treatment of category of experts). Biased decision making during the analysis may result in unequal contribution of expertise, particularly since some diversity dimensions (i.e., gender) are underrepresented in security teams. Beyond the work of risk perception of non-technical threats, no existing work has empirically studied the role of diversity in the risk analysis of technical artefacts. This paper proposes an experimental plan for identifying the key diversity factors in RA.


INTRODUCTION
Security-by-design techniques strive to avoid risks before the software is actually developed to reduce the security incurred costs in later stages of development. For instance, threat and risk analysis (RA, for short) can be performed by adopting systematic techniques to scrutinize the software system (e.g., STRIDE [32], attack trees [30], CORAS [20], to name a few). Due to development trends (Agile, DevOps), analysis automation is a much needed direction for improvement [6]. However, many automation efforts automate parts of the analysis procedure and rely on an extended manual modeling effort [35]. Key decisions in the identification, feasibility and risk analysis, and quality assessment are often left to the experts. In practice, teams of experts often analyze the system design manually Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). by holding brainstorming workshops where they discuss potential risks and decide which risks will be mitigated and how. Risk based decisions must be made in face of uncertainties [2], leaving room for biased judgement (e.g., preferential treatment of category of experts). In addition, the quality of analysis outcomes is ultimately determined by (potentially biased) expert judgement [3,16]. For instance, evidence of groupthink [44] in safety analysis sessions suggests that knowledge is not always contributed (equally) by all participants, but exact variables that effect the technical quality of the analysis are not yet understood. Further, the type of judgement bias present in risk-based decision making and more generally the role of diversity has not yet been sufficiently explored in RA (as discussed in the related work, Section 3).
In many engineering domains, there is a lack of security experts [36]. In addition, security expert teams are not diverse, especially for what concerns gender. According to a news post by the International Consortium of Minority Cyber Professionals (nonprofit) in August 2021 [7], the cybersecurity workforce in the USA is currently only 14% female. Generally speaking, there is a significant mismatch between the cultural notion of a successful technical professional and the perceived skills that are needed for success in the computer science domain. Wynn and Correll [47] show that men are more likely than women to believe they possess the stereotypical traits and skills of a successful tech employee. This mismatch can explain the low technical confidence of young female professionals that are deciding about their career path in computer science [1]. We believe there is still untapped potential in empowering women to pursue their career as security experts. This motivates us to gather empirical evidence that could help in (i) raising more interest and confidence levels of female students in cybersecurity topics, and (ii) encouraging organizations to build gender balanced security teams.
The purpose of the laid out research track is to identify the diversity dimensions (see Section 2) that play a role during RA of more sophisticated technical artefacts. To simulate RA in the experimental setup, the ACM Code of Ethics scenarios can be used as RA task when interacting with the communication science population as opposed to traditional economics/social experiments. Second, case studies with detailed information about the software design (e.g., IoT home monitoring system used in [37]) can be used as RA task when interacting with the computer science population. The experimental conditions will differ depending on the measure and target audience (as described in Section 5).
This paper discusses the challenges (see Section 4) and outlines the experimental plan (see Section 5) for identifying the role of diversity in the technical domain of RA. Our experimental plan is three-fold. First, we plan to characterize the key differentiating factors in diversity dimensions (e.g., gender) that are present to RA. Second, we plan to evaluate their effects in a non-technical population and technical population. Third, we plan to develop and validate diversity interventions (given to participants in the form of training) in realistic (technical) environments. We briefly mention the next steps in the laid out plan and present the concluding remarks in Section 7.

DIVERSITY DIMENSIONS IN RA
Diversity processes are well understood in the social science disciplines, such as, the role of diversity factors in group dynamics or effects of diversity biases in organizational processes. In contrast to the technical domain, studies have demonstrated how diversity (gender, nationality) can be beneficial for decision making and progress, once effectively incorporated [24,40,41].
Gender. Gender is an individual's own gender identity, which is typically, man and woman, but can also be non-binary. Rodriguez et al. [28] found evidence of bias against women in some software engineering communities, and sometimes negative perceptions about women working in teams. Thus, this is an interesting dimension to further investigate in the context of cybersecurity.
Education. Education is an individuals' achieved level and topic of specialization (e.g., computer security vs AI) of academic studies. Risk-based decisions have to be made in organizations by the managerial layers, who typically have a good understanding of the product, but do not necessarily posses the technical skills of security experts or engineers. Therefore, it is interesting to investigate this dimension and include participants from a different domain (e.g., with background in communication sciences). Education turned out to be a non-significant variable in the study of the impact of commercial Antivirus on people's awareness of security incidents [15]. However, it is not clear whether this dimension has an impact in performing a RA task.
Nationality & Race. Nationality is the country of origin, which is often coupled with the culture and language that categorizes social groups. Race is a social construct linked with individual's physical characteristics such as skin color and is used to categorize populations. Determining the effect of racial (and nationality) bias in cybersecurity practices is to date an open question. Thomas et al. [34] conducted semi-structured interviews with 14 Black women in computing and report that Black women experienced isolation (though it is not clear whether due to gender or race). But, few studies have focused on racial (and nationality) diversity in the software engineering discipline [28].
Age & Seniority. Age and seniority (i.e., position in the organization and years of experience) may have an impact on both the technical outcomes of a RA task and on the judgment of outcomes quality (e.g., over-favoring results submitted by senior analysts). For instance, a category of experts (e.g., junior vs senior) may underestimate the ease with which a mitigation can be implemented in reality [46]. This may hurt later-on when infeasible mitigations have to be replaced with cheaper (and perhaps) less insecure solutions. We remark that though this dimension is indeed relevant, empirically observing seniority (in a realistic setting) requires involving professionals. Our experimental plan is designed in an academic (controlled) setting with student participants. Thus we do not attempt to observe this dimension as part of this work.

RELATED WORK
There is an extensive account of existing RA techniques, their main characteristics, information sources, and limitations. We refer the interested reader to the literature reviews by Tuma et al. [35] and Ling et al. [19]. For an overview of research on diversity theory and the relevant social processes involved in organizations (e.g., group dynamics) see studies [24,40,41].

Diversity Studies in Risk Analysis
Beyond the gender theory about different risk perception [10], there was little scientific inquiry in the computer science domain about gender diversity (or other diversity processes) in risk analysis in the past 10 years. When considering non-technical scenarios (e.g., returning from work at night or natural disasters) different risk perceptions between men and women (and ethnic minorities) have been investigated, mainly in the USA but also in Sweden [25]. While the studies conducted in the USA suggest that white males perceived lower risk compared to women and minorities, the same was not observed by Olofsson and Rashid for Sweden [25]. A recent study [15] investigated the impact of commercial Antivirus on people's awareness of security incidents with 1000 respondents. They found that the only relevant variables were online insecurity, risk aversion and gender (other non-significant variables were education, age, experience with information systems). Other empirical studies [8,23] claim a diversity based on participation in simple games.
Thus, we discern that existing research has focused on measuring diversity aspects in RA when participants consider non-technical artefacts while there is still the need to study diversity factors when considering technical (e.g., cybersecurity threats and mitigations) artefacts.

Diversity Studies in Computer Science & Software Engineering
Rodriguez et al. [28] have recently conducted a systematic literature review to identify the diversity aspects that have been studied in software engineering and development. Gender has been the most widely researched dimension (61% of the papers in [28]), while others (nationality, age, race) are less explored. This does not come as a surprise, considering that computer science, engineering, and physics are less gender balanced compared to other STEM fields (e.g., biology) [5]. In addition, existing research has focused on studying diversity in the context of open source communities and developers (e.g., see the study on gender bias on GitHub [14]), while other domains are less explored. For instance the role of gender-diversity in the domain of software architecture is unexplored [27,33]. Some tools have been proposed to assess diversity dimensions of software or software development processes (Gen-derMag, InclusiveMag, and AID Tool, to name a few). However, most studies have focused on identifying bias through case studies, rather than conducting experiments or proposing new models, tools, or processes [28]. GenderMag [43] is a method to inspect software (e.g., interface design) to identify gender-inclusiveness issues. InclusiveMag [21] is a generalization of GenderMag that can be used to generate inclusiveness methods for a particular dimension of diversity. AID

Outcome
Outcome type Tool [4] is an automation of the GenderMag method to detect "inclusivity bugs" which are software features that disproportionately disadvantage particular groups of contributors in OSS. The tool was evaluated on 20 GitHub projects.

CHALLENGES
This section outlines the challenges in establishing the role of diversity in RA.

Evidence of Biased Judgement
The first challenge is capturing whether diversity dimensions have an effect on biased judgment. To observe this we plan to measure the type (i.e., gender bias, education bias, and nationality and race bias) and amount of bias the analyst may be projecting when making (subjective) judgements about another analyst's work (report of outcomes). We are interested to observe the evidence of biased judgement in studies with and without the technical artefact and in different populations (i.e., computer science and communication science). Measuring such bias is relevant considering that sometimes a preferential (managerial) choice must be made between analysis reports of two different analysts which may belong to different diversity groups (e.g., man vs woman). Lastly, capturing the interplay between the evidence of biased judgment and actual diversity effects on analysis outcomes is an unexplored direction. We may find there are no actual differences in the outcomes produced, but there is evidence of biased judgments (or vice-versa).

Evidence of Diversity Effects on Outcomes
The second challenge is to investigate whether diversity dimensions have an actual effect on the task of RA. To this aim, we will measure the effects of diversity dimensions on the RA outcomes. Since the quality of analysis lacks a formalised definition (e.g., often natural language is used to describe attack scenarios and informal notations are used for modeling [35]), we will use measures that can be easily reproduced. Namely, we can observe how diversity dimensions effect the type of analysis outcomes. Table 1 shows various outcomes that are often included in the analysis report. In what follows, we describe the outcomes and motivate their inclusion for observation. Threats. A malicious attacker (with means and motivations) poses a threat to harm the desired security properties of system assets. As part of the analysis outcomes, the identified threats (e.g., spoofing an authorized user) are accompanied by a detailed description of the attack scenarios (e.g., CAPEC-656: Voice Phishing 1 ) that realize the threat. We borrow the threat categories from the documentation of STRIDE, a threat modeling methodology developed by Microsoft [32].
Analyses conducted by experts tend to be more balanced in terms of the categories of identified threats, whereas RA novices tend to report disproportionately more tampering, denial of service and information disclosure threats [36,37]. We are interested to observe whether category distribution patterns emerge for other diversity dimensions.
Assumptions. Assumptions are statements about the system under analysis that may or may not be true [36]. While system requirements and constraints are typically explicitly defined and static, assumptions are often implicit and dynamic in nature (i.e., they can be invalidated and modified as the project evolves). Due to little support offered to the analysts regarding the assumption definition, quality assessment, and management, a recent study investigated the role of assumptions in context of STRIDE [42]. Van Landuyt and Joosen [42] find that the majority of assumptions (created by students during STRIDE) were used to either justify an existence of threats or are used to eliminate threats. In [42] a substantial subset (78%) of the assumptions was in direct reference to security-related concepts (i.e., security assumptions), however also domain assumptions (statements about component functionalities) were made.
In the safety science domain, managerial over-reliance on routine technical inspection was identified as a common underlying characteristic of human-created disasters [45]. In RA, a category of experts may make assumptions about different properties of the system and similar over-reliance on verifying assumptions could be present. We are interested to investigate the effect of other diversity dimensions on the type of assumptions that are made during RA.
Attack surface. Security analysts often rely on defining the attack surface and the required attacker profile to exploit it to determine the feasibility of an attack scenario. Intuitively, consider the feasibility of an information disclosure threat on internal communication channels to expose intercepted emails by a 'script kiddie' with little computing resources and off-the-shelf hacking tools. The attack may require gaining physical access to the organizations internal network and breaking TLS encryption. In addition, the 'script kiddie' may not have a motive to launch this attack. Determining feasibility is subjective, domain-specific and not always obvious.
Giddens et al. [9] found that when analyzing internal threats, managers will consider male employees as exhibiting greater malicious intentions (associated with computer abuse) compared to female employees). Thus, it is interesting to measure whether diversity dimensions play a role in identifying the core variables for determining threat feasibility.
Effected security objective. The attacker goal is to compromise a security objective (confidentiality, integrity, availability, and accountability) of an asset of value. Since spoofing and elevation of privilege threats may compromise multiple security objectives (e.g., a lack of authentication due to spoofing may result in loss of confidentiality, integrity, or availability), we are also interested to observe possible patterns of the threatened security objectives per diversity factor.
Threat impact. Given a successful attack, threat impact refers to the damage it causes to the organization. Estimating the type (and amount) of impact is useful for organizations to put the threat into perspective of the domain (e.g., when safety-critical systems are analyze, the safety impact can not be left unmitigated). Some impacts may be easier to estimate depending on the background and previous education. For instance, for an engineer the number of effected units may seem easier to determine compared to legal and financial loss estimates, for which other department expertise may be needed [6]. Thus, we are interested to investigate whether diversity dimensions (e.g., education) have an effect on the type of impacts that our participants identify.
Risk priority. Since the number of identified threats quickly explodes in realistic projects [36], they must be prioritized based on estimations of risk (e.g., the last step of STRIDE [32]). The most critical threats are red flags and consequently larger investments are made to develop and test their mitigations. We refer to risk as a product of threat probability and impact. How individuals assess risk priorities may be related to their risk perception which is already well understood [10].
Mitigations. Mitigations of a cybersecurity risk can be preventative (e.g., implementation of two-factor authentication), detective/reactive (e.g., using intrusion detection and access revocation techniques) and corrective (such as maintaining audit trails or restoring from a secure state). Multiple strategies can be adopted to counter a cybersecurity threat, and the final choice may depend on domain-related factors, as the cost of implementing the mitigation has to be reasonable. A category of experts (e.g., man vs woman) may underestimate the ease with which a mitigation is actually implemented, as observed in [46]. By doing so, they in fact contribute to the actual mitigation of the threat. Thus, we are interested to observe how diversity dimensions effect the type of mitigations that are identified during the analysis.

Identification of countermeasures to identified effects
Because risk decisions must be made in face of uncertainties for which no statistical bias is available [2], organizational issues play a major role in which the final technical artefact is chosen [26] (i.e., security mitigations), and thus we must put in place technical/organizational measures to counterbalance it. Our final challenge is to identify: i RA practices (e.g., methods, tools, processes borrowed from adjacent disciplines) that help foster diversity, and ii feasible and effective countermeasures for organizations to adopt.
Identifying diversity fostering practices could be partially achieved by conducting a more systematic review of the existing literature. However, to evaluate the identified countermeasures empirically more future efforts will be needed.

EXPERIMENTAL PLAN
Study of diversity effects and their prevalence in RA. There is a fairly complete account and understanding of the existing RA techniques, their main characteristics, information sources, and limitations [19,35]. On the other hand, research has shown, with regard to gender diversity, that there are differences in the perceived competence of men and women [40]. In the context of academic funding competitions, to illustrate this process, there is evidence that men and women produce ideas of similar quality, yet women are underestimated for their competence [39]. In other words, this implies that women are likely to be equally competent in identifying security threats as men, yet their assessments might not be sufficiently valued. Hence, this might have far reaching consequences for the RA as this might result in an underestimation of the security threats. It is thus pivotal to investigate whether diversity processes are at play during RA.

RQ1.
What diversity processes play a role during security threat analysis and risk assessment of IT systems?
Experimentation and validation of diversity interventions in RA. Previous studies have experimented with RA techniques with practitioner and student participants [17,18,36,37] measuring (among others) technique productivity, analysis outcomes, perceived ease of use. But, diversity relevant measures, controls and relevant training for RA techniques have not yet been defined for such experiments [28]. In contrast, Van der Lee and Ellemers [40] have identified characteristics of effective diversity interventions, and applied those to practical decision making processes. One of the key features of diversity intervention effectiveness is the degree to which they are applicable to the current situation. In other words, whether participants can use the knowledge that they gained [12]. The interventions thus need to promote ways in which analysts can use their diversity to better their decision making. A factor that might hinder this process is the perceived objectivity of the analysts, thereby paradoxically increasing rather than decreasing implicit biases rooted in stereotypes [38]. Hence, to strengthen the quality of RA the diversity interventions must be grounded in theory and tested in the context of RA.
RQ2. What experimental protocols, metrics, and controls capture and effectively incorporate diversity in the context of RA of IT systems?

Hypotheses & Methods
To answer the research questions RQ1 and RQ2, we formulate two type of hypotheses. First, we pose three hypotheses about the equivalence of the sample means for biased judgement • (corrective) engineering a worm which targets behavior resembling "hacker" activity on the effected server. (e.g., by using Two One-Sided T-Tests (TOST) [22]) in presence and absence of the technical artefact.
To test hypotheses 1 − 3 , we designed a survey experiment with two populations, computer science BSc students and communication science BSc students. The survey is targeting students taking courses taught by the experimenters. Individual participation to the survey will be voluntary. The survey will be based on a high-level scenario (such as the ACM ethics Malware Disruption case 2 ) and will require the participants to understand basic principles of security and privacy. We adopted a blocked design for the experiment with two versions of the survey which are distributed to both populations, as shown in Figure 1. The first will include a high-level executive summary of the RA outcomes (in security layman terms) including the final recommendation (i.e., threat mitigation) and a fictitious persona description of the RA analyst. The fictitious personas (also called vignettes) can be used to measure biased judgements. The second survey version will include the executive summary, the final RA recommendation, the fictitious persona, and a technical appendix with the actual RA outcomes in full detail. Table 2 shows the dependent variables used to measure the biased judgement. The level of each vignette dimension (gender, seniority, and mitigation) is randomly sampled and presented to the participant (similar to survey experiment in [11]) in a context of the ACM Code of Ethics scenario. The participant will be then asked to rate their trust in the analyst and agreement with the suggestion of the analyst. Several control questions will also follow. The independent variables are the participant gender and education background. 1 : First, we will measure whether the judgement bias (i.e., the type of bias and the effect size) in the computer science population is equivalent to the judgement bias in the communication science population in absence of the technical artefact. 2 : . + = .
If we find that the bias is in fact equivalent across the two domains, it is interesting to also investigate the equivalence of judgement bias for the communication science population with the technical artefact compared to the communication science population without the technical artefact. 3 : . + <> . If we find that the bias is in fact equivalent within the non-technical domain, we must (naturally) also find bias equivalence within the technical domain. Therefore, it is interesting to investigate the equivalence of judgement bias for the computer science population with the technical artefact compared to the computer science population without the technical artefact. If our statistical analysis does not support the equivalence hypothesis, we must further investigate how the two samples differ. For instance, if compared to the computer science population without the technical artefact the judgement bias of the computer science population with the technical artefact is greater, this is evidence of a systematic bias in the computer science community. Second, we pose hypotheses about the equivalence of the sample means for the RA outcomes in presence of the technical artefact. Figure 2 depicts hypotheses 4 : 6 . To test hypotheses 4 − 6 we will conduct controlled experiments with computer science MSc students. Concretely, we will hand out the documentation of a software system (e.g., requirements, component and deployment diagrams etc.) to individual students. After a training session, the students will be tasked to perform a RA following a prescribed technique and hand-in the RA outcomes. 4 : Regarding gender, we expect that the outcome types reported by women are equivalent to the outcome types reported by men. Studies of risk perception suggest that women perceive certain risks differently compared to men. Though we do not foresee strong differences, we might find some effects when it comes to risk priority. 5 : . 1 = .
2 Regarding education, we expect that the students of various specialization tracks report equivalent outcome types for the same system under analysis. Our population is computer science students, with some differences in the elective courses and program choices (e.g., we plan to include students from various master programs, such as IA, computer Security, and Software Engineering). We may find some outliers in the proportions of domain vs security assumptions made, depending on the previous completed courses or the master specialization. For example, possibly AI students make less security assumptions, compared to Computer Security students. Though we do not have string convictions, we may find that previous courses may have an effect on the threat impact (e.g., students with robotics and AI background may focus on safety impacts). Similarly, we may find come differences in the type of mitigations considered by students of different background. 6 : . = . We expect that the students of various race and nationality report statistically equivalent outcome types. We have not found any research that would suggest otherwise.
The findings from investigating 1 − 6 will play a key role in developing diversity interventions in the form of training. To validate the diversity training, we plan to conduct comparative experiments in a similar controlled setting, where we administer an additional diversity training to half of the participants.

POTENTIAL THREATS TO VALIDITY
Small female population. There is typically around 20% (or less) female students enrolled in computer science programs. We are aware of the validity threats caused by an unbalanced population sample, which is omnipresent in all gender diversity studies in STEM disciplines [28]. To partially mitigate this threat, we planned a sister experiment (with a less technically demanding task) with a communication science population (see Section 5). Second, we will rally female computer scientist students towards participation through local steminist groups and similar community organized channels.
The small female population in computer science is also the rationale for organizing the controlled experiments with computer science population (testing hypotheses 4 − 6) with individual students. A more realistic setup (as was done in previous studies [37]) would include dividing participants into teams as in practice RA is performed in teams of experts. But measuring the individual female contribution to the outcomes inside gender-mixed teams becomes very hard and possibly not reliable.
Generalizability of results to practitioners. We do not plan to include practitioners experienced in RA to participate in our studies. This limits us in observing the full complexity of the diversity effects (e.g., including seniority) that are actually present in organizations where RA is routinely performed. Still, studies have shown [13,29,31] that the differences between the performance of professionals and graduate students are often limited. To extend our observations we plan to conduct industrial cases studies in the future (e.g., similar to the study performed by Wang and Wagner [44]).

CONCLUSION
This paper outlines an experimental plan to discover the role of diversity in cybersecurity risk analysis. We present the relevant diversity dimensions that will be observed and discuss the challenges of capturing evidence about biased judgement during RA and about diversity effects on the actual RA outcomes. As a first step, we propose to measure the key differentiating diversity factors in RA by conducting survey experiments with a technical and non-technical population. Second, we plan to conduct controlled experiments with the computer science population to record diversity effects on the outcomes and to validate diversity interventions. Finally, an interview study with practitioners would be beneficial for identifying the most urgent diversity dimensions to be addressed first with the interventions.