$crypto_{lib}$: Comparing and selecting cryptography libraries (long version of EICC 2022 publication)

Selecting a library out of numerous candidates can be a laborious and resource-intensive task. We present the $crypto_{lib}$ index, a tool for decision-makers to choose the best fitting cryptography library for a given context. To define our index, 15 library attributes were synthesized from findings based on a literature review and interviews with decision-makers. These attributes were afterwards validated and weighted via an online survey. In order to create the index value for a given library, the individual attributes are assessed using given evaluation criteria associated with the respective attribute. As a proof of concept and to give a practical usage example, the derivation of the $crypto_{lib}$ values for the libraries Bouncy Castle and Tink are shown in detail. Overall, by tailoring the weighting of the $crypto_{lib}$ attributes to their current use case, decision-makers are enabled to systematically select a cryptography library fitting best to their software project at hand in a guided, repeatable and reliable way.


INTRODUCTION AND OUTLINE
In computer science, it is common to outsource often used functionalities to libraries. These are then included in software projects if required. A particular advantage of cryptography libraries is the professional implementation and maintenance of cryptographic functionalities by cryptography experienced developers. Non-cryptography experienced developers can use these libraries without having to deal with the rather complicated mathematical fundamentals and other obstacles of cryptography. Currently, there are numerous cryptography libraries 1 to choose from. This work defines the index to compare and select cryptography libraries * Both authors contributed equally to this research. 1 https://en.wikipedia.org/wiki/Comparison_of_cryptography_libraries Retrieved 03 Jan 2022 based on a guided evaluation of weighted attributes. This index can be used efficiently by decision-makers to select a cryptography library for their software project.
We first present related work in Section 2 that also describes contributions to library selection. Then, attributes and associated evaluation criteria that allow a rating to be made are identified. The attributes are first compiled in Section 3.1 based on a literature research and afterwards deepened in Section 3.2 based on interviews with decision-makers of software projects. In Section 3.3 these attributes are then synthesized into a manageable set. Section 3.4 describes the conducted validation of the synthesized set through an online survey. The index is introduced in Section 4 and described in detail in Section 5. The calculated index values for the libraries Bouncy Castle and Tink are presented and discussed in Section 6. We discuss our results in Section 7 before Section 8 concludes the paper and lists further open topics.

RELATED WORK
The work of Scheller et. al. [17] identifies the usability of libraries as an important quality attribute and proposes a method to evaluate it. Surpassing this limitation to one single attribute (usability) we present a quantitative comparison based on multiple weighted attributes in the work at hand.
Wilde and Amundsen [18] describe in their work, in principle libraries can be compared and contrasted in any case. Their work focuses on decentralized systems rather than on libraries as we do. However, they mention that ultimately there can be no single perfect choice. We agree on that, but we want to provide a way to make a choice tailored to a given context. Gao et. al. [5] addresses library recommendations in their work. Based on a selected library, they search for other suitable libraries. With the help of the libraries of previous similar software projects, the authors propose alternative libraries to consider. Different from the work at hand, they do not use an index for their library selection.
The work of Xie et. al. [19] deals with an algorithm for ranking different libraries for combined use. The approach is not appropriate for individual library evaluation and is solely based on the library description. Our work proposes a method to derive a quantitative index value for individual libraries through which they can be directly compared.
The German Federal Office for Information Security (BSI) carried out a comparison of several existing cryptographic libraries [9]. The goal of the comparison was to find a library for further development. The rating criteria were generated from the technical guideline BSI-TR-02102 [3]. This comparison is very specific to the project objective. The safety aspects from BSI-TR-02102 can serve as a basis for the evaluation of security. Our index, in contrast, is customizable to any project.
In the following section, attributes for our library index are obtained from additional related work not considered in this section.

ATTRIBUTES COLLECTION
In order to collect attributes known to the scientific community and possibly identify new relevant ones for our library evaluation, the following two methods were used: Literature review and interviews with decision-makers. Section 3.1 describes from which scientific sources attributes were identified by literature research. Subsequently, Section 3.2 describes how additional attributes were identified through interviews with decision-makers. These attributes confirm or complement the attributes identified from the literature review.

Attributes identified by a literature survey
The literature search was conducted in the digital publication libraries of IEEE, ACM, and Springer. From these sources, seven papers were identified that defined or described attributes for library evaluation. These seven papers are briefly presented in the remainder of this section. The attributes described by the respective authors can be taken from Table 5 with reference to their source. For example, the attribute Role Expressiveness (ID: 1.2.1) is acquired from paper [4], which is presented next.
Clarke's Cognitive Dimensions Framework [4] presents twelve factors by which developers are influenced when working with a library. The attributes were created by the authors with the use of Visual Basic classes in their minds. All twelve attributes have been adopted by us.
The authors Zghidi et. al. extend Clarke's framework in their paper [20] by another eleven attributes that were collected in a study in a software company that is not described further. In their study stakeholders brought a stronger technical perspective to the process. All eleven attributes were included in further consideration by us.
In the work of Grill et al. [7] heuristics are used to identify usability obstacles. The heuristics were determined by a study. All 14 attributes from this work were adopted by us.
Myers et al. [15] describe the usability of libraries using a usercentered design process. The work provides a platform as well as a cross-purpose guide to improving library usability and lists approaches to validating a library for this purpose. All ten attributes from this work were adopted by us.
Ease of learning libraries is of highest relevance for the authors of [10]. They derive so-called "learning barriers" from studies carried out in their work. We derived five attributes relevant for library evaluation from these "learning barriers" for further use in our work.
In [2], Bloch shows why a good library design is important for usability and how it can be considered in development. From this work, 21 attributes are extracted and used by us.
Using the method of categorization by [12], additional attributes were identified with the work of Acar et. al. [1]. Although [1] focuses on the applicability of Python libraries, they define several points that can be used for general scoring. Five attributes were adopted by us.
The papers presented above [1,2,4,7,10,15,20] focus on the usability of libraries. The works predominantly pay attention to good applicability and the necessary comprehensibility of the library. It is noticeable that they demand extensive and detailed documentation, preferably with many examples. The quality of library documentation is mentioned most frequently as an attribute and thus enjoys high relevance among the attributes.
In conclusion, the literature search produced a set of 78 unique attributes that are considered in our further process. Some attributes were discovered in several different works as is documented in Table 5.
Except for [1], the works presented here did not consider cryptography libraries. Accordingly, cryptography specific attributes were additionally identified through the interviews described in the following section.

Attributes identified by interviews with decision-makers
Additional attributes were identified via interviews with decisionmakers. The interviews intended to examine the attributes found in the literature for their actuality and additionally to identify further relevant attributes specific to cryptography libraries.
The interviews were conducted with individuals who are in positions of making technical decisions in software projects in their professional environment. Decision-makers such as technical directors, CTOs, or autonomous developers were explicitly invited to the interviews. It is important that these were people who were involved in cryptography library selection in the past or will be in the near future.
A total of 5 interviews were conducted. 4 people had already selected a cryptography library at that point. The average work experience is 8.1 years. The people were recruited from companies and the social circle of our university. The interviews were conducted in compliance with the applicable General Data Protection Regulation (GDPR) and in accordance with §24 of the Hessian Data Protection and Freedom of Information Act (HDSIG) 2 .
They were performed via different online conferencing systems. In the process, the audio was recorded with the consent of the subjects. Subsequently, the recordings were anonymized by verbatim transcription. Qualitative content analysis using a categorization method described in [11] was used to extract attributes from the interviews. Since the interviews were conducted independently from the literature review, we refrained from defining a grouping with justified evaluation aspects before categorizing. Therefore, categorization grouping was performed using Mayring's inductive category formation method [14].
The interviewer followed the interview guide (Appendix B) to ensure consistent conditions. In order to reduce initial contact inhibitions and to be able to better assess the people, demographic information and previous experience in projects were shortly discussed at the beginning of each interview. Subsequent questions addressed the library decision-making process used in the company or project in which the people were involved. Through the questions formulated in Appendix B, the interview aimed to find out whether there is a decision-making process for choosing libraries and how it is formulated or lived.
This allowed us to generate attributes of interest for scoring from possible existing specifications imposed on people in their projects. Likewise, the composition of the team or the decision-making body is interesting; such as areas of expertise and interests of the people involved. Possibly, evaluation criteria result from organizational dependencies.
To identify more attributes, the interviewer asked how this group of people find a decision and what sources are used in this process. At the end of the interview, the people were given the opportunity to ask further questions or to express themselves freely.
From the interviews, through a qualitative content analysis of the transcription, 50 attributes were identified. These are tagged in Table 5 with an "I" for interview and the number of mentions across all interviews, for example "(I:3)". In addition to the collection of attributes for scoring, it is obvious from the interviews that there is a strong personal opinion or preference among the people about the procedure. These preferences come into effect differently at evaluating and making decisions about libraries. For people without a given fixed decision-making process in the company or the project, personal opinion shapes relevant decision criteria.

Synthesis of Section 3.1 and Section 3.2
The 78 attributes obtained from the literature review and 50 from the interviews can be considered complementary sources, as they were collected using independent methods. The 128 attributes were categorized by the authors based on their descriptions or the context in which they are placed. In this process, the same, similar or related attributes were grouped under a meaningful attribute and structured hierarchically. In Table 5 this is represented by levels 1 − 4 and the ID. In the further course of this work, level 1 is referred to as attribute. Since the respective levels 2 − 4 are more finely granular than the superordinate level 1, they are suitable as evaluation criteria and will be referred to as such in the following.
This process resulted in 15 attributes which are listed in Table 1.
The description of the respective attributes listed there includes an explanation of the term. These were generated from the description of the attributes in the literature or the contexts in the interviews. The description of the attributes is intended to help developers later when weighting the attributes.

Validation of the synthesis via survey
By an online survey, the results in Table 1 ere confirmed. Participation in the survey was voluntary and in compliance with the currently applicable data protection regulations GDPR and §24 HD-SIG. People acquisition was conducted via direct messaging, social media, and email. A total of 36 people participated in the survey. Twelve people performed them completely. Appendix C lists the questionnaire. The average programming experience of the people is 7.88 years. Figure 1 shows the frequency with which people make decisions about libraries. 24% of the people have already selected a cryptography library. How complex is the API and how flexible can it be used and configured? What is the design aesthetic and how much boilerplate code (same code that needs to be repeated many times with no changes) needs to be written? Maintained How maintained is the API, meaning is it being developed further and is support being provided? Spread How widespread is the API and how big is the community? What are the opinions on the API, how is its reputation, are there successful stories or recommendations? Performance Impact How does the API impact performance and latency?

Security
Are the API and the procedures used secure? Documentation Is the API documented thoroughly and are best practices being followed or examples are given? In Figure 2, the questions 4 and 5 are answered by people using a five-point scale (not at all = 1, hardly = 2, moderately = 3, fairly = 4, extraordinarily = 5 [16]). It can be seen from the blue dots (question no. 4) that all descriptions were rated by them as matching the attributes with a tendency to fairly.
From these results, a preference of the people is deduced. It becomes clear that Ease of Use and Security are the most relevant attributes for cryptography libraries from the people's point of view. For both attributes received the most votes (four) in 15th place. Using the preferences from Table 2, a reference weighting for the index is derived in the next section.

INDEX DEFINITION
In order to create a comparable consistent weighting for the index, the number of mentions of the attributes from the literature, the interviews and the weighting from Table 2 are put together. The number of mentions in the literature, the interviews, and the rankings from Table 2 are each normalized by the rank method [8]. The average of these three sources (column no. VI in Table 3) is then ranked again (in column no. VII). Based on the rank in column no. VII, the possible values for ( ) (column no. VIII in Table 3) 1.5; 1.25; 1; 0.75; 0.5 are assigned as following: the ranks 15 -13 get 1.5 points each, the ranks 12 -10 get 1.25 points each and so on. This procedure is shown in Table 3 with the intermediate results and yields the reference weighting ( ) of the attributes in column no. VIII.
The weighting of the attributes can be chosen differently for each software project. This is realized by individual adjustments to the weighting in Table 3 column no. VIII. The sum of the individual weights should correspond to the number of attributes. This is to prevent excessive weighting.
The index of a cryptography library is calculated from the sum of all evaluation criteria of an attribute divided by the number of evaluation criteria. This sum is multiplied by the weight of the attribute and the sum of all attributes gives the index. Thus, the index is defined as follows: Where stands for evaluation criterion of attribute ; = number of in attribute ; = number of attributes and stands for the weighting of the respective attribute (see Table 3 column no. VIII).
We construct the formula in this way because, this index treats attributes with many evaluation criteria, the same as attributes with few once. This allows evaluation criteria to be easily added (or omitted) in a possible new version of this index. Multiple evaluation criteria for a single attribute ensure that the evaluation of the attribute will be more fine-grained and accurate. The weighting of the respective attribute has more influence on the index than the number of evaluation criteria of an attribute. This makes the index flexibly adaptable to the needs of software projects.

DEFINITION OF THE INDEX EVALUATION CRITERIA
The evaluation criteria defined in Appendix A are used for library assessments. The assessment results are used to create the index described in the previous section. In general, all evaluation criteria are normalized to the scale [−2, −1, 0, +1, +2]. For many evaluation criteria, the scale is not fully indicated within its textual description. In such cases, the intermediate values can be interpolated linearly. Evaluation criteria without explicit description of normalization are presented as follows via numerical values in percent. This rating is referred to as the default rating in the further text.
+2 if at least 90% of the condition of the evaluation criterion is met. +1 if at least 75% of the condition is met. 0 if at least 50% of the condition is met.
-1 if at least 25% of the condition is met. -2 if less than 25% of the condition is met.
The evaluation criteria are listed and described in detail in Appendix A.

EXEMPLARY USE
In the previous section, the evaluation criteria of the 15 attributes were introduced. These were used to evaluate the Java versions of Google's Tink 3 library and the Bouncy Castle 4 library. The two libraries were chosen because they appeal to the similar target group of developers. They are both developed and maintained by an open source community. Bouncy Castle is established. Tink is relatively new (since 2017) and claims to be easy to use for developers without cryptographic knowledge 5 . Table 4 shows the individual points achieved for the evaluation criteria of both libraries. The attributes are the average of the evaluation criteria. These values were multiplied with the reference weighting from Table 3 column no. VIII and the results summed up resulting in the index value for each. The index of Bouncy Castle is 7.08. Tink reaches an index of 16.75. With the reference weighting used, a minimum index of −29 points and a maximum index of 29 points is achievable. Two detailed assessments are shown below as examples. They are based on the respective criteria in Annex A. For the evaluation criterion 1b Default Settings, the Bouncy Castle library was given a score of 0, since obsolete algorithms can be used due to the flexible constructors. One example is the constructor for the base AES keygen KeyGen (int) where the user can input any value. The Tink library has been assigned a score of +2, because safe default values are used. For the evaluation criterion 11a Release-frequency, the Bouncy Castle library was given a score of +1, because the current 6 release cycle is every four months on average. The Tink library has achieved a rating of +1 as well because currently 6 releases are being released quarterly.
These examples, and Table 4 shows that the selected evaluation criteria are applicable. With the reference weighting generated from the online survey preferences in Section 3.4, the Tink library gets a better index. When compared in detail, the Tink library performs better in each of the three attributes weighted highest at times 1.5: Ease of Use, Code Quality, and Documentation. Moreover, with the exception for Testability, this is the same case for the attributes with 1.25-fold weights Complexity and Security. Thus, according to the reference weighting, the Tink library is preferable to the Bouncy Castle library.
Since each software project has different conditions and requirements, the weighting can be adjusted individually. This leads to index results adapted to the corresponding software project.

DISCUSSION AND LIMITATIONS
It is generally difficult to get experts for interviews. Since only 5 people participated in the interviews, it is possible that not all relevant properties for cryptographic libraries were mentioned. Due to technical advancements, some of the evaluation criteria might change their relevance. They should periodically be reviewed, therefore, to ensure that they are up to date. New criteria should be added to a new version of the index, if necessary. For example, the authors are not aware of any useful evaluation criterion applicable for the attribute Performance Impact. This might change over time.
In the same way, individual evaluation criteria show potential for improvement. For example, the evaluation criterion 12b Repositories is difficult to implement for commercial projects, since these are rarely developed on a public repository. Other evaluation criteria are better suited for libraries written in object-oriented languages. For example, the criteria 3b Exceptions, 4a Public, and 4b Interfaces.
This represents a bias against procedural languages.
Overall, the index described in this work is a first step to compare cryptography libraries with each other. The individually adaptable weighting supports the project-specific customization to meet the needs of the software project. The reference weighting provided in the work at hand enables libraries to be compared independently of specific projects, leading to a general quantitative comparison of crypto libraries.

CONCLUSION AND OUTLOOK
This is a first attempt to create a index based on literature and interviews. By evaluating the two libraries Bouncy Castle ( index of 7.08) and Tink ( index of 16.75) as examples, we have shown that this index is applicable.
Our reference weighting emphasizes that Ease of Use, Code Quality, and Documentation are the most relevant attributes. We have seen that some evaluation criteria do not equally fit to all programming languages. Furthermore, it is not always easy or even possible to find the information needed to assess the evaluation criteria. More surveys may provide new attributes or new evaluation criteria of existing attributes for our index.
Although it takes effort to evaluate many libraries using our index, in the end there will be a benefit for software quality. We envision an open access database where the index of many libraries is stored and actively maintained by a committee of experts. The ranking of individual libraries based on the reference weighting could be visualized in a graph over time to show the evolution of each library. Decision-makers could then just adapt the weightings of the attributes to fit their particular project.

A EVALUATION CRITERIA
(1) Ease of Use: (a) Readability: The length of the function calls or the number of their parameters in a library are crucial for intuitive use [15]. Function calls with more than three parameters should be well justified and still avoided if possible [13].
The default evaluation determines how many functions have niladic or monadic (less than or equal to two parameters  +2 for licenses that allow unrestricted commercial use. 0 for licenses that allow free use for non-commercial purposes or allow commercial use against payment. -2 for licenses that do not provide for commercial use and still require payment. (9) Requirements: (a) Dependencies: There must be other software, software packages, or files installed. +2 the library automatically installs all dependencies if it has any (possibly via a package manager). -2 the dependent software must be installed manually. (10) Complexity: (a) Atomic Setting: Precise or fine adjustments can be made. +2 if an API specifies settings and these can be changed. 0 wenn eine API Einstellungen vorgibt und diese nicht verändert werden können. index 0 if an API specifies settings and these cannot be changed.
-2 when parameters must all be selected manually. (b) Boilerplatecode: Recurring code overhead must be written by developers in order to use certain functionalities of the API. +2 for APIs that provide dedicated methods instead of recurring code. 0 for APIs where the largest blocks are simplified. -2 when the user has to write the same code over and over again. (11) Maintained: (a) Release-frequency: How often is a library updated or a new major version released? The question refers to the last three major versions of a library. +2 there is a fixed release schedule of the library. 0 there is no fixed release schedule, but eventually, the next release will come. -2 it is not clear if there will be another release. (b) Patch frequency: Security gaps are quickly fixed by patches.
+2 if patches have been released within 90 8 days in the past. +1 when patches are delayed for more than 90 days. 0 when it is not clear if patches will be provided. -2 if no patches are delivered. (c) Support: The library has an official support channel (e.g., a forum, wiki, or mailing lists). +2 if the project offers free support for the product. 0 with an official paid support.
-2 when there are no official information channels and sources. (12) Spread: (a) Successful Stories: There are articles on successful stories of using the library in other projects. Successful stories are articles, contributions or scientific papers on successful use, which, for example, attest to the added value of the library or describe experiences. +2 in the case of several articles or field reports in reputable trade journals such as heise.de 9 . 0 with no significant mentions in professional journals.
-2 for purchased posts on blogs or similar formats. (b) Repositories: The project is very popular on repositories (such as GitHub, GitLab, SourceForge, Bitbucket or a repository managed by the project). This can be read off from "'likes"', "'stars"' or similar. The basis for the rating is the most popular Java library on GitHub in English language at the time of the work (December 2020) (Mindustry with 7557 stars 10 ). The evaluation of this criterion is based on the default evaluation. (13) Performance Impact (14) Security (a) Standards: Are used exclusively algorithms that are on the white list [3].
(2) How often do you need to choose an API? (checkbox: never, once a year, several times a year, once a month, several times a month, weekly) (3) Have you used a crypto-API before? (yes/no) (4) Does the description match the term? (Answer options: not at all, barely, mediocrally, fairly, exceptionally) List with the attributes and descriptions from table 1. (5) Is the term suitable for the rating of an API (not in context of own projects)? (Answer options: not at all, barely, mediocrally, fairly, exceptionally) List with attributes from table 1. (6) Sorting question: Sort the following attributes according to their relevance for choosing a suitable (crypto-)API for your last project in descending order. Drack-and-drop list with the attributes from table 1. (7) Do you have any Remarks, Feedback or suggestions concerning the questions or survey?