SkillSpan: Hard and Soft Skill Extraction from English Job Postings

Skill Extraction (SE) is an important and widely-studied task useful to gain insights into labor market dynamics. However, there is a lacuna of datasets and annotation guidelines; available datasets are few and contain crowd-sourced labels on the span-level or labels from a predefined skill inventory. To address this gap, we introduce SKILLSPAN, a novel SE dataset consisting of 14.5K sentences and over 12.5K annotated spans. We release its respective guidelines created over three different sources annotated for hard and soft skills by domain experts. We introduce a BERT baseline (Devlin et al., 2019). To improve upon this baseline, we experiment with language models that are optimized for long spans (Joshi et al., 2020; Beltagy et al., 2020), continuous pre-training on the job posting domain (Han and Eisenstein, 2019; Gururangan et al., 2020), and multi-task learning (Caruana, 1997). Our results show that the domain-adapted models significantly outperform their non-adapted counterparts, and single-task outperforms multi-task learning.


Introduction
Job markets are under constant developmentoften due to developments in technology, migration, and digitization-so are the skill sets required. Consequentially, job vacancy data is emerging on a variety of platforms in big quantities and can provide insights on labor market skill demands or aid job matching (Balog et al., 2012). SE is to extract the competences necessary from unstructured text.
Previous work in SE shows promising progress, but is halted by a lack of available datasets and annotation guidelines. Two out of 14 studies release their dataset, which limit themselves to crowd-sourced labels (Sayfullina et al., 2018) or annotations from a predefined list of skills on the document-level (Bhola et al., 2020). Additionally, * Equal contribution.
You will thrive working in a Dev/Sec Ops culture . none of the 14 previously mentioned studies release their annotation guidelines, which obscures the meaning of a competence. Job markets change, as do the skills in, e.g., the European Skills, Competences, Qualifications and Occupations (ESCO; le Vrang et al., 2014) taxonomy (Section 3). Hence, it is important to cover for possible emerging skills.
We propose SKILLSPAN, a novel SE dataset annotated at the span-level for skill and knowledge components (SKCs) in job postings (JPs). As illustrated in Figure 1, SKCs can be nested inside skills. SKILLSPAN allows for extracting possibly undiscovered competences and to diminish the lack of coverage of predefined skill inventories.
Our analysis ( Figure 2) shows that SKCs contain on average longer sequences than typical Named Entity Recognition (NER) tasks. Albeit we additionally study models optimized for long spans (Joshi et al., 2020;Beltagy et al., 2020), some underperform. Overall, we find specialized domain BERT models (Alsentzer et al., 2019;Lee et al., 2020;Gururangan et al., 2020;Nguyen et al., 2020) perform better than their non-adapted counterparts. We explore the benefits of domain-adaptive pretraining on the JP domain (Han and Eisenstein, 2019;Gururangan et al., 2020). Last, given the examples from Figure 1, we formulate the task as both as a sequence labeling and a multi-task learning (MTL) problem, i.e., training on both skill and knowledge components jointly (Caruana, 1997). Table 1: Contributions of Related Work. We list the recent works of Skill Extraction. Note that (*) indicates labels that are automatically inferred from some source (e.g., a predefined skill inventory) and not manually annotated. With respect to the annotation approach, "Manual" indicates uncertainty whether they used domain experts or not. Also note that many works do not release their dataset with annotations () nor guidelines (). The list is inspired by Khaouja et al. (2021).
Contributions In this paper: 1 We release SKILLSPAN, a novel skill extraction dataset, with annotation guidelines, and our open-source code. 1 2 We present strong baselines for the task including a new SpanBERT (Joshi et al., 2020) trained from scratch, and domain-adapted variants (Gururangan et al., 2020), which we will release on the HuggingFace platform (Wolf et al., 2020). To the best of our knowledge, we are the first to investigate the extraction of skills and knowledge from job postings with state-of-the-art language models. 3 We give an analysis on single-task versus multi-task learning in the context of skill extraction, and show that for this particular task single-task learning outperforms multi-task learning.

Related Work
There is a pool of prior work relating to SE. We summarize it in Table 1, depicting state-of-the-art approaches, level of annotations, what kind of competences are annotated, the modeling approaches, the size of the dataset (if available), type of skills annotated for, baseline models, and whether they release their annotations and guidelines.
As can be seen in Table 1, many works do not release their data (apart from Sayfullina et al., 2018 andBhola et al., 2020) and none release their annotation guidelines. In addition, none of the previous studies approach SE as a span-level extraction task with state-of-the-art language models, nor 1 https://github.com/kris927b/SkillSpan did they release a dataset of this magnitude with manually annotated (long) spans of competences by domain experts.
Although Sayfullina et al. (2018) annotated on the span-level (thus being useful for SE) and release their data, they instead explored several approaches to Skill Classification. To create the data, they extracted all text snippets containing one soft skill from a predetermined list. Crowdworkers then annotated the highlighted skill whether it was a soft skill referring to the candidate or not. They show that an LSTM (Hochreiter et al., 1997) performs best on classifying the skill in the sentence. In our work, we annotated a dataset three times their size (Table 2) for both hard and soft skills. In addition, we also extract the specific skills from the sentence. Tamburri et al. (2020) classifies sentences that contain skills in the JP. The authors manually labeled their dataset with domain experts. They annotated whether a sentence contains a skill or not. Once the sentence is identified as containing a skill, the skill cited within is extracted. In contrast, we directly annotate for the span within the sentence. Bhola et al. (2020) cast the task of skill extraction as a multi-label skill classification at the document-level. There is a predefined set of unique skills given the job descriptions and they predict multiple skills that are connected to a given job description using BERT (Devlin et al., 2019). In addition, they experiment with several additional layers for better prediction performance. We instead explore domain-adaptive pre-training for SE.
The work closest to ours is by Chernova (2020), who approach the task similarly with span-level annotations (including longer spans) but approach this for the Finnish language. It is unclear whether they annotated by domain experts. Also, neither the data nor the annotation guidelines are released. For a comprehensive overview with respect to SE, we refer to Khaouja et al. (2021).

Skill & Knowledge Definition
There is an abundance of competences and there have been large efforts to categorize them. For example, the The International Standard Classification of Occupations (ISCO; Elias, 1997) is one of the main international classifications of occupations and skills. It belongs to the international family of economic and social classifications. Another example, the European Skills, Competences, Qualifications and Occupations (ESCO;le Vrang et al., 2014) taxonomy is the European standard terminology linking skills and competences and qualifications to occupations and derived from ISCO. The ESCO taxonomy mentions three categories of competences: Skill, knowledge, and attitudes. ESCO defines knowledge as follows: "Knowledge means the outcome of the assimilation of information through learning. Knowledge is the body of facts, principles, theories and practices that is related to a field of work or study." 2 For example, a person can acquire the Python programming language through learning. This is denoted as a knowledge component and can be considered a hard skill. However, one also needs to be able to apply the knowledge component to a certain task. This is known as a skill component. ESCO formulates it as: "Skill means the ability to apply knowledge and use know-how to complete tasks and solve problems." 3 In ESCO, the soft skills are referred to as attitudes. ESCO considers attitudes as skill components: "The ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations  To sum up, hard skills are usually referred to as knowledge components, and applying these hard skills to something is considered a skill component. Then, soft skills are referred to as attitudes, these are part of skill components. There has been no work, to the best of our knowledge, in annotating skill and knowledge components in JPs.

SKILLSPAN Dataset
Data 5 We continuously collected JPs via web data extraction between June 2020-September 2021. Our JPs come from the three sources: 3. TECH: The StackOverflow JP platform that consisted mostly of technical jobs (e.g., developer positions).
We release the anonymized raw data and annotations of the parts with permissible licenses, i.e., HOUSE (from a govermental agency which is our collaborator) and TECH. 6 For anonymization, we perform it via manual annotation of job-related sensitive and personal data regarding Organization, Location, Contact, and Name following the work by Jensen et al. (2021). Table 2 shows the statistics of SKILLSPAN, with 391 annotated JPs from the three sources containing 14.5K sentences and 232.2K tokens. The unlabeled JPs (only to be released as pre-trained model) consist of 126.8K posts, 3.2M sentences, and 460.5M tokens. What stands out is that there are 2-5 times as many annotated knowledge components in TECH in contrast to the other sources, despite a similar amount of JPs. We expect this to be due the numerous KCs depicted in this domain (e.g., programming languages), while we observe considerably fewer soft skills (e.g., "work flexibly"). The amount of skills is more balanced across the three sources. Furthermore, overlapping spans follow a consistent trend among splits, with the train split containing the most.

Data Annotation
We annotate competences related to SKCs in two levels as illustrated in Figure 1. We started the process in March 2021, with initial annotation rounds to construct and refine the annotation guidelines (as outlined further below). 6 Links to our data can be found at https://github. com/kris927b/SkillSpan. The annotation process spanned eight months in total. Our final annotation guidelines can be found in Appendix B. The guidelines were developed by largely following example spans given in the ESCO taxonomy. However, at this stage, we focus on span identification, and we do not take the fine-grained taxonomy codes from ESCO for labeling the spans, leaving the mapping to ESCO and taxonomy enrichment as future work.
Further Details on the Annotation Process The development of the annotation guidelines and our annotation process is depicted as follows: 1 We wrote base guidelines derived from a small number of JPs. 2 We had three pre-rounds consisting of three JPs each. After each round, we modified, improved and finalized the guidelines. 3 Then, we had three longer-lasting annotation rounds consisting of 30 JPs each. We re-annotated the previous 11 JPs in 1 and 2 . 4 After these rounds, one of the annotators (the hired linguist) annotated JPs in batches of 50. The data in 1 , 2 , and 3 was annotated by three annotators (101 JPs).
We used an open source text annotation tool named DOCCANO (Nakayama et al., 2018). There are around 57.5K tokens (approximately 4.6K sentences, in 101 job posts) that we calculated agreement on. The annotations were compared using Cohen's  (Fleiss and Cohen, 1973) between pairs of annotators, and Fleiss'  (Fleiss, 1971), which generalises Cohen's  to more than two concurrent annotations. We consider two levels of  calculations: TOKEN is calculated on the token level, comparing the agreement of annotators on each token (including non-entities) in the annotated dataset. SPAN refers to the agreement between annotators on the exact span match over the surface string, regardless of the type of SKC, i.e., we only check the position of tag without regarding the type of the entity. The observed agreements scores over the three annotators from step 3 are between 0.70-0.75 Fleiss'  for both levels of calculation which is considered a substantial agreement (Landis and Koch, 1977) and a  value greater than 0.81 indicates almost perfect agreement. Given the difficulty of this task, we consider the aforementioned  score to be strong. Particularly, we observed a large improvement in annotation agreement from the earlier rounds (step 1 and 2 ), where our Fleiss'  was 0.59 on tokenlevel and 0.62 for the span-level.
Overall, we observe higher annotator agreement for knowledge components (3-5% higher) com-  Table 3: Most Frequent Skills in the Development Data. Top-5 skill components in our data in terms of frequency on different sources. A larger example can be found in Table 8 and Table 9 (Appendix C).
pared to skills which tend to be longer. The TECH domain is the most consistent for agreement while BIG shows more variation over rounds, likely due to the broader nature of the domains of JPs.
Annotation Span Statistics A challenge of annotating spans is the length (i.e., boundary), SKCs being in different domains (e.g., business versus technical components), and frequently written differently, e.g., "being able to work together" v.s. "teamwork"). Figure 2 shows the statistics of our annotations in violin plots. For the training set, the median length (white dot) of skills is around 4 for BIG and HOUSE, for TECH this is a median of 5.
In the development set, the median stays at length 4 across all sources. Another notable statistic is the upper and lower percentile of the length of skills and knowledge, indicated with the thick bars. Here, we highlight the fact that skill components could consist of many tokens, for example, up to length 7 in the HOUSE source split (see blue-colored violins). For knowledge components, the spans are usually shorter, where it is consistently below 5 tokens (see orange-colored violins). All statistics follow a similar distribution across train, development, and sources in terms of length and distribution. This gives a further strong indication that consistent annotation length has been conducted across splits and sources.
Qualitative Analysis of Annotations Qualitative differences in SKCs over the three sources are shown (lowercased) in Table 3. With respect to skill components, all sources follow a similar usage of skills. The annotated skills mostly relate to the attitude of a person and hence mostly consist of soft skills. With respect to knowledge components, we observe differences between the three sources. First, on the source-level, the knowledge components vastly differ between BIG and TECH. BIG postings seem to cover more business related components, whereas TECH has more engineering components. HOUSE seems to be a mix of the other two sources. Lastly, note that both the skill and knowledge components between the splits diverge in terms of the type of annotated spans, which indicates a variation in the annotated components. We show the top-10 skills annotated in the train, development, and test splits for SKCs in Appendix C. From a syntactic perspective, skills frequently consist of noun phrases, verb phrases, or adjectives (for soft skills). Knowledge components usually consists of nouns or proper nouns, such as "python", "java", and so forth.

Experimental Setup
The task of SE is formulated as a sequence labeling problem. Formally, we consider a set of JPs D, where d 2 D is a set of sequences (i.e., entire JPs) with the i th input sequence .., y T } (e.g., "B-SKILL", "I-KNOWLEDGE", "O"). The goal is to use D to train a sequence labeling algorithm h : X 7 ! Y to accurately predict entity spans by assigning an output label y t to each token x t .
As baseline we consider BERT and we investigate more recent variants, and we also train models from scratch. Models are chosen due to their stateof-the-art performance, or in particular, for their strong performance on longer spans.
SpanBERT (Joshi et al., 2020) A BERT-style model that focuses on span representations as opposed to single token representations. SpanBERT is trained by masking contiguous spans of tokens and optimizing two objectives: (1) masked language modeling, which predicts each masked token from its own vector representation. (2) The span boundary objective, which predicts each masked token from the representations of the unmasked tokens at the start and end of the masked span.
We train a SpanBERT base model from scratch on the BooksCorpus (Zhu et al., 2015) and English Wikipedia using cased Wordpiece tokens (Wu et al., 2016). We use AdamW (Kingma and Ba, 2015) for Figure 3: Performance of Models. We test the models on SKILLS, KNOWLEDGE, and COMBINED. We report the span-F1 and standard deviation (error bars) of runs on five random seeds. Note that the y-axis starts from 50 span-F1. STL indicates single-task learning and MTL indicates the multi-task model. Differences can be seen on the test set: JobSpanBERT performs best on SKILLS, JobBERT is best on KNOWLEDGE, and JobBERT achieves best in COMBINED. Exact numbers of the plots are in Table 5 (Appendix E).
2.4M training steps with batches of 256 sequences of length 512. The learning rate is warmed up for 10K steps to a maximum value of 1e-4, after which it has a decoupled weight decay (Loshchilov and Hutter, 2019) of 0.1. We add a dropout rate of 0.1 across all layers. Pretraining was done on a v3-8 TPU on the GCP and took 14 days to complete. We take the official TensorFlow implementation of SpanBERT by Ram et al. (2021).
JobBERT 7 We apply domain-adaptive pre-training (Gururangan et al., 2020) to a BERT base model using the 3.2M unlabeled JP sentences (Table 2). Domain-adaptive pre-training relates to the continued self-supervised pre-training of a large language model on domain-specific text. This approach improves the modeling of text for downstream tasks within the domain. We continue training the BERT model for three epochs (default in HuggingFace) with a batch size of 16.
Experiments We have 391 annotated JPs (Table 2) that we divide across three splits: Train, dev. and test set. We use 101 JPs that all three annotators annotated as the gold standard test set with aggregated annotations via majority voting. The 101 postings are divided between the sources as: 36 BIG, 33 HOUSE, and 32 TECH. The remaining 290 JPs were annotated by one annotator. We use 90 JPs (30 from each source, namely BIG, HOUSE, and TECH) as the dev. set. The remaining 200 JPs are used as the train set. The sources in the train set are divided into 60 BIG, 60 HOUSE, and 80 TECH.
Setup The data is structured as CoNLL format (Tjong Kim Sang, 2002). For the nested annotations, the skill tags are appearing only in the first column and the knowledge tags are only appearing in the second column of the file and they are allowed to overlap with each other. We perform experiments with single-task learning (STL) on either the skill or knowledge components, MTL for predicting both skill and knowledge tags at the same time, while evaluating the MTL models also on either skills or knowledge components. We used a single joint MTL model with hard-parameter sharing (Caruana, 1997). All models are with a final Conditional Random Field (CRF; Lafferty et al.,  2021) show that BERT models with a CRF-layer improve or perform similarly to its simpler variants when comparing the overall F1 and make no tagging errors (e.g., B-tag follows I-tag). In the case of MTL we use one for each tag type (skill and knowledge). In the STL experiments we use one CRF for the given tag type.
We use the MACHAMP toolkit (van der Goot et al., 2021) for our experiments. For each setup we do five runs (i.e., five random seeds). 9 For evaluation we use span-level precision, recall, and F1, where the F1 for the MTL setting is calculated as described in Benikova et al. (2014).

Results
The results of the experiments are given in Figure 3. We show the average performance of each model in F1 and respective standard deviation over the development and test split. Exact scores on each source split and other metric details are provided in Appendix E. As mentioned before, we experiment with the following settings: SKILL, we train and predict only on skills. KNOWLEDGE, train and only predict for knowledge. COMBINED, we merge the STL predictions of both skills and knowledge. We also train the models in an MTL setting, predicting both skills and knowledge simultaneously. We evaluate the MTL model on both SKILL and KNOWLEDGE separately, and also compare it against the aggregated STL predictions.
Performance on Development Set In Figure 3, we show the results on the development set in the upper plot. We observe similar performance between the domain-adapted STL models-JobBERT and JobSpanBERT-have similar span-F1 for 9 For reproducibility, we refer to Appendix D. SKILL: 60.05±0.70 vs. 60.07±0.70. In contrast, for KNOWLEDGE, BERT base and JobBERT are closest in predictive performance: 60.44±0.58 vs. 60.66±0.43. In the COMBINED setting, JobBERT performs highest with a span-F1 of 60.32±0.39. On average, JobBERT performs best over all three settings. Surprisingly, the models for both SKILL and KNOWLEDGE perform similarly (around 60 span-F1), despite the sources' differences in properties and length Figure 2. In addition, we find that MTL is not performing better than STL across sources. For exact numbers and source-level (i.e., BIG, HOUSE, TECH), we refer to Appendix E.

Performance on Test Set
We select the best performing models in the development set evaluation and apply it to the test set. Results are in Figure 3 in the bottom plot. Since JobBERT and JobSpan-BERT are performing similarly, we apply both to the test set and BERT base . We observe a deviation from the development set to the test set: JobSpan-BERT 60.07±0.30!56.64±0.83 on SKILL, Job-BERT 60.66±0.43!63.88±0.28 on KNOWLEDGE. For COMBINED, JobBERT performs slightly worse: 60.32±0.39!59.73±0.38. Similar to the development set, we find that on all three methods of evaluation (i.e., SKILL, KNOWLEDGE, and COM-BINED), STL still outperforms MTL. For SKILL and KNOWLEDGE, STL is almost stochastically dominant over MTL (i.e., significant), and for COMBINED there is stochastic dominance of STL over MTL, indicated in the next paragraph.
Significance We compare all pairs of models based on five random seeds each using Almost Stochastic Order (ASO; Dror et al., 2019) tests with a confidence level of ↵ = 0.05. The ASO scores of the test set are indicated in Figure 4. We show that MTL-JobSpanBERT for SKILL shows almost stochastic dominance (✏ min < 0.5) over all other models. For KNOWLEDGE and COMBINED, We show that STL-JobBERT is stochastically dominant (✏ min = 0.0) over all the other models. For more details, we refer to Appendix F for ASO scores on the development set.

Discussion
What Did Not Work Additionally, we experiment whether representing the entire JP for extracting tokens yields better results than the experiments so far, which were sentence-by-sentence processing setups. To handle entire JPs and hence much longer sequences we use a pre-trained Longformer base (Beltagy et al., 2020)  . This drop in performance is difficult to attribute to a concrete reason: e.g., the Longformer is trained on more varied sources than BERT, but not specifically for JPs, which may have contributed to this gap. Since the vanilla Longformer already performs worse than BERT base overall, we did not opt to apply domainadaptive pre-training. Overall, we show that representing the full JP is not beneficial for SE, at least not in the Longformer setup tested here.
Continuous Pretraining helps SE As previously mentioned, due to the domain specialization of the domain-adapted pre-trained BERT models, they predict more skills and frequently perform better in terms of precision, recall, and F1 as compared to their non-adaptive counterparts. This is especially encouraging as we confirm findings that continuous pre-training helps to adapt models to a specific domain (Alsentzer et al., 2019;Lee et al., 2020;Gururangan et al., 2020;Nguyen et al., 2020). However, there are exceptions. Particularly in Table 5 on TEST for KNOWLEDGE, BERT base comes closer in predictive performance to JobBERT (difference of 1.5 F1) than on SKILLS. Our intuition is that knowledge components are often already in the pre-training data (e.g., Wikipedia pages of certain competences like Python, Java etc.) and therefore adaptive pre-training does not substantially boost performance.

Difference in Length of Predictions
The main motivation of selecting models optimized for long spans was the length of the annotations (Figure 2). We investigate the average length of predictions of each model ( Figure 5) to find out whether the models that are adapted to handle longer sequences truly predict longer spans. Interestingly, the average length of predicted skills are longer than the annotations over all three sources. There is a consistent trend among SKILL: BIG and TECH have similar length over predictions (>4), while HOUSE is usually lower than length 3. For both BIG and TECH, JobSpanBERT predicts the longest skill spans (4.51 and 4.48 respectively). We suspect due to the domain-adaptive pre-training on JPs, it improved the span prediction performance. In contrast, the Longformer predicts shorter spans. Note that the Longformer is not domain-adapted to JPs.
Regarding KNOWLEDGE, there is also a consistent trend: BIG has the overall longest prediction length while TECH has the lowest. The Longformer predicts the longest spans on average for BIG and TECH. Knowledge components are representative of a normal-length NER task and might not need a specialized model for long sequences. We show the exact numbers in Table 7 (Appendix E) and the number of predicted SKILL and KNOWLEDGE: JobBERT and JobSpanBERT have higher recall than the other models.
Performance per Span Length SKILLS are generally longer than KNOWLEDGE components in our dataset (Figure 2). The previous overall results on the test set ( Figure 3) show that performance on Figure 6: Average Span-F1 per Span Length. We bucket the performance of JobBERT according to the length of the spans until 10 tokens and show the performance on each length, averaged over five random seeds. Indicated per bar is the support. The model performs best on medium-length skill spans (i.e., spans with token length of [4][5]. For knowledge spans, on average, it performs best on short-length spans (i.e., spans with token length of 1-2).
SKILL is substantially lower than KNOWLEDGE. We therefore investigate whether this performance difference is attributed to the longer spans in SKILL. In Figure 6, we show the average performance of the best performing model (JobBERT) on the three sources (test set) based on the gold span length, until a length of 10.
In SKILL components (upper plot), we see much support for spans with length 1 and 2, which then lowers once the spans become longer. Spans with length of 1 shows low performance on BIG and TECH (around 40 span-F1), which influences the total span-F1. Short skills are usually soft skills, such as "passionate", which can be used as a skill or not. This might confuse the model. In contrast, performance effectively stays similar (around 60 span-F1) for span length of 2 till 7 for all sources. Afterwards, it drops in performance. Thus, the weak performance on SKILL seem to be due to lower performance on the short spans.
For the KNOWLEDGE components (lower plot), they are generally shorter. We see that there is a gap in support between the sources, TECH has a larger number of gold labels compared to BIG and HOUSE. Unlike soft skills, KCs usually consist of proper nouns such as "Python", "Java", and so forth, which connects to the high performance on TECH (around 76 span-F1). Furthermore, support for spans longer than 2 drops considerably. In this case, if the model predicts a couple of instances correctly, it would substantially increase span-F1. Contrary to SKILL, high performance of KNOWL-EDGE can be attributed to its strong performance on short spans.

Conclusion
We present a novel dataset for skill extraction on English job postings-SKILLSPAN-and domainadapted BERT models-JobBERT and JobSpan-BERT. We outline the dataset and annotation guidelines, created for hard and soft skills annotation on the span-level. Our analysis shows that domainadaptive pre-training helps to improve performance on the task for both skills and knowledge components. Our domain-adapted JobSpanBERT performs best on skills and JobBERT on knowledge. Both models achieve almost stochastic dominance over all other models for skills and knowledge extraction, whereas JobBERT in the STL setting achieves stochastic dominance over other models.
With the rapid emergence of new competences, our new approach of skill extraction has future potential, e.g., to enrich knowledge bases such as ESCO with unseen skills or knowledge components, and in general, contribute to providing insights into labor market dynamics. We hope our dataset encourages research into this emerging area of computational job market analysis. 11. If there's a knowledge/skill component in the position, we exclude it as well.
12. Only annotate the skills that are related to the position.
12.1. This includes skills that are specific for the position as well (e.g., skills of a ruminants professor versus math professor).
12.2 Also skills that the person for the position is expected to do in the future.
12.3 This does not include skills, knowledge or attitudes describing only the company, the group you will join in the department, and so on. Only annotate if it is specified or implied that the employee should possess the skill as well.
13. We annotate industries and fields (that the employee will be working in) as knowledge components.

C Type of Skills Annotated
In both Table 8 and Table 9, we show the top-10 skill and knowledge components that have been annotated. We split the top-10 among the data splits (i.e., train, development, and test set), and also between source splits (i.e., BIG, HOUSE, TECH).  We use the default hyperparameters in MACHAMP (van der Goot et al., 2021) as shown in Table 4. For more details we refer to their paper. For the five random seeds we use 3477689, 4213916, 6828303, 8749520, and 9364029. All experiments with MACHAMP were ran on an NVIDIA ® TITAN X (Pascal) 12 GB GPU and an Intel ® Xeon ® Silver 4214 CPU.

E Exact Number of Performance
In Table 5, we show the exact numbers of the plot indicated in Figure 3. In addition, we also show the results of each respective split. For the STL models, we observe differences in performances over the sources which is particularly pronounced for knowledge components: The TECH source is the easiest to process (and has most SKCs), while SKCs identification performance is the lowest for BIG. This might be due to the broad nature of this source.
In the exact results table (Table 5) we add a ( †) next to the highest span-F1 if the model is truly stochastically dominant (✏ min = 0.0) over all the other models. (*) denotes that the best model achieved almost stochastic dominance (✏ min < 0.5) over-at minimum-one other model (e.g., in TEST rows w.r.t COMBINED: MTL-JobBERT ⌫ MTL-JobSpanBERT with ✏ min = 0.06) and stochastically dominant over the rest.
In Table 6, we report the precision and recall of the models, SKILL and KNOWLEDGE show the precision and recall of the STL models. MULTI shows the precision and recall of the MTL models.
Last, in Table 7, we show the exact numbers of the length of predictions Figure 5. We also add the number of predicted SKILL and KNOWLEDGE Overall, JobBERT and JobSpanBERT predict more skills in general than the other models. This is also the case for knowledge components. We hypothesize that this might be due to the BERT models now being more specialized towards the JP domain and recognizing more SKCs.

F Significance Testing
Recently, the ASO test (Dror et al., 2019) 10 has been proposed to test statistical significance for deep neural networks over multiple runs. Generally, the ASO test determines whether a stochastic order (Reimers and Gurevych, 2018) exists between two models or algorithms based on their respective sets of evaluation scores. Given the single model scores over multiple random seeds of two algorithms A and B, the method computes a test-specific value (✏ min ) that indicates how far algorithm A is from being significantly better than algorithm B. When distance ✏ min = 0.0, one can claim that A stochastically dominant over B with a predefined significance level. When ✏ min < 0.5 one can say A ⌫ B. On the contrary, when we have ✏ min = 1.0, this means B ⌫ A. For ✏ min = 0.5, no order can be determined. We took 0.05 for the predefined significance level. In Figure 7, we show the ASO scores on the development set.     Table 8: Most Frequent Skills in the Data. Top-10 skill components in our data in terms of frequency.