<jupyter_start><jupyter_text>Specific examples of transitions data  0. Import dependencies and inputs<jupyter_code>%run ../../notebook_preamble_Transitions.ipy
# Location to store transitions data
outputs_folder = f'{useful_paths.data_dir}processed/transitions/specific_examples/'

# File name to use for the specific examples
file_name = 'Data_example'

# First batch of transitions
path_to_val_data = data_folder + 'restricted/validation/nesta_output_16Dec.csv'
batch_results = pd.read_csv(path_to_val_data)<jupyter_output><empty_output><jupyter_text>1. Prepare transitions data  1.0 Specify transitions  Example 1: Generate transitions<jupyter_code># Example 1: Generate a collection of transitions (e.g. all transitions from hotel porter and hotel concierge)
transitions_df = trans_utils.get_transitions(origin_ids=[732, 329], destination_ids='report')
transitions_df = transitions_df[transitions_df.is_safe_desirable].reset_index(drop=True)
transitions_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 21 entries, 0 to 20
Data columns (total 25 columns):
 #   Column                      Non-Null Count  Dtype  
---  ------                      --------------  -----  
 0   origin_id                   21 non-null     int64  
 1   origin_label                21 non-null     object 
 2   destination_id              21 non-null     int64  
 3   destination_label           21 non-null     object 
 4   similarity                  21 non-null     float64
 5   is_jobzone_ok               21 non-null     bool   
 6   is_earnings_ok              21 non-null     bool   
 7   is_not_high_risk            21 non-null     bool   
 8   is_safer                    21 non-null     bool   
 9   is_strictly_safe            21 non-null     bool   
 10  job_zone_dif                21 non-null     float64
 11  earnings_ratio              21 non-null     float64
 12  risk_dif                    21 non-null     float64
 13  prop_dif                    21 non-nu[...]<jupyter_text>Example 2: Use the validation dataset<jupyter_code># Example 2: Add transition data using a pre-specified list of occupation pairs
validation_data = pd.read_csv(useful_paths.data_dir + 'processed/validation/Transitions_to_validate_BATCH_1.csv')
transition_pairs = [(row.origin_id, row.destination_id) for j, row in validation_data.iterrows()]
validation_data.head(5)
transition_pairs[0:5]
transitions_df = trans_utils.get_transition_data(transition_pairs)
transitions_df.head(5)<jupyter_output><empty_output><jupyter_text>1.1 Table of transitions<jupyter_code># Select only the desired columns
transitions_table = transitions_df[[
    'origin_id', 'origin_label', 'destination_id', 'destination_label',
    'similarity', 'W_essential_skills', 'W_optional_skills', 'W_activities', 'W_work_context',
    'sim_category', 'is_viable', 'is_desirable', 'is_safe_desirable']]

transitions_table.sample(5)<jupyter_output><empty_output><jupyter_text>Notes: The column `similarity` stores the combined similarity measure, i.e. the average of the four different similarity measures described in the Mapping Career Causeways report. These are also provided in the columns `W_{x}`).Column `sim_category` indicates whether the combined similarity is above 0.4 ('highly viable') or only between 0.3 and 0.4 ('minimally viable').The columns `is_viable`, `is_desirable`, `is_safe_desirable` should be clear (see the report for more details).  Export transitions<jupyter_code>transitions_table.to_csv(f'{outputs_folder}{file_name}_Transitions.csv', index=False)<jupyter_output><empty_output><jupyter_text>1.2 Table of occupational profiles<jupyter_code># Select all IDs involved in the transitions
all_ids = set(transitions_table.origin_id.to_list()).union(set(transitions_table.destination_id.to_list()))
# Occupational profiles
occ_profiles = data.occ_report[data.occ_report.id.isin(all_ids)]
occ_profiles.sample(5)<jupyter_output><empty_output><jupyter_text>Please select the columns, which you find are the most relevant. For more information about the column data, please see [here](https://github.com/nestauk/mapping-career-causeways/tree/main/supplementary_online_data/transitionsnumber-of-transition-options-for-esco-occupations).<jupyter_code>occ_profiles.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 954 entries, 1 to 1625
Data columns (total 31 columns):
 #   Column                    Non-Null Count  Dtype  
---  ------                    --------------  -----  
 0   id                        954 non-null    int64  
 1   concept_type              954 non-null    object 
 2   concept_uri               954 non-null    object 
 3   preferred_label           954 non-null    object 
 4   isco_level_1              954 non-null    int64  
 5   isco_level_2              954 non-null    int64  
 6   isco_level_3              954 non-null    int64  
 7   isco_level_4              954 non-null    int64  
 8   is_top_level              954 non-null    bool   
 9   job_zone                  954 non-null    float64
 10  education_level           954 non-null    float64
 11  related_work_experience   954 non-null    float64
 12  on_the_job_training       954 non-null    float64
 13  annual_earnings           954 non-null    float64
 14  total_pai[...]<jupyter_text>Export profiles<jupyter_code>occ_profiles.to_csv(f'{outputs_folder}{file_name}_Occupation_profiles.csv', index=False)<jupyter_output><empty_output><jupyter_text>1.3 Skills matching data for specific transitions<jupyter_code># Choose a row of the transition table
row = 0

# Get skills matches
skills_matching = trans_utils.show_skills_overlap(transitions_table.loc[row].origin_id,
                                                  transitions_table.loc[row].destination_id,
                                                  skills_match='optional',
                                                  verbose=True)

# Export the skills matching table
skills_matching.to_csv(f'{outputs_folder}{file_name}_Skills_match_{row}.csv', index=False)

skills_matching

# Extra: You may also want to get some reference data on the skills (links to the ESCO database)
all_skills_ids = set(skills_matching.origin_skill_id.to_list()).union(set(skills_matching.destination_skill_id.to_list()))
data.skills[data.skills.id.isin(all_skills_ids)]<jupyter_output><empty_output><jupyter_text>2. Data Sample export<jupyter_code>transition_mean = batch_results.groupby('subject_ids')['feasibility_1-5'].mean()
transitions = batch_results.reset_index().drop_duplicates('subject_ids').set_index('subject_ids')

export_jobs = ['user interface developer', 'lottery cashier', 
               'community health worker', 'sales support assistant', 
               'travel agent', 'postman/postwoman', 'real estate surveyor',
               'investment clerk', 'financial analyst', 'packaging production manager'
              ]

sample_transitions = (transitions
                      .reset_index()
                      .set_index('origin_label')
                      .loc[export_jobs]
                      .reset_index()
                      .set_index('subject_ids'))
sample_transitions = sample_transitions.merge(transition_mean, left_index=True, 
                                              right_index=True, how='inner', 
                                              suffixes=('_raw', '_mean')
                                             )

sample_transitions = sample_transitions[sample_transitions['feasibility_1-5_mean'] > 2]


with open(data_folder + 'restricted/validation/data_sample_ids.txt', 'w') as f:
    for l in [str(tuple(r)) for r in sample_transitions[['origin_id', 'destination_id']].to_numpy()]:
        f.write(l + '\n')
from ast import literal_eval
with open(data_folder + 'restricted/validation/data_sample_ids.txt', 'r') as f:
    sample_ids = f.read().splitlines()
    
sample_ids = [literal_eval(l) for l in sample_ids]<jupyter_output><empty_output><jupyter_text>2.1 Sample Transitions<jupyter_code>sample_transitions_df = trans_utils.get_transition_data(sample_ids)
sample_transitions_df = sample_transitions_df.reset_index().rename(columns={'index': 'transition_id'})
sample_transitions_df<jupyter_output><empty_output><jupyter_text>2.2 Sample Job Profiles<jupyter_code># Select all IDs involved in the transitions
all_ids = set(sample_transitions_df.origin_id.to_list()).union(set(sample_transitions_df.destination_id.to_list()))
# Occupational profiles
sample_occ_profiles = data.occ_report[data.occ_report.id.isin(all_ids)]
sample_occ_profiles.sample(5)<jupyter_output><empty_output><jupyter_text>2.3 Sample Skills<jupyter_code>rows = sample_transitions_df[['transition_id', 'origin_id', 'destination_id']].iterrows()

skills_matches = []

for _, (i, origin_id, destination_id) in rows:
    skills_matching = trans_utils.show_skills_overlap(origin_id, destination_id, 
                                                      skills_match='optional', verbose=False)
    skills_matching['transition_id'] = i
    skills_matches.append(skills_matching)
    
skills_matches = pd.concat(skills_matches, axis=0)
skills_matches.head()
# Extra: You may also want to get some reference data on the skills (links to the ESCO database)
sample_skills_ids = (set(skills_matches.origin_skill_id.to_list())
                     .union(set(skills_matches.destination_skill_id.to_list())))

sample_skill_profiles = data.skills[data.skills.id.isin(sample_skills_ids)]
sample_skill_profiles.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09278350515463918, 'fraction_numerical': 0.022406900904691773, 'mean_word_length': 3.2785778577857787, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 54, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '19551', 'license': 'MIT', 'n_tokens_mistral': 3041, 'n_tokens_neox': 2833, 'n_words': 614, 'path': 'codebase/notebooks/05_transition_analyses/specific_examples/Generate_transitions_examples.ipynb', 'repo_name': 'maxdata/mapping-career-causeways', 'size': '9506'}