<jupyter_start><jupyter_text>This is a service notebook to create csvs to use as dfs in other notebooks.  NB:At the very end of this notebook, we generate a votelist with coded respondent metadata.This votelist needs to be re-generated if the coding changes.<jupyter_code>import pandas as pd
exportdate = 20180327
projectname = 'repract'
number_of_papers = 435<jupyter_output><empty_output><jupyter_text>DataFrame with Evaluation Counts and Paper Metadata<jupyter_code>evaluation_counts = pd.read_csv(f'../analysis/{exportdate}{projectname}_evaluation_counts.csv')
xls = pd.read_csv('../data/all_conferences_from_googledrive.csv', sep=';',
                 usecols=['Unipark ID', 'Title', 'Authors', 'Venue', 'Year', 'Nb. pages',
       'Academic vs. industry', 'Industry track', 'Summary'],
                 nrows=435)
xls = xls[['Unipark ID', 'Venue', 'Year', 'Nb. pages', 'Industry track', 'Academic vs. industry',
       'Title', 'Authors', 'Summary']]
papermeta = evaluation_counts.merge(xls, left_on='PaperID', right_on='Unipark ID').drop([
    'Unipark ID'], axis=1)
# To avoid malformed csvs (due to ONE carriage return...)
papermeta['Title'] = [x.replace('\r', '') for x in list(papermeta['Title'].values)]
# Renaming for more comfortable coding
papermeta = papermeta.rename({'Nb. pages': 'TotalPages', 
               'Industry track': 'IndTrack', 
               'Academic vs. industry': 'AcadVsInd'}, axis=1)
#papermeta.to_csv(
#    f'../analysis/{exportdate}{projectname}_evaluation_counts_with_papermeta.csv', index=False)<jupyter_output><empty_output><jupyter_text>Long-Form df with List of Votes<jupyter_code>df = pd.read_csv(f'../data/{exportdate}{projectname}.csv')
votelist = pd.DataFrame(columns=['EvID', 'PaperID', 'Vote'])
ratings = ['Essential', 'Worthwhile', 'Unimportant', 'Unwise']
for row in df[['v_'+str(x) for x in range(7039,7039+number_of_papers)]].iterrows():
    for idx, elem in enumerate(row[1]):
        if elem in ratings:
            minidf = pd.DataFrame({'EvID':[row[0]], 'PaperID':[idx+1], 'Vote':[elem]})
            votelist = votelist.append(minidf)
votelist = votelist.reset_index().drop('index', axis=1)
#votelist.to_csv(f'../analysis/{exportdate}repract_votelist.csv', index=False)<jupyter_output><empty_output><jupyter_text>DataFrame with One Row per Evaluation and Participant Metadata * v_5: primary working area* v_6: free text in case of other in v5* v_11: free text years of primary working area experience* v_118-v_121: intensity of involvement in RE (v118 most intense)* v_12: CS degree* v_14: team size* v_15: class of systems in project scope* v_16: free text in case of other in v15* v_19: free text industry sector* v_124: country<jupyter_code>demographics = ['v_5', 'v_6', 'v_11', 'v_12', 'v_14', 'v_15', 'v_16', 'v_19', 'v_124']
reqinvolvement = ['v_118', 'v_119', 'v_120', 'v_121']
votelist_with_respondentmeta = votelist.merge(
    df[demographics+reqinvolvement].reset_index().rename({'index':'EvID'}, axis=1))
votelist_with_respondentmeta.head(5)
#votelist_with_respondentmeta.to_csv(
#    f'../analysis/{exportdate}{projectname}_votelist_with_respondentmeta.csv', 
#                                    index=False)<jupyter_output><empty_output><jupyter_text>Here a version with coded short free-text answers:<jupyter_code>votelist = pd.read_csv(f'../analysis/{exportdate}repract_votelist.csv')
df_coded = pd.read_csv(f'../analysis/{exportdate}{projectname}_with_shorttext_integration.csv')
df_coded.shape
# these columns are not present in the original data
df_coded.columns.values.flatten()[-6:]
demographics_coded = ['v_5', 'v_6', 'v_6_coded', 'v_5_6_merged', 
                      'v_11', 'v_11_coded', 
                      'v_12', 'v_14', 
                      'v_15', 'v_16', 'v_16_coded', 'v_15_16_merged',
                      'v_19', 'v_19_coded', 'v_124']
reqinvolvement = ['v_118', 'v_119', 'v_120', 'v_121']
votelist_with_respondentmeta_coded = votelist.merge(
    df_coded[demographics_coded+reqinvolvement].reset_index().rename({'index':'EvID'}, axis=1))
votelist_with_respondentmeta_coded.head(5)
#votelist_with_respondentmeta_coded.to_csv(
#    f'../analysis/{exportdate}{projectname}_votelist_with_respondentmeta_coded.csv', 
#                                    index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14631873252562907, 'fraction_numerical': 0.0309878844361603, 'mean_word_length': 4.77792732166891, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '560325', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1634, 'n_tokens_neox': 1472, 'n_words': 314, 'path': 'notebooks/02_csv_preparation.ipynb', 'repo_name': 'mendezfe/repract2018', 'size': '4292'}