<jupyter_start><jupyter_text>How many datasets have CC0 (or CCBY for forked repositories) and all 8 "Terms of Use" fields filled? How many have 7? How many have 6? Etc...<jupyter_code>import csv
import numpy as np
import pandas as pd

data = pd.read_csv('terms_metadata.tab', sep='\t', na_filter = False)
# Get only metadata for the latest versions of each dataset
latestversion = data.iloc[data.groupby('persistentUrl')['datasetVersionId'].agg(pd.Series.idxmax)].sort_values(by=['publisher'], inplace=False, ascending=True).reset_index(drop=True, inplace=False)

# Replace any blank values with NaN
latestversion = latestversion.replace(r'^\s*$', np.nan, regex=True)

# Replace publsiher "Root" with Dartmouth
latestversion['publisher'] = latestversion['publisher'].replace(['Root'],'Dartmouth')

latestversion_termsofuse = latestversion.drop(columns=[
    'publisher', 'datasetVersionId', 'majorVersionNumber', 'minorVersionNumber', 'termsOfUse',
    'termsOfAccess', 'availabilityStatus', 'contactForAccess', 'dataaccessPlace',
    'originalArchive', 'sizeOfCollection', 'studyCompletion'])

latestversion_termsofuse.license.unique()
latestversionCC0 = (latestversion_termsofuse
    .query('license == "CC0" or license == "CCBY" or license == "CC BY"')
#     .reset_index(drop = False, inplace = True)
    .set_index('persistentUrl')
)
latestversionCC0.head(5)
len(latestversionCC0)
countsByDataset = (latestversionCC0
    .drop(columns=['license'])
    .count(axis=1)
    .to_frame('count')
    .query('count > 0')
)
countsByDataset.head(5)
len(countsByDataset)<jupyter_output><empty_output><jupyter_text>Looks like there are 498 datasets with both:1. CC0 waiver or CCBY license and2. One or more Terms of Use fields filled (exluding the Terms of Use field)Let's join the countsByDataset dataframe to the latestversionCC0 dataframe:<jupyter_code>countsByDataset_merged = (pd
    .merge(countsByDataset, latestversion, how='left', on='persistentUrl')
    .drop(columns=[
        'termsOfUse', 'termsOfAccess', 'availabilityStatus',
        'contactForAccess', 'dataaccessPlace', 'originalArchive', 'sizeOfCollection', 'studyCompletion'])
)

countsByDataset_merged.head(5)
len(countsByDataset_merged)<jupyter_output><empty_output><jupyter_text>For now, I'm going to export the results to a CSV file since it's faster for me to group these counts using pivot table wizards (e.g. in Excel and Google Sheets).<jupyter_code># Export to CSV
file = 'countsByDataset_merged.csv'
countsByDataset_merged.to_csv(file, index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12225705329153605, 'fraction_numerical': 0.008228840125391849, 'mean_word_length': 6.62089552238806, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 14, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '238884', 'license': 'CC0-1.0', 'n_tokens_mistral': 848, 'n_tokens_neox': 808, 'n_words': 203, 'path': 'notebooks/Terms_metadata_in_Dataverse_repositories/Count of Terms of Use metadata.ipynb', 'repo_name': 'jggautier/ux-research-data-analysis', 'size': '2552'}