<jupyter_start><jupyter_text>Preprocessing Rubenstein and Goodenough datasetThis notebook preprocesses the Rubenstein and Goodenough dataset. The dataset comes from their 1965 paper _Contextual correlates of synonymy_. I'm using a machine-readable version from <NAME>'s [sim-eval](https://github.com/alexanderpanchenko/sim-eval/) tool. The file looks faithfully transcribed from the 1965 paper.<jupyter_code>import pandas as pd
raw_data = 'raw_rg-65.csv'
df = pd.read_csv(raw_data, sep=';', header=None)
df.head()
df.columns = ['word1', 'word2', 'similarity_out_of_4']<jupyter_output><empty_output><jupyter_text>Participants were asked to rate word pairs on a scale from 0 to 4, with 4 being most similar. To standardise across datasets, I convert this to a scale from 0 to 10, with 10 being most similar.<jupyter_code>df['similarity'] = 10 * df['similarity_out_of_4']/4
outfile = '../rg-65.csv'
df.to_csv(outfile, index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1044932079414838, 'fraction_numerical': 0.02821316614420063, 'mean_word_length': 6.312977099236641, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '476575', 'license': 'MIT', 'n_tokens_mistral': 338, 'n_tokens_neox': 300, 'n_words': 98, 'path': 'evaluate/data/rg-65/preprocessing/preprocessing_rg-65.ipynb', 'repo_name': 'geoffbacon/evaluate', 'size': '970'}