<jupyter_start><jupyter_text>Downloading and preprocessing the SimLex datasetThis notebook downloads and preprocesses the SimLex dataset.<jupyter_code>%env URL=https://www.cl.cam.ac.uk/~fh295/SimLex-999.zip
! wget $URL
!unzip SimLex-999.zip
!ls
!ls SimLex-999
import pandas as pd
df = pd.read_csv('SimLex-999/SimLex-999.txt', sep='\t')
df.head()<jupyter_output><empty_output><jupyter_text>Important notes from the README:- POS is for both words, meaning that SimLex only has word pairs with same POS.- Similarity ratings were collected on a 0-6 scale, but have been linearly mapped to 0-10.- Concreteness ratings are from the Nelson norms, and are on a 1-7 scale.- Concreteness quartile also from Nelson norms (unsure how it is calculated from two numbers)- Assoc(USF) = Nelson norms- Binary variable indicating whether the word pair is in the top 333 (third) of association ratings, as per the Nelson norm column<jupyter_code>df.columns = ['word1', 'word2', 'POS', 'similarity', 'word1_concreteness', 'word2_concreteness',
              'concreteness_quartile', 'nelson_norms', 'top_333_in_nelson', 'similarity_sd']
df.head()
outfile = '../simlex.csv'
df.to_csv(outfile, index=False)<jupyter_output><empty_output><jupyter_text>Remove everything except this file to save space.<jupyter_code>!find . -not -name 'preprocessing_simlex.ipynb' -print0 | xargs -0 rm --
! rm -rf SimLex-999<jupyter_output>rm: "." and ".." may not be removed
rm: ./.ipynb_checkpoints: is a directory
rm: ./SimLex-999: is a directory
{'fraction_non_alphanumeric': 0.11324503311258279, 'fraction_numerical': 0.02847682119205298, 'mean_word_length': 5.068273092369478, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '160331', 'license': 'MIT', 'n_tokens_mistral': 551, 'n_tokens_neox': 498, 'n_words': 162, 'path': 'evaluate/data/simlex/preprocessing/preprocessing_simlex.ipynb', 'repo_name': 'geoffbacon/evaluate', 'size': '1510'}