<jupyter_start><jupyter_text>Fix some annotation mistakes (ad-hoc) In the beginning, some annotators didn't understand the instructions and input some bogus names. We'll need to filter them out.<jupyter_code>problematic_names = ['An asian country', 'A political figure', 'car maker brand', 'A company name', 
                     'No idea, the sentences work withouth this word', 'approval',
                     'social aid organisation (red crescent, red cross)', 'political party',
                     'Name of a journalist', 'devil', 'State of the USA']
anns[anns['Guessed name'].isin(problematic_names)][['Document', 'Guessed name', 'Guessed characteristics']]
anns[anns['Guessed name'].isin(problematic_names)] = float('nan')<jupyter_output><empty_output><jupyter_text>Mid-way through, I found out that the name of the speaker (typically broadcasters) are shown in the document and annotators just copied them into the answer sheet. This is not intended because I want to check if annotators know about the entities already before the task, not what they learn. Also, they typically don't know anything about the broadcasters so the names would be useless anyway. I'll filter them out.<jupyter_code>speaker_names = ['<NAME>', '<NAME>', '<NAME>', '<NAME>']
anns[anns['Guessed name'].isin(speaker_names)][['Document', 'Guessed name', 'Guessed characteristics']]
anns[anns['Guessed name'].isin(speaker_names)] = float('nan')<jupyter_output><empty_output><jupyter_text>Stats<jupyter_code>len(anns)
anns['Guessed name'].isna().sum() / len(anns)
(anns['Guessed name'].isna() & anns['Guessed characteristics'].isna()).sum() / len(anns)
anns['Guessed name'].value_counts()
anns.groupby('Username').agg({'Timestamp': len, 'Guessed name': 'count', 'Guessed characteristics': 'count'})
anns[anns.Username == 'student3']['Guessed characteristics']<jupyter_output><empty_output><jupyter_text>Generate spreadsheet for annotation<jupyter_code>def extract_conll_path(path):
    with open(path) as f:
        html = f.read()
    conll_path, = re.findall(r'name="conll_file" value="(.+)"', html)
    return conll_path
def file_contains(path, s):
    with open(path) as f:
        return s in f.read()
    
def file_equal(path1, path2):
    name_pattern = r'\b_[A-Z][A-Z0-9]+_\b'
    with open(path1) as f1, open(path2) as f2:
        tokens1 = re.findall(name_pattern, f1.read())
        tokens2 = re.findall(name_pattern, f2.read())
        return tokens1 == tokens2
    
def add_urls(row):
    doc = row['Document']
    token = row['Token']
    if not pd.isna(doc):
        paths = glob('../data/annotations/documents/*/' + doc)
        paths = [p for p in paths if file_contains(p, token)]
        if paths:
            if any(not file_equal(paths[0], p) for p in paths[1:]):
                paths_related_to_name = [p for p in paths if 'name' in p]
                if len(paths_related_to_name) == 1:
                    paths = paths_related_to_name
                else:
                    print('Warning: found two or more non-equivalent files: %s' %paths)
            for i, path in enumerate(paths):
                path = os.path.abspath(path)
                url = 'file://%s' %(urllib.parse.quote(path))
                row['url%d' %i] = url
            conll_paths = [extract_conll_path(p) for p in paths]
            orig_conll_paths = [re.sub(r'/no-(?:external|internal)/', '/orig/', p) for p in conll_paths]
            orig_conll_paths = [p.replace('auto_conll', 'gold_conll') for p in orig_conll_paths]
            orig_conll_paths = [p.replace('transformed.v2', 'transformed') for p in orig_conll_paths]
            orig_html_paths = [p + '.html' for p in orig_conll_paths]
            orig_html_paths = list(set(orig_html_paths))
            for i, path in enumerate(orig_html_paths):
                path = os.path.abspath(os.path.join('..', path))
                url = 'file://%s' %(urllib.parse.quote(path))
                row['orig%d' %i] = url
        else:
            print('Warning: no document found for %s - %s' %(doc, token))
    return row
anns = anns.apply(add_urls, axis=1)
cols = ([c for c in anns.columns if 'url' in c or 'orig' in c] + 
        ['Token', 'Guessed name', 'Guessed characteristics'])
anns[cols].to_csv('../output/entity-guessing-score-sheet.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13912037037037037, 'fraction_numerical': 0.004629629629629629, 'mean_word_length': 3.321, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '342793', 'license': 'MIT', 'n_tokens_mistral': 1414, 'n_tokens_neox': 1365, 'n_words': 379, 'path': 'notebooks/process-name-quiz.ipynb', 'repo_name': 'natemalek/molen-pater-nathan-rma-thesis-coreference-with-singletons', 'size': '4346'}