<jupyter_start><jupyter_text>Hash StatThis is the notebook for some statistics of the number of returned references of hashes.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
df_stat = pd.read_pickle('./pickle/hash_stat.pickle')
df_stat.head()
# In thousands
df_stat['k_group'] = df_stat['counts'].apply(lambda x: (x//1000))
df_stat.groupby('k_group')['counts'].count()
# In hundreds, for those below 1,000
df_stat['h_group'] = df_stat['counts'].apply(lambda x: (x//100))
df_stat[df_stat.counts<1000].groupby('h_group')['counts'].count()
# In tens, for those below 100
df_stat['t_group'] = df_stat['counts'].apply(lambda x: (x//10))
df_stat[df_stat.counts<100].groupby('t_group')['counts'].count()
# In ones, for those below 10
df_stat['d_group'] = df_stat['counts'].apply(lambda x: (x//1))
df_stat[df_stat.counts<10].groupby('d_group')['counts'].count()
# 1000+
df_stat[df_stat.k_group>0].index.values<jupyter_output><empty_output><jupyter_text>**Brief Analysis:**1. `venezia` or `venetia` is quite common.2. `cultura` means 'culture' and `storia` means 'history'.3. `sanmarco` is one of the six sestieri of Venice.4. 'atti del convegno' means 'conference proceedings'. `atticonvegno` is parsed as part of title.5. 'a cura di' means 'edited by'. `cura` is included in the hashtag as part of author due to parsing error.6. `etamoderna` means 'modern age'; `moyenage`, `middleage` means 'middle age'. 7. 'catalogo della mostra' means 'exhibition catalog'. `catalogomostra` is parsed as part of title.8. `xvisecolo`, `secolixvi`, `sixteenthcentury`,  `xviiisecolo`  means '16th century' or '18th century'.9. `dizionariobiografico`, `biograficoitaliani` are part of 'Dizionario biografico degli Italiani', which is a biographical dictionary.10. `palazzoducale` is a palace built in Venetian Gothic style.11. `convegnointernazionale` means 'international conference'.12. `santamaria` always showes up as part of church name.13. `cittanobilissima`, `sansovinocittanobilissima` points to the reference 'Venetia città nobilissima et singolare' that is commonly cited.14. `originicaduta` points to the reference 'Storia di Venezia nella vita privata dalle origini alla caduta della Repubblica' that is commonly cited.15. `paolosarpi` is an Italian historian.16. `cozzirepubblicavenezia` is a combination of author 'cozzi' and title bigram 'repubblicavenezia' which is the favorite topic of this author.**Black List for this part:**:- Meaningless: `cura...`- **??Too common??**: `venezia`, `veneta`, `palazzoducale`, `paolosarpi`, etc.- Not necessay : `secolo`, `secoli`, `century` ... date information which is not main part of title- Publication name or similar: `convegnointernazionale`, `dizionariobiografico`, `biograficoitaliani`<jupyter_code># 500-900+
tags = df_stat[(df_stat.counts<1000) & (df_stat.h_group>=5)].index.values
# remove those with 'cura' and with 'century'
np.array([t for t in tags if  not t.startswith('cura#') 
                 and not (t.startswith('secoli#') or t.endswith('#secolo') or t.endswith('#century'))])<jupyter_output><empty_output><jupyter_text>Try to Build Black List - whether include in those too common ones?<jupyter_code># with cura
hash_cura = [h for h in df_stat.index.values if h.startswith('cura#')]
# only bigrams with century information
hash_seco = [h for h in df_stat.index.values 
             if (h.startswith('secoli#') or h.endswith('#secolo') or h.endswith('#century'))
                and (len(h.split('#'))<=2)
            ]
# part of a publication where the reference from

# TODO #
hash_public = ['studi#onore', 'archivio#veneto', 'lettere#arti', 'scienze#lettere',
               'dizionario#biografico', 'biografico#italiani', 'convegno#internazionale',
               'storia#patria', 
              ]
# hashes produced only by one ref
# avoid for useless search
hash_one = df_stat[df_stat['counts']==1].index.values<jupyter_output><empty_output><jupyter_text>Match Result without Hash Blacklist IF do the match **without building hash blacklist**, and the result of $\frac{3}{4}$ of all full refs:<jupyter_code>def number_level(n):
    if n//1000>0:
        return '1000+'
    if n//500>0:
        return '500+'
    if n//200>0:
        return '200+'
    if n//100>0:
        return '100+'
    if n//50>0:
        return '50+'
    if n//20>0:
        return '20+'
    if n//10>0:
        return '10+'
    if n==1:
        return '1'
    return '1+'
level = ['1000+', '500+', '200+', '100+', '50+', '20+', '10+', '1+', '1']
df_1 = pd.read_pickle("./pickle/hash_ref1.pickle")

df_1['len'] = df_1['match'].apply(len)

df_1['len'].describe()
df_1['len'].apply(number_level).value_counts()[level]
df_2 = pd.read_pickle("./pickle/hash_ref2.pickle")

df_2['len'] = df_2['match'].apply(len)

df_2['len'].describe()
df_2['len'].apply(number_level).value_counts()[level]
df_3 = pd.read_pickle("./pickle/hash_ref3.pickle")

df_3['len'] = df_3['match'].apply(len)

df_3['len'].describe()
df_3['len'].apply(number_level).value_counts()[level]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15325443786982249, 'fraction_numerical': 0.03195266272189349, 'mean_word_length': 4.848904267589389, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '535400', 'license': 'MIT', 'n_tokens_mistral': 1892, 'n_tokens_neox': 1795, 'n_words': 473, 'path': 'code/Hash Stat.ipynb', 'repo_name': 'TaoSunVoyage/Semester-Project-References-Disambiguation', 'size': '5070'}