<jupyter_start><jupyter_text>Journal NamesQuestions in the context of fuzzy matching.* How many journal names appear more than once?* What is the average length of the duplicated names vs the unique names?Input file is a single larger JSON, mapping names to issns.```json{  "Acta Orientalia.": [    "0001-6438"  ],  "Acta Orientalia (København)": [    "0001-6438"  ],..```<jupyter_code>import json
import pandas as pd
with open("../data/name_to_issn.json") as f:
    mapping = json.load(f)<jupyter_output><empty_output><jupyter_text>We have about 3M keys.<jupyter_code>len(mapping)
df = pd.DataFrame(((k, len(v)) for k, v in mapping.items()), columns=["name", "issn_count"])
len(df)
df.head()
unique_name = df[df.issn_count == 1]
repeated_names = df[df.issn_count > 1]
len(repeated_names)
len(repeated_names) / len(df)<jupyter_output><empty_output><jupyter_text>About 19% (or 586466) names are repeated.<jupyter_code>repeated_names.describe()<jupyter_output><empty_output><jupyter_text>Which name is shared most?<jupyter_code>repeated_names.iloc[repeated_names.issn_count.argmax()] # Annual report.<jupyter_output><empty_output><jupyter_text>It is the "Annual report."<jupyter_code>mapping["Annual report."][:10]<jupyter_output><empty_output><jupyter_text>On average a repeated name will point to 3 ISSN. Over 20k names point to more than 3 ISSN.<jupyter_code>len(repeated_names[repeated_names.issn_count > 3])
repeated_names[repeated_names.issn_count > 3].sample(n=10)
mapping["Philosophica."]
repeated_names[repeated_names.issn_count > 3].issn_count.hist(bins=20)
repeated_names[(repeated_names.issn_count > 3) & (repeated_names.issn_count < 50)].issn_count.hist(bins=10)
repeated_names[(repeated_names.issn_count > 3) & (repeated_names.issn_count < 20)].issn_count.hist(bins=10)
repeated_names[(repeated_names.issn_count > 3) & (repeated_names.issn_count < 8)].issn_count.hist()
repeated_names[repeated_names.issn_count > 1000].issn_count.hist(bins=10)
repeated_names[repeated_names.issn_count > 1000]
repeated_names[repeated_names.issn_count > 500]
repeated_names[repeated_names.issn_count > 200]
repeated_names[repeated_names.issn_count > 100]
repeated_names<jupyter_output><empty_output><jupyter_text>If a name matches a repeated name exactly or fuzzy matches to a repeated name and there is not other information available, the match status must be ambigious.<jupyter_code>mapping["Nigerian Journal of Wildlife Management"]<jupyter_output><empty_output><jupyter_text>These are two ISSN refering to the same journal.<jupyter_code>import requests
requests.get("https://portal.issn.org/resource/ISSN/2735-928X?format=json").json()<jupyter_output><empty_output><jupyter_text>Interestingly, most (80%) journal do not seem to have the distinction between electronic and print. But it may be that names are not used consistently.<jupyter_code>len(unique_name) / len(df)
unique_name.sample(n=10)<jupyter_output><empty_output><jupyter_text>Example "Fieldiana"* "Fieldiana. Anthropology"* https://www.jstor.org/journal/fieldianaanthro<jupyter_code>mapping["Fieldiana. Anthropology"]<jupyter_output><empty_output><jupyter_text>However, jstor reports two ISSN: INSSP: 0071-4739, EISSN: 2162-4321 - but ISSN.org does not know about it?<jupyter_code>unique_issn = set([item for v in mapping.values() for item in v])
"2162-4321" in unique_issn
for k, v in mapping.items():
    if "2162-4321" in v:
        print(k, v)<jupyter_output>Fieldiana. ['2162-4267', '2162-4321', '2162-4291', '0096-2651', '0015-0746', '0097-3572', '2163-7105', '0071-4739', '0015-0754', '2158-5520', '0096-0438', '2162-4348']
Fieldiana. Anthropology ['2162-4321', '0071-4739']
Fieldiana. Anthropology (Online) ['2162-4321', '0071-4739']<jupyter_text>As an example: when given a string like "Fieldiana" we would return ambiguous. But "Fieldiana. (Online)" might be matched to '2162-4321'.   Other Examples<jupyter_code>repeated_names.sample(n=10)
mapping["Russian journal of coordination chemistry."]<jupyter_output><empty_output><jupyter_text>A new question: How many journals are listed under different names, yet still refer to the same journal?<jupyter_code>mapping["San Bernardino County popular street atlas"]<jupyter_output><empty_output><jupyter_text>Journal of ...<jupyter_code>df[df.name.str.startswith("Journal of")]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12699514226231784, 'fraction_numerical': 0.05829285218598196, 'mean_word_length': 7.511811023622047, 'pattern_counts': {'":': 2, '<': 59, '<?xml version=': 0, '>': 68, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '369457', 'license': 'MIT', 'n_tokens_mistral': 1729, 'n_tokens_neox': 1538, 'n_words': 330, 'path': 'notebooks/Journal_Names.ipynb', 'repo_name': 'bnewbold/fuzzycat', 'size': '4323'}