<jupyter_start><jupyter_text>Parsing Citations With [AnyStyle.io](http://anystyle.io) and Crossref's [search api](http://labs.crossref.org/resolving-citations-we-dont-need-no-stinkin-parser/)<jupyter_code>import pandas as pd
citations = pd.read_csv("cites.csv")
citations
citations.iloc[0]
len(citations)<jupyter_output><empty_output><jupyter_text>ok, we got a pile of citations. But they aren't in shape. When we look at the cites, they are a collection of 1,505 strings, this isn't very useful for doing data analysis. We need to *get them into shape*, that is, we need to break them up into their component parts.<jupyter_code>citations.iloc[0:5]<jupyter_output><empty_output><jupyter_text>Note, these citations are not always formatted in the same way, for example, lets looks at some from a different part of the piles.<jupyter_code>citations.iloc[390:395]<jupyter_output><empty_output><jupyter_text>Parsing citaitons is a whole area of research to be discussed at another time. I am going to use [AnyStyle.io](http://anystyle.io) to try and parse these citations because it has a nicely designed API.<jupyter_code>import requests
import os
import json
import numpy as np
import time
# get the API key for AnyStyle.io from a text file in this directory
with open('anystyle_key.txt','r') as f:
    api_key = f.read()
# I want to figure out what cite is causing the error

parsed_cites = []

for cite in list(segment['cite']):
   

    
    payload = {"format": "json",
               "access_token": api_key,
               "references": cite}
    headers = {"Content-Type": "application/json;charset=UTF-8"}
    #print("Payload Build, requesting")
    
    r = requests.post("http://anystyle.io/parse/references",
                  headers=headers,
                  data=json.dumps(payload))
    #print("Got response", r)
    if r.status_code == 400:
        print(cite)
    
    #parsed_cites.append(r.json())

parsed_cites = []

for segment in np.array_split(citations,5):
    print("Segment Length: ",len(segment))
    cite_pile = list(segment['cite'])

    
    payload = {"format": "json",
               "access_token": api_key,
               "references": cite_pile}
    headers = {"Content-Type": "application/json;charset=UTF-8"}
    print("Payload Build, requesting")
    
    r = requests.post("http://anystyle.io/parse/references",
                  headers=headers,
                  data=json.dumps(payload))
    print("Got response", r)
    parsed_cites.append(r.json())

print(len(parsed_cites))
parsed_cites_master = [cite for cites in parsed_cites for cite in cites]
parsed_cites_master[0:10]<jupyter_output><empty_output><jupyter_text>Sweet!<jupyter_code>with open("parsed_cites.json",'w') as f:
    print(json.dumps(parsed_cites_master, indent=4), file=f )
df_citations = pd.DataFrame(parsed_cites_master)
df_citations
df_citations.to_csv("parsed_cites.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11801029159519726, 'fraction_numerical': 0.007890222984562607, 'mean_word_length': 4.0625, 'pattern_counts': {'":': 8, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '637261', 'license': 'CC0-1.0', 'n_tokens_mistral': 981, 'n_tokens_neox': 930, 'n_words': 246, 'path': 'notebooks/parsing-citations.ipynb', 'repo_name': 'mcburton/ischools-core', 'size': '2915'}