<jupyter_start><jupyter_text>Notebook to process output from thecontentmineThis notebook illustrates the basic process of processing each resource in each paper and allocates a score.<jupyter_code>import pandas
import json

import process_urls
with open('amw_token', 'r') as f:
    github_token = f.read()
#Read JSON data into the datastore variable
with open('../sample_data/small_dict_of_papers.json', 'r') as f:
    dict_of_papers = json.load(f)
resources_list = process_urls.process_papers_dict(dict_of_papers, verbose=True, 
                                                  github_token=github_token)<jupyter_output>processing paper with doi 10.1038/s41467-018-03297-7
check licence for URL https://github.com/YeatmanLab/AFQ-Browser_data
URL https://github.com/YeatmanLab/AFQ-Browser_data does not have a licence
check licence for URL https://github.com/YeatmanLab/AFQ-Browser_data/blob/master/AFQ-Browser_ALSexample/Reproducing-Sarica2017-Figure3.ipynb
skipping licence check for URL https://github.com/YeatmanLab/AFQ-Browser_data/blob/master/AFQ-Browser_ALSexample/Reproducing-Sarica2017-Figure3.ipynb
check licence for URL https://github.com/YeatmanLab/AFQ-Browser_data/blob/master/AFQ-Browser_ALSexample/Figure6.ipynb
skipping licence check for URL https://github.com/YeatmanLab/AFQ-Browser_data/blob/master/AFQ-Browser_ALSexample/Figure6.ipynb
check licence for URL https://github.com
URL https://github.com does not have a licence
check licence for URL https://github.com/YeatmanLab/AFQ-Browser
URL https://github.com/YeatmanLab/AFQ-Browser has a licence
check licence for URL https://github.com/YeatmanLab/AFQ[...]<jupyter_text>Analysis of the dataWe can store the data in a flat SQL database, but here we stuff it into a Pandas dataframe to allow some analysis.<jupyter_code>url_df = pandas.DataFrame.from_dict(resources_list)
url_df<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09057005860415557, 'fraction_numerical': 0.017048481619605753, 'mean_word_length': 4.961904761904762, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '494497', 'license': 'MIT', 'n_tokens_mistral': 654, 'n_tokens_neox': 588, 'n_words': 144, 'path': 'notebooks/resolvre_and_check_resources.ipynb', 'repo_name': 'gitter-badger/code-cite', 'size': '1877'}