<jupyter_start><jupyter_text>Fix ArXiv dataThis python package includes an option to download ArXiv metadata and then use that data to train a simple model for tracking rejected articles with CrossRef. The ArXiv data used here is incomplete in that not all ArXiv preprints correctly link to the right DOI. This notebook runs a simple procedure to use that model to complete the ArXiv data.<jupyter_code># load ArXiv data
import os
import pandas as pd
from rejected_article_tracker.src.ML.ArXivOAIPMH import ArXivOAIPMH

arxiv_data_generator = ArXivOAIPMH().yield_json()
arxiv = pd.DataFrame(arxiv_data_generator)
arxiv.shape
# limit to missing entries
arxiv = arxiv[arxiv['doi']=='']
# convert columns to match expected input
# we've based the input for the tracker on Scholar One column-headings
# so we need to convert our ArXiv column headings to match.
from rejected_article_tracker.src.ML.ArXivAuthorNames import ArXivAuthorNames
arxiv['Journal Name']  = 'ArXiv'

arxiv = arxiv.rename(columns={'created':'Submission Date',
                            'id':'Manuscript ID',
                            'title':'Manuscript Title',
                            'authors':'Author Names',
                            })
# add a column for decision date (not important)
arxiv['Decision Date'] = arxiv['Submission Date']
# pretend they all got rejected
arxiv['Accept or Reject Final Decision'] = 'Reject'
# preprocess author list
def pre_authors(authors):

    author_ls = [x.strip() for x in authors.split(',')]
    author_ls_out = []
    for author in author_ls:
        first_last = author.split('+')
        first = first_last[0]
        try:
            last = first_last[1]
            author_name = last+', '+first
        except:
            author_name = first
        
        author_ls_out.append(author_name)
    return '; '.join(author_ls_out)
arxiv['Author Names'] = arxiv['Author Names'].map(lambda x: pre_authors(x))
arxiv.head(2)
# use package to search for missing entries
# code from readme.md
from rejected_article_tracker import ScholarOneRejectedArticlesMatch
import pandas as pd

allowed_cols = [
    'Journal Name',
    'Manuscript ID',
    'Manuscript Title',
    'Author Names',
    'Submission Date',
    'Decision Date',
    'Accept or Reject Final Decision'
]

# for testing
articles = arxiv.sample(20)[allowed_cols].to_dict('records')

# Which might look like:
"""  
articles = [
{
      "Journal Name": "The International Journal of Robotics Research",
      "Manuscript Title": "Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection",
      "Author Names": "<NAME>; <NAME>; <NAME>; <NAME>; <NAME>",
      "Accept or Reject Final Decision": "",
      "Decision Date": "2019-01-01T13:29:58.999Z", 
      "Submission Date": "2018-10-01T13:29:58.999Z",
      "Manuscript ID": "ABC-18-070",
    }
]
"""

# @see below for configuration details.
search_config = {
    "threshold": 70, # Filters out matches which are less than this nubmer  
}

# The CrossRef API requires an email address for lookups.    
email = os.environ.get('MY_EMAIL','')<jupyter_output><empty_output><jupyter_text>TestCheck to see how long it takes to retrieve results from a batch of articles.<jupyter_code># %%time
# # Define a 'results' list.
# results = []

# # Run match
# ScholarOneRejectedArticlesMatch(
#     articles=articles,
#     config=search_config,
#     email=email,
#     results=results
# ).match()

# print(len(results))<jupyter_output><empty_output><jupyter_text>Test took 2mins to retrieve data for 20 articles. In the past, we've had average times of ~3s per article, but this is more like 6s per article. At 6s per article, the entire 40k dataset will take ~65hrs to retrieve. In practice, you can use a server / VM to run the process constantly. But another option is to do it in stages.<jupyter_code>from rejected_article_tracker.src.ML.CrossRefUtils import CrossRefUtils
from rejected_article_tracker.src.ML.config import Config as config
from tqdm import tqdm

data_path = config.ml_data_dir
data_file_path = os.path.join(data_path,'arxiv_results.csv')

if os.path.exists(data_file_path):
    results_df = pd.read_csv(data_file_path)
else:
    results_df = pd.DataFrame()

print('Starting with dataframe, shape:', results_df.shape)

# run the whole dataset
articles = arxiv[allowed_cols].to_dict('records')

# drop any that we've already retrieved
n_results_retrieved = results_df.shape[0]
articles = articles[n_results_retrieved:]

# split into batches so that we can write out periodically
batches = list(CrossRefUtils().chunks(articles, 30))

for batch in tqdm(batches):
    # Define a 'results' list.
    results = []
    # Run match
    ScholarOneRejectedArticlesMatch(
        articles=batch,
        config=search_config,
        email=email,
        results=results
    ).match()

    results_df = pd.concat([results_df,pd.DataFrame(results)])
    results_df.to_csv(data_file_path, index=False)<jupyter_output>Starting with dataframe, shape: (1590, 24)
100%|██████████| 1253/1253 [00:00<00:00, 625724.84it/s]
  2%|▏         | 20/1253 [1:32:00<85:09:50, 248.65s/it]<jupyter_text>What did we find?<jupyter_code>results_df.head(2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10408708938120703, 'fraction_numerical': 0.02215431627196333, 'mean_word_length': 3.9127579737335836, 'pattern_counts': {'":': 8, '<': 23, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '142923', 'license': 'MIT', 'n_tokens_mistral': 1779, 'n_tokens_neox': 1592, 'n_words': 495, 'path': 'rejected_article_tracker/src/ML/notebooks/track_arxiv.ipynb', 'repo_name': 'sagepublishing/rejected_article_tracker_pkg', 'size': '5278'}