<jupyter_start><jupyter_text>Semantic Scholar Scraping for Summarisation Literature Review This has been integrated into a python script. url: http://s2-public-api-prod.us-west-2.elasticbeanstalk.com/corpus/download/<jupyter_code>from tqdm import tqdm
import gzip
import json
from langdetect import detect
manifestFile = open("../data/sample/semantic scholar manifest 2020-03.txt", "r")
manifestFile = manifestFile.read()
manifestFileList = manifestFile.split('\n')
manifestFileList = [file for file in manifestFileList if 's2' in file]<jupyter_output><empty_output><jupyter_text>Extract Data<jupyter_code>s2CorpusUrl = '../data/s2-corpus-000.gz'    #'s2-corpus/sample-S2-records.gz'
def unzipS2Contents(url):
    f = gzip.open(url, 'rb')
    file_content = f.read().decode('utf-8')
    f.close()
    fileContentsList = file_content.split('\n')
    return fileContentsList
%%time
fileContentsList = unzipS2Contents(s2CorpusUrl)
len(fileContentsList)
fileContentsList[3]
keyTermsList = ['summarisation', 'summarization', 'nlg', 'extractive', 'summeries']    # spelling problems...; removed automatic as it's too general when it matches by itself
%%time
# Process documents and keep only the english ones.
fileContentsListEnglish = []
fileCount = 1
noFilesToProcess = 100000

for file in fileContentsList:
    try:
        fileJSON = json.loads(file)

        if detect(fileJSON["title"]) == 'en':
            fileContentsListEnglish.append(file)
            
            if fileCount % (noFilesToProcess/10) == 0:
                print(f'File No.: {fileCount} - Year: {fileJSON["year"]} - Title: {fileJSON["title"]}')

            fileCount += 1   # only counting for english documents.
            
    except Exception as e:
        print(f'File No.: {fileCount} - ERROR: {e}')
        
    if fileCount == noFilesToProcess+1:
        break<jupyter_output>File No.: 1962 - ERROR: No features in text.
File No.: 3408 - ERROR: No features in text.
File No.: 3591 - ERROR: No features in text.
File No.: 7111 - ERROR: No features in text.
File No.: 9483 - ERROR: No features in text.
File No.: 10000 - Year: 1989 - Title: Epidemiological approaches for identifying risk factors in human congenital malformations : Abstracts of Papers Presented at the 29th Annual Meeting of the Japanese Teratology Society, Yamagata, Japan July 13 - 14
File No.: 10129 - ERROR: No features in text.
File No.: 13320 - ERROR: No features in text.
File No.: 16084 - ERROR: No features in text.
File No.: 17267 - ERROR: No features in text.
File No.: 17351 - ERROR: No features in text.
File No.: 19752 - ERROR: No features in text.
File No.: 20000 - Year: 2009 - Title: Indian Adaptations in Flooded Regions of South America: Introduction
File No.: 22540 - ERROR: No features in text.
File No.: 24989 - ERROR: No features in text.
File No.: 25199 - ERROR: No features in text.
Fi[...]<jupyter_text>Keep documents that contain key termsprint(f'Number of english documents: {len(fileContentsListEnglish)}')<jupyter_code>%%time
for file in fileContentsListEnglish:
    
    fileJSON = json.loads(file)
    
    keyTermsMatched = set(fileJSON["title"].lower().split(' ')).intersection(set(keyTermsList))
    
    if (0 < len(keyTermsMatched)) :
        print(f'\nDOC MATCHED: {fileJSON["title"]}\n')
    else:
#         print(fileJSON["title"])
        pass<jupyter_output>DOC MATCHED: An Effective Joint Framework for Document Summarization


DOC MATCHED: Developments in Physical Chemistry and Basic Principles of Extractive Metallurgy in 1980


DOC MATCHED: Political and economic implications of extractive industries


DOC MATCHED: Belief as summarization and meta-support


DOC MATCHED: Effect of Sea Cucumber Extractive on Erythrocyte Membrane Fluidity in Rats with Overtraining Syndrome


DOC MATCHED: Automatic Generation of Summeries for the Web


DOC MATCHED: A video summarization approach based on the emulation of bottom-up mechanisms of visual attention


DOC MATCHED: Summarization Experiments in DUC 2004


DOC MATCHED: QA@INEX Track 2011: Question Expansion and Reformulation Using the REG Summarization System


DOC MATCHED: Summarization of news speech with unknown topic boundary


DOC MATCHED: Attend to the beginning: A study on using bidirectional attention for extractive summarization


DOC MATCHED: Topical Summarization on the Mayor and Schoolm[...]
{'fraction_non_alphanumeric': 0.09354171454837969, 'fraction_numerical': 0.031027350034474835, 'mean_word_length': 3.9737142857142858, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 11, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '435738', 'license': 'MIT', 'n_tokens_mistral': 1442, 'n_tokens_neox': 1266, 'n_words': 466, 'path': 'project/notebooks/Data Extraction.ipynb', 'repo_name': '4theKnowledge/literaturesieve', 'size': '4351'}