<jupyter_start><jupyter_text>Setting up Kaggle API in Google Colaboratory<jupyter_code>!pip install -q kaggle
f = open("kaggle.json", "w")
f.write('{"username":"pranjalya","key":"fa4deb4f89e2fe282a3e29c166413ecf"}')
f.close()
!mkdir ~/.kaggle
!cp kaggle.json ~/.kaggle/
!chmod 600 ~/.kaggle/kaggle.json
!kaggle datasets download -d skylord/coronawhy<jupyter_output>Downloading coronawhy.zip to /content
100% 12.8G/12.8G [03:28<00:00, 36.9MB/s]
100% 12.8G/12.8G [03:28<00:00, 65.6MB/s]<jupyter_text>Finding papers containing pollution keywords in correlation with COVID-19 lethal outcomes **Task at hand** :  https://trello.com/c/SA27hbNo/35-finding-papers-on-correlation-between-pollution-and-lethal-outcomes<jupyter_code>!pip install spacy_langdetect
!unzip coronawhy.zip v6_text/*
!unzip coronawhy.zip clean_metadata.csv
# Importing required libraries

import os
import pandas as pd
import numpy as np

os.system('pip install scholarly')

import scholarly

PATH = "v6_text/v6_text/"<jupyter_output><empty_output><jupyter_text>Here are the keywords (currently, we have trigrams) to search for in the papers.<jupyter_code># Pollution n-grams

pollution = ['air pollution and', 'indoor air pollutants', 'indoor air pollution', 'household air pollution', 
             'air pollution is', 'between air pollution', 'of air pollution', 'particulate air pollution',
             'pollution and the', 'air pollutant data']<jupyter_output><empty_output><jupyter_text>The output is supposed to be in this format.<jupyter_code>df_output = pd.DataFrame(columns=['Risk Factor', 'Keyword/Ngram', 'Paper ID', 'URL', 
                                  'Sentences from Result', 'Sentences from Method', 
                                  'Design Methodology', 'Correlation', 'Authors', 
                                  'No of Citations', 'No of keyword occurence in Paper'])<jupyter_output><empty_output><jupyter_text>**Searching for the keywords only in method and result section of the papers, as the most relevant papers can be found there only.**(This cell may take some time)<jupyter_code>firstpass = True

for pkl in os.listdir(PATH):
  df = pd.read_pickle(PATH+pkl, compression='gzip')
  if(firstpass):
    df_method = df[df['section']=='methods']
    df_result = df[df['section']=='results']
    firstpass = False
  else:
    df_method = pd.concat([df_method, df[df['section']=='methods']])
    df_result = pd.concat([df_result, df[df['section']=='results']])
print("No of unique papers in method section : ", df_method['paper_id'].nunique(), " out of ", 
      len(df_method), " rows in dataframe")
print("No of unique papers in result section : ", df_result['paper_id'].nunique(), " out of ", 
      len(df_result), " rows in dataframe")
df_method.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 187090 entries, 288 to 370559
Data columns (total 36 columns):
 #   Column                           Non-Null Count   Dtype 
---  ------                           --------------   ----- 
 0   paper_id                         187090 non-null  string
 1   language                         187090 non-null  object
 2   section                          187090 non-null  object
 3   sentence                         187090 non-null  string
 4   lemma                            187090 non-null  object
 5   UMLS                             187090 non-null  object
 6   GGP                              187090 non-null  object
 7   SO                               187090 non-null  object
 8   TAXON                            187090 non-null  object
 9   CHEBI                            187090 non-null  object
 10  GO                               187090 non-null  object
 11  CL                               187090 non-null  object
 12  DNA           [...]<jupyter_text>*Just some function I thought I would use, but never did, actually. Maybe someone could find them useful.<jupyter_code>def isPresentInList(A, B):
  '''
  A, B : Lists
  Returns : Boolean value for whether A appears as sequence in B
  '''
  return ', '.join(map(str, A)) in ', '.join(map(str, B))<jupyter_output><empty_output><jupyter_text>***Now we extract sentences from the method and result sections of papers, which contain the keywords.***<jupyter_code>df_method_p = df_method[df_method['sentence'].str.contains('|'.join(pollution), case=False) == True]
df_result_p = df_result[df_result['sentence'].str.contains('|'.join(pollution), case=False) == True]

# Merging the method and result section sentences into single dataframe
df_real = pd.concat([df_method_p, df_result_p])
print("There are {} sentences containing keywords/ngrams in Method section.".format(len(df_method_p)))
print("There are {} sentences containing keywords/ngrams in Result section.".format(len(df_result_p)))
print("Total unique papers in Method section : {}".format(df_method_p['paper_id'].nunique()))
print("Total unique papers in Result section : {}".format(df_result_p['paper_id'].nunique()))
print("Total unique papers in combined section : {}".format(df_real['paper_id'].nunique()))<jupyter_output>Total unique papers in Method section : 17
Total unique papers in Result section : 6
Total unique papers in combined section : 20<jupyter_text>Hence, we have 20 papers which mention mention the specified keywords.<jupyter_code>metadata = pd.read_csv('clean_metadata.csv')
metadata.rename(columns={'sha':'paper_id'}, inplace = True)
metadata['paper_id'] = metadata['paper_id'].astype("string")
metadata.info()
df_real.head(2)
metadata.head(1)<jupyter_output><empty_output><jupyter_text>Merging the given papers with their metadata, which contains relevant data.<jupyter_code>df_real = df_real.merge(metadata[['paper_id', 'title', 'abstract', 'publish_time', 'authors', 'url']], on='paper_id', how='left')<jupyter_output><empty_output><jupyter_text>Keeping only the fields which are relevant to us.<jupyter_code>df_real = df_real[['paper_id', 'language', 'section', 'sentence', 'lemma', 'UMLS', 'sentence_id', 'title', 'abstract', 'publish_time', 'authors', 'url']]
df_real.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Int64Index: 46 entries, 0 to 45
Data columns (total 12 columns):
 #   Column        Non-Null Count  Dtype 
---  ------        --------------  ----- 
 0   paper_id      46 non-null     string
 1   language      46 non-null     object
 2   section       46 non-null     object
 3   sentence      46 non-null     string
 4   lemma         46 non-null     object
 5   UMLS          46 non-null     object
 6   sentence_id   46 non-null     string
 7   title         45 non-null     object
 8   abstract      45 non-null     object
 9   publish_time  45 non-null     object
 10  authors       42 non-null     object
 11  url           45 non-null     object
dtypes: object(9), string(3)
memory usage: 4.7+ KB<jupyter_text>**TO WORK ON : ** If we need to also get all the papers which show lethality, among the papers containing risk factor keywords.<jupyter_code>lethal = ["death", "fatality", "mortality","lethal", "lethality", "morbidity"]
def keywordcounter(sentences):
  '''
  Input : List of sentences
  Returns : Keywords present in sentences, Total count of all keywords present in Input
  '''
  keyword = {}
  sent = " ".join(sentences)
  for pol in pollution:
    counter = sent.lower().count(pol)
    if (counter > 0):
      keyword[pol] = counter
  return list(keyword.keys()), sum(keyword.values())<jupyter_output><empty_output><jupyter_text>***Grouping the sentences by their Paper IDs.***<jupyter_code>grouped = df_real.groupby('paper_id')<jupyter_output><empty_output><jupyter_text>Function to format all the code in specified output format.Work left : -> No of citations for each paper-> Correlation (Unclear)-> Design Methodology (Unclear)<jupyter_code>def aggregation(item):
  '''
  Input : Dataframe of sentences of a paper
  Return : Datframe in Standard Output format
  '''
  dfo = {}

  dfo['Risk Factor'] = 'Pollution'
  dfo['Title'] = item['title'].iloc[0]
  dfo['Keyword/Ngram'], dfo['No of keyword occurence in Paper'] = keywordcounter(item['sentence'].tolist())
  dfo['Paper ID'] = item['paper_id'].iloc[0]
  dfo['URL'] = item['url'].iloc[0]
  dfo['Sentences from Method'] = item[item['section']=='methods']['sentence'].tolist()
  dfo['Sentences from Result'] = item[item['section']=='results']['sentence'].tolist()
  dfo['Authors'] = item['authors'].iloc[0]

  # For papers which do not have title (not in metadata) we have to resolve exceptions
  try:
    dfo['No of Citations'] = next(scholarly.search_pubs_query(item['title'].iloc[0])).citedby
  except:
    dfo['No of Citations'] = 0

  dfo['Correlation'] = 'Not yet'
  dfo['Design Methodology'] = 'None'

  return dfo<jupyter_output><empty_output><jupyter_text>Applying function to get the final output dataframe.<jupyter_code>for key, item in grouped:
  df_output = pd.concat([df_output, pd.DataFrame([aggregation(item)])])
df_output = df_output.reset_index()
del df_output['index']
df_output<jupyter_output><empty_output><jupyter_text>Saving the extracted papers.<jupyter_code>df_output.to_csv('pollution_papers.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12786238632628466, 'fraction_numerical': 0.026624301522953874, 'mean_word_length': 3.596173212487412, 'pattern_counts': {'":': 2, '<': 61, '<?xml version=': 0, '>': 63, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '386842', 'license': 'MIT', 'n_tokens_mistral': 3054, 'n_tokens_neox': 2822, 'n_words': 777, 'path': 'notebooks/Pollution_relation_with_Covid_19.ipynb', 'repo_name': 'rajgupt/task-risk', 'size': '9127'}