<jupyter_start><jupyter_text>Great! We now have **PubMed IDs** that could contain Id's that have papers related to term *[Open science] AND Kenya.* This is the same way you'd request for information at pubmed in NCBI. Let go ahead and get the information the full paper if possible in the next step.<jupyter_code># We just need to change our handle to get for instance a summary of 
# the data the papers we need
handle2 = Entrez.esummary(db="pubmed", id = "30123385")
# Let's bring our result back from NCBI
record2 = Entrez.read(handle2)

# see what we are capable of subsetting
record2
print("Extract interesting entries in the data")
print("")
print(record2[0]['Id'])
print("")
print(record2[0]['Title'])
print("")
print(record2[0]['AuthorList'])
print("")
print(record2[0]['FullJournalName'])
print("")
print(record2[0]['EPubDate'])
# We can't extract everything we want so let's just get the full paper
# if we can
?Entrez.efetch
# As Caleb said this will give us XML output which we continue to parse
# Notice, the documentation of each argument need more annotation
handle4 = Entrez.efetch(db="pubmed", id = "30123385", rettype="gb",retmode="text")

# fetching the result from the database
# print(handle4.read())
# Checking what type of object is return XML maybe?
# remove the delete the hash tag to see for yourself
# type(handle4.read())
# Using store magic command to store the output to a file
# Storing in variables for later is not allowed
%store handle4.read() >> file.txt
# you can use shell commands directly on the cell of jupyter
%cat file.txt<jupyter_output>1. Open AIDS J. 2018 Jul 19;12:53-67. doi: 10.2174/1874613601812010053. eCollection 
2018.

HIV Prevention in Adolescents and Young People in the Eastern and Southern
African Region: A Review of Key Challenges Impeding Actions for an Effective
Response.

<NAME>(1), <NAME>(1), <NAME>(2), <NAME>G(3), <NAME>(4), Bains
A(4).

Author information: 
(1)Health Economics and HIV and AIDS Research Division, University of
KwaZulu-Natal, Durban, South Africa.
(2)School of Applied Human Sciences, University of KwaZulu-Natal, Durban, South
Africa.
(3)Department of Psychology, Middle Tennessee State University, Murfreesboro,
United States of America.
(4)UNICEF, Eastern and Southern Africa Regional Office, Nairobi, Kenya.

The global commitment to ending the AIDS epidemic by 2030 places HIV prevention
at the centre of the response. With the disease continuing to disproportionately 
affect young populations in the Eastern and Southern African Region (ESAR),
particularly adoles[...]<jupyter_text>We have to write code 20 times! Nope there's an efficient way to solve our this problem, at least partially. It's .... you guessed it write a function. Go back to the code we wrote we put everything together and change just a few things and we are golden right? RIGHT?<jupyter_code># write a function to automate rewriting the code over and over again
# This is hack function you could use a loop to feed into the other using a generator that could traverse the list
# generated and give you the result try and fix this function
def paper_retriever(email, searchterm, pubmedid):
    '''The paper retriever function takes your email which uses the same name email as an 
    argument, pubmedid you can get this from the previous function, searchterm take the NCBI type of query as a string
    and renamefile just changing your file names to avoid confusion.
    
    Return the full paper depending on if it's open access or not.
    '''
    # Enter your own email
    Entrez.email = email
    
    # search NCBI from the particular search term with method esearch
    handle = Entrez.esearch(db="pubmed", term=searchterm)
    
    # get the results
    record = Entrez.read(handle)
    
    # the method efetch does and fetches the information you need brings it back to your Ipython session
    handle2 = Entrez.efetch(db="pubmed", id = pubmedid, rettype="gb",retmode="text")
    
    # seeing the results
#     print("Extract interesting entries in the data")
#     print("")
#     print(record[0]['Id'])
#     print("")
#     print(record[0]['Title'])
#     print("")
#     print(record[0]['AuthorList'])
#     print("")
#     print(record[0]['FullJournalName'])
#     print("")
#     print(record[0]['EPubDate'])
    
    # using cell magic in a function in the jupyter notebook
    return handle2.read()
# calling the function as a test
print (paper_retriever(email="<EMAIL>", searchterm="[Open science] AND Kenya",pubmedid=30123385))
paper = paper_retriever(email="<EMAIL>", searchterm="[Open science] AND Kenya",pubmedid=30123385)
paper
%store paper >> file2.txt
%cat file2.txt
%%writefile paper_retriever.py
from Bio import Entrez
def paper_retriever(email, searchterm, pubmedid):
    '''The paper retriever function takes your email which uses the same name email as an 
    argument, pubmedid you can get this from the previous function, searchterm take the NCBI type of query as a string.
    
    Return the full paper depending on if it's open access or not.
    '''
    # Enter your own email
    Entrez.email = email
    
    # search NCBI from the particular search term with method esearch
    handle = Entrez.esearch(db="pubmed", term=searchterm)
    
    # get the results
    record = Entrez.read(handle)
    
    # the method efetch does and fetches the information you need brings it back to your Ipython session
    handle2 = Entrez.efetch(db="pubmed", id = pubmedid, rettype="gb",retmode="text")
    
    # using cell magic in a function in the jupyter notebook
    return handle2.read()

paper1 = paper_retriever(email="<EMAIL>", searchterm="[Open science] AND Kenya",pubmedid=30123385)
#%store paper >> papers1.txt
print(paper1)
from Bio import Entrez
def paper_parser(term, identity):
    handle = Entrez.esearch(db="pubmed", term=identity)
    record = Entrez.read(handle)
    print (record)
    handle2 = Entrez.esummary(db="pubmed", id = identity)
    record2 = Entrez.read(handle2)
    print("Extract interesting entries in the data")
    print("")
    print(record2[0]['Id'])
    print("")
    print(record2[0]['Title'])
    print("")
    print(record2[0]['AuthorList'])
    print("")
    print(record2[0]['FullJournalName'])
    print("")
    print(record2[0]['EPubDate'])
    return record

print(paper_parser(term="[Open science] AND Kenya", identity=30123385))
%%writefile paper_parser.py
from Bio import Entrez
def paper_parser(term, identity):
    Entrez.email = "<EMAIL>" #use your email
    handle = Entrez.esearch(db="pubmed", term=identity)
    record = Entrez.read(handle)
    print (record)
    handle2 = Entrez.esummary(db="pubmed", id = identity)
    record2 = Entrez.read(handle2)
    print("Extract interesting entries in the data")
    print("")
    print(record2[0]['Id'])
    print("")
    print(record2[0]['Title'])
    print("")
    print(record2[0]['AuthorList'])
    print("")
    print(record2[0]['FullJournalName'])
    print("")
    print(record2[0]['EPubDate'])
    return record2

paper2 = paper_parser(term="[Open science] AND Kenya", identity=30123385)
#%store paper2 >> papers2.txt run in ipython session
print(paper2)
%run ../Code/paper_retriever.py
%run ../Code/paper_parser.py
# stores the interesting NCBI results
%store record["IdList"] >> NCBIids.txt<jupyter_output><empty_output><jupyter_text>>group publications by dateimport the datado some cleaning that is, parsing datetime objectsselect the necessary columnsgroupby date and publications columns<jupyter_code>%cat kenyan_papers_details.txt
# import data manipulation library 
import pandas as pd
# importing data using the read_csv function delimiter set to tabs
df = pd.read_csv('../Data/kenyan_papers_details.txt', delimiter="\t")
# .head() allows you to see the first observations 
df.head()
# see the column names of the dataframe
df.columns
# gives a concise summary of the dataframe
df.info()
# dimensions of the dataframe (rows, columns)
df.shape<jupyter_output><empty_output><jupyter_text>Columns of interest include: EpubDate<jupyter_code>df.PubDate[1]
# df['EPubDate'] = df['EPubDate']
# counting the number of times a publication was 
# posted for this sample 2018 Jan 18 was common
df.EPubDate.value_counts()[:10]
# a lot of publications were posted in 2018 for this 
# sample
df.PubDate.value_counts()[:10]
# answers the question what are the popular journals researchers post their work
# This could coicide with the type of research that's done mostly namely the malaria journal 
df_group_pubs = df.groupby(by=["FullJournalName"]).count().sort_values(by="Source", ascending=False)
df_group_pubs
# grouping by publication date and fulljournalname, papers were posted recently in PloS one
# to get all the results i recommend subsetting the column and writing to a file
df_group_pubs2 = df.groupby(by=["PubDate","FullJournalName"]).count().sort_values(by="Source", ascending=False)
df_group_pubs2
# counting the number of occurences of a paricular journals as confirmation that our function works
# wellcome open research has 27 papers
df_group_pubs.EPubDate.sort_values(ascending=False)
# same thing but taking publication date into consideration 
df_group_pubs.PubDate.sort_values(ascending=False)
df_group_pubs2.sort_values(by="FullJournalName",ascending=False)<jupyter_output><empty_output><jupyter_text>group pubs by journalsimport the datado some cleaning that is, parsing datetime objectsselect the necessary columnsgroupby date and publications columns<jupyter_code>df["FullJournalName"].value_counts()
def summary (filename):
    '''The function summary takes a file and outputs important summary statistics namely:
    * The number of records
    * The most popular times of the year a journal is published
    * group publications by date
    * group publication by Journals
    
    Returns tables with the above information.'''
    df = pd.read_csv(filename, delimiter="\t")
    msg = "The number of publications from the sample gotten from NCBI (rows,columns) {}"
    msg.format(df.shape); print("=" * 100)
    print ("Common electronic publication dates" + str(df.EPubDate.value_counts()[:10]))
    print ("Common Publication dates" + str(df.PubDate.value_counts()[:10]))
    print("=" * 100)
    print("Grouping publications by date and journal")
    df_group_pubs2 = df.groupby(by=["PubDate","FullJournalName"]).count().sort_values(by="Source", ascending=False) 
    print(df_group_pubs2)
    print("="*100)
    print("Sorting the publications found by journals")
    print(df_group_pubs.PubDate.sort_values(ascending=False))
    
    return "Done"
print (summary('../Data/kenyan_papers_details.txt'))
# taking the df column with author list to determine the most frequent author in the 2000 paper sample
df.AuthorList.value_counts()[1:100]

# Found 4 occurrences of the following authors 
# '<NAME>', '<NAME>', '<NAME>', '<NAME>', 'Foba CN', '<NAME>', '<NAME>'
# 'Williams PCM', '<NAME>'
# commands used find the common and least common author names our sample
# %cut -f2 kenyan_papers_details.txt | head -10
# %cut -f2 kenyan_papers_details.txt > authorList2000papers.txt<jupyter_output>UsageError: Cell magic `%%cut` not found.<jupyter_text>common names in publications|name| occurence of name|| ------------- |-----:||Van| 114| |Wang |60||Otieno| 52||Zhang |49||Bukusi| 46||Cheng|45| Above are the common names that were found in the kenyan_papers_details.txt file. People with the name Van and Wang were the most common followed by Otieno, Zhang, Bukusi and Cheng. These were determined by the wordcloud application using the file authorList2000papers.txt [here]( https://gettingappy.shinyapps.io/wordcloudunigrams/). In the image the size of the word corresponds to the frequency of the text. ![The wordcloud and how the parameters were set](plots/wordcloud_result.png)<jupyter_code>?pd.read_csv
# load in the dataframe and give the columns names
df2 = pd.read_csv("PMID_PMC_Journal_Year.txt",delimiter="\t",names=["pmid","pmd","journal","year"])
# no rows are lost
# pandas loads in an all or nothing approach
df2.shape
# concise summary of the dataset
df2.info()
# Use R programming language in the notebook 
# https://www.linkedin.com/pulse/interfacing-r-from-python-3-jupyter-notebook-jared-stufft/
# import rpy2.rinterface 
# %load_ext rpy2.ipython
# %%R -i df2
# head(df2)
# convert the txt file into a csv for visualization in R
df2.to_csv("PMID_PMC_Journal_Year.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10409967845659164, 'fraction_numerical': 0.0204983922829582, 'mean_word_length': 4.773085846867749, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 51, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '500532', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 3872, 'n_tokens_neox': 3650, 'n_words': 1356, 'path': 'Notebooks/data-import-NCBI.ipynb', 'repo_name': 'BioinfoNet/data-mining', 'size': '12536'}