<jupyter_start><jupyter_text>NLM data explorationby <NAME> and <NAME> _**Introduction**_The National Library of Medicine (NLM) produces a resource named HealthReach (http://healthreach.nlm.nih.gov) that provides access to free low literacy, mulitlingual health information to be used by healthcare providers with patients/clients. NLM collects the materials and indexes with controlled vocabulary.Users can access the HealthReach collection through the website but our interest is to make our collection only available through an application programming interface (API). We curently have a closed API that we share with another NLM resource, and a hospital in Kenya that is pulling English, French, and Swahili materials into their patient electronic health records (EHR). The HealthReach program manager wants everyone to be able to pull the exact materials they want (determined by metadata) and use as they need it, when they need it, and where they need it. _**Statement of Need**_The majority of our users are in the healthcare service field and do not feel they are "savvy" in technology. Therefore dicussions of APIs and data often stall conversations about how they can customize the use of the HealthReach data. This is unfortunate as the true strength of the collection would be at point of need in the EHR...not stored in a web database that must be accessed, searched, and retrieved. To make our current users more comfortable with thinking about the multilingual collection of HealthReach as "data," we created a visualization of the collection in Tableau https://healthreach.nlm.nih.gov/data-viz. To create the visualizations in Tableau the progammer and program manager have to put in siginificant effort into accurately determining, combining, and displaying the data. The data is extracted directly from the database and users are provided .csv files of the data under an "accessibility" section. The program manager is responsible for checking the Tableau displays to ensure the "numbers" are correct. This is not inconsequential - not only does the program manager not want to do this, but the data that is displayed cannot easily be checked from the .csv because some "massaging" of the data has been done, and the data changes over time. For all of its transparency faults and diffculty in creating, Tableau provides an interactive and approachable interface for low-data literacy consumers.To best serve the ~~lazy~~ program manager, we will use Python to automate a task she doesn't want to do - check the "numbers" in the visualization. _**Demonstration**_In this presentation we will replicate one visualization on the HealthReach Tableau.To begin, we pulled data from the HealthReach database and exported to an common-seperated value (.CSV) spreadsheet. We will use this data taken on Novermeber 28, 2018 for our visualization:https://github.com/BIOF309/final-project-nlm/blob/master/HealthReach_Data_11_28_2018.csvWe can view our the .CSV file, but that is a alot of data for a non-programmer to view. We have over 8,000 rows and 21 columns. We will use **Python for data anyalsis** or **pandas** for short. Let us begin by importing **pandas** so we can use it for our visualizations.<jupyter_code>#import pandas as pd
import pandas as pd<jupyter_output><empty_output><jupyter_text>Next we will bring in our data from our .csv sheet. First we need to download our .csv.<jupyter_code>#download .csv
url = 'https://raw.githubusercontent.com/BIOF309/final-project-nlm/master/HealthReach_Data_11_28_2018.csv'<jupyter_output><empty_output><jupyter_text>We need to tell pandas to read our .csv. We will create a dataframe named df1.<jupyter_code>df1 = pd.read_csv(url)<jupyter_output><empty_output><jupyter_text>Now our .csv file is loaded and ready to display our .csv data in this Juypter Notebook<jupyter_code>df1<jupyter_output><empty_output><jupyter_text>Now that we have our data loaded, let's answer a question! *How many items are in each "Format" in HealthReach?*This question can be answered with one column of data. We have a metadata field "Format" which has 3 options:- Document- Audio- VideoTo answer this question we need to see what rows assigned the controlled vocabulary term (docuemnt, audio, or video) in the metadata column "Format," and display our results showing the 3 options. For this I am choosing to display in a pie chart. Let's begin by calculating the number of rows associated with the vocabulary assigned:<jupyter_code>import numpy as np
#Code borrowed from "basic pieplot with pandas" by Python graph gallery
#import matplotlib
#import matplotlib as plt
import matplotlib.pyplot as plt
%matplotlib inline
#download .csv
url = 'https://raw.githubusercontent.com/BIOF309/final-project-nlm/master/HealthReach_Data_11_28_2018.csv'
df=pd.read_csv(url)
df
#Determine the count of rows for each option in Format column
df2 = df1.set_index(['Format'])

df_video_size = len(df2.loc['Video'])
df_doc_size = len(df2.loc['Document'])
df_audio_size = len(df2.loc['Audio'])


print('doc: {}'.format(df_doc_size))

print('video: {}'.format(df_video_size))

print('audio: {}'.format(df_video_size)) 

#1.	here is an example of picking a subset of data
#df[df.Format=='Audio'][df.Language=='English']
#please notice the == 
#2.	To get the count for each format:


#AudioCount = len(df[df.Format=='Audio'])
#VideoCount = len(df[df.Format=='Video'])
#DocCount = len(df[df.Format=='Document'])
#print(AudioCount)
#print(VideoCount)
#print(DocCount)

#Check to make sure numbers are accurate
print (6533+1143+1143)
#create pie chart
#Code borrowed from "basic pieplot with pandas" by Python graph gallery
# --- dataset 1: just 3 values for 3 groups: 


df = pd.DataFrame([6533,1143,1143], index=['Document', 'Audio', 'Video'], columns=['Collection by Format']) 

# make the plot 
df.plot(kind='pie', autopct='%.1f%%', subplots=True, figsize=(6, 6), title='HealthReach Collection by Format');<jupyter_output><empty_output><jupyter_text>Try to replicate the word cloud..... In the meantime, let's identify the number of unique languages by cleaning and analyzing the language column.<jupyter_code>#Starting from the beginning, import pandas as pd
import pandas as pd
#Also import numpy as np
import numpy as np
#download the raw data .csv file and save as 'url'
url = 'https://raw.githubusercontent.com/BIOF309/final-project-nlm/master/HealthReach_Data_11_28_2018.csv'
#Create a dataframe of the raw data in the .csv file and name it df2:
df2= pd.read_csv(url)
#Create a new dataframe of the Language column as 'dflang':
dflang = pd.DataFrame(df2,columns=['Language'])
print (dflang)
dflang['Language'].value_counts()
#Glancing at dflang, many of the rows are duplicates and there are multiple languages in one row.
#We have some cleaning to do.
#First, let's drop the duplicates to get an initial sense of how many unique languages there could be:
print (dflang.drop_duplicates())
#We're down to 93 rows, but there are multiple languages in one row. 

#Let's separate the multiple languages in a row into individual rows and call this new series 's'.
#Code borrowed from: https://stackoverflow.com/questions/17116814/pandas-how-do-i-split-text-in-a-column-into-multiple-rows

import pandas as pd
s = dflang['Language'].str.split(';').apply (pd.Series, 1).stack()
s.index = s.index.droplevel(-1)
print(s)

#Note: The numbers below are the index, not the frequency of language occurence
#Drop duplicates in series 's'
#Code borrowed from: https://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.Series.drop_duplicates.html

s.drop_duplicates()
#The Length is 88 but the public HealthReach website lists 61 unique languages as of 12/1/2018.
#Public HealthReach website: https://healthreach.nlm.nih.gov/data-viz

#Need to investigate inconsistency further.
#Let's look at the actual strings and print.

print(pd.unique(s))
#It looks like there are leading and trailing spaces that are causing languages to appear more than once (e.g. 'English' & ' English').

#Let's strip the leading and trailing spaces and call the new data series 's2' and print

s2 = s.str.strip()
print(s2)
#Lets drop the duplicates from s2 to get unique languages and call it 's3' and print.

s3 = s2.drop_duplicates()
print(s3)
#Yay! After dropping the duplicates, the Length = 61 which is the number of languages listed on the public HealthReach website.

#Count the number of unique rows in s3 to verify the final count

len(s3)
#For a clean, ordered list, print the distinct languages in ascending alphabetic order

print(s3.sort_values(ascending=True))

#Note: The numbers below are the index, not the frequency of language occurence<jupyter_output>5923                                    Albanian
1                                        Amharic
2                                         Arabic
1827                                    Armenian
2455                                     Bengali
1149                                     Bosnian
4                                        Burmese
6480                         Cape Verdean Creole
986       Chinese, Simplified (Mandarin dialect)
1444    Chinese, Traditional (Cantonese dialect)
2935                                    Chuukese
5                                           Dari
3                                       Dzongkha
0                                        English
6                                          Farsi
987                                       French
2524                                      German
1798                                    Gujarati
1445                              Haitian Creole
7                                     Hakha Chin
1799                [...]<jupyter_text>Back to trying to replicate the language word cloud. For now, I'll use the **s2 data series** above in as the data for the word cloud and other visualizations since it **lists every instance of every language used in the .csv file** on a separate row.- The HealthReach webpage counts the number of **Titles** for the word cloud, which is **not enumerated in s2 data**.- Cleaning the data to isolate the number of titles by language is beyond what I can do right now because the Title needs to be duplicated for each language that is separated into its own row when grouped with other languages in the Language column (e.g. English, Arabic).- It would be great to duplicate the Format for each language also for further analysis and visualization of the data.After trying the word cloud without much progress, this section will be on creating a horizontal bar graph of the languages using the s2 data series above. - The numbers and language order in the bar graph here are **not** the same as the HealthReach data since the raw data differs for each.<jupyter_code>#Convert data series s2 to a dataframe with column name 'Language'
dflang2 = pd.DataFrame(s2).reset_index()
dflang2.columns=['Index','Language']
dflang2
#Count the instance of each distinct language in the 'Language' column
s4 = dflang2['Language'].value_counts()
s4
#Import python packages (numpy, pandas, matplotlib, pillow, and wordcloud) to create word cloud and bar graph.
#https://www.datacamp.com/community/tutorials/wordcloud-python

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('bmh')
%matplotlib inline
from os import path
from PIL import Image
#pip install wordcloud
#from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
#import pillow as pillow


#Convert s4 data into dataframe to clean up for bar graph.
dflang3 = pd.DataFrame(s4).reset_index()
dflang3.columns=['Language','Occurrence'] #Added column headers. Used 'Occurrence' instead of 'Count' since Count could be mistaken for Title Count.
dflang3
#Keep only the top 11 languages
#Code borrowed from: https://erikrood.com/Python_References/dropping_rows_cols_pandas.html
dflang3.drop(dflang3.index [11:], inplace=True)
dflang3

#I couldn't get it to drop English and everything after 10 simultaneously (e.g. [0, 11:] didn't work)
#Drop English since we want top 10 non-English languages
dflang3.drop(dflang3.index[0], inplace=True)
dflang3
#Reverse the order of the rows in dataframe
#Code borrowed from: https://stackoverflow.com/questions/39127900/how-to-reverse-the-order-of-rows-in-my-dataframe-using-python-but-not-reverse-t
dflang4 = dflang3.iloc[::-1]
dflang4

#I didn't touch the index, which gave me the result I wanted.
#Drop Index Column from dataframe
#Code borrowed from: https://stackoverflow.com/questions/37351172/how-to-remove-index-from-a-created-dataframe-in-python
dflang4.set_index('Language', inplace=True)
dflang4
#Create a horizontal bar chart of the top 10 languages
ax=dflang4.plot(kind='barh')
ax.set_xlabel('Language Occurrence')
ax.set_title('Top 10 non-English Languages in HealthReach Collection')

#Remove the legend
ax.get_legend().remove()
#Let's try changing the color of each bar
#Use multiple colors for the bars in the graph
#Code borrowed from: https://stackoverflow.com/questions/18973404/setting-different-bar-color-in-matplotlib-python

my_colors = 'rgbkymc'  #red, green, blue, black, etc.

ax=dflang4.plot(kind='barh', color=my_colors)
ax.set_xlabel('Language Occurrence')
ax.set_title('Top 10 non-English Languages in HealthReach Collection')


#Use 10 different colors not just 7
#my colors = ['r','g,'b','k','y','m','c','maroon','darkorange','hotpink']  

#Adding more than 7 colors isn't working
#Perhaps keeping it monochromatic is best anyway.

#Remove the legend
ax.get_legend().remove()<jupyter_output><empty_output><jupyter_text>**Reminder:** The numbers and language order in the bar graph here are **not** the same as the HealthReach data since the raw data differs for each. HealthReach data uses Titles and the Occurrence here is the occurrence of each language after being parsed into individual rows. Duplicate titles for languages exist in the raw data for this bar graph, so numbers here are likely to be higher than on the HealthReach website.<jupyter_code>#To make graphs interactive, import plotly but I get an error right now.

#import plotly.plotly as py
#import plotly.tools as tls<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07730356133408706, 'fraction_numerical': 0.019431882419446014, 'mean_word_length': 4.0063671736823485, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 13, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 23, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '399645', 'license': 'MIT', 'n_tokens_mistral': 4160, 'n_tokens_neox': 3872, 'n_words': 1736, 'path': 'NLM data_FINAL_plus_IHC.ipynb', 'repo_name': 'ihcihci/final-project-nlm', 'size': '14164'}