<jupyter_start><jupyter_text>A quick look into the flow of Human Blood in 2016 (Imports and Exports)by <NAME>, 6th of January 2018 This is a personal project mostly to practice Python3 Data Science skills. I will be looking at the extremes of the import and export of human blood during 2016.The full description of the commodity code relating to human blood is: "3002 - Human blood; animal blood for therapeutic, prophylactic or diagnostic uses; antisera, other blood fractions, immunological products, modified or obtained by biotechnologial processes, vaccines, toxins, cultures of micro-organisms (excluding yeasts) ect"<jupyter_code>"""This imports required libraries/packages for the data analysis. Warnings to "filter" the warnings displayed. Pandas is 
needed for the dataframes. matplotlib is for the graphs. numpy is needed for scientific computing."""
import warnings
warnings.simplefilter('ignore', FutureWarning)

from pandas import *
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Obtaining and preparing the data I obtained the data from the [United Nations Comtrade](https://comtrade.un.org/data/) website, by selecting the following configuration: - Type of Product: Goods - Frequency: Annual - Periods: All of 2016 - Reporter: All - Partners: World - Trade flows: All - HS (as reported) commodity codes: 3002 - Human bloodThe data was downloaded using the *Download CSV* button, and the file was named appropriately.<jupyter_code>#This global variable sets the location of the csv file to be imported.
LOCATION = 'comtrade_humanblood_world_2016.csv'

#This saves the data from a csv file in the location "LOCATION" as a dataframe called bloodTotal.
bloodTotal = read_csv(LOCATION)
#bloodtotal.head() shows the first 5 rows of the dataframe. (Only here to make sure the data is imported correctly/at all)
bloodTotal.head()<jupyter_output><empty_output><jupyter_text>After the import I have noticed there are a lot of columns that are not relevent to my research and some that could be named better as well.So here I will start cleaning up the dataframe.<jupyter_code>#These set up the global variables for the heading names in the dataframes.
YEAR = 'Year'
COUNTRY = 'Reporter'
FLOW = 'Trade Flow'
VALUE = 'Trade Value (US$)'
NETWEIGHT = 'Netweight (kg)'

#I set the headings variable to a list of the global variables above.
headings = [YEAR, FLOW, COUNTRY, VALUE, NETWEIGHT]
#Here I make it so the headings in the bloodTotal dataframe are only the ones I have saved to the headings variable.
bloodTotal = bloodTotal[headings]
#I change the 'Reporter' column name to 'Country'
bloodTotal = bloodTotal.rename(columns={'Reporter' : 'Country'})
#After I changed the 'Reporter' coulumn's name to 'Country' I change the global variable to that as well so I can use it later.
COUNTRY = 'Country'
bloodTotal.head()<jupyter_output><empty_output><jupyter_text>We can see that there are missing values in the Netweight (kg) column, I plan on just leaving it there to show what little extra data we have while not using it for analysis.I will however quickly run a check that none of the other columns have any missing data.<jupyter_code>#This checks everything in bloodTotal to see if it is null, and shows me the total in each column.
bloodTotal.isnull().sum()<jupyter_output><empty_output><jupyter_text>Great, the rest of the columns at least has a value. There are way to fix or change the data to deal with the null values, for example: - I could drop every row that has a null value in it. But this will cause us to lose 32 rows of data, and as we arent going to be doing anything wit the netweight data, other than show it where availiable, keeping all the rows in the datafame seems the best best. For the above I could just use "bloodTotal.dropna()" which will drop any row with at least one null value in it.- I could also fill every null with a value I set. Or possibly if it suited the dataset a mean of the above and below values in the column. The issue with this is you are creating data, its not real and can end up leading to some wrong conclusions.An example of the above is "bloodTotal['Netweight (kg)'] = bloodTotal['Netweight (kg)'].fillna(10)" - Which would replace all null values in the bloodTotal Netweight column with 10.For the best estimate I would think that working out how much each country paid per kg of blood, both import and export, then getting the mean of that per kg and dividing a countries trade value by that number. This could and most likely would still cause massive errors though as we are looking at all human blood trades, and some is bound to be more valuable that others, so we will never be exactly correct.  Aims and ObjectivesAs mentioned before I will be looking at the extremes of this data for the purpose of satisfying my curiosity. For the same reasons I will take a quick look at the data of some countries that interest me.So in the objectives are:1. Find the total value of all Human Blood exports.- Find the total value of all Human Blood imports.- Find the top 5 importers of Human Blood. (Display in simple graph).- Find the top 5 exporters of Human Blood. (Display in simple graph).- Find the 5 lowest importers of Human Blood. (Display in simple graph).- Find the 5 lowest exporters of Human Blood. (Display in simple graph).- Find the top 5 importers AND exporters of Human Blood. (Display in simple graph).- Find the lowest 5 importers AND exporters of Human Blood. (Display in simple graph).- Find the data on the United Kingdom, USA and China. (Display in a pivot table).  Making import and export only dataframesWe will first need to make some additional dataframes to make completing the above objectives easier.<jupyter_code>"""Here we are grouping the data based on the value in the 'Trade Flow' column. This allows us to save every row of data with
'Import' in that coulumn to its own group. Same for 'Export'. We are then assigning it to its own dataframe"""
grouped = bloodTotal.groupby('Trade Flow')
bloodImport = grouped.get_group('Import')
bloodExport = grouped.get_group('Export')

"""Below I reset the index of the new dataframes, as they would otherwise still be using the old dataframes index. The 
inplace=True means to apply the reset to the dataframe rather than make a copy."""
bloodImport.reset_index(inplace=True)
bloodExport.reset_index(inplace=True)<jupyter_output><empty_output><jupyter_text>Objective 1 - Find the total value (in US$) of all Human Blood Exports<jupyter_code>#I sum up the total of the bloodExport trade value column.
bloodExport[VALUE].sum()<jupyter_output><empty_output><jupyter_text>The total value of the worlds Human Blood exports is $130,032,923,258  Objective 2 - Find the total value (in US$) of all Human Blood Imports<jupyter_code>#I sum up the total of the bloodImport trade value column.
bloodImport[VALUE].sum()<jupyter_output><empty_output><jupyter_text>The total value of the worlds Human Blood imports is $131,519,813,186The slight difference, of nearly $1.5 billion, is due to not working with the re-export or re-import data.  Objective 3 - Find the top 5 importers of Human Blood (Display in graph)<jupyter_code>#Here I make a new dataframe containing only the top 5 values, after sorting obviously, from the "main" bloodImport dataframe.
bloodImportTopFive = bloodImport.sort_values('Trade Value (US$)', ascending=False).reset_index(drop=True).head()
#This is to delete the extra, and irrelevant, index column that gets added to the new dataframe.
del bloodImportTopFive['index']
bloodImportTopFive.head()<jupyter_output><empty_output><jupyter_text>The table below shows the top 5 importers of Human Blood.<jupyter_code>#Basic graph plotting - Set the X and Y values, what type of graph, to show the grid, and then the size
bloodImportTopFive.plot(x='Country', y='Trade Value (US$)', kind='bar', grid=True, figsize=(10,4))<jupyter_output><empty_output><jupyter_text>Objective 4 - Find the top 5 exporters of Human Blood (Display in graph) The table below shows the top 5 exporters of Human Blood.<jupyter_code>#All of this is the objective 3 code but for the exports not imports.
bloodExportTopFive = bloodExport.sort_values('Trade Value (US$)', ascending=False).reset_index(drop=True).head()
del bloodExportTopFive['index']
bloodExportTopFive.head()
#Basic graph plotting
bloodExportTopFive.plot(x='Country', y='Trade Value (US$)', kind='bar', grid=True, figsize=(10,4))<jupyter_output><empty_output><jupyter_text>Objective 5 - Find the lowest 5 importers of Human Blood (Display in graph)<jupyter_code>#Same code as objective 3, just reversed the ascending.
bloodImportBottomFive = bloodImport.sort_values('Trade Value (US$)', ascending=True).reset_index(drop=True).head()
del bloodImportBottomFive['index']
bloodImportBottomFive.head()<jupyter_output><empty_output><jupyter_text>Interesting extra information:Kiribati, officially the Republic of Kiribati, is a small nation in the centeral pacific ocean. It has a permanent population of just over 100,000 (acording to statistics from 2015), which is spread out over 33 atolls and one raised coral island. (More that half of the population live on Tarawa Atoll). The total land area is around 800 square kilometers (310sq mi), and the islands are dispersed over 3.5 million square kilometers (1.3 million square miles).The Republic of Kiribati became independant from the United Kingdom in 1979. The capital and most populated area is South Tarawa. The country is a member of the Commonwealths of Nations, the IMF and the World Bank, and became a full member of the United Nations in 1999.[1][1] [Wikipedia](https://en.wikipedia.org/wiki/Kiribati) - Sourced 7th January 2018<jupyter_code>#Basic graph plotting
bloodImportBottomFive.plot(x='Country', y='Trade Value (US$)', kind='bar', grid=True, figsize=(10,4))<jupyter_output><empty_output><jupyter_text>Objective 6 - Find the lowest 5 exporters of Human Blood (Display in graph)<jupyter_code>#Same code as objective 3, just reversed the ascending.
bloodExportBottomFive = bloodExport.sort_values('Trade Value (US$)', ascending=True).reset_index(drop=True).head()
del bloodExportBottomFive['index']
bloodExportBottomFive.head()
#Basic graph plotting
bloodExportBottomFive.plot(x='Country', y='Trade Value (US$)', kind='bar', grid=True, figsize=(10,4))<jupyter_output><empty_output><jupyter_text>Objective 7 - Find the top 5 importers and exporters of Human Blood (Display in graph)<jupyter_code>"""Here I use grouping to add the countries total imports, exports, re-imports and re-exports together. I save the results to a
dataframes called bloodImportAndExport"""
grouped = bloodTotal.groupby('Country')
bloodImportAndExport = grouped.aggregate(sum)
#As Ive summed up all the iformation on each country into one row the year value gets added as well. We dont need it so I
#remove it.
del bloodImportAndExport['Year']
#Here I do the usual sort, reset index, save only the head (top 5) to a new dataframe.
bloodImportAndExportTopFive = bloodImportAndExport.sort_values('Trade Value (US$)', ascending=False).head().reset_index(drop=False)
bloodImportAndExportTopFive.head()
#Basic graph plotting
bloodImportAndExportTopFive.plot(x='Country', y='Trade Value (US$)', kind='bar', grid=True, figsize=(10,4))<jupyter_output><empty_output><jupyter_text>Objective 8 - Find the lowest 5 importers and exporters of Human Blood (Display in graph)<jupyter_code>#Here I do the usual sort, reset index, save only the head (top 5) to a new dataframe.
bloodImportAndExportBottomFive = bloodImportAndExport.sort_values('Trade Value (US$)', ascending=True).head().reset_index(drop=False)
bloodImportAndExportBottomFive.head()
#Basic graph plotting
bloodImportAndExportBottomFive.plot(x='Country', y='Trade Value (US$)', kind='bar', grid=True, figsize=(10,4))<jupyter_output><empty_output><jupyter_text>Objective 9 - Find the data on the United Kingdom, USA and China. Display in a pivot table.<jupyter_code>"""Here I pull out all rows of data from the "main" dataframe,(bloodTotal where all the raw data was imported and cleaned up), 
that are the countries needed for my pivot table."""
bloodObjectiveNine = bloodTotal[(bloodTotal['Country'] == "United Kingdom") | (bloodTotal['Country'] == "USA") | 
                     (bloodTotal['Country'] == "China")]
#I used .head(20) to see how many results we actually have in the table. The number in the () sets how many to show.
bloodObjectiveNine.head(20)
"""
This is the layout of the pivot table. We use the dataframe that contains only the countries we are intereted in - 
bloodObjectiveNine. We set the index to the country so we can read all values and netweight of one countries data across all 
the columns.

The fill_value=0 makes it so all the null/NaN values are replaced with a 0. This makes it much more readable. It could 
cause problems in some datasets though, as giving something a value might cause errors or people coming to the wrong 
conclusions
"""
pivot_table(bloodObjectiveNine,
           index=COUNTRY,
           values=[VALUE, NETWEIGHT],
           columns=FLOW,
           fill_value=0,
           )<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07602516486015311, 'fraction_numerical': 0.011748654589555067, 'mean_word_length': 5.355491329479769, 'pattern_counts': {'":': 0, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '96705', 'license': 'MIT', 'n_tokens_mistral': 3842, 'n_tokens_neox': 3553, 'n_words': 1732, 'path': 'Project 1 - Human Blood/Human Blood V1.ipynb', 'repo_name': 'DanielPDWalker/Human-Blood-Trade-2016', 'size': '13200'}