<jupyter_start><jupyter_text>Introduction to Python Programming for Medical Researchers, University of Bern, <NAME>  2. Example of a small medical data analysis with Python(Use 45 minutes to go through this notebook on colab)  In this notebook we will step by step see how to do a simple data analysis with Python. We will see how to* read in in comma separated value data (text or excel files)* use Dataframes for preprocessing and cleaning of data* use Dataframes for descriptive statistics* use Dataframes for plotting* use Dataframes for hypothesis testing* save data and results to a fileYou are not supposed to understand everything right away. It is an **introductory tutorial** which intends to show how a medical researcher would work and use Python for data analysis.In the next session we will then cover basic Python.  DocumentationWhen working with Python, one always has to consult the online documentaion and google examples and answers, often given on stackoverflow. For this notebook we need the [online documentation](https://pandas.pydata.org/docs/reference/frame.html) for the Pandas library.------  Read in a small medical dataset with blood pressure measurementsSome colleagues provided this blood pressure [dataset](https://drive.google.com/file/d/1d8x5ZSwa9Yn-8J8NxiygFDaoMO8PRNWc/view?usp=sharing). Follow the link and take a look. If you don't understand what the columns mean, you can study this [webpage](https://www.heart.org/en/health-topics/high-blood-pressure/understanding-blood-pressure-readings). In Python not all software libraries, called modules, are in memory. So we need to import those we need for our data analysis.<jupyter_code># The data analysis module containing DataFrames
import pandas as pd
# The .....

# Create a variable with the download link of the data file on google drive
# and create a dataframe varible (object) into which we read the data
url  = 'https://drive.google.com/uc?export=download&id=1d8x5ZSwa9Yn-8J8NxiygFDaoMO8PRNWc'
data = pd.read_csv(url)
data

pd.set_option('max_rows',None)
#data
pd.set_option('max_rows',20)
data.info()
data.describe()
round(data,2)<jupyter_output><empty_output><jupyter_text>Data CleaningWe seem to have a problem with the blood pressure columns. They contain rows with NULL. Pandas dataframes have ways to handle that.<jupyter_code>clean_data = data.replace('#NULL!',float(0))
clean_data['BP_SYS'] = pd.to_numeric(clean_data['BP_SYS'])
clean_data['BP_DIA'] = pd.to_numeric(clean_data['BP_DIA'])
clean_data['BP_SYS'] = clean_data['BP_SYS'].replace(float(0),clean_data['BP_SYS'].mean())
clean_data['BP_DIA'] = clean_data['BP_DIA'].replace(float(0),clean_data['BP_DIA'].mean())
clean_data<jupyter_output><empty_output><jupyter_text>PlottingNow that we can study the summary statistics we also want to study the distributions.<jupyter_code># Plot all histograms fast
import matplotlib.pyplot as plt
clean_data.hist()
# Make one plot a bit nicer
plt.figure()
ax = clean_data['Age'].plot(kind='hist',fill=False,histtype='step',color='b',label='Age')
ax.set_xlabel('Age / yr')
ax.set_ylabel('Count')
ax.set_title('Age Distribution')
plt.legend()
plt.show()
# Plot all correlations fast
from pandas.plotting import scatter_matrix
scatter_matrix(clean_data, alpha=0.2, figsize=(8, 8), diagonal='hist')
plt.show()
# Plot the boxplots by sex
color = dict(boxes='DarkGreen', whiskers='DarkOrange',
             medians='DarkBlue', caps='Gray')
clean_data.groupby('Sex').plot.box(color=color)<jupyter_output><empty_output><jupyter_text>Descriptive Statistics<jupyter_code>round(clean_data.describe(),2)
# Histogram distributions and correlations and box plots as above<jupyter_output><empty_output><jupyter_text>Inferential StatisticsNormality test, test if female and males are different?<jupyter_code>from scipy import stats
male_data=clean_data[clean_data['Sex'] == 1.0]
female_data=clean_data[clean_data['Sex'] == 0.0]
male_data.hist()
# Check for normality of the distributions
k2, p = stats.normaltest(male_data['BP_SYS']) # D Agostino-Pearson test
print("p-value = %1.2f" % p)
k2, p = stats.normaltest(male_data['BP_DIA']) # D Agostino-Pearson test
print("p-value = %1.2f" % p)
k2, p = stats.normaltest(female_data['BP_SYS']) # D Agostino-Pearson test
print("p-value = %1.2f" % p)
k2, p = stats.normaltest(female_data['BP_DIA']) # D Agostino-Pearson test
print("p-value = %1.2f" % p)

# Check if male and females are different
print(stats.mannwhitneyu(male_data['BP_SYS'],female_data['BP_SYS']))
stats.mannwhitneyu(male_data['BP_DIA'],female_data['BP_DIA'])<jupyter_output>MannwhitneyuResult(statistic=884.0, pvalue=0.006147669597082928)<jupyter_text>Saving ResultsSave raw and cleaned data.<jupyter_code>data.to_csv('my_rawdata.csv')
clean_data.to_csv('my_cleandata.csv')
!ls -l<jupyter_output>total 12
-rw-r--r-- 1 root root 2272 Sep  1 17:20 my_cleandata.csv
-rw-r--r-- 1 root root 2014 Sep  1 17:20 my_rawdata.csv
drwxr-xr-x 1 root root 4096 Aug 25 13:35 sample_data
{'fraction_non_alphanumeric': 0.11187260632936907, 'fraction_numerical': 0.020157226365652085, 'mean_word_length': 6.275659824046921, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '68394', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1697, 'n_tokens_neox': 1585, 'n_words': 483, 'path': 'Solutions/02-Example-1-Medical-Analysis-Sol.ipynb', 'repo_name': 'sigvehaug/Introduction-to-Python-Programming-For-Medical-Researchers', 'size': '4965'}