<jupyter_start><jupyter_text>Matplotlib-The Power of Plots by <NAME>  Observations and Insights<jupyter_code># Dependencies and Setup
import matplotlib.pyplot as plt
import pandas as pd
import scipy.stats as st
import numpy as np

# Study data files
mouse_metadata_path = "data/Mouse_metadata.csv"
study_results_path = "data/Study_results.csv"

# Read the mouse data and the study results
mouse_metadata = pd.read_csv(mouse_metadata_path)
study_results = pd.read_csv(study_results_path)<jupyter_output><empty_output><jupyter_text>Let's have a look at the **mouse_metadata**<jupyter_code>mouse_metadata.head()<jupyter_output><empty_output><jupyter_text>Let's have a look at **study_results**<jupyter_code>study_results.head()<jupyter_output><empty_output><jupyter_text>Now let's combine the 2 data frames into 1 single dataframe<jupyter_code>merge_df_unclean = pd.merge(mouse_metadata, study_results, on = 'Mouse ID', how = 'inner')
merge_df_unclean.head()<jupyter_output><empty_output><jupyter_text>Now let's check the number of mice in this dataframe. Since there are multiple records of the same mice for different *Timepoint*, a simple len function on the whole dataset would give us incorrect result. Hence, we need to group them first by *Mouse ID* and then find its length<jupyter_code>grouped_df = merge_df_unclean.groupby(['Mouse ID'])
len(grouped_df)<jupyter_output><empty_output><jupyter_text>In order to get duplicates, we can run duplicated function on the merged dataframe. This will give us all the data for the duplicate *Mouse ID*<jupyter_code>merge_df_duplicate = merge_df_unclean[merge_df_unclean.duplicated()]
merge_df_duplicate<jupyter_output><empty_output><jupyter_text>In order to clean the dataframe and drop the duplicate row, we can use drop_duplicates method on **merge_df_duplicate** dataframe<jupyter_code>merge_df = merge_df_unclean.drop_duplicates()
merge_df.head()<jupyter_output><empty_output><jupyter_text>In order to check the number of mice in the clean dataframe and if our dataset is indeed clean, we need to first find the total number on mice in unclean dataframe and then in the crean dataframe<jupyter_code>print(f"Number of mice in unclean dataframe is {len(merge_df_unclean)}, and in the cleaned dataframe is {len(merge_df)}.")<jupyter_output>Number of mice in unclean dataframe is 1893, and in the cleaned dataframe is 1892.<jupyter_text>Summary Statistics Let's first get the names of *Drug Regimen* and store them in an array. This array will be used several time in our code<jupyter_code>drug_unique = merge_df['Drug Regimen'].unique()
drug_unique<jupyter_output><empty_output><jupyter_text>In order to get Mean, Median, Variance, Dtandard Deviation and SEM for each Drug Regimen, we can use a for loop that will take a item form the **drug_unique** array, compare it to merge_df's Drug Regimen name extrace all values realated to that name and perform those mathematical operations.Let' first create lists which will hold the data that will be generated by our for loop. We will then appennd the values to these lists and then convert it all to a dataframe<jupyter_code>drug_name_ar = []
mean_ar = []
median_ar = []
var_ar = []
standard_ar = []
sem_ar = []
for drug in drug_unique:
    drug_name = merge_df.loc[merge_df['Drug Regimen'] == drug, :]
    mean_ar_1 = drug_name['Tumor Volume (mm3)'].mean()
    median_ar_1 = drug_name['Tumor Volume (mm3)'].median()
    var_ar_1 = drug_name['Tumor Volume (mm3)'].var()
    standard_ar_1 = drug_name['Tumor Volume (mm3)'].std()
    sem_ar_1 = drug_name['Tumor Volume (mm3)'].sem()
    drug_name_ar.append(drug)
    mean_ar.append(mean_ar_1)
    median_ar.append(median_ar_1)
    var_ar.append(var_ar_1)
    standard_ar.append(standard_ar_1)
    sem_ar.append(sem_ar_1)
data_summary = [drug_name_ar, mean_ar, median_ar, var_ar, standard_ar, sem_ar]
data_summary_df = pd.DataFrame(data_summary)
data_summary_df = data_summary_df.transpose()
data_summary_df.columns = ['Drug Regimen', 'Mean', 'Median', 'Variance', 'Standard Deviation', 'SEM']
data_summary_df = data_summary_df.sort_values('Drug Regimen', ascending=True)
data_summary_df = data_summary_df.set_index('Drug Regimen')
data_summary_df<jupyter_output><empty_output><jupyter_text>For this method, we can use a groupby function on *Drug Regimen* and then apply the mathematical operations on *Tumor Volume (mm3)*<jupyter_code>mean_1 = merge_df.groupby('Drug Regimen').mean()['Tumor Volume (mm3)']
median_1 = merge_df.groupby('Drug Regimen').median()['Tumor Volume (mm3)']
variance_1 = merge_df.groupby('Drug Regimen').var()['Tumor Volume (mm3)']
standard_1 = merge_df.groupby('Drug Regimen').std()['Tumor Volume (mm3)']
sem_1 = merge_df.groupby('Drug Regimen').sem()['Tumor Volume (mm3)']

data_summary_df_2a = pd.DataFrame(mean_1)
data_summary_df_2 = data_summary_df_2a.rename(columns={"Tumor Volume (mm3)": "Mean"})

data_summary_df_2["Median"] = median_1
data_summary_df_2["Variance"] = variance_1
data_summary_df_2["Standard Deviation"] = standard_1
data_summary_df_2["SEM"] = sem_1

data_summary_df_2<jupyter_output><empty_output><jupyter_text>Bar Plots Now, we need to plot this data using Bar Plots and pandas<jupyter_code>x_axis_data_1 = merge_df.groupby('Drug Regimen').count()['Mouse ID']
x_axis_data = pd.DataFrame(x_axis_data_1)
x_axis_data.plot.bar(rot=90)
plt.ylabel("Number of Mice")
plt.xlabel("Drug Regimen")
plt.title("Number of mice per time point for each treatment")
plt.show()<jupyter_output><empty_output><jupyter_text>Here, we need to plot this data using Bar Plots and pyplot<jupyter_code>x_axis = np.arange(len(x_axis_data_1))
tick_locations = [x for x in x_axis]
plt.figure(figsize=(6,4))
bar_2 = x_axis_data.reset_index()
plt.bar(x_axis, x_axis_data['Mouse ID'], align="center")
plt.xticks(tick_locations, bar_2['Drug Regimen'],rotation="vertical")
plt.xlim(-0.75, len(x_axis_data_1)-.25)
plt.ylim(0, 250)
plt.title("Number of mice per time point for each treatment")
plt.xlabel("Drug Regimen")
plt.ylabel("Number of Mice")
plt.show()<jupyter_output><empty_output><jupyter_text>Pie Plots In order to find gender distribution, we first need to group the mice by their Sex<jupyter_code>gender = mouse_metadata.groupby('Sex').count()
gender<jupyter_output><empty_output><jupyter_text>We can now plot the data using Pie Plot<jupyter_code>colors = ["gold", "lightskyblue"]
plt.pie(gender['Mouse ID'], labels=list(gender.index), colors=colors, autopct="%1.1f%%", shadow=True, startangle=180)
plt.title('Male vs Female Mouse Population')
plt.ylabel('Sex')
plt.show()<jupyter_output><empty_output><jupyter_text>Here, we have to plot the same Pie Plot but using pyplot this time<jupyter_code>colors = ["gold", "lightskyblue"]
secondpie = gender["Mouse ID"].plot.pie(y='sizes', figsize=(4, 4), autopct="%1.1f%%", startangle=180, shadow=True,colors=colors)
plt.title('Male vs Female Mouse Population')
plt.ylabel('Sex')
plt.show()<jupyter_output><empty_output><jupyter_text>Quartiles, Outliers and Boxplots In order to calculate IQR and potential outliers, let's create a function. This function will take the drug name is input, and calculate IQR and potential outliers. It will then print the results for the *Drug Regimen*<jupyter_code># Function
def timepoint_calc(drug_name):
    drug_df_1c = merge_df.loc[merge_df["Drug Regimen"] == drug_name, :]
    drug_df_2c = drug_df_1c.groupby('Mouse ID').max()['Timepoint']
    drug_df_3c = pd.DataFrame(drug_df_2c)
    drug_df_3c_merge = pd.merge(drug_df_3c, merge_df, on=("Mouse ID","Timepoint"),how="left")

    tumors = drug_df_3c_merge["Tumor Volume (mm3)"]

    quartiles = tumors.quantile([.25,.5,.75])
    lowerq = quartiles[0.25]
    upperq = quartiles[0.75]
    iqr = (upperq-lowerq)

    lower_bound = lowerq - (1.5*iqr)
    upper_bound = upperq + (1.5*iqr)
    print(f"{drug_name}'s potential outliers could be values below {lower_bound} and values above {upper_bound}, and it's IQR is {iqr}.")

for drug_name in drug_unique:
    print(timepoint_calc(drug_name))<jupyter_output>Ramicane's potential outliers could be values below 17.912664470000003 and values above 54.30681135, and it's IQR is 9.098536719999998.
None
Capomulin's potential outliers could be values below 20.70456164999999 and values above 51.83201549, and it's IQR is 7.781863460000004.
None
Infubinol's potential outliers could be values below 36.83290494999999 and values above 82.74144559000001, and it's IQR is 11.477135160000003.
None
Placebo's potential outliers could be values below 30.15582196999999 and values above 90.92136789000001, and it's IQR is 15.191386480000006.
None
Ceftamin's potential outliers could be values below 25.355449580000002 and values above 87.66645829999999, and it's IQR is 15.577752179999997.
None
Stelasyn's potential outliers could be values below 27.53557243999998 and values above 94.04496758000002, and it's IQR is 16.62734878500001.
None
Zoniferol's potential outliers could be values below 24.77952216500001 and values above 92.00293628499998, and it's IQR is 16.8058[...]<jupyter_text>In order to find four of the most promising treatment regimens, we need to find the decrease/increase in tumor volume per drug. To get that, we need to know the size of each tumor at the end of recorded *Timepoint*<jupyter_code>best_drug_1 = merge_df.groupby('Mouse ID').max()['Timepoint']
best_drug_2 = pd.DataFrame(best_drug_1)
best_drug_2.head()<jupyter_output><empty_output><jupyter_text>Now that we have the above information, we can merge the above dataframe with merge_df using left joint. We'll also sort the values by *Tumor Volume (mm3)*<jupyter_code>best_drug_3 = pd.merge(best_drug_2, merge_df, on=("Mouse ID","Timepoint"),how="left")
best_drug_3 = best_drug_3.sort_values('Tumor Volume (mm3)')
best_drug_3.head()<jupyter_output><empty_output><jupyter_text>Now we need to find the best 4 performing drugs. We can write a for loop that will take 1 element in drug_unique (which we created above), compare that to the best_drug_3 dataframe that we just created, find the minimum value of tumor volume and save the name of the corrosponding drug and it's tumor volume in a dataframe<jupyter_code>drug_name_a = []
tumor_volume = []
for b in drug_unique:
    best_drug_4 = best_drug_3.loc[best_drug_3['Drug Regimen'] == b, :]
    best_drug_5 = best_drug_4['Tumor Volume (mm3)'].min()
    drug_name_a.append(b)
    tumor_volume.append(best_drug_5)
best_drug_6 = pd.DataFrame(tumor_volume, drug_name_a)
best_drug_6.reset_index(inplace=True)
index = ['Drug Name', 'Tumor Volume (mm3)']
best_drug_6.columns = index
best_drug_6 = best_drug_6.sort_values(['Tumor Volume (mm3)', 'Drug Name'], ascending=True)
best_drug_6<jupyter_output><empty_output><jupyter_text>In order to get the top 4 drugs, we can use .head(4) on Drug Name column of best_drug_6 dataframe<jupyter_code>best_drug_7 = best_drug_6['Drug Name'].head(4)
best_drug_7<jupyter_output><empty_output><jupyter_text>Now, let's make a function that will again take the drug name is input, find the max timepoint related to that drug, merge it with the original dataframe using left joint and return a dataframe for all the mice treated with that drug and final tumor volume.<jupyter_code># Function
def timepoint_calc_1(drug_name):
    drug_df_new_1 = merge_df.loc[merge_df["Drug Regimen"] == drug_name, :]
    drug_df_new_1 = drug_df_new_1.groupby('Mouse ID').max()['Timepoint']
    drug_df_new_2 = pd.DataFrame(drug_df_new_1)
    drug_df_new = pd.merge(drug_df_new_2, merge_df, on=("Mouse ID","Timepoint"),how="left")
    tumors = drug_df_new["Tumor Volume (mm3)"]
    return(drug_df_new)<jupyter_output><empty_output><jupyter_text>We now need to have the column Tumor Volume (mm3) for each of the best 4 performing drug<jupyter_code>tumors1a = timepoint_calc_1(best_drug_7.iloc[0])
tumors1 = tumors1a['Tumor Volume (mm3)']

tumors2a = timepoint_calc_1(best_drug_7.iloc[1])
tumors2 = tumors2a['Tumor Volume (mm3)']

tumors3a = timepoint_calc_1(best_drug_7.iloc[2])
tumors3 = tumors3a['Tumor Volume (mm3)']

tumors4a = timepoint_calc_1(best_drug_7.iloc[3])
tumors4 = tumors4a['Tumor Volume (mm3)']<jupyter_output><empty_output><jupyter_text>We can now plot our boxplot using all this information<jupyter_code>fig1, ax1 = plt.subplots()
ax1.set_title('Tumors')
ax1.set_ylabel('Final Tumor Volume (mm3)')
ax1.set_xlabel('Drug Regimen')

data_to_plot = [tumors1, tumors2, tumors3, tumors4]

ax1.boxplot(data_to_plot, labels=[best_drug_7.iloc[0],best_drug_7.iloc[1],best_drug_7.iloc[2],best_drug_7.iloc[3]])
plt.show()<jupyter_output><empty_output><jupyter_text>Line and Scatter Plots Let's find the *Mouse ID* for the **Capomulin** drug which had the best results<jupyter_code>best_drug_3d = best_drug_3.loc[best_drug_3['Drug Regimen'] == 'Capomulin', :]
best_mouse_id = best_drug_3d['Mouse ID'].iloc[0]
best_mouse_id<jupyter_output><empty_output><jupyter_text>We can use **timepoint_calc_1** function that we created above to get all the data for the choosen *Mouse ID* and Drug<jupyter_code>value_1 = timepoint_calc_1('Capomulin')
best_result_df = merge_df.loc[merge_df["Mouse ID"] == best_mouse_id,:]
best_result_df.head()<jupyter_output><empty_output><jupyter_text>Let us plot this using Line Plot<jupyter_code>x_axis_lp = best_result_df["Timepoint"]
lp_df = best_result_df["Tumor Volume (mm3)"]

plt.title(f"Line plot for Capomulin treatmeant of mouse {best_mouse_id}.")
plt.plot(x_axis_lp, lp_df, linewidth=2, markersize=12)
plt.xlabel('Timepoint (Days)')
plt.ylabel('Tumor Volume (mm3)')
plt.show()<jupyter_output><empty_output><jupyter_text>Let's plot a scatter plot<jupyter_code>sp_mean = value_1.groupby(['Mouse ID']).mean()
plt.scatter(sp_mean['Weight (g)'],sp_mean['Tumor Volume (mm3)'])
plt.xlabel('Weight (g)')
plt.ylabel('Average Tumor Volume (mm3)')
plt.title(f"Scatter plot for Capomulin treatmeant of mouse {best_mouse_id}.")
plt.show()<jupyter_output><empty_output><jupyter_text>Correlation and Regression Let's calculate Pearson's coefficent, slope, intercept, rvalue, pvalue, stderr, regress_values and plot the results<jupyter_code>rpearson_coefficent = st.pearsonr(value_1['Weight (g)'], value_1['Tumor Volume (mm3)'])[0]
slope, intercept, rvalue, pvalue, stderr = st.linregress(value_1['Weight (g)'],value_1['Tumor Volume (mm3)'])
regress_values = slope * ((value_1['Weight (g)']) + intercept)

plt.scatter(sp_mean['Weight (g)'],sp_mean['Tumor Volume (mm3)'])
plt.plot(sp_mean['Weight (g)'], regress_values, "-r")
plt.xlabel('Weight (g)')
plt.ylabel('Average Tumore Volume (mm3)')
plt.title('Linear Regression Model')
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11220986128279083, 'fraction_numerical': 0.03728883395138031, 'mean_word_length': 6.975355969331873, 'pattern_counts': {'":': 1, '<': 116, '<?xml version=': 0, '>': 116, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '632354', 'license': 'ADSL', 'n_tokens_mistral': 5664, 'n_tokens_neox': 5130, 'n_words': 1337, 'path': 'Matplotlib-The Power of Plots.ipynb', 'repo_name': 'sushant822/Assignment-5', 'size': '14573'}