<jupyter_start><jupyter_text>Model & Synthesize DataThe purpose of this Jupyter Notebook is to complete the task of modeling and synthesizing a data set.  To choose a real-world phenomenon that can be measured and for which you can collect at least one-hundred data points across at least four different variables. Idea GenerationMy focus for this project is in relation to the Vermont Oxford Network.  Vermont Oxford Network (VON) is a nonprofit voluntary collaboration of health care professionals working together as an interdisciplinary community to change the landscape of neonatal care. Founded in 1988, VON is now comprised of teams of health professionals representing neonatal intensive care units and level I and II care centers around the world, in support of the mission to improve the quality and safety of medical care for newborn infants and their families through a coordinated program of research, education, and quality improvement projects. (ref: https://public.vtoxford.org/about-us/)The National Perinatal Epidemiology Centre (NPEC) is based in the UCC Anu Research Centre in Cork University Maternity Hospital. The overall objective of the Centre is to collaborate with Irish maternity services to translate clinical audit data and epidemiological evidence into improved maternity care for families in Ireland. NPEC collect data and contribute to the Vermont Oxford Network.  The National Perinatal Epidemiology Centre, UCC, published a report in 2018 "Mortality Risk Amongst Very Low Birth Weight Infants Born in the Republic of Ireland" (ref: https://www.ucc.ie/en/media/research/nationalperinatalepidemiologycentre/annualreports/MortalityRiskAmongstVeryLowBirthWeightInfantsintheRepublicofIrelandReport2014-2016.pdf).In 2014-2016, 19 neonatal units submitted data to the VON’s VLBW database. The inclusion criteria were as follows:'Any infant who is born alive at your hospital and whose birth weight is between 401 and 1500 grams OR whose gestational age is between 22 weeks 0 days and 29 weeks 6 days (inclusive), regardless of where in your hospital the infant receives care'This notebook aims to sythesise data to reflect the key variables & findings in this report  The Summary ApproachIn order to complete the task of synthesizing data which assesses the mortality risk amongst very low birth weight babies, I have decided to take the following approach: - Explain the Variables, their likely distributions, and their relationships with each other. - Generate a Data Frame Structure - Create the DF variables - Analyise & plot the dataset and highlight key findings  The Variables - Explained Gestational WeeksGestational age is the common term used during pregnancy to describe how far along the pregnancy is. It is measured in weeks, from the first day of the woman's last menstrual cycle to the current date. A normal pregnancy can range from 38 to 42 weeks. Infants born before 37 weeks are considered premature. (ref:https://medlineplus.gov/ency/article/002367.htm)VON criteria states that VLBW baby meets the criteria if the gestational age is between 22 and 30 weeks.  As per the NPEC report, the likely outcome of this variable will be between 21 - 33 weeks. It is possible that the VLBW baby may be low in birthweight but high in Gestational age.  In this dataset the Gestational weeks has a spread from 21 weeks to 33 weeks. With the majority of babies born around weeks 28, 29, 30. Based on this, I think it is best to generate an array from a normal distribution. BirthweightBirth weight is the first weight of your baby, taken just after he or she is born. A low birth weight is less than 2,640 grams. A high birth weight is more than 4,000 grams. The expected result of this variable will see most weights less than 2640 grams.  Again please note, It is possible that the VLBW baby may be high in birthweight but low in Gestational age.  In the NPEC report, the gestational ages of the VLBW (Very Low Birth Weight) infants ranged from 21 weeks to 33 weeks and their birth weights ranged from 360g to 2,640g(ref:https://www.ucc.ie/en/media/research/nationalperinatalepidemiologycentre/annualreports/MortalityRiskAmongstVeryLowBirthWeightInfantsintheRepublicofIrelandReport2014-2016.pdf)The birth weights of very low birth weight infants ranged from 360g to 2,640g. The data here forms a uniform distribution. Infants are born at many different birth weights. No specific weight more than any other weight. All pregnancies are different. Congenital Anomaly Congenital anomalies are also known as birth defects, congenital disorders or congenital malformations. If the infant was born with a congenital anomaly, it was recored. About 9% of babies born in Ireland have a congenital anomaly. This figure is high in comparison to VON data as abortion was/is not in Ireland. Does the baby have a congenital anomaly? Yes or No? Disposition at 1 YearDid the infant live or die.  The status of the infant is recorded one year after birth.  There are 4 possible outcomes:- Died: Unfortunately self explanatory  - Home: The infant has been discharged from hospital and is at home- Still Hospitalised: The infant is still under medical care in hospital- Unkown: no contact with the infant's family so status of infant is unknown   Relationships of the Variables Gestational Weeks and DeathOne of the key findings of the report, Mortality Risk Amongst Very Low Birth Weight Infants Born in the Republic of Ireland, is that the Gestational weeks of the VLBW infant directly effects the chance at lifehttps://www.ucc.ie/en/media/research/nationalperinatalepidemiologycentre/annualreports/MortalityRiskAmongstVeryLowBirthWeightInfantsintheRepublicofIrelandReport2014-2016.pdfThe below table depicts the key finding of the report.  Based on Gestational weeks, the chance of death, within one year of birth. ![Gestational%20Age.Chance%20of%20Death.PNG](attachment:Gestational%20Age.Chance%20of%20Death.PNG)  Gestational Weeks and BirthweightIf Gestational weeks is above 23 weeks, then the birthweight needs to be below 1500g for the infant to be eligible for this report.If the Birthwegith is above 1500g, then the Gestational weeks needs to be below 30.  How to create a Dataset<jupyter_code># create a dataframe structure by starting with the column headings
import pandas as pd #use pandas to create the dataframe
import numpy as np # use numpy to generate the random variables


df = pd.DataFrame(columns= ["Gestational Weeks", "Birthweight", "Sex", "Congenital Anomaly", "Disposition at 1 Year"]) #gives headings to each of the columns
df # create the dataframe<jupyter_output><empty_output><jupyter_text>The code has generated a dataframe with the headings I require<jupyter_code># Testing to populate the data 
data = [['Alex',10, 6, 2, 5],['Bob',12, 6 ,2, 5],['Clarke',13, 6 ,2, 5]]
df = pd.DataFrame(data, columns = ['Gestational Weeks', 'Birthweight', 'Sex', 'Congenital Anomaly', 'Disposition at 1 Year'])
df<jupyter_output><empty_output><jupyter_text>The above table is the basic structure which I will use as the foundations for systhesizing my dataset.The key learning here is how to structure code in order to populate the rows  Gestational WeeksIn this dataset the Gestational weeks has a spread from 21 weeks to 33 weeks. With the majority of babies born around weeks 28 and 29.  Based on this, I think it is best to generate an array from a normal distribution. Below is table showing the actual gestational weeks of VLBW babies in the report.https://www.ucc.ie/en/media/research/nationalperinatalepidemiologycentre/annualreports/MortalityRiskAmongstVeryLowBirthWeightInfantsintheRepublicofIrelandReport2014-2016.pdf![Gestational%20Age.PNG](attachment:Gestational%20Age.PNG)<jupyter_code># Create a Bar chart of data to highlight key mean and distribution
import matplotlib.pyplot as plt # importing matplotlib to plot the data
y = [6, 53, 77, 112, 124, 115, 145, 252, 268, 197, 166, 109, 141]
x = [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]
width = 1/1.5 # put a space between the bars
plt.bar(x, y, width, color="blue")
plt.ylabel("No of Infants") # name the y label
plt.xlabel("Gestational Age") # name the x label
plt.title("Gestational Age of Infant") # give the plot a title
plt.show()
#The above bar chart helps to set Parameters for gestational weeks.  Majority of babies born weeks 28 and 29
# mean and standard deviation
gw = np.random.normal(28, 3, 100).round(0)  # 28 is the mean, 3 is standard deviation, 100 is the size of the sample
gw

# took research to find how to round the floats to integers
# https://stackoverflow.com/questions/33160367/numpy-how-to-generate-a-normally-distributed-set-of-integers?rq=1
plt.hist(gw) #plot the histogram of the random data to view the distribution of data<jupyter_output><empty_output><jupyter_text>From the above analysis, there is consistency between the real data and the synthsized data  BirthweightThe birth weights of very low birth weight infants ranged from 360g to 2,640g.  I have tried and tested various distributions here and I think the logisitc distribution reflects the real life data more than any other distribution.. Infants are born at many different birth weights.  Most infants were born between 1200g and 1400g. Reviewing the random data generated from this code, I am confident it reflects the result of the actual report. Below is table showing the actual gestational weeks of VLBW babies in the report.https://www.ucc.ie/en/media/research/nationalperinatalepidemiologycentre/annualreports/MortalityRiskAmongstVeryLowBirthWeightInfantsintheRepublicofIrelandReport2014-2016.pdf![Birthweight.PNG](attachment:Birthweight.PNG)<jupyter_code># Create a Bar chart of data to highlight key mean and distribution
import matplotlib.pyplot as plt # importing matplotlib to plot the data
y = [5, 65, 101, 130, 122, 118, 142, 164, 183, 201, 223, 235, 75]
x = [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
plt.bar(x, y, width, color="blue")
plt.ylabel("No of Infants") # name the y label
plt.xlabel("Birthweight '00 grams") # name the x label
plt.title("Birthweight of Infant") # give the plot a title
plt.show
#The above bar chart helps to Set Parameters for birthweight, Mean = 1200g, median = 300, Size=100
# I have tried various distributions here to reflect the distribution of the real data and I think the log
bw = np.random.logistic(1400, 200, 100).round(0) #location/mean, median, and size
bw


plt.hist(bw) #plot the histogram of the random data to view the distribution of data<jupyter_output><empty_output><jupyter_text>From the above analysis, there is consistency between the real data and the synthsized data  SexIs the infant male or female.  There is a 50/50 chance of being either male or female.<jupyter_code>#using random.choice, I have ensure that there is a 50/50 chance of the infant being male or female
s = np.random.choice(["Male", "Female"], 100, p=[0.5, 0.5])
s<jupyter_output><empty_output><jupyter_text>The chance of the VLBW infant being male or female is 50/50.  The 'p' in this function enables me to put in an even probablity of the outcome being male or female  Congenital AnomalyAbout 9% of babies have a congenital anomaly.  This figure is high in comparison to VON data as abortion was/is not in Ireland.  Does the baby have a congenital anomaly? Yes or No?<jupyter_code># using random.choice, I have implmented a bias to reflect the real world
ca = np.random.choice(["Yes","No"], 100, p=[0.08, 0.92])
ca<jupyter_output><empty_output><jupyter_text>The 'p' in this function enables me to impose the 9% chance of yes being the outcome.  Disposition at 1 Year<jupyter_code># 1 year after the infants recorded birth, data is once again gathered to establish the health of the infant.
dip = np.random.choice(["Died", "At Home", "Hospital", "Unknown"], 100, p=[0.2, 0.65, 0.05, 0.1]) 
# I have used an estiamted the probabilty based on the report's key findings and used this to generate this array
dip<jupyter_output><empty_output><jupyter_text>Creating the Dataset<jupyter_code># Populating the Dataset
# using my trial and error example above, I have used this structure to create the dataset

df = pd.DataFrame({"Gestational Weeks" : gw, "Birthweight" : bw, "Sex" : s, "Congenital Anomaly" : ca, "Disposition" : dip})
df
import pandas as pd
print(df.shape) # shape - this tells quantity of rows and columns in the dataset
print(df.head(10)) # head - prints first ten rows of the dataset
# description - prints keys stats on the dataset
print(df.describe().round(0))
df.describe(include='all') #Describing all columns of a DataFrame regardless of data type.<jupyter_output><empty_output><jupyter_text>Relationship between VariablesAccording to the findings from the NPEC report, there is a relationship between the Gestation Age and the likelihood of death.  The lower the gestational age, the more the chance of death.  The problem this poses is how to show this relationship in synthesized data. ![Gestational%20Age.Chance%20of%20Death.PNG](attachment:Gestational%20Age.Chance%20of%20Death.PNG)<jupyter_code>import matplotlib.pyplot as plt # importing matplotlib to plot the data
# Create a Bar chart of data to highlight key mean and distribution
y = [100, 100, 70, 42, 25, 23, 7, 9, 6, 4, 6, 6, 6]
x = [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]
width = 1/1.5 # put a space between the bars
plt.bar(x, y, width, color="blue")
plt.ylabel("% Chance of Death") # name the y label
plt.xlabel("Gestational Age") # name the x label
plt.title("Chance of Death v Gestational Age of Infant ") # give the plot a title
plt.show()<jupyter_output><empty_output><jupyter_text>The chart clearly shows that the lower the gestational age, the higher the chance of death.  How to reflect this in the synthesised data?  I have tried but failed at creating code that can generate a correlation between these two varables.  Below if my a reflection of the effort.<jupyter_code>#Research tells me that I need to create a covariance matrix
# ref https://stackoverflow.com/questions/45952895/generating-correlated-numbers-in-numpy-pandas

#Research also tells me that to use np.random.multivariate_normal will generate the correlation that I need

#dip = np.random.multivariate_normal(mean = [28, 1249], cov=cov, size=100)

#mean = [1500]
#cov = [[1, 0], [0, 100]]  # diagonal covariance
#Diagonal covariance means that points are oriented along x or y-axis:


#import matplotlib.pyplot as plt
#x, y = np.random.multivariate_normal(mean, cov, 100).T
#plt.plot(x, y, 'x')
#plt.axis('equal')
#plt.show()<jupyter_output><empty_output><jupyter_text>Gestational Weeks and BirthweightIf Gestational weeks is above 23 weeks, then the birthweight needs to be below 1500g for the infant to be eligible for this report.If the Birthweight is above 1500g, then the Gestational weeks needs to be below 30The challenge here is to receate the dataset with these conditions included<jupyter_code># ref: https://docs.scipy.org/doc/numpy-1.10.4/reference/generated/numpy.where.html 
#I am going to examine numpy.where(condition[, x, y])

gw[np.where( gw > 30 )] # singling out the infants who are born at gestation week 30 or more

bw[np.where(bw < 1500)] # singling out the infants who are born less than 1500g

#must introduce a condition: when gw[np.where( gw > 30 )] then bw[np.where(bw < 1500)] is this

gw_condition = gw[np.where( gw > 30 )], bw[np.where(bw < 1500)]
gw_condition






bw_condition = bw[np.where( bw > 1500 )], gw[np.where(gw < 30 )]
bw_condition
# Populating the Dataset
# using my trial and error example above, I have used this structure to create the dataset




#conditions = [
    #(df['gw'] == 'gw_condition') & (df['Type'] == 'A'),
    #(df['Set'] == 'Z') & (df['Type'] == 'B'),
    #(df['Type'] == 'B')]
#choices = ['yellow', 'blue', 'purple']
#df['color'] = np.select(conditions, choices, default='black')


#unfortunately I wasnt successful at generating a dataframe with these conditions attached.
df.describe(include=[np.object]) #Including only string columns in a DataFrame description
 df.describe(exclude=[np.number]) #Excluding numeric columns from a DataFrame description - same result as code above
df.groupby('Sex').size()

df.groupby('Congenital Anomaly').size()

df.groupby('Disposition').size()


#Create a DataFrame for each Disposition Outcome
# idea from https://www.kaggle.com/abhishekkrg/python-iris-data-visualization-and-explanation
# Gather data by analysing the variables that give a certain outcome (disposition)
AtHome=df[df['Disposition']=='At Home']
Died=df[df['Disposition']=='Died']
Hospital=df[df['Disposition']=='Hospital']
Unknown=df[df['Disposition']=='Unknown']

print(AtHome.describe())
print(Died.describe())
print(Hospital.describe())
print(Unknown.describe())

plt.plot(gw) # Plot the spread of gestational weeks of each infant
plt.ylabel("Gest. Weeks") # name the y label
plt.title("Gestational Weeks of Infant") # give the plot a title
plt.plot(bw) # Plot the spread of birth weight of each infant
plt.ylabel("Birthweight") # name the y label
plt.title("Birthweight of Infant") # give the plot a title
# better to read this data on a histogram
plt.hist(gw)
plt.xlabel("Gest. Weeks") # name the y label
plt.ylabel("No. of Infants")
plt.title("Gestational Weeks of Infant") # give the plot a title
# better to read this data on a histogram
plt.hist(bw)
plt.xlabel("Birthweight") # name the y label
plt.ylabel("No. of Infants")
plt.title("Birthweight of Infant") # give the plot a title
#view 2 plots at once - Birthweight & Gestational Weeks
plt.subplot(2, 1, 1)
plt.hist(gw)
plt.xlabel("Gest. Weeks") # name the y label
plt.ylabel("No. of Infants")


plt.subplot(2, 1, 2)
plt.hist(bw)
plt.xlabel("Birthweight") # name the y label
plt.ylabel("No. of Infants")

# Plotting for each 'sub dataframe'
# Frist an overall look at the Master DF
# ref: https://github.com/ClaireDrummond/GMITProject2018/blob/master/Graphs.py
from pandas.plotting import scatter_matrix
# box plots
df.plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False)
plt.show()

# histograms 
df.hist()
plt.show()

#scatter plot matrix
scatter_matrix(df)
plt.show()




# box plots
# Graphs for the 'At Home Dataframe'
AtHome.plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False)
plt.show()

# histograms 
AtHome.hist()
plt.show()

#scatter plot matrix
scatter_matrix(AtHome)
plt.show()


# box plots
# Graphs for the 'Hospital Dataframe'
Hospital.plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False)
plt.show()

# histograms 
Hospital.hist()
plt.show()

#scatter plot matrix
scatter_matrix(Hospital)
plt.show()


# box plots
# Graphs for the 'Unkown Outcome Dataframe'
Unknown.plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False)
plt.show()

# histograms 
Unknown.hist()
plt.show()

#scatter plot matrix
scatter_matrix(Unknown)
plt.show()


# Plotting for each 'sub dataframe'
# Graphs for the 'Died Dataframe'
# ref: https://github.com/ClaireDrummond/GMITProject2018/blob/master/Graphs.py
from pandas.plotting import scatter_matrix
# box plots
Died.plot(kind='box', subplots=True, layout=(2,2), sharex=False, sharey=False)
plt.show()

# histograms 
df.hist()
plt.show()

#scatter plot matrix
scatter_matrix(df)
plt.show()
#Scatterplot showing Birthweight and Disposition of Infant after 1 Year
# ref https://datacamp-community-prod.s3.amazonaws.com/f9f06e72-519a-4722-9912-b5de742dbac4
import seaborn as sns
sns.stripplot(dip, bw)
plt.xlabel("Disposition") # name the y label
plt.ylabel("Birthweight")
plt.title("Birthweight v Disposition") # give the plot a title
# #Scatterplot showing Gestational Age and Disposition of Infant after 1 Year
sns.stripplot(dip, gw)
plt.xlabel("Disposition") # name the y label
plt.ylabel("Gestational Age")
plt.title("Gestational Age v Disposition") # give the plot a title
#A simple count plot showing the actual number of infants and their disposition at 1 year
sns.countplot(dip, palette="Greens_d")
plt.xlabel("Disposition") # name the y label
plt.ylabel("No. of Infants")
plt.title("Disposition") # give the plot a title
sns.countplot(ca, palette="Greens_d")
plt.xlabel("Congenital Anomoly") # name the y label
plt.ylabel("No. of Infants")
plt.title("Congenital Anomoly") # give the plot a title
# Count plot showing the number of males verus females in the dataframe
sns.countplot(s, palette="Greens_d")
plt.xlabel("Sex") # name the y label
plt.ylabel("No. of Infants")
plt.title("Sex") # give the plot a title
# count plot showing what portion of infants had congenital anomolies and their disposition after one year
ax = sns.countplot(x="Disposition", hue="Congenital Anomaly", data=df)
plt.title("Disposition/Congential Anomoly") #Title to chart
# count plot showing what portion of infants were male/female and their disposition after one year
ax = sns.countplot(x="Disposition", hue="Sex", data=df)
plt.title("Disposition/Sex") #Title to chart
# Plot showing how many males/females had congenital anomoly
ax = sns.countplot(x="Congenital Anomaly", hue="Sex", data=df)
plt.title("Congential Anomoly/Sex") #Title to chart<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08459500378501135, 'fraction_numerical': 0.028671461014383042, 'mean_word_length': 5.185835528241147, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 14, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '516445', 'license': 'MIT', 'n_tokens_mistral': 6985, 'n_tokens_neox': 6049, 'n_words': 2702, 'path': 'Model and Synthesize data using Python.ipynb', 'repo_name': 'ClaireDrummond/Programming-Project-2018', 'size': '21136'}