<jupyter_start><jupyter_text>Programming for Data Analysis - Project By <NAME>Create a data set by simulating a real-world phenomenon of your choosing. Tasks:Choose a real-world phenomenon that can be measured and collect at least one hundred data points across at least four different variables.Investigate the types of variables involved, their likely distributions, and their relationships with each other.Synthesise/simulate a data set as closely matching their properties as possible.Detail the research and implement the simulation in a Jupyter notebook.  1. Chosen Real Life Phenomenon The data that we are going to analyze are property prices in Ireland as per week starting Monday November 23rd 2020.I have manually collected the following data from the website www.daft.ie - 100 randomly chosen data points accross 4 cities containing 2 variables - property type and number of bedrooms. I would like to investigate the question so as to be able to simulate a dataset that would link property type, number of bedrooms and location and correlate these values to the property price.I have created a csv file with the above mentioned data and placed it in my working folder.  LocationData set contains data for 4 cities - Dublin, Limerick, Galway and Cork. As the cities vary in size, I have taken different amount of data for each city.City   &emsp; &emsp; &emsp; &emsp;        Population    &emsp;&emsp;        Data points  Dublin        &emsp; &emsp; &emsp; &emsp; 1,024K &emsp; &emsp; &emsp; &emsp; 41 Cork    &emsp;    &emsp; &emsp;  &emsp; &emsp;    190K &emsp;  &emsp; &emsp; &emsp; 25  Limerick      &emsp; &emsp; &emsp; &emsp;    90K  &emsp; &emsp; &emsp; &emsp; 16 Galway       &emsp; &emsp;    &emsp; &emsp;  71K &emsp;  &emsp; &emsp; &emsp; 18   Reference: "Population of cities in Ireland"    https://worldpopulationreview.com/countries/cities/ireland<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
# Calling the csv file that contains the data set
df = pd.read_csv("Ireland_prices.csv")
df<jupyter_output><empty_output><jupyter_text>Let us make our data framer easier to understand<jupyter_code>df_new = df.replace({"a":"apartment","d":"detached","t":"terraced","dp":"duplex","sd":"semi","tn":"townhouse"})
df_new<jupyter_output><empty_output><jupyter_text>Distribution of number of bedrooms<jupyter_code># Distribution classified by number of bedrooms.
sns.catplot(x="Bedrooms", kind="count", palette="ch:.25", data=df_new);

# Reference: https://seaborn.pydata.org/tutorial/categorical.html<jupyter_output><empty_output><jupyter_text>Distribution Property type<jupyter_code># Distribution classified by property type
sns.catplot(x="Property type", kind="count", palette="ch:.25", data=df_new);<jupyter_output><empty_output><jupyter_text>Distribution Location<jupyter_code># Distribution by location
sns.catplot(x="Location", kind="count", palette="ch:.25", data=df_new);<jupyter_output><empty_output><jupyter_text>Let us look at data for each city separately  Limerick *****<jupyter_code>Limk = df_new.loc[(df_new['Location'] == "Limerick")] 
Limk
# Applying descriptive stats to look into the data in more details
Limk["Bedrooms"].describe().round(0)<jupyter_output><empty_output><jupyter_text>Calculating probability of occurence by "Bedrooms" value<jupyter_code>bedrooms_prob = Limk.groupby('Bedrooms').size().div(len(Limk)).round(2)

# Reference: https://stackoverflow.com/questions/37818063/how-to-calculate-conditional-probability-of-values-in-dataframe-pandas-python
bedrooms_prob<jupyter_output><empty_output><jupyter_text>Plotting "Bedrooms" value over 100 data points, taking into account probability<jupyter_code>Property_type = [2,3,4]
limk_bedrooms100 = np.random.choice(Property_type, 100, p=[0.19, 0.37, 0.44])

# Reference: https://stackoverflow.com/questions/57435469/how-to-generate-random-categorical-data-in-python-according-to-a-probability-dis
# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.choice.html#numpy.random.choice
limk_bedrooms100
# Plotting a histogram
plt.hist(limk_bedrooms100);
# Applying descriptive stats to look into the data in more details
Limk["Property type"].describe()<jupyter_output><empty_output><jupyter_text>Calculating probability of "Property type" value<jupyter_code>property_prob = Limk.groupby('Property type').size().div(len(Limk)).round(2)

# Reference: https://stackoverflow.com/questions/37818063/how-to-calculate-conditional-probability-of-values-in-dataframe-pandas-python
property_prob<jupyter_output><empty_output><jupyter_text>Plotting "Property type" value over 100 data points, taking into account probability<jupyter_code>Property_type = ['apartment', 'detached', 'duplex', 'semi','terraced']
Limk_property100 = np.random.choice(Property_type, 100, p=[0.25, 0.32, 0.06, 0.25,0.12])

# Reference: https://stackoverflow.com/questions/57435469/how-to-generate-random-categorical-data-in-python-according-to-a-probability-dis
# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.choice.html#numpy.random.choice
Limk_property100
# Creating a data set with that information

np.random.seed(12) # Seed() function - the numbers will stay the same at re-run

NewDF = pd.DataFrame({'Property type':Limk_property100, 'Location':'Limerick', 'Bedrooms':limk_bedrooms100})
NewDF # Print new data frame<jupyter_output><empty_output><jupyter_text>Lower end price for an apartment in Limerick is about 130,000 euros.Average price for a 3 bedroom house in Limerick is about 205,000 euros.Higher end price for a house in Limerick is about 550,000 euros.https://www.limerickpost.ie/2020/06/29/average-limerick-city-house-prices-rise-2-5-in-past-year/https://www.limerickpost.ie/2020/06/29/average-limerick-city-house-prices-rise-2-5-in-past-year/ Let us draw 100 samples from a triangular distribution. The triangular distribution is a continuous probability distribution with lower limit left, peak at mode, and upper limit right.<jupyter_code>np.random.seed(35) # Using seed option so that the numbers are the same 
limkprice = np.random.triangular(130000,205000,550000,100).round(0)

# Reference: https://numpy.org/doc/stable/reference/random/generated/numpy.random.triangular.html
limkprice
# Converting array into dataset
Price = pd.DataFrame(limkprice)
# Calling out the data set
Price<jupyter_output><empty_output><jupyter_text>Combining all data for Limerick together<jupyter_code>LimerickDF = NewDF.join(Price)

# Reference:  https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html
LimerickDF
# Giving a name "Price" to a newly added column
LimerickDF.rename(columns={0: 'Price'})<jupyter_output><empty_output><jupyter_text>Now we are going to create the same datasets for other cities  Dublin*****<jupyter_code>Dublin = df_new.loc[(df_new['Location'] == "Dublin")] 
Dublin
# Applying descriptive stats to look into the data in more details
Dublin["Bedrooms"].describe().round(0)<jupyter_output><empty_output><jupyter_text>Calculating probability of occurence by "Bedrooms" value<jupyter_code>bedrooms_prob_d = Dublin.groupby('Bedrooms').size().div(len(Dublin)).round(2)

# Reference: https://stackoverflow.com/questions/37818063/how-to-calculate-conditional-probability-of-values-in-dataframe-pandas-python
bedrooms_prob_d<jupyter_output><empty_output><jupyter_text>Plotting "Bedrooms" value over 100 data points, taking into account probability<jupyter_code>Property_type_d = [1,2,3,4,5,6]
Dublin_bedrooms100 = np.random.choice(Property_type_d, 100, p=[0.02,0.12,0.39,0.32, 0.10, 0.05])

# Reference: https://stackoverflow.com/questions/57435469/how-to-generate-random-categorical-data-in-python-according-to-a-probability-dis
# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.choice.html#numpy.random.choice
Dublin_bedrooms100
# Plotting a histogram
plt.hist(Dublin_bedrooms100);
# Applying descriptive stats to look into the data in more details
Dublin["Property type"].describe()<jupyter_output><empty_output><jupyter_text>Calculating probability of "Property type" value<jupyter_code>property_prob_d = Dublin.groupby('Property type').size().div(len(Dublin)).round(2)

# Reference: https://stackoverflow.com/questions/37818063/how-to-calculate-conditional-probability-of-values-in-dataframe-pandas-python
property_prob_d<jupyter_output><empty_output><jupyter_text>Plotting "Property type" value over 100 data points, taking into account probability<jupyter_code>Property_type_d = ['apartment', 'detached', 'duplex', 'semi','terraced','townhouse']
Dublin_property100 = np.random.choice(Property_type_d, 100, p=[0.22,0.22, 0.05, 0.47, 0.02,0.02])

# Reference: https://stackoverflow.com/questions/57435469/how-to-generate-random-categorical-data-in-python-according-to-a-probability-dis
# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.choice.html#numpy.random.choice
Dublin_property100
# Creating a data set with that information

np.random.seed(32) # Seed reset - the numbers will stay the same at re-run

NewDF_d = pd.DataFrame({'Property type':Dublin_property100, 'Location':'Dublin', 'Bedrooms':Dublin_bedrooms100})
NewDF_d # Print new data frame<jupyter_output><empty_output><jupyter_text>Lower end price for an apartment in Dublin is about 220,000 euros.Average price for a 3 bedroom house in Dublin is about 381,000 euros.Higher end price for a house in Dublin is about 680,000 euros. Reference:  https://www.dublinlive.ie/news/property/dublin-cheap-house-property-sale-15779001 https://www.daft.ie/property-for-sale/dublin-city Let us draw 100 samples from a triangular distribution. The triangular distribution is a continuous probability distribution with lower limit left, peak at mode, and upper limit right.<jupyter_code>np.random.seed(25) # Using seed option so that the numbers are the same 
dublinprice = np.random.triangular(220000,381000,680000,100).round(0)

# Reference: https://numpy.org/doc/stable/reference/random/generated/numpy.random.triangular.html
dublinprice<jupyter_output><empty_output><jupyter_text>Converting array into dataset<jupyter_code>Price_d = pd.DataFrame(dublinprice)
Price_d<jupyter_output><empty_output><jupyter_text>Combining all data for Dublin together<jupyter_code>DublinDF = NewDF_d.join(Price_d)

# Reference:  https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html
DublinDF<jupyter_output><empty_output><jupyter_text>Giving a name "Price" to a newly added column<jupyter_code>DublinDF.rename(columns={0: 'Price'})<jupyter_output><empty_output><jupyter_text>Cork*****<jupyter_code>Cork = df_new.loc[(df_new['Location'] == "Cork")]
Cork
# Applying descriptive stats to look into the data in more details
Cork["Bedrooms"].describe().round(0)<jupyter_output><empty_output><jupyter_text>Calculating probability of occurence by "Bedrooms" value<jupyter_code>bedrooms_prob_c = Cork.groupby('Bedrooms').size().div(len(Cork)).round(2)

# Reference: https://stackoverflow.com/questions/37818063/how-to-calculate-conditional-probability-of-values-in-dataframe-pandas-python
bedrooms_prob_c<jupyter_output><empty_output><jupyter_text>Plotting "Bedrooms" value over 100 data points, taking into account probability<jupyter_code>Property_type_c = [2,3,4,5,7]
Cork_bedrooms100 = np.random.choice(Property_type_c, 100, p=[0.12,0.44,0.32,0.08, 0.04])

# Reference: https://stackoverflow.com/questions/57435469/how-to-generate-random-categorical-data-in-python-according-to-a-probability-dis
# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.choice.html#numpy.random.choice
Cork_bedrooms100
# Plotting a histogram
plt.hist(Cork_bedrooms100);
# Applying descriptive stats to look into the data in more details
Cork["Property type"].describe()<jupyter_output><empty_output><jupyter_text>Calculating probability of "Property type" value<jupyter_code>property_prob_c = Cork.groupby('Property type').size().div(len(Cork)).round(2)

# Reference: https://stackoverflow.com/questions/37818063/how-to-calculate-conditional-probability-of-values-in-dataframe-pandas-python
property_prob_c<jupyter_output><empty_output><jupyter_text>Plotting "Property type" value over 100 data points, taking into account probability<jupyter_code>Property_type_c = ['apartment', 'detached', 'duplex', 'semi','terraced']
Cork_property100 = np.random.choice(Property_type_c, 100, p=[0.12,0.24, 0.08, 0.52, 0.04])

# Reference: https://stackoverflow.com/questions/57435469/how-to-generate-random-categorical-data-in-python-according-to-a-probability-dis
# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.choice.html#numpy.random.choice
Cork_property100
# Creating a data set with that information

np.random.seed(40) # Seed reset - the numbers will stay the same at re-run

NewDF_c = pd.DataFrame({'Property type':Cork_property100, 'Location':'Cork', 'Bedrooms':Cork_bedrooms100})
NewDF_c # Print new data frame<jupyter_output><empty_output><jupyter_text>Lower end price for an apartment in Cork is about 180,000 euros. Average price for a 3 bedroom house in Cork is about 265,000 euros. Higher end price for a house in Cork is about 590,000 euros.Reference: https://www.daft.ie/property-for-sale/cork/houseshttps://www.echolive.ie/corknews/Cork-house-prices-drop-by-7-d0cec175-b7d8-42df-a25f-8bb4a48be304-ds:~:text=According%20to%20that%20latest%20housing,%E2%82%AC265%2C637%2C%20down%207.1%20percent. Let us draw 100 samples from a triangular distribution. The triangular distribution is a continuous probability distribution with lower limit left, peak at mode, and upper limit right.<jupyter_code>np.random.seed(259) # Using seed option so that the numbers are the same 
corkprice = np.random.triangular(180000,265000,600000,100).round(0)

# Reference: https://numpy.org/doc/stable/reference/random/generated/numpy.random.triangular.html
corkprice<jupyter_output><empty_output><jupyter_text>Converting array into dataset<jupyter_code>Price_c = pd.DataFrame(corkprice)
Price_c<jupyter_output><empty_output><jupyter_text>Combining all data for Cork together<jupyter_code>CorkDF = NewDF_c.join(Price_c)

# Reference:  https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html
CorkDF<jupyter_output><empty_output><jupyter_text>Giving a name "Price" to a newly added column<jupyter_code>CorkDF.rename(columns={0: 'Price'})<jupyter_output><empty_output><jupyter_text>Galway*****<jupyter_code>Galway = df_new.loc[(df_new['Location'] == "Galway")]
Galway
# Applying descriptive stats to look into the data in more details
Galway["Bedrooms"].describe().round(0)<jupyter_output><empty_output><jupyter_text>Calculating probability of occurence by "Bedrooms" value<jupyter_code>bedrooms_prob_g = Galway.groupby('Bedrooms').size().div(len(Galway)).round(2)

# Reference: https://stackoverflow.com/questions/37818063/how-to-calculate-conditional-probability-of-values-in-dataframe-pandas-python
bedrooms_prob_g<jupyter_output><empty_output><jupyter_text>Plotting "Bedrooms" value over 100 data points, taking into account probability<jupyter_code>bedrooms_prob_g = [2,3,4,5,6]
Galway_bedrooms100 = np.random.choice(bedrooms_prob_g, 100, p=[0.17,0.33,0.38,0.06, 0.06])

# Reference: https://stackoverflow.com/questions/57435469/how-to-generate-random-categorical-data-in-python-according-to-a-probability-dis
# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.choice.html#numpy.random.choice
Galway_bedrooms100
# Plotting a histogram
plt.hist(Galway_bedrooms100);
# Applying descriptive stats to look into the data in more details
Galway["Property type"].describe()<jupyter_output><empty_output><jupyter_text>Calculating probability of "Property type" value<jupyter_code>property_prob_g = Galway.groupby('Property type').size().div(len(Galway)).round(2)

# Reference: https://stackoverflow.com/questions/37818063/how-to-calculate-conditional-probability-of-values-in-dataframe-pandas-python
property_prob_g<jupyter_output><empty_output><jupyter_text>Plotting "Property type" value over 100 data points, taking into account probability<jupyter_code>Property_type_g = ['apartment', 'detached', 'semi','terraced']
Galway_property100 = np.random.choice(Property_type_g, 100, p=[0.17,0.28, 0.49, 0.06])

# Reference: https://stackoverflow.com/questions/57435469/how-to-generate-random-categorical-data-in-python-according-to-a-probability-dis
# https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.random.choice.html#numpy.random.choice
Galway_property100
# Creating a data set with that information

np.random.seed(20) # Seed reset - the numbers will stay the same at re-run

NewDF_g = pd.DataFrame({'Property type':Galway_property100, 'Location':'Galway', 'Bedrooms':Galway_bedrooms100})
NewDF_g # Print new data frame<jupyter_output><empty_output><jupyter_text>Lower end price for an apartment in Galway is about 190,000 euros. Average price for a 3 bedroom house in Galway is about 306,000 euros. Higher end price for a house in Galway is about 610,000 euros.Reference: https://www.google.com/search?q=galway+house+prices&oq=galway+house+prices&aqs=chrome..69i57j0j0i22i30l6.9103j0j4&sourceid=chrome&ie=UTF-8 Let us draw 100 samples from a triangular distribution. The triangular distribution is a continuous probability distribution with lower limit left, peak at mode, and upper limit right.<jupyter_code>np.random.seed(25) # Using seed option so that the numbers are the same 
galwayprice = np.random.triangular(190000,306000,610000,100).round(0)

# Reference: https://numpy.org/doc/stable/reference/random/generated/numpy.random.triangular.html
galwayprice<jupyter_output><empty_output><jupyter_text>Converting array into dataset<jupyter_code>Price_g = pd.DataFrame(galwayprice)
Price_g<jupyter_output><empty_output><jupyter_text>Combining all data for Galway together<jupyter_code>GalwayDF = NewDF_g.join(Price_g)

# Reference:  https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html
GalwayDF<jupyter_output><empty_output><jupyter_text>Giving a name "Price" to a newly added column<jupyter_code>GalwayDF.rename(columns={0: 'Price'})<jupyter_output><empty_output><jupyter_text>Concatenating 4 separate data sets into single one<jupyter_code>Combining = [DublinDF,LimerickDF,CorkDF,GalwayDF]
FinalDF = pd.concat(Combining)
FinalDF
FinalDF.tail(20)<jupyter_output><empty_output><jupyter_text>With tail() function above we called last 20 rows of the dataset and we can see that after concatenation the index of the total dataset is not continuous throughout, it ends in 99 end restarts with 0 again. Let us rectify that by calling index_reset() function.<jupyter_code># Now the index will be continuous - 0 to 399.
FinalDF.reset_index(drop=True)
# Reference: https://stackoverflow.com/questions/35528119/pandas-recalculate-index-after-a-concatenation<jupyter_output><empty_output><jupyter_text>Now we are going the shuffle to rows randomly within the dataset using sample() function.Sample() function will shuffle order of the DataFrame's rows, but values within a row are the same, only order of the rows will change (row sampling without replacement).<jupyter_code>import numpy as np
from numpy import random
Final = FinalDF.sample(frac=1).reset_index(drop=True)

# The frac keyword argument specifies the fraction of rows to return in the random sample, so frac=1 means return all rows (in random order).
# Reset_index() function resets index, specifying drop=True prevents .reset_index from creating a column containing the old index entries.

# Reference: https://stackoverflow.com/questions/29576430/shuffle-dataframe-rows
Final
# Renaming last column to "Price"
Final.rename(columns={0: 'Price'})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11656813703198424, 'fraction_numerical': 0.04027083017533222, 'mean_word_length': 6.722200546234881, 'pattern_counts': {'":': 6, '<': 170, '<?xml version=': 0, '>': 170, 'https://': 43, 'lorem ipsum': 0, 'www.': 8, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '42', 'dir': 'jupyter-structured-clean-dedup', 'id': '423746', 'license': 'MIT', 'n_tokens_mistral': 7245, 'n_tokens_neox': 6513, 'n_words': 1455, 'path': 'Property-Prices-in-Ireland.ipynb', 'repo_name': 'dmitrijsg123/PDA_Project', 'size': '19805'}