<jupyter_start><jupyter_text>Programming Project52465 - Programming For Data Analysis: Building a Data set.  IntroductionIn this project I will:1. Build a Data set based that will demonstrate a sale of 20% on four different type of shoes and amount that is available during this sale. I have collected one-hundred data points across six diﬀerent variables. Those variables include: shoe size, shoe type, original prize, price of 20% dicount, sale price, and amount avaialble.2. I will investigate the types of variables involved, their likely distributions, and their relationships with each other.   3. I will simulate a data set as closely matching their properties as possible.Before starting the project I wanted to do some research on what are the most important aspects involved in building a Dataset. I have found various articles that gave good advice on how to prepare data before begining the data set and that the end goal has to be clear, especially I found a [Blog](https://www.altexsoft.com/blog/datascience/preparing-your-dataset-for-machine-learning-8-basic-techniques-that-make-your-data-better/) on 8 basic techniques that make your Data better very useful.  To have a clear picture of the data points I will use I did a research on variables.  Shoe SizeI decided to use European women’s shoe size which goes from size 35 to size 42. For reference please see website which describes [SIZE GUIDE]( https://m.asos.com/Women/footwear-size-guide/?szgid=8&szgtid=1&r=2). Shoe TypeI have done some research on few website and I came to conclusion the shoe types which are commonly on sales are Trainers, High Heels, Boots and Sandals. I took this information from visiting Sale section of different websites like [Office](https://www.office.co.uk/sale). As we are in December, [research](http://brokeandchic.com/when-is-the-best-time-to-buy-boot/) shows that best time to buy boots is from September to December.  Original PrizeI have decided to set a price range that is affordable. As all my previous research was based on middle class classification I decided to take a price range from a shoe website [Topshop](http://eu.topshop.com/en/tseu/category/shoes-485095).  The prize will vary from 5euro to 195euro. Discount of 20%I have decided that since we are in December months the discount on average has to match the year sale cycle which is usually higher at the end of the year. [MoneyGuideIreland](http://www.moneyguideireland.com/sales-starting-december-26th.html) wrote that Schuh shoe shop start their sale in December with up to 45 per cent off styles from the shoe shop’s autumn-winter collections. As my dataset is not exclusively autumn-winter collection but more a variety of collections I have decided to assign a discount of 20%. Sale PrizeThis will give me the sale prize of the shoe after 20% discount is taken away from the original prize Amount AvailableLastly, this data will give us an idea of the amount of shoe pairs in a particular size and shoe type there is in the shop. To build the dataset I watched a lot of videos on pandas DataFrame and blogs. Reference to those resources can be found at the bottom of the Jupyter Notebook under Reference section.<jupyter_code>#Libraries
#DataFrames in Python come with the Pandas library
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib<jupyter_output>Using matplotlib backend: Qt5Agg<jupyter_text>I will begin building the Dataset by setting the variables first and concentrate on code that will help me build it. I will use randomly generated numbers for now.<jupyter_code>datadraft = pd.DataFrame(np.random.randn(10, 4), columns=list('abcd')) #expolring code that can be used for my datset
datadraft #columns=list only generates 4 columns
#and random.randn is not a function I want
datadraft = pd.DataFrame(np.random.randint(0,100,size=(10, 4)), columns=list('ABCD')) #randint accepts the size
#creates an output array of size (10,4) with random integer elements between (0,100)
datadraft #still not exactly what I am looking for.
# I want each column to be named differently and have still relative data within its true range
datadraft = pd.DataFrame(np.random.randint(0,100,size=(10, 6)), columns=list('ABCDEF'))
datadraft # more columns by increaing the size to 6 and setting the names ABCDEF
# Here I want to assign the column names
datadraft = pd.DataFrame(np.random.randint(0,100,size=(10, 6)), columns=('shoe size','shoe type','original_price','dicount_20', 'sale_price', 'amount_available'))
datadraft 
# Here we have 100 data points randomly generated and 6 variables in only 10 columns
#Im going to focus on fist column first 'shoe size'
#As this is women size it starts from 35 to 42
#this will allocate the sizes at random
data = pd.DataFrame({"shoe_size": np.random.randint(35,43,100)}) 
#this will generate 100 random numbers between smallest size of shoe 35 and largest 42.
data.tail(10) #shows last 10 data points and to show the first 10 data we can use data.head(10)
#Now I want to focus on the shoe type, this code will have to generate four different shoe types
#the shoe types I choosed are trainers, sandals, high heels and boots.
types = ['trainers', 'sandals', 'boots', 'high_heels']

data = pd.DataFrame({"shoe_size": np.random.randint(35,43,100), "shoe_type": np.random.shuffle(types)})
data.head(10)
#I received an error on shoe_type
#I believe it is because the amount of times has to be identified and shuffle is not a correct function 
#random.shuffle
#random.choice
data = pd.DataFrame({"shoe_size": np.random.randint(35,43,100), "shoe_type": np.random.choice(types, 100)})
data.head(20)
#I used random choice as it allows multiple elements from a sequence with replacement (duplicates are possible)<jupyter_output><empty_output><jupyter_text>As I am concentrating on middle class range of shoes I checked a website called [Topshop](http://eu.topshop.com/en/tseu/category/shoes-485095/N-a2eZdgf?No=0&Nrpp=20&siteId=%2F13058&Nf=nowPrice%7CBTWN%2B5%2B195) which sells shoes and the prices vary from 5 euro to 195 euro. That is the range I will allocate for my shoes before the discount.<jupyter_code>#randomly allocating a range of numbers between 5 euro and 195 euro.
dataprice = pd.DataFrame({"original_price": np.random.randint(5,195,100)}) 
dataprice.head(15)
#now lets put it into the draft dataset:
data = pd.DataFrame({"shoe_size": np.random.randint(35,43,100), 
                     "shoe_type": np.random.choice(types, 100), "original_price": np.random.randint(5,195,100)})
data.head()
#this column will show what is 20% dicount from the orginal prize
data['discount_20'] = data['original_price'] * 20 // 100 
data
data.head()
#this additional column will show the sale price after 20% discount is applied on the original price
#Here I researched how to subtract two columns in a dataframe
data['sale_price'] = data['original_price']-data['discount_20']
data.head()
#I wanted to have a link between the type of shoe type and amount available by mapping two columns:
#data['setamount_available'] = data['shoe_type'].map({'sandals': 3, 'trainers': 10, 'boots': 20, 'high_heels': 5})
#data.loc[0:10, ['shoe_type', 'setamount_available']]
#I did not end up using mapping as it turned out later it is not the best method when analysing the data later
#I have then decided to randomly assign number between 1 to 10 pairs available at random.
data['amount_available'] = np.random.randint(1,10,100) #random amount from 1 to 10 pairs left on sale?
data.head()
#Now this is the output of the overall data
data<jupyter_output><empty_output><jupyter_text>Relationship and distribution between the variables in the dataset Now I will focus on representing the relationship between columns of the dataset:<jupyter_code>#memory footprint and datatypes
data.info()
#First lets have a statistical overview of intergers 
data.describe()
#Here we get the different statistical data of the dataset like mean, median etc.
# we can see that the average of the original price is 100.33 euro and after 20% discount its 80.68
#on average the customer is saving aoung 19.65 euro.
#visual representation of the original sale price, the price after discount and 20% dicount amount.
df = data.loc[:,['original_price', 'sale_price', 'discount_20']].mean()
df.plot(kind='bar')
plt.title('Average Price of original, sale and 20% discount price')
#Here we can just see the boxplot of the original price
sns.boxplot(x=data["original_price"])
plt.title('Average of original price')
#we can also focus on the type of shoe and see its variables
shoe_type = data[data.shoe_type == 'boots']
shoe_type
#all shoe type: 'boots' available, this shows us the sizes available, amount of each size and the prices.
#first column represents numbers 1 to 100 which also represents the row or the shoe.
#for example number 13 shoe will show you all the variables of that shoe 'shoe_size, shoe_type, original_price,etc' 
#if we want to see the sizes avaialble throught a graph we can plot it like this:
df = shoe_type.loc[:,['shoe_size',]]
df.plot(kind='bar')
#this code can be used on any type of show we want to see information on for example the sandals:
shoe_type = data[data.shoe_type == 'sandals']
shoe_type
#shoe_type = data[data.shoe_type == 'high_heels'] 
#shoe_type = data[data.shoe_type == 'trainers']
#shoes available in size 40 or above 
#this demonstrate that we can search by shoe size and see availability
size = data[data.shoe_size >=40]
size
#this can be done for any size
#the output gave a complete information on the amount and type of shoes size 40 or above<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07824011621874027, 'fraction_numerical': 0.026356750025941685, 'mean_word_length': 5.111604311984781, 'pattern_counts': {'":': 9, '<': 16, '<?xml version=': 0, '>': 17, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '125004', 'license': 'Apache-2.0', 'n_tokens_mistral': 2940, 'n_tokens_neox': 2698, 'n_words': 1310, 'path': 'RealworldPhenomenon.ipynb', 'repo_name': 'MartynaMisk/ProgrammingProject', 'size': '9637'}