<jupyter_start><jupyter_text>MCIS6273 Data Mining (Prof. Maull) / Fall 2020 / HW1  Student Name :  <NAME> Student ID   : 999901000 Section   : 029 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++  ASSIGNMENT TASKS  (35%) Perform Exploratory Data Analysis (EDA) and apply basic statistical concepts to understand the underlying data   Loading the diamonds data &167;  In the first task, we are going to load the data into a DataFrame using the standard [`pd.read_csv()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html?highlight=read_csvpandas.read_csv).<jupyter_code>#Load pandas library
import pandas as pd 
#Reading the diamonds data
diamonds_database = pd.read_csv('https://git.io/JUGqS')
database =pd.DataFrame(diamonds_database)
database
database = database.dropna()
database<jupyter_output><empty_output><jupyter_text>After loading the dataset, answer the following basic questions using the data:  1. What is the median price of a diamond over the entire dataset? The mean is the average of the numbers<jupyter_code>database['price'].median()<jupyter_output><empty_output><jupyter_text>2. What is the mean? The mean is the average of the numbers<jupyter_code>database.mean()
database.mean().price<jupyter_output><empty_output><jupyter_text>3. Of the diamonds between 1.50 and 3 carats, what is the median price?<jupyter_code>database.loc[(database.carat >=1.50) & (database.carat <=3.00)].median().price<jupyter_output><empty_output><jupyter_text>4. How many _Premium_ cut diamonds with _VVS1_, _I1_ and _IF_ clarity are there (regardless of color)?<jupyter_code>database.loc[ ( database.clarity.str.contains('VVS.', regex=True) | 
    (database.clarity.str.contains('.I1', regex=True)) | (database.clarity == 'IF') ) &
    ((database.cut == 'Premium')) ].carat.count()<jupyter_output><empty_output><jupyter_text>5. For diamonds between $3500- $ 7000, what is the mean carat size?<jupyter_code>database.loc[(database.price >=3500) & (database.price <=7000)].mean().carat<jupyter_output><empty_output><jupyter_text>Multi-Index DataFrame  * in your Multindex DataFrame  the row indices are nested with _cut_ in the outer index, _color_ in the inner index.  * the columns will be the _clarity_  * the values for a given `(cut, color, clarity)` will the _mean_ value for that combination.  For example,  1. Please make sure you output the multi-index DataFrame so that I can see it<jupyter_code>group_mean_price = database.groupby(['cut','color','clarity']).mean().price
group_mean_price
group_mean_price.index
df_multi_idx = pd.MultiIndex.from_tuples(group_mean_price.unstack().index, names=['cut', 'color'])
database_multi_index_df = pd.DataFrame(group_mean_price.unstack(),index=df_multi_idx, columns=group_mean_price.unstack().columns)
database_multi_index_df
database_multi_index_df.loc['Very Good']
database_multi_index_df.loc[('Very Good','H')]
database_multi_index_df.loc[('Very Good','H')].VS1
database_multi_index_df.loc[['Ideal','Very Good','Premium']]<jupyter_output><empty_output><jupyter_text>Now that you have the DataFrame. there is one last thing to do so that you can answer some more interesting questions.  SEABORN CODE```python   import seaborn as sns   cm = sns.light_palette("green", as_cmap=True)   df_multi_idx.astype('float').style.background_gradient(cmap=cm)```<jupyter_code>import warnings
warnings.filterwarnings("ignore")
import seaborn as sns

cm = sns.light_palette("green", as_cmap=True)
database_multi_index_df.astype('float').style.background_gradient(cmap=cm)<jupyter_output><empty_output><jupyter_text>Now answer the following questions:  1. If we are looking at _J color, Ideal cut_ diamonds, what seems unusual about the _I1 clarity_ mean price? Recall, _J color_ are considered the color right before faint yellow and are considered the least quality color rating.<jupyter_code>cut_j_i1_val = database_multi_index_df.loc[('Ideal','J')].I1
cut_j_i1_val
database_multi_index_df.astype('float').style.background_gradient(cmap=cm).highlight_max(axis=0, color='yellow')<jupyter_output><empty_output><jupyter_text>2. Can you make a general observation about the average prices trends?  What are you noticing, especially about the average prices as the color gets worse? Ans:- Based on general observation the praph has the light green color is the average prices trends.  3. Which _cut, color_ and _clarity_ has the highest overall mean price?<jupyter_code>#color
list(database_multi_index_df.idxmax().max())[0]

list(database_multi_index_df.idxmax().max())[1]
#Clarity
database_multi_index_df.max().idxmax()
#highest overall mean price
max(database_multi_index_df.max())
print(list(database_multi_index_df.idxmax().max())[0]," cut,",list(database_multi_index_df.idxmax().max())[1] ," color and",database_multi_index_df.max().idxmax(),"clarity has the highest overall mean price",max(database_multi_index_df.max()))<jupyter_output>Very Good  cut, D  color and IF clarity has the highest overall mean price 10298.260869565218<jupyter_text>§ Now that you have seen a trend on prices, instead of computing the mean price,  compute the mean carat size of the diamond and perform the same tabular gradient enhancement.<jupyter_code>#Carat mean price
carat_mean_price_group = database.groupby(['cut','color','clarity']).mean().carat
#Multiindex 
df_carat_multi_idx = pd.MultiIndex.from_tuples(carat_mean_price_group.unstack().index, names=['cut', 'color'])
#Create a dataframe
database_carat_multi_index_df = pd.DataFrame(carat_mean_price_group.unstack(),index=df_carat_multi_idx, columns=carat_mean_price_group.unstack().columns)
database_carat_multi_index_df
database_carat_multi_index_df.shape
#seaborn  Code
database_carat_multi_index_df.astype('float').style.background_gradient(cmap=sns.light_palette("green", as_cmap=True))<jupyter_output><empty_output><jupyter_text>Answer the following questions looking at the tables you've generated:  1. What seems to be the trend in carat size, especially as it relates to cut and color?Ans:-<jupyter_code>database_carat_multi_index_df
database_carat_multi_index_df.max().max()
database_carat_multi_index_df.loc[('Premium', 'J')].max()<jupyter_output><empty_output><jupyter_text>2. Which cut (regardless of color) has the largest mean carat size?  Is this a surprising outcome?  Explain in a sentence or two what you think is going on?<jupyter_code>print(database_carat_multi_index_df.idxmax().max()[0]," has the larget mean carat size",database_carat_multi_index_df.max().max())<jupyter_output>Premium  has the larget mean carat size 1.9934782608695656<jupyter_text>Plot a scatterplot of all the raw data with price on the $y$-axis and carats on the $x$-axis usingthis as a template: ```python      your_dataframe.loc[:,['price','carat']].plot(kind='scatter', x='carat', y='price')```  1. Does your plot support the claim that price increases as carats increase?  Ans:    The Price increases then automatically based on the carat diamond from the above plot.<jupyter_code>database.loc[:,['price','carat']].plot(kind='scatter', x='carat', y='price')<jupyter_output><empty_output><jupyter_text>(35%) Explore PDF and CDF with Pandas and Seaborn The first thing want to do is look at the cumulative distribution of one variable -- carat.   Cumulative Histogram:<jupyter_code>#cumulative histogram
database['clarity'].hist( cumulative = True,normed=1 )<jupyter_output><empty_output><jupyter_text>1. Produce a count of the diamond clarities using the [`pandas.DataFrame.value_counts()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.value_counts.html?highlight=value_countspandas.DataFrame.value_counts) method.  Show the count of diamonds on a bar graph using [`plot(kind="bar")`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.plot.html?highlight=plotpandas.DataFrame.plot).<jupyter_code>import matplotlib.pyplot as plt
plt.title('Count of Diamonds on a Bar Graph')
plt.xlabel('Diamond Clarities')
plt.ylabel('Count of Diamond Clarities')

database['clarity'].value_counts().sort_values().plot(kind='bar',x='Diamond Clarities',y='Count of Diamond Clarities')<jupyter_output><empty_output><jupyter_text>2. What observation can you make about the trend in the count of diamonds in each clarity category?<jupyter_code>print("Clarity max",database['clarity'].value_counts().index[0],"and max value is ",database['clarity'].value_counts()[0])
print("Clarity min",database['clarity'].value_counts().index[len(database['clarity'].value_counts())-1],"and min value is ",database['clarity'].value_counts()[len(database['clarity'].value_counts())-1])<jupyter_output>Clarity min I1 and min value is  741<jupyter_text>3. How do the numbers of diamonds in the "SI" and "IF/FL" groups compare?<jupyter_code>database.loc[(database['clarity'] == 'FL')]
sns.countplot(x="clarity", data=database.loc[(database['clarity'] == 'IF') | (database['clarity'] == 'SI1') | (database['clarity'] == 'FL')])<jupyter_output><empty_output><jupyter_text>&167;  Now that we have an understanding of how many diamonds we have in each clarity group, let's compare thedistribution of carats for each.  To do this we are going to turn to Seaborn's `distplot()` method.Read about `distplot()` [here](https://seaborn.pydata.org/generated/seaborn.distplot.htmlseaborn.distplot), butthis template of code should help you plot what you need:```python    import seaborn as sns    import matlplotlib.pyplot as plt    plt.title("Comparing Cumulative Distribution of Carat for I1, IF, SI1 and SI2")    kwargs = {'cumulative': True}    sns.distplot(the_dataframe_with_clarity_X, hist=False, hist_kws=kwargs, kde_kws=kwargs, label="clarity_X")     repeat for the other clarities you want compare``` You now have a cumulative distribution for each clarity. Figure 5 shows you an example of what **one** of the plotsmight look like. ![Cumulative distribution plot example](./example_distc_plot_240.png)You will be reminded that the $y$-axis shows usthe cumulative probability while the $x$-axis shows us the carat sizes.  The smoothed curve shows the densityof the carat for a given probability and uses a smoothing technique called _kernel density estimate_.<jupyter_code>import seaborn as sns
import matplotlib.pyplot as plt

plt.title("A Cumulative Distribution")
plt.ylabel("Cumulative Probability")
plt.xlabel("Carat Size")
kwargs = {'cumulative': True}

sns.distplot(database.loc[(database['clarity'] == 'SI1')].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='blue', label="CLARITY_SI1",axlabel="CARAT")

plt.title("Cumulative Distribution for each Clarity")
plt.ylabel("Cumulative Probability")
plt.xlabel("Carat Size")
sns.distplot(database.loc[(database['clarity'] == 'I1') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='red', label='I1',axlabel="CARAT")
sns.distplot(database.loc[(database['clarity'] == 'IF') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='black', label='IF',axlabel="CARAT")
sns.distplot(database.loc[(database['clarity'] == 'SI1') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='tomato', label='SI1',axlabel="CARAT")
sns.distplot(database.loc[(database['clarity'] == 'SI2') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='aqua', label='SI2',axlabel="CARAT")
sns.distplot(database.loc[(database['clarity'] == 'VS1') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='green', label='VS1',axlabel="CARAT")
sns.distplot(database.loc[(database['clarity'] == 'VS2') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='blue', label='VS2',axlabel="CARAT")
sns.distplot(database.loc[(database['clarity'] == 'VVS1') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='pink', label='VVS1',axlabel="CARAT")
sns.distplot(database.loc[(database['clarity'] == 'VVS2') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='midnightblue', label='VVS2',axlabel="CARAT")<jupyter_output><empty_output><jupyter_text>After displaying the plot from the code above in your notebook, answer the following questions:  1. Which clarity has the largest relative distribution of high carat diamonds?<jupyter_code>database_carat_multi_index_df.max()
database_carat_multi_index_df.max().max()
database_carat_multi_index_df.max().idxmax()
 print(database_carat_multi_index_df.max().idxmax(),'has the largest relative distribution of high carat diamonds ')<jupyter_output>I1 has the largest relative distribution of high carat diamonds<jupyter_text>2. Which has the largest relative distribution of small diamonds?<jupyter_code>database_carat_multi_index_df.max()
database_carat_multi_index_df.max().min()
print(database_carat_multi_index_df.max().idxmin(),'has the largest relative distribution of small diamonds.')<jupyter_output>IF has the largest relative distribution of small diamonds.<jupyter_text>3. Based on the prior two questions, is this a surprising outcome?<jupyter_code># Yes, The Prior two questions, as show the surprising outcome
plt.title("Comparing Cumulative Distribution of Carat for IF and VS2.")
plt.xlabel("CARAT SIZE")
plt.ylabel("Cumulative Distribution")
sns.distplot(database.loc[(database['clarity'] == 'IF') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='red', label="MAX: clarity_IF")
sns.distplot(database.loc[(database['clarity'] == 'VS2') ].carat, hist=False, hist_kws=kwargs, kde_kws=kwargs, color='yellow', label="MIN: clarity_VS2")

import warnings
warnings.filterwarnings("ignore")
# Yes, The Prior two questions, as show the surprising outcome
plt.title("Comparing Cumulative Distribution of Carat for IF and VS2.")
plt.xlabel("CARAT SIZE")
plt.ylabel("Cumulative Distribution")
sns.distplot(database_carat_multi_index_df['IF'], hist=False, hist_kws=kwargs, kde_kws=kwargs, color='red', label="MAX: clarity_IF")
sns.distplot(database_carat_multi_index_df['VS2'], hist=False, hist_kws=kwargs, kde_kws=kwargs, color='yellow', label="MIN: clarity_VS2")<jupyter_output><empty_output><jupyter_text>4. Which two distributions look most similar? Ans:-    SI1 snd SI2 are two distributions look most similar.<jupyter_code># Yes, The Prior two questions, as show the surprising outcome
plt.title("Most Similar Two Distributions")
plt.xlabel("CARAT SIZE")
plt.ylabel("Cumulative Distribution")
sns.distplot(database_carat_multi_index_df['SI1'], hist=False, hist_kws=kwargs, kde_kws=kwargs, color='red', label="Most Similar: clarity_SI1")
sns.distplot(database_carat_multi_index_df['SI2'], hist=False, hist_kws=kwargs, kde_kws=kwargs, color='yellow', label="Most Similar: clarity_SI2")<jupyter_output><empty_output><jupyter_text>&167;  We're now at the point where we'd like to graph this same data in two dimensions, with what is called the _kernel density estimate plot_ or KDE plot. The KDE plot takes our bivariate data as input and outputs a smoothed 2D contour plot.  You see, in the example above, we only showed a single feature of thedata.  While interesting, it is far more interesting when we compare two variables.  Indeed, since diamonds are generally considered a luxury item, and not typically considered "inexpensive", we might liketo know how _carat_ size influences _price_.  This makes things a bit more interesting and useful.We're going to use the KDE plot to visually compare the densities of _carat_ and _price_. ```python    plt.title("KDE Plot of Price to Carat for X and Y")     dataset X    sns.kdeplot(        x_carat,        x_price,        cmap="Reds",        shade=True,        shade_lowest=False, label="x_label")     dataset Y    sns.kdeplot(        y_carat,        y_price,        cmap="Blues",        shade=True,        shade_lowest=False, label="y_label")    plt.legend()    plt.savefig("your_plot_can_be_saved.png")``` Your plot will look something like that in Figure 3 below.  If you notice in the figure the densities show that SI2 has a broad distribution between a depth of 58 and 65, while the price varies greatly reaching well above $10,000.  You can see IF diamonds are much tighter both in terms of price and depth.![KDE Plot for depth vs. price (SI2 and IF clarity)](./2d_kde_plot.png)You may want to only plot each alone or no more than 2 per plot so you can see the data.After you have the plots, make sure you save them and include them in your zipped submission.<jupyter_code>plt.title("KDE Plot of Price to Depth for IF and SI2")
# dataset X
sns.kdeplot(
    database.loc[(database['clarity'] == 'SI2')]['depth'],
    database.loc[(database['clarity'] == 'SI2')]['price'],
    cmap="Reds",
    color='r',
    shade=True,
    shade_lowest=False, label="SI2")

# dataset Y
sns.kdeplot(
    database.loc[(database['clarity'] == 'IF')]['depth'],
    database.loc[(database['clarity'] == 'IF')]['price'],
    color='b',
    cmap="Blues",
    shade=True,
    shade_lowest=False, label="IF")

plt.legend()
plt.savefig("your_plot_can_be_saved.png")<jupyter_output><empty_output><jupyter_text>Then answer the questions below based on your plots:  1. How do the density plots compare to one another?  What general observation can you make about the typical SI1 and SI2 diamonds?  What are their similarities?<jupyter_code>plt.title("KDE Plot of Price to Depth for SI1 and SI2")
# dataset X
sns.kdeplot(
    database.loc[(database['clarity'] == 'SI1')]['depth'],
    database.loc[(database['clarity'] == 'SI1')]['price'],
    cmap="Reds",
    color='r',
    shade=True,
    shade_lowest=False, label="SI1")

# dataset Y
sns.kdeplot(
    database.loc[(database['clarity'] == 'SI2')]['depth'],
    database.loc[(database['clarity'] == 'SI2')]['price'],
    color='b',
    cmap="Blues",
    shade=True,
    shade_lowest=False, label="SI2")

plt.legend()
plt.savefig("your_plot_can_be_saved.png")<jupyter_output><empty_output><jupyter_text>2. What about the density plots for I1 and IF?  How are they different?  Please provide as much description to support your answer.<jupyter_code>plt.title("KDE Plot of Price to Depth for I1 and IF")
# dataset X
sns.kdeplot(
    database.loc[(database['clarity'] == 'I1')]['depth'],
    database.loc[(database['clarity'] == 'I1')]['price'],
    cmap="Reds",
    color='r',
    shade=True,
    shade_lowest=False, label="I1")

# dataset Y
sns.kdeplot(
    database.loc[(database['clarity'] == 'IF')]['depth'],
    database.loc[(database['clarity'] == 'IF')]['price'],
    color='b',
    cmap="Blues",
    shade=True,
    shade_lowest=False, label="IF")

plt.legend()
plt.savefig("your_plot_can_be_saved.png")<jupyter_output><empty_output><jupyter_text>3. From the plots alone, what can you make of the price density of 1-carat SI1 diamonds versus those of SI2 diamonds?<jupyter_code>plt.title("KDE Plot of Price to Carat for SI1 and SI2")

# dataset X
sns.kdeplot(
    database.loc[(database['carat'] == 1.0) & (database['clarity'] == 'SI1')]['carat'],
    database.loc[(database['carat'] == 1.0) & (database['clarity'] == 'SI1')]['price'],
    color='b',
    cmap="Blues",
    shade=True,
    shade_lowest=False, label="SI1")

# dataset Y
sns.kdeplot(
    database.loc[(database['clarity'] == 'SI2')]['carat'],
    database.loc[(database['clarity'] == 'SI2')]['price'],
    cmap="Reds",
    color='r',
    shade=True,
    shade_lowest=False, label="SI2")



plt.legend()
plt.savefig("your_plot_can_be_saved.png")<jupyter_output><empty_output><jupyter_text>4. What would you say of the claim: "If clarity is of great concern to you, the your best value diamonds are in found in the I1 clarity."?  What challenges could you make to this claim based on the KDE plots?  What are the supports for this claim?<jupyter_code>plt.title("KDE Plot of Price to Depth for IF and I1")
# dataset X
sns.kdeplot(
    database.loc[(database['clarity'] == 'IF')]['carat'],
    database.loc[(database['clarity'] == 'IF')]['price'],
    color='b',
    cmap="Blues",
    shade=True,
    shade_lowest=False,
    label="IF")

#plt.title("KDE Plot of Price to Depth for I1")
# dataset Y
sns.kdeplot(
    database.loc[(database['clarity'] == 'I1')]['carat'],
    database.loc[(database['clarity'] == 'I1')]['price'],
    cmap="Reds",
    color='r',
    shade=True,
    shade_lowest=False,
    label="I1")

plt.legend()
plt.savefig("your_plot_can_be_saved.png")<jupyter_output><empty_output><jupyter_text>5. Write a query using the [`DataFrame.query()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html)   method to show the descriptive statistics for _clarity I1_ and _clarity SI1_ diamonds.  You can use the [`DataFrame.describe()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html?highlight=describepandas.DataFrame.describe) method to get the descriptive statistics.  The query method will restrict diamonds between .95 and 1.05 carats for each color. Statistics for clarity I1 Diamonds: Condition:  clarity == "I1" and carat >= 0.95 and carat <= 1.05<jupyter_code>database.query(' clarity == "I1" and carat >= 0.95 and carat <= 1.05 ').describe()<jupyter_output><empty_output><jupyter_text>Statistics for clarity SI1 DiamondsCondition: clarity == "SI1" and carat >= 0.95 and carat <= 1.05<jupyter_code>database.query('clarity == "SI1" and carat >= 0.95 and carat <= 1.05 ').describe()<jupyter_output><empty_output><jupyter_text>6. Based on that query how can it be used as evidence to support the original claim in question 4 previously?<jupyter_code>database.loc[(database['clarity'] == 'IF')]['price'].describe()
database.loc[(database['clarity'] == 'I1')]['price'].describe()<jupyter_output><empty_output><jupyter_text>Compare to IF and I1 Clarity, the best value diamond is I1 becaouse the mean value maximun number 3924.168691 compare to IF clarity  (30%) Explore distance and similarity measures in Pandas and Scikit-learn  Normalize the data:<jupyter_code>from sklearn.preprocessing import normalize
import pandas as pd
import numpy as np


#Mapping Function
def mapping(data,feature):
    featureMap=dict()
    count=0
    for i in sorted(data[feature].unique(),reverse=True):
        featureMap[i]=count
        count=count+1
    data[feature]=data[feature].map(featureMap)
    return data
database = pd.read_csv('https://git.io/JUGqS')
database = pd.DataFrame(database)
print(database)<jupyter_output>carat        cut color clarity  depth  table  price     x     y     z
0       0.23      Ideal     E     SI2   61.5   55.0    326  3.95  3.98  2.43
1       0.21    Premium     E     SI1   59.8   61.0    326  3.89  3.84  2.31
2       0.23       Good     E     VS1   56.9   65.0    327  4.05  4.07  2.31
3       0.29    Premium     I     VS2   62.4   58.0    334  4.20  4.23  2.63
4       0.31       Good     J     SI2   63.3   58.0    335  4.34  4.35  2.75
...      ...        ...   ...     ...    ...    ...    ...   ...   ...   ...
53935   0.72      Ideal     D     SI1   60.8   57.0   2757  5.75  5.76  3.50
53936   0.72       Good     D     SI1   63.1   55.0   2757  5.69  5.75  3.61
53937   0.70  Very Good     D     SI1   62.8   60.0   2757  5.66  5.68  3.56
53938   0.86    Premium     H     SI2   61.0   58.0   2757  6.15  6.12  3.74
53939   0.75      Ideal     D     SI2   62.2   55.0   2757  5.83  5.87  3.64

[53940 rows x 10 columns]<jupyter_text>1. Once the data is normalized and your sample has been made, please save the resulting DataFrame to a CSV file called `normalized_datasample_5400k.txt`.  Also make sure the head of this dataset is also displayed in your notebook.<jupyter_code>database = mapping(database,"cut")
database = mapping(database,"color")
database = mapping(database,"clarity")
database
database.to_csv('normalized_datasample_5400k.txt')
pd.read_csv("normalized_datasample_5400k.txt")<jupyter_output><empty_output><jupyter_text>2. NOTE: We have  sample data rows [here in `sample_data.csv`](https://github.com/kmsaumcis/mcis6273_f20_datamining/tree/master/homework/hw1/sample_data.csv) for the next questions.  You might find it just as easy to insert these data into your original dataset and normalize when answering the questions, but there are other ways of doing the same thing.<jupyter_code>database = pd.read_csv("sample_data.csv")
database =pd.DataFrame(database)
database_df =database
database_df = mapping(database_df,"cut")
database_df = mapping(database_df,"color")
database_df = mapping(database_df,"clarity")
database_df<jupyter_output><empty_output><jupyter_text>3. For diamond 2 (the 0.38 carat, Ideal, G, VS1, $759), please find the 5 most similar diamonds from the sample   set that you have.  You will need to learn to use the [`sklearn.metrics.pairwise_distances`](https://Scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise_distances.html) method   obtain the distances and the [`numpy.argsort()`](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.argsort.html) method to determine the indices of the 5 closest diamonds.   Use the default Euclidean metric to perform the distances calculation.<jupyter_code>from sklearn.metrics import pairwise_distances
euclidean = list()
tempData="yes"

for df_data in database_df.itertuples():
    if(df_data.carat == 0.38):
        tempData="no"
        row1=(df_data.carat,df_data.cut,df_data.color,df_data.clarity,df_data.depth,df_data.table,df_data.x,df_data.y,df_data.z)

for df_data in database_df.itertuples():
    if(df_data.carat == 0.38):
        tempData="no"
        row1=(df_data.carat,df_data.cut,df_data.color,df_data.clarity,df_data.depth,df_data.table,df_data.x,df_data.y,df_data.z)
        euclidean.append(0)
    else:
        rowDf=(df_data.carat,df_data.cut,df_data.color,df_data.clarity,df_data.depth,df_data.table,df_data.x,df_data.y,df_data.z)
        X=[ list(rowDf)]
        Y=[list(row1)]
        distance_val= pairwise_distances(X, Y, metric='euclidean')[0][0]
        euclidean.append(distance_val)

database_df['euclidean'] = euclidean
print(database_df)

#Max to min
database_df= database_df.iloc[database_df['euclidean'].values.argsort(axis=0)[::-1]]
#top 5 euclidean distances based on the second row
database_df.head(5)<jupyter_output><empty_output><jupyter_text>4. Run normalization again, **but drop `price` from the sample before normalizing**.  Find the 5 closest diamonds. The price per carat for the diamond we were looking at is $1946.15.  How does this compare with the price per carat of the 5 most similar diamonds?  You will obviously need to keep the original dataset in order to determine the prices of the 5 most similar that you find.<jupyter_code>#drop price column
database_df=database_df.drop(["price"],axis=1)
database_df
# Find the 5 closest diamonds
database_df.iloc[(database['price']-1946.15).abs().argsort()[:5]]<jupyter_output><empty_output><jupyter_text>5. When looking at the top 5, what commonalities do they have with the sample diamond 2?  Differences? Ans:-<jupyter_code>top5=database_df.head()
top5<jupyter_output><empty_output><jupyter_text>6. Perform the same analysis on diamond 14, (1.13 carat, Ideal, F, VS2, $6283).  Again how does the price per carat compare?  Use the data you have as evidence to back up your answer.<jupyter_code>##14, (1.13 carat, Ideal, F, VS2, $6283).
db_results = database.loc[(database['carat'] == 1.13) & (database['cut'] == 2) & (database['color'] == 2) & (database['clarity'] == 1)]
price_value = int(db_results.price)
if(price_value == 6283):
    print("True Statement for diamond #14, (1.13 carat, Ideal, F, VS2, $6283).")
else:
    print("False Statement for diamond #14, (1.13 carat, Ideal, F, VS2, $6283).")<jupyter_output>False Statement for diamond #14, (1.13 carat, Ideal, F, VS2, $6283).<jupyter_text>7. What are the similarities / differences amongst the top 5?  Please be specific in your answer.<jupyter_code>database.head()
database.head().diff()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13030488463338252, 'fraction_numerical': 0.02483711889420827, 'mean_word_length': 5.549269212635549, 'pattern_counts': {'":': 0, '<': 157, '<?xml version=': 0, '>': 157, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '300733', 'license': 'CC0-1.0', 'n_tokens_mistral': 9663, 'n_tokens_neox': 9138, 'n_words': 2468, 'path': 'mcis6273_f20_datamining/homework/hw1/hw1.ipynb', 'repo_name': 'Manideepak1/DataMining', 'size': '27791'}