<jupyter_start><jupyter_text>Introducing the diamond prices dataset**Data Set Information:**A dataset containing the prices and other features of almost 54,000 diamonds.**Number of Attributes: 10**Feature Information: A data frame with 53,940 rows and 10 variables:price: price in US dollarscarat: weight of the diamondcut: quality of the cut (Fair, Good, Very Good, Premium, Ideal)color: diamond colour, from J (worst) to D (best)clarity: a measurement of how clear the diamond is (I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best))x: length in mmy: width in mmz: depth in mmdepth: total depth percentage = z / mean(x, y) = 2 * z / (x + y)table: width of top of diamond relative to widest point<jupyter_code>import numpy as np
import pandas as pd
import os<jupyter_output><empty_output><jupyter_text>Loading the data<jupyter_code>DATA_DIR = '../data'
FILE_NAME = 'diamonds.csv'
data_path = os.path.join(DATA_DIR, FILE_NAME)
diamonds = pd.read_csv(data_path)
diamonds.shape
diamonds.head()<jupyter_output><empty_output><jupyter_text>Checking summary statistiscs for numerical columns<jupyter_code>diamonds.describe()<jupyter_output><empty_output><jupyter_text>Checking the values of x equal to zero<jupyter_code>diamonds.loc[diamonds['x']==0]
diamonds = diamonds.loc[(diamonds['x']>0) | (diamonds['y']>0)]
diamonds.loc[11182]
diamonds.loc[11182, 'x'] = diamonds['x'].median()
diamonds.loc[diamonds['x']==0].shape
diamonds.loc[diamonds['y']==0]
diamonds.loc[diamonds['z']==0]
diamonds.describe()
diamonds.loc[(diamonds['y'] > 30) | (diamonds['z'] > 30)]
diamonds = diamonds.loc[~((diamonds['y'] > 30) | (diamonds['z'] > 30))]
diamonds.describe()<jupyter_output><empty_output><jupyter_text>One-hot encoding with pandas<jupyter_code>pd.get_dummies(diamonds['cut'], prefix='cut').head()
pd.get_dummies(diamonds['cut'], prefix='cut', drop_first=True).head()<jupyter_output><empty_output><jupyter_text>Adding the one-hot encoded categorical features<jupyter_code>diamonds = pd.concat([diamonds, pd.get_dummies(diamonds['cut'], prefix='cut', drop_first=True)], axis=1)
diamonds = pd.concat([diamonds, pd.get_dummies(diamonds['color'], prefix='color', drop_first=True)], axis=1)
diamonds = pd.concat([diamonds, pd.get_dummies(diamonds['clarity'], prefix='clarity', drop_first=True)], axis=1)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15278377211911956, 'fraction_numerical': 0.021148036253776436, 'mean_word_length': 8.384615384615385, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '607905', 'license': 'MIT', 'n_tokens_mistral': 893, 'n_tokens_neox': 808, 'n_words': 186, 'path': 'Chapter02/ch2-1-diamond-prices.ipynb', 'repo_name': 'arifmudi/Hands-On-Predictive-Analytics-with-Python', 'size': '2317'}