<jupyter_start><jupyter_text>Challenge  3Task: Perform data cleaning Data src: [house-prices-advanced-regression-techniques](https://www.kaggle.com/c/house-prices-advanced-regression-techniques)Github src for downloading raw data: [github.com/moyphilip](https://raw.githubusercontent.com/moyphilip/Ames-Housing/master/train.csv)Google Colab for this notebook: [link](https://colab.research.google.com/drive/1BDv0x-mYqTJAHfS2XttTGvcffTfrtcRW?usp=sharing)By: <NAME> Loading libraries<jupyter_code>import os
import pandas as pd
import seaborn as sns
import warnings
from pathlib import Path

warnings.filterwarnings('ignore')
pd.set_option('display.precision', 2)<jupyter_output><empty_output><jupyter_text>Loading data<jupyter_code>wget 'https://raw.githubusercontent.com/moyphilip/Ames-Housing/master/train.csv'
cwd = os.getcwd()
print(cwd)
PATH = Path(cwd)<jupyter_output>D:\PYTHON_PROJECTS\ML\SaturdaysAI\saturdaysai\notebooks<jupyter_text>1. Leer los datos del CSV creando un data frame de pandas<jupyter_code>df = pd.read_csv(PATH/'train.csv')
df.head()<jupyter_output><empty_output><jupyter_text>2.  ¿Cuántas columnas hay en el data frame?<jupyter_code>print(f'Theare are {len(df.columns)} columns')<jupyter_output>Theare are 81 columns<jupyter_text>3. Calcula los descriptores estadísticos básicos para las columnas numéricas<jupyter_code>df.describe().T<jupyter_output><empty_output><jupyter_text>4. ¿Cuántas columnas numéricas y cuántas columnas categóricas hay?<jupyter_code># df = df.convert_dtypes()
df.info()
df.convert_dtypes().info()
# df = df.convert_dtypes()
numeric_cols = df.select_dtypes(include='number').columns
numeric_cols
cat_cols = df.select_dtypes(exclude='number').columns
cat_cols
print(f'Total numeric columns:', len(numeric_cols))
print(f'Total categorical columns: ', len(cat_cols))<jupyter_output>Total numeric columns: 38
Total categorical columns:  43<jupyter_text>5. ¿Cuáles variables tienen más del 50% de valores NaN?<jupyter_code>null_cols = df.isnull().sum()/len(df) * 100
null_cols = df.isnull().sum()/len(df) * 100
null_cols[null_cols > 50.].sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>6. ¿Cuántos valores faltantes tiene cada una de las variables categóricas?<jupyter_code># From question 4: using cat_cols as a filter
null_cols[cat_cols].sort_values(ascending=False)
ax = sns.heatmap(df[cat_cols].isnull().T,
                 xticklabels=False,
                 cbar=False, cmap='viridis_r')
ax.set_title('Missing values for categorical columns');<jupyter_output><empty_output><jupyter_text>7. ¿Cuántos valores faltantes tiene cada una de las variables numéricas?<jupyter_code># From question 4: using numeric_cols as a filter
null_cols[numeric_cols].sort_values(ascending=False)/100 * len(df)
df[numeric_cols].isnull().sum().sort_values(ascending=False)
ax = sns.heatmap(df[numeric_cols].isnull().T, 
                 xticklabels=False,
                 cbar=False, cmap='viridis_r')
ax.set_title('Missing values for numerical columns');<jupyter_output><empty_output><jupyter_text>8. Sustituye los valores faltantes de la columna YrSold con la media de esa columna<jupyter_code>print(df['YrSold'].value_counts(dropna=False))
print('\nThere are not missing values in YrSold column')<jupyter_output>2009    338
2007    329
2006    314
2008    304
2010    175
Name: YrSold, dtype: int64

There are not missing values in YrSold column<jupyter_text>9. Sustituye los valores faltantes de la columna LotFrontage con la mediana de esa columna<jupyter_code>df['LotFrontage'].fillna(df['LotFrontage'].median(), inplace=True)
df['LotFrontage'].value_counts(dropna=False)<jupyter_output><empty_output><jupyter_text>10. Codifica la columna SaleCondition utilizando One-Hot-Encoding<jupyter_code>df['SaleCondition'].value_counts()
df = pd.get_dummies(data=df,
                    columns=['SaleCondition'],
                    drop_first=True
                    )
df.iloc[:, -6:]<jupyter_output><empty_output><jupyter_text>11. Crea una columna TotalBath a partir de la suma de baños completos e incompletos (debes descubrir cuales columnas sumar). Hint: los cantidad de baños incompletos debe multiplicarse por 0.5 antes de sumarse.```BsmtFullBath: Basement full bathroomsBsmtHalfBath: Basement half bathroomsFullBath: Full bathrooms above gradeHalfBath: Half baths above grade```<jupyter_code>df.eval(
    'TotalBath = BsmtFullBath + 0.5 * BsmtHalfBath + FullBath + 0.5 * HalfBath',
    inplace=True
    )

df['TotalBath'].value_counts()
df.head()<jupyter_output><empty_output><jupyter_text>12. Grafica la distribución de la columna LotArea<jupyter_code>ax = sns.distplot(df['LotArea'], hist=False)
ax.set_xscale('log')<jupyter_output><empty_output><jupyter_text>13. Grafica la distribución de la columna SalePrice<jupyter_code>df['SalePrice'].plot(kind='hist',)
df['SalePrice'].plot.kde()
ax = sns.distplot(df['SalePrice'], hist=False)
ax.set_xscale('log')<jupyter_output><empty_output><jupyter_text>14. Elimina la columna Id del dataset.<jupyter_code>df.drop(['LotArea', 'Street'], axis=1, inplace=True)
df.head()
del df['Id']
df.head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12053832650672908, 'fraction_numerical': 0.017554125219426564, 'mean_word_length': 6.541176470588235, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 63, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '207826', 'license': 'MIT', 'n_tokens_mistral': 1976, 'n_tokens_neox': 1868, 'n_words': 355, 'path': 'notebooks/03_data_preprocessing.ipynb', 'repo_name': 'christianvadillo/saturdaysai', 'size': '5138'}