<jupyter_start><jupyter_text>Imports<jupyter_code>%load_ext autoreload
%autoreload 2

%matplotlib inline
from fastai.imports import *
from fastai.structured import *

from pandas_summary import DataFrameSummary
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from IPython.display import display

from sklearn import metrics

pd.set_option('max_colwidth',200)
pd.set_option("display.max_rows",101)
pd.set_option("display.max_columns",101)
PATH = "/data/msnow/kaggle/house_prices/"
!ls {PATH}<jupyter_output>test.csv  train.csv<jupyter_text>Data Exploration  Explore the dataset before loading all the rows Determine the number of lines in the csv file<jupyter_code>tst_filename = f'{PATH}train.csv'
sum(1 for line in open(tst_filename))
train_raw = pd.read_csv(f'{PATH}train.csv', nrows = 10**2)
trh = train_raw.head().T
trh['dtypes'] = train_raw.dtypes
trh
any(x%1 for x in train_raw.LotFrontage)
train_raw.describe().T<jupyter_output><empty_output><jupyter_text>Load the full dataset<jupyter_code>train_raw = pd.read_csv(f'{PATH}train.csv')
train_raw.describe().T
data_dict = {"SalePrice": "the property's sale price in dollars. This is the target variable that you're trying to predict.","MSSubClass": "The building class","MSZoning": "The general zoning classification","LotFrontage": "Linear feet of street connected to property","LotArea": "Lot size in square feet","Street": "Type of road access","Alley": "Type of alley access","LotShape": "General shape of property","LandContour": "Flatness of the property","Utilities": "Type of utilities available","LotConfig": "Lot configuration","LandSlope": "Slope of property","Neighborhood": "Physical locations within Ames city limits","Condition1": "Proximity to main road or railroad","Condition2": "Proximity to main road or railroad (if a second is present)","BldgType": "Type of dwelling","HouseStyle": "Style of dwelling","OverallQual": "Overall material and finish quality","OverallCond": "Overall condition rating","YearBuilt": "Original construction date","YearRemodAdd": "Remodel date","RoofStyle": "Type of roof","RoofMatl": "Roof material","Exterior1st": "Exterior covering on house","Exterior2nd": "Exterior covering on house (if more than one material)","MasVnrType": "Masonry veneer type","MasVnrArea": "Masonry veneer area in square feet","ExterQual": "Exterior material quality","ExterCond": "Present condition of the material on the exterior","Foundation": "Type of foundation","BsmtQual": "Height of the basement","BsmtCond": "General condition of the basement","BsmtExposure": "Walkout or garden level basement walls","BsmtFinType1": "Quality of basement finished area","BsmtFinSF1": "Type 1 finished square feet","BsmtFinType2": "Quality of second finished area (if present)","BsmtFinSF2": "Type 2 finished square feet","BsmtUnfSF": "Unfinished square feet of basement area","TotalBsmtSF": "Total square feet of basement area","Heating": "Type of heating","HeatingQC": "Heating quality and condition","CentralAir": "Central air conditioning","Electrical": "Electrical system","1stFlrSF": "First Floor square feet","2ndFlrSF": "Second floor square feet","LowQualFinSF": "Low quality finished square feet (all floors)","GrLivArea": "Above grade (ground) living area square feet","BsmtFullBath": "Basement full bathrooms","BsmtHalfBath": "Basement half bathrooms","FullBath": "Full bathrooms above grade","HalfBath": "Half baths above grade","Bedroom": "Number of bedrooms above basement level","Kitchen": "Number of kitchens","KitchenQual": "Kitchen quality","TotRmsAbvGrd": "Total rooms above grade (does not include bathrooms)","Functional": "Home functionality rating","Fireplaces": "Number of fireplaces","FireplaceQu": "Fireplace quality","GarageType": "Garage location","GarageYrBlt": "Year garage was built","GarageFinish": "Interior finish of the garage","GarageCars": "Size of garage in car capacity","GarageArea": "Size of garage in square feet","GarageQual": "Garage quality","GarageCond": "Garage condition","PavedDrive": "Paved driveway","WoodDeckSF": "Wood deck area in square feet","OpenPorchSF": "Open porch area in square feet","EnclosedPorch": "Enclosed porch area in square feet","3SsnPorch": "Three season porch area in square feet","ScreenPorch": "Screen porch area in square feet","PoolArea": "Pool area in square feet","PoolQC": "Pool quality","Fence": "Fence quality","MiscFeature": "Miscellaneous feature not covered in other categories","MiscVal": "Dollar Value of miscellaneous feature","MoSold": "Month Sold","YrSold": "Year Sold","SaleType": "Type of sale","SaleCondition": "Condition of sale"}<jupyter_output><empty_output><jupyter_text>Initial Processing Convert the outcome of interest to the metric asked for, i.e., RMSLE of SalePrice<jupyter_code>train_raw.SalePrice = np.log(train_raw.SalePrice)<jupyter_output><empty_output><jupyter_text>Look at columns with missing data<jupyter_code>train_raw.isnull().sum().sort_index()/len(train_raw)<jupyter_output><empty_output><jupyter_text>Convert strings to pandas categories<jupyter_code>train_cats(train_raw)<jupyter_output><empty_output><jupyter_text>Replace categories with numeric codes, handle missing continuous values and split off dependent variable<jupyter_code>df_train, y, nas = proc_df(train_raw, 'SalePrice')
nas<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12849371600075032, 'fraction_numerical': 0.0052522978803226416, 'mean_word_length': 8.837638376383763, 'pattern_counts': {'":': 80, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '205169', 'license': 'MIT', 'n_tokens_mistral': 1660, 'n_tokens_neox': 1566, 'n_words': 484, 'path': 'kaggle/house_prices.ipynb', 'repo_name': 'mseinstein/side_projects', 'size': '5331'}