<jupyter_start><jupyter_text>We mostly have entries for all the variables. We're missing:* 8 entries about MasVnr (apparently Masonry Veneer), which is nothing big, * about 40 entires about basements,* 1 entry in Electrical,* around 80 entries about garageswhich are all fine, and (we'll check) this probably means these features just aren't present in the given houses and we can just replace the nan with 'No" like we did for fireplaces.Unfortunately, we're also missing about 260 entries about LotFrontage, which is a substantial chunk of our data, and also could concievably be important in determining house price. We can take a look at the observations that are missing LotFrontage, maybe a missing value just means the lot actually doesn't have any linear frontage<jupyter_code>train.loc[pd.isnull(train.LotFrontage)]<jupyter_output><empty_output><jupyter_text>Unfortunately, it looks like the frontage data is legitimately missing (but should exist). With just a cursory glance over these data we can see that HouseID 67 has a lot area of almost 20000 sqft in a regular LotShape, so it should almost certainly have some frontage. House ID 1363 is described in LotConfig as having frontage on 2 sides, so it can't be the case that it just doesn't have any frontage for some reason. So, we shouldn't just replace the missing data with 0, since we know there's probably a real value it should have. It would also be painful to just discard all these houses from our data set, since they account for around 1/7 of our training data. It looks like the best we can do is make some educated guessing about what the values might be, and replace the NaNs with, say, the mean of the LotFrontage data we do have.<jupyter_code>#Instead of just using the mean of LotFrontage, first let's take a look and see if there's any reasonable relation we can detect between LotArea and LotFrontage.
#We have all the LotArea data, so maybe we can use that to make better guesses about the missing frontage.

plt.scatter(train.LotArea, train.LotFrontage)
plt.show()

#Let's zoom in to get a better sense of what's happening away from the outliers
plt.scatter(train.LotArea, train.LotFrontage)
plt.xlim(0,30000)
plt.ylim(0,200)
plt.show()

#There's clearly a general trend upwards, but it's obviously still very noisy. It's probably not worth it at this point to try anything fancier than just using the mean
#Let's look at a summary of the LotFrontage data
train.LotFrontage.describe()
#Looks like the mean is 70 and the median is 69. A pretty nice distribution. Let's go ahead and replace the missing values with the mean (removing NaNs from the computation)

train.LotFrontage= train.LotFrontage.fillna(np.nanmean(train.LotFrontage))

#Now let's do a quick check for the other missing values, to see whether the NaNs just correspond to houses without the given features,
#so we can replace them with some definite 'none' category

#Again, we're missing
#* 8 entries about MasVnr,
#* about 40 entires about basements,
#* 1 entry in Electrical, which we'll handle on its own
#* around 80 entries about garages

#Count the houses with MasVnr
np.array((sum(train.MasVnrArea >0), sum(train.TotalBsmtSF>0), sum(train.GarageArea >0)))<jupyter_output><empty_output><jupyter_text>Looks like we're right about the Basement and Garage issues, the NaNs just correspond to houses without either basements or garages (in categorical variables), so we can just add a category for 'none'. Something else is going on with the Masonry Veneer, since only 591 houses are recorded as having more than 0 sqft Masonry Veneer, but 1452 houses are described as having a Masonry Veneer Type (and 8 are missing a type). Let's see what's going on with the houses that are missing the type.<jupyter_code>train.loc[pd.isnull(train.MasVnrType)]
train.MasVnrType.unique()
#It looks like the MasVnrType field already inclues a 'None' category for those houses that don't have Masonry Veneer
#and the few house that are missing entries are also the houses missing entries in the MasVnrArea field.

#Since only 591 houses actually have Masonry Veneer, let's just assume the houses that don't have anything listed also don't have it,
#and we'll go ahead and fill out the MasVnrArea column with 0 and the MasVnrType column with 'None'

train.MasVnrArea = train.MasVnrArea.fillna(0)
train.MasVnrType = train.MasVnrType.fillna('None')
#Let's take a look at the house without Electrical data
train.loc[pd.isnull(train.Electrical)]

#Seems like a reasonable house: Built in 2006, remodeled in 2007, sold in 2008 for $167500. It has central air and all public Utilities. 
#It's doubtful that it just doesn't have electricity. We can just fill in the Electrical value with whatever the most common kind of Electrical is. 
#Take a look at our Electrical field
train.Electrical.describe()

#The overwhelming majority of houses just have SBrkr - Standard Circuit Breakers & Romex, so let's go ahead and say this probably has the same. 
#In any case, if the Electrical data is so uniform it's quite doubtful that it will have an effect on the model we build. 

train.Electrical = train.Electrical.fillna('SBrkr')<jupyter_output><empty_output><jupyter_text>Now let's just clean up the Garage and Basement issues. We'll replace the missing qualities with 'None'.<jupyter_code>#The basement fields that have NaNs are:
#BsmtQual
#BsmtCond
#BsmtExposure
#BsmtFinType1
#BsmtFinType2

#One thing we have to note is that 2 of the fields, BsmtExposure and BsmtFinType2 are missing 1 more entry than all the others 
#(and in fact, there are 1423 entries with non-zero basements, so at least one of these is missing these fields)
#For the other fields, we'll just fill the missing values in with 'None'
np.array((train.BsmtQual.unique(), train.BsmtCond.unique(), train.BsmtExposure.unique(), train.BsmtFinType1.unique(), train.BsmtFinType2.unique()))

train.BsmtQual = train.BsmtQual.fillna('None')
train.BsmtCond = train.BsmtCond.fillna('None')
train.BsmtFinType1 = train.BsmtFinType1.fillna('None')
#Let's see the house that missing BsmtFinType2, but not BsmtFinType 1
train.loc[pd.isnull(train.BsmtFinType2) & (train.TotalBsmtSF>0)]

#It seems this house actually has almost all the basement information listed (it even an entry for BsmtExposure, so there's a different house that's missing that)

#Let's take a look at BsmtFinType2 to see what makes sense to fill in here
train.BsmtFinType2.describe()
#It looks like the vast majority of houses with basements have BsmtFinType2 as 'Unfinished'
#That's a little weird, given that BsmtFinType2 is supposed to be the rating of basement *finished* area,
#but maybe it's just how they record unfinished sections of basement
#Let's check
sum((train.BsmtFinType2=='Unf') & (train.BsmtFinSF2<train.BsmtUnfSF))

#Ok, so actually most of the houses have 'Unfinished' BsmtFinType2, and and that doesn't actually count all the unfinished basement square footage.
#It seems like a weird way to do it, but that's fine.
#In this case, it probably makes the most sense to go ahead and set the missing BsmtFinType2 to 'Unfinished' as well
train.loc[332,'BsmtFinType2'] = 'Unf'

#Let's take a look at the last missing basement data point
train.loc[pd.isnull(train.BsmtExposure) & (train.TotalBsmtSF>0)]
#It look like this house has a totally unfinished basement, so let's go ahead and assume it also doesn't have basement exposure

train.loc[948, 'BsmtExposure'] = 'No'
#Now the rest of the BsmtFinType2 and BsmtExposure NaNs are just houses without basements, so we can go ahead and fill those in with the category 'None'

train.BsmtFinType2 = train.BsmtFinType2.fillna('None')
train.BsmtExposure = train.BsmtExposure.fillna('None')

#Let's take a last look at our data, and make sure the only remaining null entries are in our data on garages
train.info()

#Recall that we had 1379 houses with GarageArea greater than 0, so the missing values are just houses without garages
#So, let's just fill in all the missing data with 'None'
#or, for GarageYrBlt, which is a number, lets use '0', as something out of the acceptable range (years in the Common Era) for garages to have been built.
#Presumably, more modern garages are worth more, so not having a garage should map onto roughly the same idea as having a very very very old one. 

#We'll fill the missing entries in the Year Built field with zeroes first
train.GarageYrBlt = train.GarageYrBlt.fillna(0)

#Now the only missing entries in the table at all are those in the garage fields under consideration, so we can just fill the rest of the table
train = train.fillna('None')<jupyter_output><empty_output><jupyter_text>Beautiful! All our data is nice and clean now. Let's start working on actually making models and predictions. First let's split off our target variable, and then separate the data we have into a training set and a cross-validation set so we can test things out for ourselves before submitting.<jupyter_code>target = train.loc[:,'SalePrice']
train = train.drop('SalePrice', axis=1)
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(train, target, test_size=0.3, random_state=40)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06434083251820455, 'fraction_numerical': 0.013802847516574285, 'mean_word_length': 5.151069518716578, 'pattern_counts': {'":': 0, '<': 22, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '538061', 'license': 'MIT', 'n_tokens_mistral': 2827, 'n_tokens_neox': 2625, 'n_words': 1311, 'path': 'houseprices.ipynb', 'repo_name': 'icherbert/Housing-Kaggle', 'size': '9201'}