<jupyter_start><jupyter_text>[House Prices: Advanced Regression Techniques | Kaggle](https://www.kaggle.com/c/house-prices-advanced-regression-techniques) Code Link: [Github](https://github.com/AutuanLiu/Kaggle-Compettions/tree/master/House%20Prices)<jupyter_code>%matplotlib inline
%reload_ext autoreload
%autoreload 2
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
import pandas as pd
import numpy as np
import xgboost as xgb
import seaborn as sns
import matplotlib.pyplot as plt
import re
from collections import Counter
from sklearn.preprocessing import LabelEncoder
sns.set_style('whitegrid')
sns.set_palette('Set1')<jupyter_output><empty_output><jupyter_text>1 导入数据<jupyter_code>datadir = 'dataset/'
# read data from file
train_data = pd.read_csv(f'{datadir}train.csv')
test_data = pd.read_csv(f'{datadir}test.csv')<jupyter_output><empty_output><jupyter_text>2 数据基本信息<jupyter_code>set(train_data.columns) - set(test_data.columns)
train_data.columns.shape
test_data.columns.shape
train_data.head()
test_data.head()
train_data.shape
test_data.shape
train_data.describe()
test_data.describe()
train_data.info()
test_data.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 1460 entries, 0 to 1459
Data columns (total 81 columns):
Id               1460 non-null int64
MSSubClass       1460 non-null int64
MSZoning         1460 non-null object
LotFrontage      1201 non-null float64
LotArea          1460 non-null int64
Street           1460 non-null object
Alley            91 non-null object
LotShape         1460 non-null object
LandContour      1460 non-null object
Utilities        1460 non-null object
LotConfig        1460 non-null object
LandSlope        1460 non-null object
Neighborhood     1460 non-null object
Condition1       1460 non-null object
Condition2       1460 non-null object
BldgType         1460 non-null object
HouseStyle       1460 non-null object
OverallQual      1460 non-null int64
OverallCond      1460 non-null int64
YearBuilt        1460 non-null int64
YearRemodAdd     1460 non-null int64
RoofStyle        1460 non-null object
RoofMatl         1460 non-null object
Exterior1st      1460 non-n[...]<jupyter_text>* 为了防止更改原始数据，这里复制数据<jupyter_code>train_set = train_data.copy()
test_set = test_data.copy()
del train_data, test_data<jupyter_output><empty_output><jupyter_text>* 数据清理时一起进行，所以将其合并<jupyter_code>dataset = pd.concat([train_set, test_set], axis=0, sort=False, ignore_index=True)
dataset.shape
dataset.head()
dataset.tail()
dataset.loc[:2, 'MSZoning']<jupyter_output><empty_output><jupyter_text>* 查看空值信息<jupyter_code>dataset.shape
a = dataset.isna().sum()
data_na = a[a > 0]
data_na<jupyter_output><empty_output><jupyter_text>* 空值所占的比例<jupyter_code>per = data_na/dataset.shape[0]
per<jupyter_output><empty_output><jupyter_text>* 超过一半都是空值的列<jupyter_code>col50 = per[per > 0.5]
col50<jupyter_output><empty_output><jupyter_text>3 数据清洗<jupyter_code>dataset.columns<jupyter_output><empty_output><jupyter_text>3.1 分类变量 |var|define||---|---||MSSubClass| 待售房子类型、风格等||MSZoning|房子的用途、工业、农业等||Street|街道道路类型||Alley|小巷的道路类型，NA表示没有小巷||LotShape|房子的形状||LandContour|房子的平整度，坡道等||Utilities|公共设施的可使用情况||LotConfig|房子的配置||LandSlope|房子的倾斜程度||Neighborhood|城市范围内的位置||Condition1|房子的条件，临街、铁路等||Condition2|房子的条件，临街、铁路等，有多个||BldgType|建筑类型，独栋等||HouseStyle|住宅类型，一层还是一层半等||OverallQual|房子的整体评价与装修||OverallCond|整体评分||YearBuilt|建造时间||YearRemodAdd|重装时间，NA表示与建成时间相同||RoofStyle|屋顶样式||RoofMatl|屋顶材料||Exterior1st|房子的外墙材料类型||Exterior2nd|房子的外墙材料类型，超过一种||MasVnrType|砖的类型，None表示没有||ExterQual|外墙的材料质量等评分||ExterCond|外墙材料的评价||Foundation|基础条件的评价，砖、煤等||BsmtQual|地下室的高度，NA表示没有地下室||BsmtCond|地下室条件的评价||BsmtExposure|地下室墙壁情况||BsmtFinType1|已完工地下室等级，适合居住么||BsmtFinType2|已完工地下室类型2如果有多个的话||Heating|取热类型，煤等||HeatingQC|取热质量评价||CentralAir|有没有中央空调||Electrical|电力系统的类型||BsmtFullBath|地下室满浴室数量||BsmtHalfBath|地下室半浴室数量||FullBath|满浴室数量||HalfBath|半浴室数量||Bedroom|楼上卧室数量||Kitchen|厨房数量||KitchenQual|厨房质量评价||TotRmsAbvGrd|房间数量，不包括浴室||Functional|房间的功能性||Fireplaces|壁炉的数量||FireplaceQu|炉的质量||GarageType|车库位置，NA表示没有||GarageYrBlt|车库建成时间||GarageFinish|车库装修情况||GarageCars|车库的容纳car数量||GarageQual|车库的质量评价||GarageCond|车库的条件评价||PavedDrive|铺设车道||PoolQC|游泳池质量评价，NA表示没有游泳池||Fence|栅栏质量， NA表示没有||MiscFeature|其他类别中未包含的杂项功能||MoSold|月销量||YrSold|年销量||SaleType|销售类型||SaleCondition|销售条件|  3.1.1 第一组数据 |var|define||---|---||MSSubClass| 待售房子类型、风格等||MSZoning|房子的用途、工业、农业等||Street|街道道路类型||Alley|小巷的道路类型，NA表示没有小巷||LotShape|房子的形状|<jupyter_code>cat1 = ['MSSubClass', 'MSZoning', 'Street', 'Alley', 'LotShape']
dataset[cat1].isna().sum()  # 不存在空值且为分类变量
df = dataset.loc[:, cat1]
df.info()
df.head()
def t_type(l):
    for i in l:
        dataset[i].astype('float64')
def t_type2(l):
    for i in l:
        dataset[i].astype('str')
t_type(['MSSubClass'])
t_type2(['MSZoning', 'Street', 'Alley', 'LotShape'])<jupyter_output><empty_output><jupyter_text>* 创造一个新的变量，表示是否有 Alley<jupyter_code># dataset['isAlley'] = 1   # 初始化
# dataset['isAlley'].loc[dataset['Alley'].isna()] = 0
dataset.loc[dataset.MSZoning.isna()]
for label in cat1:
    print(label)
    print(dataset[label].value_counts(), '\n-----')<jupyter_output>MSSubClass
20     1079
60      575
50      287
120     182
30      139
160     128
70      128
80      118
90      109
190      61
85       48
75       23
45       18
180      17
40        6
150       1
Name: MSSubClass, dtype: int64 
-----
MSZoning
RL         2265
RM          460
FV          139
RH           26
C (all)      25
Name: MSZoning, dtype: int64 
-----
Street
Pave    2907
Grvl      12
Name: Street, dtype: int64 
-----
Alley
Grvl    120
Pave     78
Name: Alley, dtype: int64 
-----
LotShape
Reg    1859
IR1     968
IR2      76
IR3      16
Name: LotShape, dtype: int64 
-----<jupyter_text>* MSZoning 使用众数填充* MSSubClass 暂时不用处理* Alley 使用 None 填充<jupyter_code>mode_v = dataset.MSZoning.mode().iloc[0]
dataset.MSZoning.fillna(mode_v, inplace=True)
dataset.Alley.fillna('None', inplace=True)
# dataset.loc[[1915, 2216, 1414], cat1+['isAlley']]<jupyter_output><empty_output><jupyter_text>3.1.2 第二组数据 |var|define||---|---||LandContour|房子的平整度，坡道等||Utilities|公共设施的可使用情况||LotConfig|房子的配置||LandSlope|房子的倾斜程度||Neighborhood|城市范围内的位置|<jupyter_code>def get_info(cat):
    print('空值信息')
    print(dataset[cat].isna().sum())
    for label in cat:
        print(label, ':')
        print(dataset[label].value_counts(), '\n-----')
cat2 = ['LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Neighborhood']
get_info(cat2)
dataset.loc[:, cat2].head()
t_type2(cat2)
dataset.loc[dataset.Utilities.isna()]<jupyter_output><empty_output><jupyter_text>* 使用众数 AllPub 公共区域填充 Utilities 空值<jupyter_code>dataset.Utilities.fillna('AllPub', inplace=True)
dataset.loc[[1915, 1945], cat2]<jupyter_output><empty_output><jupyter_text>3.1.3 第三组数据 |var|define||---|---||Condition1|房子的条件，临街、铁路等||Condition2|房子的条件，临街、铁路等，有多个||BldgType|建筑类型，独栋等||HouseStyle|住宅类型，一层还是一层半等||OverallQual|房子的整体评价与装修|<jupyter_code>cat3 = ['Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'OverallQual']
get_info(cat3)
dataset.loc[:, cat3].head()<jupyter_output><empty_output><jupyter_text>* 这里不涉及空值的问题* Condition1 和 Condition2 要单独处理* 增加变量n_condition表示具备条件的数量（把Norm当做一种条件）<jupyter_code># dataset['n_condition'] = 1 # 初始化
# dataset['n_condition'].loc[dataset['Condition2'] != 'Norm'] += 1
# dataset.loc[5:10, cat3+['n_condition']]<jupyter_output><empty_output><jupyter_text>3.1.4 第四组数据 |var|define||---|---||OverallCond|整体评分||YearBuilt|建造时间||YearRemodAdd|重装时间，NA表示与建成时间相同||RoofStyle|屋顶样式||RoofMatl|屋顶材料|<jupyter_code>cat4 = ['OverallCond', 'YearBuilt', 'YearRemodAdd', 'RoofStyle', 'RoofMatl']
get_info(cat4)<jupyter_output>空值信息
OverallCond     0
YearBuilt       0
YearRemodAdd    0
RoofStyle       0
RoofMatl        0
dtype: int64
OverallCond :
5    1645
6     531
7     390
8     144
4     101
3      50
9      41
2      10
1       7
Name: OverallCond, dtype: int64 
-----
YearBuilt :
2005    142
2006    138
2007    109
2004     99
2003     88
1977     57
1920     57
1976     54
1999     52
2008     49
1958     48
2000     48
2002     47
1998     46
1968     45
1959     43
1954     43
1910     43
1970     42
1967     41
1972     40
1956     39
1971     39
1978     39
1993     39
1950     38
1994     37
1960     37
1940     36
1966     35
       ... 
1982      7
1931      7
1985      7
1942      6
1912      5
1919      5
1932      5
1880      5
1934      5
2010      3
1905      3
1895      3
1917      3
1908      2
1885      2
1901      2
1892      2
1882      1
1896      1
1898      1
1902      1
1907      1
1904      1
1906      1
1913      1
1911      1
1875      1
1879      1
1893      1
1872      1
Name:[...]<jupyter_text>* 这组数据不存在空值* 关键在于处理年份信息<jupyter_code>dataset.YearBuilt.dtype
# 是否有异常值
dataset.YearBuilt.loc[dataset.YearBuilt > 2018]
dataset.YearRemodAdd.loc[dataset.YearRemodAdd > 2018]<jupyter_output><empty_output><jupyter_text>* 增加建造到重建之间的年限 re_years<jupyter_code># dataset['re_years'] = dataset.YearRemodAdd - dataset.YearBuilt<jupyter_output><empty_output><jupyter_text>* 增加是否重建的变量 is_rebuild<jupyter_code># dataset['is_rebuild'] = 1
# dataset['is_rebuild'].loc[dataset['re_years'] == 0] = 0
# dataset['re_years'].sample(5)
# dataset['is_rebuild'].sample(5)<jupyter_output><empty_output><jupyter_text>* 将建造年份\重建年份更改为分类数据<jupyter_code>dataset['YearBuilt'].plot(kind='kde')
dataset['YearRemodAdd'].plot(kind='kde')
# dataset['YearBuiltbin'] = pd.qcut(dataset['YearBuilt'], 4)
# dataset['YearRemodAddbin'] = pd.qcut(dataset['YearRemodAdd'], 4)
# dataset['YearBuiltbin'].sample(10)
# dataset['YearRemodAddbin'].sample(10)<jupyter_output><empty_output><jupyter_text>* 删除原始label列<jupyter_code># dataset.drop(labels=['YearRemodAdd', 'YearBuilt'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>3.1.5 第五组数据 |var|define||---|---||Exterior1st|房子的外墙材料类型||Exterior2nd|房子的外墙材料类型，超过一种||MasVnrType|砖的类型，None表示没有||ExterQual|外墙的材料质量等评分||ExterCond|外墙材料的评价|<jupyter_code>cat5 = ['Exterior1st', 'Exterior2nd', 'MasVnrType', 'ExterQual', 'ExterCond']
get_info(cat5)<jupyter_output>空值信息
Exterior1st     1
Exterior2nd     1
MasVnrType     24
ExterQual       0
ExterCond       0
dtype: int64
Exterior1st :
VinylSd    1025
MetalSd     450
HdBoard     442
Wd Sdng     411
Plywood     221
CemntBd     126
BrkFace      87
WdShing      56
AsbShng      44
Stucco       43
BrkComm       6
CBlock        2
Stone         2
AsphShn       2
ImStucc       1
Name: Exterior1st, dtype: int64 
-----
Exterior2nd :
VinylSd    1014
MetalSd     447
HdBoard     406
Wd Sdng     391
Plywood     270
CmentBd     126
Wd Shng      81
BrkFace      47
Stucco       47
AsbShng      38
Brk Cmn      22
ImStucc      15
Stone         6
AsphShn       4
CBlock        3
Other         1
Name: Exterior2nd, dtype: int64 
-----
MasVnrType :
None       1742
BrkFace     879
Stone       249
BrkCmn       25
Name: MasVnrType, dtype: int64 
-----
ExterQual :
TA    1798
Gd     979
Ex     107
Fa      35
Name: ExterQual, dtype: int64 
-----
ExterCond :
TA    2538
Gd     299
Fa      67
Ex      12
Po       3
Name: ExterCond[...]<jupyter_text>* 使用众数VinylSd填充Exterior1st* 使用Exterior1st的值填充Exterior2nd* 使用 'None' 填充MasVnrType表示没有<jupyter_code>dataset.Exterior1st.fillna('VinylSd', inplace=True)
idx = dataset[dataset.Exterior2nd.isna()].index[0]
dataset.Exterior2nd.fillna(dataset.Exterior1st.iloc[idx], inplace=True)
dataset.MasVnrType.fillna('None', inplace=True)<jupyter_output><empty_output><jupyter_text>* 创建新的变量表示外墙使用了多少种材料* has_MasVnr 是否有砖<jupyter_code>dataset.loc[:, ['Exterior1st', 'Exterior2nd']].sample(10)
# dataset['n_Exterior'] = 1
# dataset['n_Exterior'].loc[dataset['Exterior2nd'] != dataset['Exterior1st']] += 1
# dataset['has_MasVnr'] = 1
# dataset['has_MasVnr'].loc[dataset.MasVnrType == 'None'] = 0
# dataset['n_Exterior'].sample(10)<jupyter_output><empty_output><jupyter_text>3.1.6 第六组数据 |var|define||---|---||Foundation|基础条件的评价，砖、煤等||BsmtQual|地下室的高度，NA表示没有地下室||BsmtCond|地下室条件的评价||BsmtExposure|地下室墙壁情况||BsmtFinType1|已完工地下室等级，适合居住么|<jupyter_code>cat6 = ['Foundation', 'BsmtQual', 'BsmtCond', 'BsmtExposure', 'BsmtFinType1']
get_info(cat6)<jupyter_output>空值信息
Foundation       0
BsmtQual        81
BsmtCond        82
BsmtExposure    82
BsmtFinType1    79
dtype: int64
Foundation :
PConc     1308
CBlock    1235
BrkTil     311
Slab        49
Stone       11
Wood         5
Name: Foundation, dtype: int64 
-----
BsmtQual :
TA    1283
Gd    1209
Ex     258
Fa      88
Name: BsmtQual, dtype: int64 
-----
BsmtCond :
TA    2606
Gd     122
Fa     104
Po       5
Name: BsmtCond, dtype: int64 
-----
BsmtExposure :
No    1904
Av     418
Gd     276
Mn     239
Name: BsmtExposure, dtype: int64 
-----
BsmtFinType1 :
Unf    851
GLQ    849
ALQ    429
Rec    288
BLQ    269
LwQ    154
Name: BsmtFinType1, dtype: int64 
-----<jupyter_text>* 使用 'None'填充BsmtQual, BsmtCond, BsmtFinType1, BsmtExposure表示没有* 创造has_Exposure变量表示墙壁接触* 创造has_finshed1变量表示有没有完工地下室1* 有没有地下室存在差异，使用81* 有没有地下室 1 has_Bsmt1<jupyter_code>label = ['BsmtCond', 'BsmtQual', 'BsmtFinType1', 'BsmtExposure']
def fill_None(label):
    for v in label:
        dataset[v].fillna('None', inplace=True)
fill_None(label)
# dataset['has_Exposure'] = 1
# dataset['has_Exposure'].loc[dataset['BsmtExposure'] == 'No'] = 0
# dataset['has_finshed1'] = 1
# dataset['has_finshed1'].loc[dataset['BsmtFinType1'] == 'Unf'] = 0
# dataset['has_Bsmt1'] = 1
# dataset['has_Bsmt1'].loc[dataset['BsmtFinType1'] == 'None'] = 0
dataset.sample(10)<jupyter_output><empty_output><jupyter_text>3.1.7 第七组数据 |var|define||---|---||BsmtFinType2|已完工地下室类型2如果有多个的话||Heating|取热类型，煤等||HeatingQC|取热质量评价||CentralAir|有没有中央空调||Electrical|电力系统的类型|<jupyter_code>cat7 = ['BsmtFinType2', 'Heating', 'HeatingQC', 'CentralAir', 'Electrical']
get_info(cat7)<jupyter_output>空值信息
BsmtFinType2    80
Heating          0
HeatingQC        0
CentralAir       0
Electrical       1
dtype: int64
BsmtFinType2 :
Unf    2493
Rec     105
LwQ      87
BLQ      68
ALQ      52
GLQ      34
Name: BsmtFinType2, dtype: int64 
-----
Heating :
GasA     2874
GasW       27
Grav        9
Wall        6
OthW        2
Floor       1
Name: Heating, dtype: int64 
-----
HeatingQC :
Ex    1493
TA     857
Gd     474
Fa      92
Po       3
Name: HeatingQC, dtype: int64 
-----
CentralAir :
Y    2723
N     196
Name: CentralAir, dtype: int64 
-----
Electrical :
SBrkr    2671
FuseA     188
FuseF      50
FuseP       8
Mix         1
Name: Electrical, dtype: int64 
-----<jupyter_text>* 使用众数'SBrkr'填充Electrical* 使用 'None'填充BsmtFinType2表示没有* 新变量n_Bsmt表示地下室个数* 新变量 has_Bsmt2 表示有没有地下室2* 创造has_finshed2变量表示有没有完工地下室2* 删除 has_Bsmt1 and has_Bsmt2<jupyter_code>dataset.loc[dataset.BsmtFinType2.isna(), ['BsmtFinType1', 'BsmtFinType2']]
dataset.Electrical.fillna('SBrkr', inplace=True)
dataset.BsmtFinType2.fillna('None', inplace=True)
# dataset['has_finshed2'] = 1
# dataset['has_finshed2'].loc[dataset['BsmtFinType1'] == 'Unf'] = 0
# dataset['has_Bsmt2'] = 1
# dataset['has_Bsmt2'].loc[dataset['BsmtFinType2'] == 'None'] = 0
# dataset['n_Bsmt'] = dataset['has_Bsmt1'] + dataset['has_Bsmt2']
# dataset.drop(['has_Bsmt1', 'has_Bsmt2'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>3.1.8 第八组数据 |var|define||---|---||BsmtFullBath|地下室满浴室数量||BsmtHalfBath|地下室半浴室数量||FullBath|满浴室数量||HalfBath|半浴室数量||BedroomAbvGr|楼上卧室数量|<jupyter_code>dataset.sample(10)
cat8 = ['BsmtFullBath', 'BsmtHalfBath', 'FullBath', 'HalfBath', 'BedroomAbvGr']
get_info(cat8)<jupyter_output>空值信息
BsmtFullBath    2
BsmtHalfBath    2
FullBath        0
HalfBath        0
BedroomAbvGr    0
dtype: int64
BsmtFullBath :
0.0    1705
1.0    1172
2.0      38
3.0       2
Name: BsmtFullBath, dtype: int64 
-----
BsmtHalfBath :
0.0    2742
1.0     171
2.0       4
Name: BsmtHalfBath, dtype: int64 
-----
FullBath :
2    1530
1    1309
3      64
0      12
4       4
Name: FullBath, dtype: int64 
-----
HalfBath :
0    1834
1    1060
2      25
Name: HalfBath, dtype: int64 
-----
BedroomAbvGr :
3    1596
2     742
4     400
1     103
5      48
6      21
0       8
8       1
Name: BedroomAbvGr, dtype: int64 
-----<jupyter_text>* 使用中位数填充<jupyter_code>dataset.BsmtFullBath.fillna(dataset.BsmtFullBath.median(), inplace=True)
dataset.BsmtHalfBath.fillna(dataset.BsmtHalfBath.median(), inplace=True)
dataset.loc[:, cat8].sample(10)<jupyter_output><empty_output><jupyter_text>3.1.9 第九组数据 |var|define||---|---||KitchenAbvGr|厨房数量||KitchenQual|厨房质量评价||TotRmsAbvGrd|房间数量，不包括浴室||Functional|房间的功能性||Fireplaces|壁炉的数量|<jupyter_code>cat9 = ['KitchenAbvGr', 'KitchenQual', 'TotRmsAbvGrd', 'Functional', 'Fireplaces']
get_info(cat9)<jupyter_output>空值信息
KitchenAbvGr    0
KitchenQual     1
TotRmsAbvGrd    0
Functional      2
Fireplaces      0
dtype: int64
KitchenAbvGr :
1    2785
2     129
0       3
3       2
Name: KitchenAbvGr, dtype: int64 
-----
KitchenQual :
TA    1492
Gd    1151
Ex     205
Fa      70
Name: KitchenQual, dtype: int64 
-----
TotRmsAbvGrd :
6     844
7     649
5     583
8     347
4     196
9     143
10     80
11     32
3      25
12     16
15      1
13      1
14      1
2       1
Name: TotRmsAbvGrd, dtype: int64 
-----
Functional :
Typ     2717
Min2      70
Min1      65
Mod       35
Maj1      19
Maj2       9
Sev        2
Name: Functional, dtype: int64 
-----
Fireplaces :
0    1420
1    1268
2     219
3      11
4       1
Name: Fireplaces, dtype: int64 
-----<jupyter_text>* 使用众数填充<jupyter_code>dataset.Functional.loc[dataset.Functional.isna()]
dataset.KitchenQual.fillna(dataset.KitchenQual.mode()[0], inplace=True)
dataset.Functional.fillna(dataset.Functional.mode()[0], inplace=True)<jupyter_output><empty_output><jupyter_text>3.1.10 第十组数据 |var|define||---|---||FireplaceQu|炉的质量||GarageType|车库位置，NA表示没有||GarageYrBlt|车库建成时间||GarageFinish|车库装修情况||GarageCars|车库的容纳car数量|<jupyter_code>cat10 = ['FireplaceQu', 'GarageType', 'GarageYrBlt', 'GarageFinish', 'GarageCars']
get_info(cat10)<jupyter_output>空值信息
FireplaceQu     1420
GarageType       157
GarageYrBlt      159
GarageFinish     159
GarageCars         1
dtype: int64
FireplaceQu :
Gd    744
TA    592
Fa     74
Po     46
Ex     43
Name: FireplaceQu, dtype: int64 
-----
GarageType :
Attchd     1723
Detchd      779
BuiltIn     186
Basment      36
2Types       23
CarPort      15
Name: GarageType, dtype: int64 
-----
GarageYrBlt :
2005.0    142
2006.0    115
2007.0    115
2004.0     99
2003.0     92
1977.0     66
2008.0     61
1998.0     58
2000.0     55
1999.0     54
2002.0     53
1950.0     51
1976.0     50
1993.0     49
1968.0     48
1997.0     44
1958.0     42
1978.0     41
2001.0     41
1956.0     41
1996.0     40
1966.0     39
1994.0     39
1960.0     37
1954.0     37
1967.0     36
1959.0     36
1979.0     35
1964.0     35
1962.0     35
         ... 
1922.0      8
1924.0      8
1935.0      8
1915.0      7
1928.0      7
1936.0      7
1937.0      6
1916.0      6
1923.0      6
1942.0      6
1900.0      6
1927.0      5
1921.0     [...]<jupyter_text>* 使用 'None' 填充表示没有<jupyter_code>fill_None(['FireplaceQu', 'GarageType', 'GarageYrBlt', 'GarageFinish'])
dataset.loc[dataset.GarageCars.isna(), cat10+['Neighborhood']]
ab = dataset.loc[dataset.Neighborhood == 'IDOTRR', ['Neighborhood', 'GarageType', 'GarageCars']]
cd = ab.loc[ab.GarageType == 'Detchd']
cd
cd.GarageCars.mode()<jupyter_output><empty_output><jupyter_text>* 缺失的 车库类型为 Detchd，附近区域， 使用该车库容纳车辆的众数填充<jupyter_code>dataset.GarageCars.fillna(1.0, inplace=True)<jupyter_output><empty_output><jupyter_text>* has_fire 表示有没有火炉* has_Garage 表示有没有车库* has_finish 车库有没有完成<jupyter_code># dataset['has_fire'] = 1
# dataset['has_Garage'] = 1
# dataset['has_finishGarage'] = 1
# dataset['has_finishGarage'].loc[dataset.GarageFinish == 'Unf'] = 0
# dataset['has_fire'].loc[dataset.FireplaceQu == 'None'] = 0
# dataset['has_Garage'].loc[dataset.GarageType == 'None'] = 0<jupyter_output><empty_output><jupyter_text>* 使用 附近同区域的众数数值填充<jupyter_code>a1 = dataset.loc[dataset.GarageYrBlt == 'None', ['Neighborhood', 'GarageType', 'GarageYrBlt']]
a1.loc[a1.GarageType != 'None']
a2 = dataset.loc[dataset['Neighborhood'] == 'OldTown', ['Neighborhood', 'GarageType', 'GarageYrBlt']]
a3 = a2.loc[dataset.GarageType == 'Detchd']
a3
a3.GarageYrBlt.mode()
dataset.GarageYrBlt.iloc[2126] = 1920
a4 = dataset.loc[dataset['Neighborhood'] == 'IDOTRR', ['Neighborhood', 'GarageType', 'GarageYrBlt']]
a5 = a4.loc[dataset.GarageType == 'Detchd']
a5
Counter(a5.GarageYrBlt).most_common(3)
dataset.GarageYrBlt.iloc[2576] = 1950
dataset.GarageYrBlt.loc[dataset.GarageYrBlt != 'None'].plot(kind='kde')
df1 = dataset.GarageYrBlt.loc[dataset.GarageYrBlt != 'None']
df2 = dataset.loc[dataset.GarageYrBlt != 'None']
df2.head()
df2.GarageYrBlt[df2.GarageYrBlt > 2018].index
df2.loc[df2.GarageYrBlt > 2018., ['GarageType', 'GarageYrBlt', 'GarageFinish', 'Neighborhood']]<jupyter_output><empty_output><jupyter_text>* 这里 2592 应该是一个异常值，尝试插值<jupyter_code>a6 = dataset.loc[dataset['Neighborhood'] == 'Timber', ['Neighborhood', 'GarageType', 'GarageYrBlt', 'GarageFinish']]
a7 = a6.loc[a6.GarageType == 'Attchd']
a8 = a7.loc[a7.GarageFinish == 'RFn']
a8
a8.GarageYrBlt.median()
dataset.GarageYrBlt.iloc[2592] = 1999
# pd.qcut(df1, 4)
# dataset['GarageYrBltbin'] = 'None'
# dataset['GarageYrBltbin'].iloc[df1.index] = pd.qcut(df1, 4)
# dataset.drop('GarageYrBlt', axis=1, inplace=True)
dataset.loc[2900:2912, ['Neighborhood', 'GarageType', 'GarageYrBltbin']]<jupyter_output>d:\softwares\miniconda\envs\torch36\lib\site-packages\pandas\core\indexing.py:1472: FutureWarning: 
Passing list-likes to .loc or [] with any missing label will raise
KeyError in the future, you can use .reindex() as an alternative.

See the documentation here:
https://pandas.pydata.org/pandas-docs/stable/indexing.html#deprecate-loc-reindex-listlike
  return self._getitem_tuple(key)<jupyter_text>3.1.11 第十一组数据 |var|define||---|---||GarageQual|车库的质量评价||GarageCond|车库的条件评价||PavedDrive|铺设车道||PoolQC|游泳池质量评价，NA表示没有游泳池||Fence|栅栏质量， NA表示没有|<jupyter_code>cat11 = ['GarageQual', 'GarageCond', 'PavedDrive', 'PoolQC', 'Fence']
get_info(cat11)
fill_None(cat11)<jupyter_output><empty_output><jupyter_text>* 新变量has_Pool, has_Fence 表示有没有泳池和围墙<jupyter_code># dataset['has_Pool'] = 1
# dataset['has_Fence'] = 1
# dataset['has_Pool'].loc[dataset['PoolQC'] == 'None'] = 0
# dataset['has_Fence'].loc[dataset['Fence'] == 'None'] = 0<jupyter_output><empty_output><jupyter_text>3.1.12 第十二组数据 |var|define||---|---||MiscFeature|其他类别中未包含的杂项功能||MoSold|月销量||YrSold|年销量||SaleType|销售类型||SaleCondition|销售条件|<jupyter_code>cat12 = ['MiscFeature', 'MoSold', 'YrSold', 'SaleType', 'SaleCondition']
get_info(cat12)
dataset['YrSold'].plot(kind='kde')
dataset.SaleType.mode()
dataset.SaleType.fillna('WD', inplace=True)
fill_None(['MiscFeature'])
# dataset['has_MiscFeature'] = 1
# dataset['has_MiscFeature'].loc[dataset['MiscFeature'] == 'None'] = 0<jupyter_output><empty_output><jupyter_text>3.2 连续变量 |var|define||---|---||LotFrontage|距离街道的距离||LotArea|地段面积||MasVnrArea|砖砌的面积||BsmtFinSF1|已完工地下室面积，NA表示没有地下室||BsmtFinSF2|已完工地下室面积，多个||BsmtUnfSF|未完工的地下室面积||TotalBsmtSF|地下室总面积||1stFlrSF|一楼的面积||2ndFlrSF|二楼的面积||LowQualFinSF|低质量的房屋面积||GrLivArea|地面以上居住面积||GarageArea|车库面积||WoodDeckSF|木质甲板面积||OpenPorchSF|开放式门廊的面积||EnclosedPorch|封闭式门廊面积||3SsnPorch|三季门廊面积||ScreenPorch|屏风廊面积||PoolArea|泳池面积||MiscVal|杂项功能的价值|  3.2.1 第一组<jupyter_code>cat13 = ['LotFrontage', 'LotArea', 'MasVnrArea', 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF', '1stFlrSF', '2ndFlrSF']
get_info(cat13)
dataset['1stFlrSF'].plot(kind='kde')
df2 = dataset.loc[dataset.LotFrontage.isna(), ['LotFrontage', 'Neighborhood']]
df2
idx1 = df2.index
n_type = list(dataset.Neighborhood.unique())
def get_med(n_type, col):
    med = {}
    for v in n_type:
        df1 = dataset.loc[dataset.Neighborhood == v, col]
        med[v] = df1.median()
    return med
med_dict = get_med(n_type, 'LotFrontage')
med_dict
for x in idx1:
    dataset.LotFrontage.iloc[x] = med_dict[dataset.Neighborhood.iloc[x]]
df3 = dataset.loc[dataset.MasVnrArea.isna(), ['MasVnrArea', 'Neighborhood']]
df3
idx2 = df3.index
med_dict1 = get_med(n_type, 'MasVnrArea')
med_dict1
for x in idx2:
    dataset.MasVnrArea.iloc[x] = med_dict[dataset.Neighborhood.iloc[x]]
dataset['MasVnrArea'].plot(kind='kde')
dataset.OpenPorchSF.plot(kind='kde')
# pd.cut(dataset['MasVnrArea'], 4)
# dataset['MasVnrAreabin'] = pd.cut(dataset['MasVnrArea'], 4)
dataset.loc[dataset.BsmtFinSF1.isna(), ['BsmtFinSF1', 'Neighborhood']]
dataset.loc[dataset.BsmtFinSF2.isna(), ['BsmtFinSF2', 'Neighborhood']]
dataset.loc[dataset.BsmtUnfSF.isna(), ['BsmtUnfSF', 'Neighborhood']]
dataset.loc[dataset.TotalBsmtSF.isna(), ['TotalBsmtSF', 'Neighborhood']]
p = [ 'BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'TotalBsmtSF']
for i in p:
    dataset[i].fillna(get_med(n_type, i)['BrkSide'], inplace=True)
get_info(cat13)
# dataset.drop('MasVnrArea', axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>* 根据情况判断是否删除<jupyter_code># dataset.drop(columns=['BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>3.2.2 第二组<jupyter_code>cat14 = ['LowQualFinSF', 'GrLivArea', 'GarageArea', 'WoodDeckSF', 'OpenPorchSF', 'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal']
get_info(cat14)
dataset.loc[dataset.GarageArea.isna(), ['GarageArea', 'Neighborhood']]
dataset['GarageArea'].fillna(get_med(n_type, 'GarageArea')['IDOTRR'], inplace=True)
get_info(cat14)
dataset.WoodDeckSF.plot(kind='kde')
# dataset['wood_a'] = pd.cut(dataset.WoodDeckSF, 4)
# dataset['wood_a'].sample(10)
# dataset.drop('WoodDeckSF', axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>4 再次检查<jupyter_code>dataset.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 2919 entries, 0 to 2918
Data columns (total 81 columns):
Id               2919 non-null int64
MSSubClass       2919 non-null int64
MSZoning         2919 non-null object
LotFrontage      2919 non-null float64
LotArea          2919 non-null int64
Street           2919 non-null object
Alley            2919 non-null object
LotShape         2919 non-null object
LandContour      2919 non-null object
Utilities        2919 non-null object
LotConfig        2919 non-null object
LandSlope        2919 non-null object
Neighborhood     2919 non-null object
Condition1       2919 non-null object
Condition2       2919 non-null object
BldgType         2919 non-null object
HouseStyle       2919 non-null object
OverallQual      2919 non-null int64
OverallCond      2919 non-null int64
YearBuilt        2919 non-null int64
YearRemodAdd     2919 non-null int64
RoofStyle        2919 non-null object
RoofMatl         2919 non-null object
Exterior1st      2919 non[...]<jupyter_text>* 对数变换<jupyter_code>dataset.SalePrice = np.log(dataset.SalePrice)<jupyter_output><empty_output><jupyter_text>5 整理数据<jupyter_code>le = LabelEncoder()
dataset.head(10)
all_col = list(dataset.columns)
dataset.columns.shape
all_col
uncertion = ['BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF']
uncertion1 = ['BsmtFinSF1', 'BsmtFinSF2', 'BsmtUnfSF', 'MiscFeature', 'MiscVal']
# no_label = ['Id', 'SalePrice', 'LotArea', '1stFlrSF', 'TotalBsmtSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'GarageArea', 'OpenPorchSF', 
#             'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'MoSold', 'YearBuilt', 'YearRemodAdd', 'GarageYrBlt', 'MasVnrArea', 'WoodDeckSF']
no_label = ['Id', 'SalePrice', 'LotArea', '1stFlrSF', 'TotalBsmtSF', '2ndFlrSF', 'LowQualFinSF', 'GrLivArea', 'GarageArea', 'OpenPorchSF', 
            'EnclosedPorch', '3SsnPorch', 'ScreenPorch', 'PoolArea', 'MiscVal', 'MoSold','MasVnrArea', 'WoodDeckSF']<jupyter_output><empty_output><jupyter_text>* 数据类型的问题<jupyter_code>dataset.info()
need_label = list(set(all_col) - set(uncertion) -set(no_label))
dataset['GarageYrBlt'].replace('None', 0, inplace=True)
need_label1 = list(set(need_label) - set(cf))
t1 = ['Ex', 'Fa', 'Gd', 'TA']
t2 = ['Min1', 'Min2', 'Maj1', 'Maj2', 'Sev', 'Typ', 'Mod']
t3 = ['Othr', 'Gar2', 'TenC', 'Shed']
v1 = ['KitchenQual', 'Functional', 'MiscFeature']
for v, t in zip(v1, [t1, t2, t3]):
    dataset.loc[[not x for x in dataset[v].isin(t)] , v] = 'None'
for x in need_label:
    print(x)
    dataset[x] = le.fit_transform(dataset[x])
# one-hot编码
def dummies(col,data):
    data_dum = pd.get_dummies(data[col])
    data = pd.concat([data, data_dum], axis=1)
    data.drop(col, axis=1, inplace=True)
    return data
# dataset = dummies(need_label, dataset)
# dataset.loc[:, cf]
dataset['target'] = dataset.SalePrice
dataset.drop(['SalePrice'], axis=1, inplace=True)<jupyter_output><empty_output><jupyter_text>* 重新分割数据集并保存<jupyter_code>train_set.shape
dataset.target.plot(kind='box')
dataset.target.plot(kind='kde')
dataset.replace('None', 0, inplace=True)
dataset.info()
train = dataset[:1460]
test = dataset[1460:]
test = test.drop(columns=['target'], axis=1, inplace=False)
fig, ax = plt.subplots()
ax.scatter(x = train['GrLivArea'], y = train['target'])
plt.ylabel('target', fontsize=13)
plt.xlabel('GrLivArea', fontsize=13)
# 删除最右边的异常值
train = train.drop(train[(train['GrLivArea'] > 4000) & (train['target'] < 300000)].index)
fig, ax = plt.subplots()
ax.scatter(x = train['GrLivArea'], y = train['target'])
plt.ylabel('target', fontsize=13)
plt.xlabel('GrLivArea', fontsize=13)
train.to_csv(f'{datadir}train_process.csv', index=False)
test.to_csv(f'{datadir}test_process.csv', index=False)
dataset.drop(columns=uncertion, axis=1, inplace=True)
train1 = dataset[:1460]
test1 = dataset[1460:]
test1 = test1.drop(columns=['target'], axis=1, inplace=False)
train1.to_csv(f'{datadir}part_train_process.csv', index=False)
test1.to_csv(f'{datadir}part_test_process.csv', index=False)
t = dataset.isna().sum()
t[t>0]
uncertion1 = ['MiscFeature', 'MiscVal']
dataset.drop(columns=uncertion1, axis=1, inplace=True)
train1 = dataset[:1460]
test1 = dataset[1460:]
test1 = test1.drop(columns=['target'], axis=1, inplace=False)
train1.to_csv(f'{datadir}part_train_process1.csv', index=False)
test1.to_csv(f'{datadir}part_test_process1.csv', index=False)
dataset.sample(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14523809523809525, 'fraction_numerical': 0.0716297786720322, 'mean_word_length': 4.62979044742307, 'pattern_counts': {'":': 0, '<': 193, '<?xml version=': 0, '>': 200, 'https://': 3, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '50', 'dir': 'jupyter-structured-clean-dedup', 'id': '533461', 'license': 'MIT', 'n_tokens_mistral': 14486, 'n_tokens_neox': 13628, 'n_words': 2076, 'path': 'House Prices/data pre-processing.ipynb', 'repo_name': 'AutuanLiu/Kaggle-Compettions', 'size': '29820'}