<jupyter_start><jupyter_text>0. loading data<jupyter_code>original_train_values = pd.read_csv('data/train.csv')
original_train_labels = pd.read_csv('data/train_labels.csv')
test = pd.read_csv('data/test.csv')
original_train = pd.merge(original_train_values, original_train_labels, on='id')
original_train.date_recorded = pd.to_datetime(original_train.date_recorded)
original_train.columns
train = original_train.copy()<jupyter_output><empty_output><jupyter_text>1. data understanding / preparation / cleaning  1.1 meaningless features  There are few columns that contain the same information but on different granularity level. In each case, we decided to keep columns that have more granular data and in case we needed to group them later, we would be able to do that. Columns in bold are those that we keep:1. waterpoint_type / waterpoint_type_group2. source / source_type / source_class3. quantity / quantity_group4. water_quality / quality_group5. payment / payment_type6. management / management_group7. extraction_type / extraction_type_group / extraction_type_class8. scheme_management / scheme_name-----  Besides, we decided to drop the following features because they don't contain useful information:1. recorded_by: contains only one value, which is organization that recorded the data2. id: surrogate key, column representing id of the observation in numerical format, doesn't contain any information.-----  We also decided to drop 4 features related to location (region, district_code, region_code, subvillage) for the following reasons:1. combination of the features district_code, region_code gives us zip code which identifies the location, but we can get the same information in more detailed way from geograhic coordinates (longitude, latitude)2. we have feature lga that contains similar information to region_code, region but on more granular level3. we will use latitude and longitude for creating clusters of regions with more granularity, therefore, we can drop subvillage too-----  We are dropping public_meeting too for two reasons:1. 86% of values are the same2. According to research on water issue in Tanzania, public meeting at installation is not good predictor<jupyter_code>columns_to_drop = ['id','waterpoint_type_group', 'source_type', 'source_class', 
                   'quantity_group', 'quality_group', 'payment_type', 
                   'management_group', 'extraction_type_group', 'extraction_type_class', 
                   'scheme_name', 'recorded_by', 'district_code', 
                   'region_code', 'region','subvillage', 'public_meeting']

train = train.drop(columns_to_drop, axis=1)
test = test.drop(columns_to_drop, axis=1)<jupyter_output><empty_output><jupyter_text>1.2 missing values  features with various names for missing valueThe dataset is quite dirty in a sense that missing values are represented in various forms:1. simply NaN / missing cells2. contains value "unknown", "Not known", "not known", "-", "No", "no", "0", "unknown", "none".We tried to find those values that might represent missing value in each column and standardize them by replacing all those columns with "unknown"<jupyter_code>train = OurSimpleImputer(coords=False).fit_transform(train)<jupyter_output><empty_output><jupyter_text>longitude and latitudeAfter checking longitude and latitude, we found that we have some of the values to be 0. Given that this is Tanzania, latitude and longitude of 0 or close to 0 are simply impossible. We decided to replace those values with mean for each LGA, which has quite detailed granularity (125 LGAs).<jupyter_code>train = OurSimpleImputer(categorical=False).fit_transform(train)<jupyter_output><empty_output><jupyter_text>populationAlmost 48% of observations have value "population" equal to 0 or 1. According to the description, this feature indicates the population around the particular waterpoint. It is hard to believe that the waterpoint was built for 0 or 1 person. Hence, we believe these two values indicate missing values or incorrect recording. We initially wanted to take population from other dataset and merge it with this one, however, the definition of population feature in this dataset is slightly different. Given that this is a population around a particular waterpoint, we decided to use population average of each coordinates-based cluster to impute missing values.  (Seems like 50 is good number of K-Means clusters)<jupyter_code>locations = train[['latitude','longitude']]
locations_transformed = StandardScaler().fit_transform(locations)

n_clusters = [10, 20, 30, 40, 60, 80, 100, 150]
inertia = []

for num in n_clusters:
    kmeans_model = KMeans(n_clusters=num, n_jobs=-1).fit(locations_transformed)
    wss = kmeans_model.inertia_
    inertia.append(wss)
    
pd.DataFrame({'n_clusters':n_clusters, 'inertia': inertia}).plot(x='n_clusters', y='inertia');
train = Clustering().fit_transform()
train = OurAdvancedImputer().fit_transform()<jupyter_output><empty_output><jupyter_text>1.3 data correctness  Installer and Funder<jupyter_code>train = DataCorrection(installer=True, funder=True).fit_transform(train)<jupyter_output><empty_output><jupyter_text>2. feature engineering Ideas:1. Clustering based on lat/lon2. Distance to centers (manhattan and eucledian)3. Bucketization for population4. Interactions    1. scheme_management + payment    2. basin + source    3. source + waterpoint_type    4. extraction_type + waterpoint_type    5. source + extraction_type    6. water_quality + quantity    7. extaction_type + payment5. Dry season from date_recorded (True/False)6. Replace num_private to True/False7. CountEncoder only: funder, installer, wpt_name, lga, ward8. 1Hot / CountEncoder: basin(9), scheme_management(13), management(12), source(10), payment(7), extraction_type(18), waterpoint_type(7), water_quality(8), quantity(5)9. Age feature: year the data was recorded - construction year10. Construction year and amount_tsh?<jupyter_code>meaningless_cols = ['id','waterpoint_type_group', 'source_type', 'source_class', 
                   'quantity_group', 'quality_group', 'payment_type', 
                   'management_group', 'extraction_type_group', 'extraction_type_class', 
                   'scheme_name', 'recorded_by', 'district_code', 
                   'region_code', 'region','subvillage', 'public_meeting']

features_to_drop = ['latitude', 'longitude', 'date_recorded', 'num_private']

transformation_pipeline = Pipeline([
    ('meaningless_features', DropColumns(meaningless_cols)),
    ('simple_imputer', OurSimpleImputer(permit=False)),
    ('government', DataCorrection(installer=True, funder=True)),
    ('geo_clusters', GeoClustering()),
    ('advanced_imputer', OurAdvancedImputer(population_bucket=False)),
    ('distance', Distance()),
    ('interactions', Interactions()),
    ('other_features', OtherFeatures(num_private=False)),
    ('drop', DropColumns(features_to_drop))
])
train_prep = transformation_pipeline.fit_transform(original_train)<jupyter_output><empty_output><jupyter_text>3. feature selection  3.1 information gain<jupyter_code>def information_gain(df, feature, target):
    p_target = df[target].value_counts(normalize=True)
    H_target = np.nansum(-np.log2(p_target) * p_target)
    
    p_feature_target = df.groupby(feature)[target].value_counts(normalize=True)
    p_features = df[feature].value_counts(normalize=True)
    
    H_feature = 0
    for value in df[feature].unique():
        H_value = p_features[value] * np.nansum(-np.log2(p_feature_target[value]) * p_feature_target[value])
        H_feature += H_value
    
    return H_target - H_feature 
features = train_prep.columns.drop('status_group')
information_gain_features = {}
for feature in features:
    information_gain_features[feature] = information_gain(train_prep, feature, 'status_group')
sorted(information_gain_features.items(), key=lambda tup: -tup[1])<jupyter_output><empty_output><jupyter_text>Notes:1. permit: if NA are imputed with True, information gain is 0.00083. If with False, then 0.00075. We are dropping this feature.2. num_private: very low value of 0.00016, we decided to drop this feature3. population_binned: very low value of 0.00226, we decided not to create this feature in next iteration and pipeline4. installer: not corrected version of installer gives 0.189, corrected/grouped version gives 0.1435. funder: not corrected version of installer gives 0.176, corrected/grouped version gives 0.1486. cluster: k=50 clusters gives 0.084  3.2 LogisticRegression L1 selection<jupyter_code>X, y = train_prep[train_prep.columns.drop("status_group")], train['status_group']

transformation_pipeline = Pipeline([
    ('count_encoder', CountEncoder(handle_unknown=0)),
    ('scaler', StandardScaler())
])

X = transformation_pipeline.fit_transform(X)

logistic = LogisticRegressionCV(Cs=[3, 10, 30, 100], penalty="l1", 
                                multi_class='multinomial',solver='saga', cv=3, n_jobs=-1).fit(X, y);
model = SelectFromModel(logistic, prefit=True);

X_new = model.transform(X);

selected_features = pd.DataFrame(model.inverse_transform(X_new), 
                                 index=train_prep.index,
                                 columns=train_prep.drop('status_group', axis=1).columns)

dropped_columns = selected_features.columns[selected_features.var() == 0]
dropped_columns<jupyter_output><empty_output><jupyter_text>Notes:Lasso selection didn't remove any features, so we can continue with modeling and hyperparameter tuning  4. modeling and evaluation  4.1.1 Random Forest without PCA<jupyter_code>meaningless_cols = ['id','waterpoint_type_group', 'source_type', 'source_class', 
                   'quantity_group', 'quality_group', 'payment_type', 
                   'management_group', 'extraction_type_group', 'extraction_type_class', 
                   'scheme_name', 'recorded_by', 'district_code', 
                   'region_code', 'region','subvillage', 'public_meeting']

features_to_drop = ['latitude', 'longitude', 'date_recorded', 'num_private', 'permit', 'water_quality']

transformation_pipeline = Pipeline([
    ('meaningless_features', DropColumns(meaningless_cols)),
    ('simple_imputer', OurSimpleImputer(permit=False)),
    ('government', DataCorrection(installer=False, funder=False)),
    ('geo_clusters', GeoClustering()),
    ('advanced_imputer', OurAdvancedImputer(population_bucket=False)),
    ('distance', Distance(strategy='eucledian')),
    ('interactions', Interactions()),
    ('other_features', OtherFeatures(num_private=False, dry_season=False)),
    ('drop', DropColumns(features_to_drop)),
    ('encoder', CountEncoder(handle_unknown=0))
])

class_to_num = {'functional': 2, 'non functional': 0, 'functional needs repair': 1}
num_to_class = {0:'non functional', 1: 'functional needs repair', 2: 'functional'} 

X = original_train.drop('status_group', axis=1)
y = original_train.status_group.replace(class_to_num)

X_prep = transformation_pipeline.fit_transform(X)

rf = RandomForestClassifier(n_jobs=-1)

# setting grid
param_grid_forest = {
    'max_depth': [10, 25, 50, None],
    'n_estimators': [50, 100]
}

grid_search_forest = GridSearchCV(rf, cv=5, param_grid=param_grid_forest,n_jobs=-1).fit(X_prep, y)<jupyter_output><empty_output><jupyter_text>evaluation<jupyter_code>print('Train performance on Random Forest w/o PCA: ', grid_search_forest.best_estimator_.score(X_prep, y))
params = ['rank_test_score','mean_test_score', 'param_max_depth', 'param_n_estimators']
pd.DataFrame(grid_search_forest.cv_results_)[params].sort_values('rank_test_score')<jupyter_output><empty_output><jupyter_text>feature importance<jupyter_code>sorted([*zip(X_prep.columns, grid_search_forest.best_estimator_.feature_importances_)], key=lambda tup: -tup[1])<jupyter_output><empty_output><jupyter_text>error analysis* Rows: actual class* Columns: predicted class<jupyter_code>y_pred_cv = cross_val_predict(grid_search_forest.best_estimator_, X_prep, y, cv=5)
conf_mx = confusion_matrix(y, y_pred_cv)
row_sums = conf_mx.sum(axis=1, keepdims=True) 
norm_conf_mx = conf_mx / row_sums
np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
plt.show()<jupyter_output><empty_output><jupyter_text>Notes:Our model is overfitting based on the difference between train performance (0.99) and cross-validation performance (0.81). Feature importance shows that wee need to drop two features "dry_season" and "water_quality". After removing those two features and reducing max_depth to 100, our generalization didn't improve. We figured out also that using "Eucledian" distance gives slightly better performance than "Manhattan" in transformer "Distance()". Based on error analysis, we can see that Random Forest is doing a lot of mistakes in predicting class 1 which is "non functional needs repair". This is understable given the heavy imbalance of this class (around 7% of classes). The best parameters based on CV: 1. n_estimators = 1002. max_depth = 25  submission creation<jupyter_code>test_prep = transformation_pipeline.transform(test)
y_pred = grid_search_forest.best_estimator_.predict(test_prep)

test_id = pd.read_csv('data/test.csv').id
submission = pd.DataFrame({"id": test_id, "status_group": y_pred})
submission = submission.replace({'status_group': num_to_class})
submission.to_csv('submission_rf.csv', index=False)<jupyter_output><empty_output><jupyter_text>4.1.2 Random Forest with PCA<jupyter_code>meaningless_cols = ['id','waterpoint_type_group', 'source_type', 'source_class', 
                   'quantity_group', 'quality_group', 'payment_type', 
                   'management_group', 'extraction_type_group', 'extraction_type_class', 
                   'scheme_name', 'recorded_by', 'district_code', 
                   'region_code', 'region','subvillage', 'public_meeting']

features_to_drop = ['latitude', 'longitude', 'date_recorded', 'num_private', 'permit', 'water_quality']

transformation_pipeline = Pipeline([
    ('meaningless_features', DropColumns(meaningless_cols)),
    ('simple_imputer', OurSimpleImputer(permit=False)),
    ('government', DataCorrection(installer=False, funder=False)),
    ('geo_clusters', GeoClustering()),
    ('advanced_imputer', OurAdvancedImputer(population_bucket=False)),
    ('distance', Distance(strategy='eucledian')),
    ('interactions', Interactions()),
    ('other_features', OtherFeatures(num_private=False, dry_season=False)),
    ('drop', DropColumns(features_to_drop)),
    ('encoder', CountEncoder(handle_unknown=0)),
    ('scaler', StandardScaler()),
    ('boxcos', PowerTransformer()),
    ('PCA', PCA())
])

class_to_num = {'functional': 2, 'non functional': 0, 'functional needs repair': 1}
num_to_class = {0:'non functional', 1: 'functional needs repair', 2: 'functional'} 

X = original_train.drop('status_group', axis=1)
y = original_train.status_group.replace(class_to_num)

X_prep = transformation_pipeline.fit_transform(X)

rf_pca = RandomForestClassifier(n_jobs=-1)

# setting grid
param_grid_forest = {
    'max_depth': [10, 25, 50, None],
    'n_estimators': [50, 100]
}

grid_search_forest_pca = GridSearchCV(rf_pca, cv=5, param_grid=param_grid_forest,n_jobs=-1).fit(X_prep, y)<jupyter_output><empty_output><jupyter_text>evaluation<jupyter_code>print('Train performance on Random Forest w/ PCA: ', grid_search_forest_pca.best_estimator_.score(X_prep, y))
params = ['rank_test_score','mean_test_score', 'param_max_depth', 'param_n_estimators']
pd.DataFrame(grid_search_forest_pca.cv_results_)[params].sort_values('rank_test_score')<jupyter_output><empty_output><jupyter_text>error analysis* Rows: actual class* Columns: predicted class<jupyter_code>y_pred_cv = cross_val_predict(grid_search_forest_pca.best_estimator_, X_prep, y, cv=5)
conf_mx = confusion_matrix(y, y_pred_cv)
row_sums = conf_mx.sum(axis=1, keepdims=True) 
norm_conf_mx = conf_mx / row_sums
np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
plt.show()<jupyter_output><empty_output><jupyter_text>Notes:Similar issues with Random Forest run without PCA, except for the performance on cross-validation got worse (from 0.81 to 0.77).    4.2 XGBoost<jupyter_code>meaningless_cols = ['id','waterpoint_type_group', 'source_type', 'source_class', 
                   'quantity_group', 'quality_group', 'payment_type', 
                   'management_group', 'extraction_type_group', 'extraction_type_class', 
                   'scheme_name', 'recorded_by', 'district_code', 
                   'region_code', 'region','subvillage', 'public_meeting']

features_to_drop = ['latitude', 'longitude', 'date_recorded', 'num_private', 'permit', 'water_quality']

transformation_pipeline = Pipeline([
    ('meaningless_features', DropColumns(meaningless_cols)),
    ('simple_imputer', OurSimpleImputer(permit=False)),
    ('government', DataCorrection(installer=False, funder=False)),
    ('geo_clusters', GeoClustering()),
    ('advanced_imputer', OurAdvancedImputer(population_bucket=False)),
    ('distance', Distance(strategy='eucledian')),
    ('interactions', Interactions()),
    ('other_features', OtherFeatures(num_private=False, dry_season=False)),
    ('drop', DropColumns(features_to_drop)),
    ('encoder', CountEncoder(handle_unknown=0))
])

class_to_num = {'functional': 2, 'non functional': 0, 'functional needs repair': 1}
num_to_class = {0:'non functional', 1: 'functional needs repair', 2: 'functional'} 

X = original_train.drop('status_group', axis=1)
y = original_train.status_group.replace(class_to_num)

X_prep = transformation_pipeline.fit_transform(X)


xgb_model = XGBClassifier(n_jobs=-1)

# setting grid
param_grid_xgb = {
    'max_depth': [20],
    'learning_rate': [0.03, 0.1, 0.3]
}

grid_search_xgb = GridSearchCV(xgb_model, cv=5, param_grid=param_grid_xgb, n_jobs=-1).fit(X_prep, y)<jupyter_output><empty_output><jupyter_text>evaluation<jupyter_code>print('Train performance on XGB: ', grid_search_xgb.best_estimator_.score(X_prep, y))
params = ['rank_test_score','mean_test_score', 'param_max_depth', 'param_learning_rate']
pd.DataFrame(grid_search_xgb.cv_results_)[params].sort_values('rank_test_score')<jupyter_output><empty_output><jupyter_text>error analysis<jupyter_code>y_pred_cv = cross_val_predict(grid_search_xgb.best_estimator_, X_prep, y, cv=5)
conf_mx = confusion_matrix(y, y_pred_cv)
row_sums = conf_mx.sum(axis=1, keepdims=True) 
norm_conf_mx = conf_mx / row_sums
np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
plt.show()<jupyter_output><empty_output><jupyter_text>Notes:The best parameters based on CV: 1. learning_rate = 0.12. max_depth = 20  submission creation<jupyter_code>test_prep = transformation_pipeline.transform(test)
y_pred = grid_search_xgb.best_estimator_.predict(test_prep)

test_id = pd.read_csv('data/test.csv').id
submission = pd.DataFrame({"id": test_id, "status_group": y_pred})
submission = submission.replace({'status_group': num_to_class})
submission.to_csv('submission_gbm.csv', index=False)<jupyter_output><empty_output><jupyter_text>4.3 LDA Trial<jupyter_code>meaningless_cols = ['id','waterpoint_type_group', 'source_type', 'source_class', 
                   'quantity_group', 'quality_group', 'payment_type', 
                   'management_group', 'extraction_type_group', 'extraction_type_class', 
                   'scheme_name', 'recorded_by', 'district_code', 
                   'region_code', 'region','subvillage', 'public_meeting']

features_to_drop = ['latitude', 'longitude', 'date_recorded', 'num_private', 'permit', 'water_quality']

transformation_pipeline = Pipeline([
    ('meaningless_features', DropColumns(meaningless_cols)),
    ('simple_imputer', OurSimpleImputer(permit=False)),
    ('government', DataCorrection(installer=False, funder=False)),
    ('geo_clusters', GeoClustering()),
    ('advanced_imputer', OurAdvancedImputer(population_bucket=False)),
    ('distance', Distance(strategy='eucledian')),
    ('interactions', Interactions()),
    ('other_features', OtherFeatures(num_private=False, dry_season=False)),
    ('drop', DropColumns(features_to_drop)),
    ('encoder', CountEncoder(handle_unknown=0)),
    ('scaler', StandardScaler()),
    ('boxcos', PowerTransformer())
])

class_to_num = {'functional': 2, 'non functional': 0, 'functional needs repair': 1}
num_to_class = {0:'non functional', 1: 'functional needs repair', 2: 'functional'} 

X = original_train.drop('status_group', axis=1)
y = original_train.status_group.replace(class_to_num)

X_prep = transformation_pipeline.fit_transform(X)

lda = LDA()

# setting grid
param_grid_lda = {
    'n_components': [2, 10]
}

grid_search_forest_LDA = GridSearchCV(lda, cv=5, param_grid=param_grid_lda,n_jobs=-1).fit(X_prep, y)<jupyter_output><empty_output><jupyter_text>evaluation<jupyter_code>print('Train performance on LDA: ', grid_search_forest_LDA.best_estimator_.score(X_prep, y))
params = ['rank_test_score','mean_test_score']
pd.DataFrame(grid_search_forest_LDA.cv_results_)[params].sort_values('rank_test_score')<jupyter_output><empty_output><jupyter_text>4.4 LightGBM<jupyter_code>meaningless_cols = ['id','waterpoint_type_group', 'source_type', 'source_class', 
                   'quantity_group', 'quality_group', 'payment_type', 
                   'management_group', 'extraction_type_group', 'extraction_type_class', 
                   'scheme_name', 'recorded_by', 'district_code', 
                   'region_code', 'region','subvillage', 'public_meeting']

features_to_drop = ['latitude', 'longitude', 'date_recorded', 'num_private', 'permit', 'water_quality']

transformation_pipeline = Pipeline([
    ('meaningless_features', DropColumns(meaningless_cols)),
    ('simple_imputer', OurSimpleImputer(permit=False)),
    ('government', DataCorrection(installer=False, funder=False)),
    ('geo_clusters', GeoClustering()),
    ('advanced_imputer', OurAdvancedImputer(population_bucket=False)),
    ('distance', Distance(strategy='eucledian')),
    ('interactions', Interactions()),
    ('other_features', OtherFeatures(num_private=False, dry_season=False)),
    ('drop', DropColumns(features_to_drop)),
    ('encoder', CountEncoder(handle_unknown=0))
])

class_to_num = {'functional': 2, 'non functional': 0, 'functional needs repair': 1}
num_to_class = {0:'non functional', 1: 'functional needs repair', 2: 'functional'} 

X = original_train.drop('status_group', axis=1)
y = original_train.status_group.replace(class_to_num)

X_prep = transformation_pipeline.fit_transform(X)

lgb = LGBMClassifier(objective='multiclass',num_class=3)

param_grid_lgb = {
    'num_leaves': [64,128],
    'learning_rate': [0.3, 1],
    'max_depth': [16],
    'min_data_in_leaf': [200, 500]
}

grid_search_lgb = GridSearchCV(lgb, cv=5, param_grid=param_grid_lgb,n_jobs=-1).fit(X_prep, y)<jupyter_output><empty_output><jupyter_text>evaluation<jupyter_code>print('Train performance on LightGBM: ', grid_search_lgb.best_estimator_.score(X_prep, y))
params = ['rank_test_score','mean_test_score', 'param_num_leaves', 'param_learning_rate','param_min_data_in_leaf']
pd.DataFrame(grid_search_lgb.cv_results_)[params].sort_values('rank_test_score')<jupyter_output><empty_output><jupyter_text>error analysis<jupyter_code>y_pred_cv = cross_val_predict(grid_search_lgb.best_estimator_, X_prep, y, cv=5)
conf_mx = confusion_matrix(y, y_pred_cv)
row_sums = conf_mx.sum(axis=1, keepdims=True) 
norm_conf_mx = conf_mx / row_sums
np.fill_diagonal(norm_conf_mx, 0)
plt.matshow(norm_conf_mx, cmap=plt.cm.gray)
plt.show()<jupyter_output><empty_output><jupyter_text>Notes:Best parameters based on CV:1. num_leaves: 1282. learning_rate: 0.33. min_data_in_leaf: 2004. max_depth: 16  submission creation<jupyter_code>test_prep = transformation_pipeline.transform(test)
y_pred = grid_search_lgb.best_estimator_.predict(test_prep)

test_id = pd.read_csv('data/test.csv').id
submission = pd.DataFrame({"id": test_id, "status_group": y_pred})
submission = submission.replace({'status_group': num_to_class})
submission.to_csv('submission_lgb.csv', index=False)<jupyter_output><empty_output><jupyter_text>4.5 Voting Classifier<jupyter_code>meaningless_cols = ['id','waterpoint_type_group', 'source_type', 'source_class', 
                   'quantity_group', 'quality_group', 'payment_type', 
                   'management_group', 'extraction_type_group', 'extraction_type_class', 
                   'scheme_name', 'recorded_by', 'district_code', 
                   'region_code', 'region','subvillage', 'public_meeting']

features_to_drop = ['latitude', 'longitude', 'date_recorded', 'num_private', 'permit', 'water_quality']

transformation_pipeline = Pipeline([
    ('meaningless_features', DropColumns(meaningless_cols)),
    ('simple_imputer', OurSimpleImputer(permit=False)),
    ('government', DataCorrection(installer=False, funder=False)),
    ('geo_clusters', GeoClustering()),
    ('advanced_imputer', OurAdvancedImputer(population_bucket=False)),
    ('distance', Distance(strategy='eucledian')),
    ('interactions', Interactions()),
    ('other_features', OtherFeatures(num_private=False, dry_season=False)),
    ('drop', DropColumns(features_to_drop)),
    ('encoder', CountEncoder(handle_unknown=0))
])

class_to_num = {'functional': 2, 'non functional': 0, 'functional needs repair': 1}
num_to_class = {0:'non functional', 1: 'functional needs repair', 2: 'functional'} 

X = original_train.drop('status_group', axis=1)
y = original_train.status_group.replace(class_to_num)

X_prep = transformation_pipeline.fit_transform(X)

lgb_clf = LGBMClassifier(objective='multiclass',num_class=3, min_data_in_leaf=200, num_leaves=128, learning_rate=0.3, max_depth=16)
xgb_clf = XGBClassifier(n_jobs=-1, max_depth=20, learning_rate=0.1)
rf_clf = RandomForestClassifier(max_depth=25, n_estimators=100, n_jobs=-1)

voting_clf = VotingClassifier(estimators=[
    ('rf', rf_clf), 
    ('lgb', lgb_clf),
    ('xgb', xgb_clf)], voting='soft')

voting_clf.fit(X_prep, y)
test_prep = transformation_pipeline.transform(test)
y_pred = voting_clf.predict(test_prep)

test_id = pd.read_csv('data/test.csv').id
submission = pd.DataFrame({"id": test_id, "status_group": y_pred})
submission = submission.replace({'status_group': num_to_class})
submission.to_csv('submission_voting_soft.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12327776509096446, 'fraction_numerical': 0.014424906751922052, 'mean_word_length': 6.220390217092608, 'pattern_counts': {'":': 8, '<': 113, '<?xml version=': 0, '>': 113, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '626062', 'license': 'MIT', 'n_tokens_mistral': 8900, 'n_tokens_neox': 8358, 'n_words': 1774, 'path': 'Assignment 2 - Multiclass classification/multiclass_classification_DrivenData_Tanzania.ipynb', 'repo_name': 'AdiletGaparov/mbd-machine-learning-II', 'size': '26274'}