<jupyter_start><jupyter_text>Kenya Hack - Tanzania Tourism Prediction Challenge by Nairobi Women In Machine Learning Data Science  Problem statement The objective of this hackathon was to develop a machine learning model to predict what a tourist will spend when visiting Tanzania.The model can be used by different tour operators and the Tanzania Tourism Board to automatically help tourists across the world estimate their expenditure before visiting Tanzania. The full problem statement can be found on the [competition](https://zindi.africa/hackathons/kenya-hack-tanzania-tourism-prediction-challenge) page.<jupyter_code>import pandas as pd
import numpy as np
import requests
from io import StringIO 
from sklearn.model_selection import train_test_split
from sklearn.svm import SVR, NuSVR
from sklearn.neighbors import KNeighborsRegressor
from xgboost import XGBRegressor, XGBRFRegressor
from sklearn.linear_model import LinearRegression, Ridge, Lasso, BayesianRidge
from sklearn.experimental import enable_hist_gradient_boosting
from sklearn.ensemble import RandomForestRegressor, StackingRegressor,HistGradientBoostingRegressor, ExtraTreesRegressor
from sklearn.metrics import mean_absolute_error
import lightgbm
from lightgbm import LGBMRegressor
import catboost
from catboost import CatBoostRegressor
from sklearn.cluster import KMeans
import warnings
from rgf.sklearn import RGFRegressor
warnings.filterwarnings('ignore')
pd.set_option('display.max_rows', 999)
pd.set_option('display.max_columns', 999)
# Loading data
train=pd.read_csv('Train.csv')
test_1=pd.read_csv('Test.csv')
train.head()
test_1.head()<jupyter_output><empty_output><jupyter_text>Dealing with missing values<jupyter_code>data=pd.concat([train,test_1],sort=False).reset_index(drop=True)
data.isnull().sum()

# For the travel_with and most_impressing columns, fill in with another category, None
data.travel_with.fillna('None',inplace=True)
data.most_impressing.fillna('None',inplace=True)
# For the female and male columns, fill them with their respective means
data.total_female.fillna(data.total_female.mean(),inplace = True)
data.total_male.fillna(data.total_male.mean(),inplace = True)<jupyter_output><empty_output><jupyter_text>Baseline model > Due to time costraints, I tried Catboost, LGBM and XGBoost on the data to find out the best performing model, using the feature transformation and engineering on the starter notebook, Catboost was the best.<jupyter_code># Feature engineering and exploring feature interactions
data["total_persons"] = data["total_female"] + data["total_male"]
data['total_persons'] = data['total_persons'].astype('int')


data["total_nights_spent"] = data["night_mainland"] + data["night_zanzibar"]


# categorical features encoding
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
data['age_group'] = le.fit_transform(data['age_group'])
data['package_transport_int'] = le.fit_transform(data['package_transport_int'])
data['package_accomodation'] = le.fit_transform(data['package_accomodation'])
data['package_food'] = le.fit_transform(data['package_food'])
data['package_transport_tz'] = le.fit_transform(data['package_transport_tz'])
data['package_sightseeing'] = le.fit_transform(data['package_sightseeing'])
data['package_guided_tour'] = le.fit_transform(data['package_guided_tour'])
data['package_insurance'] = le.fit_transform(data['package_insurance'])
data['first_trip_tz'] = le.fit_transform(data['first_trip_tz'])
data['country'] = le.fit_transform(data['country'])
columns_to_transform = ['tour_arrangement','travel_with','purpose','main_activity','info_source','most_impressing','payment_mode']
data = pd.get_dummies( data,columns = columns_to_transform,drop_first=True)


## convert float dtypes to int
data["total_female"] = data['total_female'].astype('int')
data["total_male"] = data['total_male'].astype('int')
data["night_mainland"] = data['night_mainland'].astype('int')
data["night_zanzibar"] = data['night_zanzibar'].astype('int')



## separate data into train and test
train=data[data.total_cost.notnull()].reset_index(drop=True)
test_1=data[data.total_cost.isna()].reset_index(drop=True)

X = train.drop(columns=['ID', 'total_cost'])
y = train['total_cost']
test=test_1.drop(columns=['ID', 'total_cost'])
# Since our scoring function is Mean absolute error, it's a very important hyperparameter to set as the default
# loss function being optimised by all default models is squared error which penalizes outliers more.
# I also tried an array of different number of folds, 4 yielded the best score.

error=[]
pred_test = np.zeros(len(test))
from sklearn.model_selection import KFold
fold=KFold(n_splits=4)

for train_index, test_index in fold.split(X,y):
    X_train, X_test = X.iloc[train_index], X.iloc[test_index]
    y_train, y_test = y.iloc[train_index], y.iloc[test_index]
    cat  = CatBoostRegressor(verbose= False,random_seed= 1234, use_best_model=True, loss_function='MAE',n_estimators=1000, learning_rate=0.05)
    cat.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_test, y_test)])
    preds=cat.predict(X_test)
    print("err: ",mean_absolute_error(y_test,preds))
    error.append(mean_absolute_error(y_test,preds))
    p2 = cat.predict(test)
    pred_test+=p2
np.mean(error)<jupyter_output>err:  5191896.980019618
err:  4781667.20669109
err:  5144549.742672514
err:  3688515.0385065204<jupyter_text>Feature importance plot<jupyter_code>#Feature importance plot.

feat_importances = pd.Series(cat.feature_importances_, index=X.columns)
feat_importances.nlargest(10).plot(kind='barh')<jupyter_output><empty_output><jupyter_text>> The feature importance plot is very important when deciding which feature interactions to explore. For example, there might be a relationship between totalnumber of visitors and their age groups. Multiplying those two into another column would add more categories and potentially a better score.> It's not a science though, it's an art. after every trial, I'd upload the file and see if it worked. > I finally came up with 4 features that improved my score on the public leaderboard to 509, which was the best. > I however knew that the public leaderboard was just 50% of the test data and a better generalized model would potentially perform better.  Best solution<jupyter_code># Feature engineering and exploring feature interactions
data["total_persons"] = data["total_female"] + data["total_male"]
data['total_persons'] = data['total_persons'].astype('int')

data['mgf2'] = data['payment_mode'] + data['purpose']
data['mgf3'] = data['country'] + data['tour_arrangement']
data['mgf4'] = data['total_persons'] * data['age_group']
data["total_nights_spent"] = data["night_mainland"] + data["night_zanzibar"]
data['mf1'] = data['total_nights_spent']*data['total_persons']

# categorical features encoding
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
data['age_group'] = le.fit_transform(data['age_group'])
data['package_transport_int'] = le.fit_transform(data['package_transport_int'])
data['package_accomodation'] = le.fit_transform(data['package_accomodation'])
data['package_food'] = le.fit_transform(data['package_food'])
data['package_transport_tz'] = le.fit_transform(data['package_transport_tz'])
data['package_sightseeing'] = le.fit_transform(data['package_sightseeing'])
data['package_guided_tour'] = le.fit_transform(data['package_guided_tour'])
data['package_insurance'] = le.fit_transform(data['package_insurance'])
data['first_trip_tz'] = le.fit_transform(data['first_trip_tz'])
data['country'] = le.fit_transform(data['country'])
columns_to_transform = ['tour_arrangement','travel_with','purpose','main_activity','info_source','most_impressing','payment_mode', 'mgf2','mgf3', 'mgf4']
data = pd.get_dummies( data,columns = columns_to_transform,drop_first=True)


## convert float dtypes to int
data["total_female"] = data['total_female'].astype('int')
data["total_male"] = data['total_male'].astype('int')
data["night_mainland"] = data['night_mainland'].astype('int')
data["night_zanzibar"] = data['night_zanzibar'].astype('int')



## separate data into train and test
train=data[data.total_cost.notnull()].reset_index(drop=True)
test_1=data[data.total_cost.isna()].reset_index(drop=True)

X = train.drop(columns=['ID', 'total_cost'])
y = train['total_cost']
test=test_1.drop(columns=['ID', 'total_cost'])<jupyter_output><empty_output><jupyter_text>Stacking and Blending > The basic architecture was two layers, each with models with varying hyperparameters, then stacked to create a set of predictions on the test data using catboost as the meta learner, then averging those two predictions together and then with another set of predictions by a lone Catboost model.> setting the evaluation metric / loss function of each model to MAE was also very important.> This technique achieved a public LB score of 506.<jupyter_code># Stacking with linear models, SVM, KNN and Decision trees and many variations of them. 
# The variations are different values of key parameters of each model. 
# While I did not have the time to tune parameters of each model, except the meta learner Catboost, educated guesses on 
# the parameters were made to have as much variability as possible.

estimators_1 = [
    ('XGBM', XGBRegressor(random_state=2020, learning_rate=0.05, eval_metric='mae', n_jobs=-1)),
    ('LR', LinearRegression()),
    ('LGBM', LGBMRegressor(learning_rate=0.2, random_state=2020, objective='mae')),
    ('SVM', SVR(degree=2)),
    ('Lasso', Lasso(random_state=2020)),
    ('KNN', KNeighborsRegressor(n_neighbors=4, n_jobs=-1)),
    ('CatBoost', CatBoostRegressor(logging_level='Silent', random_state=2020, loss_function='MAE'))
]

predictions_1 = StackingRegressor(n_jobs=-1, cv=5, estimators=estimators_1, final_estimator=CatBoostRegressor(logging_level='Silent', depth=6, bagging_temperature=5, loss_function='MAE', random_state=2020)).fit(X, y).predict(test)

estimators_2 = [
    ('XGB', XGBRegressor(learning_rate=0.2, random_state=2020, eval_metric='mae')),
    ('LR', LinearRegression()),
    ('RF', RandomForestRegressor(random_state=2020, criterion='mae')),
    ('LGBM', LGBMRegressor(learning_rate=0.05, random_state=2020, objective='mae')),
    ('SVM', SVR(degree=5)),
    ('KNN', KNeighborsRegressor(n_neighbors=6, n_jobs=-1)),
    ('CatBoost', CatBoostRegressor(logging_level='Silent', random_state=2020, loss_function='MAE'))
]

predictions_2 = StackingRegressor(n_jobs=-1, cv=4, estimators=estimators_2, final_estimator=CatBoostRegressor(logging_level='Silent', depth=6, bagging_temperature=5, loss_function='MAE', random_state=2020)).fit(X, y).predict(test)

predictions_cat_1 = CatBoostRegressor(logging_level='Silent', depth=6, loss_function='MAE', bagging_temperature=5, random_state=2020).fit(X, y).predict(test)


# Further averaging to generalise well
# While the ratios are greater than one, it still works a treat. This is definitely one of the parameters to tune to achieve great results.
stack = [x*0.3 + y*0.7 for x, y in zip(predictions_1, predictions_2)]
stack_2 = [x*0.4 + y*0.65 for x, y in zip(stack, predictions_cat_1)]

# submission file
submission_df = pd.DataFrame({'ID': test_1.ID, 'total_cost': stack_2})
submission_df.to_csv('final.csv',index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12203570476951772, 'fraction_numerical': 0.018030020428101964, 'mean_word_length': 7.8801261829652995, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '536759', 'license': 'MIT', 'n_tokens_mistral': 3798, 'n_tokens_neox': 3545, 'n_words': 867, 'path': '.ipynb_checkpoints/Best Solution-checkpoint.ipynb', 'repo_name': 'KuriaDan/Zindi-Kenya-Hack-Tanzania-Tourism-Prediction-Challenge', 'size': '11259'}