<jupyter_start><jupyter_text>Lets fill the missing data with the mean value<jupyter_code>missing_values = ["Age", "Fare"]
for i in missing_values:
    mean = train_df.loc[:, i].mean()
    train_df.loc[:, i].fillna(mean, inplace =True)
    test_df.loc[:, i].fillna(mean, inplace =True)
#Non int case for embarked, only 2 values missing here so we drop them
train_df = train_df.dropna(axis = 'index')
    
#train_df.loc[:, "Age"].fillna(train_df.loc[:,"Age"].mean(), inplace =True)


#Scikit method. Will not implement atm.
#ageimp = Imputer(missing_values = 'NaN', strategy = 'mean', axis = 1)
#x_train[:, [3,7]] = imp.transform(x_train[:,[3,7]])
#x_train[:, [3,7]] = imp.transform(x_train[:,[3,7]])
#Only works on seperate columns
#imp = imp.fit(x_train[:,[3,7]])
#x_train[:, [3,7]] = imp.transform(x_train[:,[3,7]])
#x_train[:,3] = np.asarray(imp.fit_transform(np.asmatrix(x_train[:,7])))<jupyter_output><empty_output><jupyter_text>Confirm the imputation succeeded<jupyter_code>train_df["Age"].head(n=6)
train_df.describe()
test_df.describe()
import seaborn as sns
sns.set(style="whitegrid", color_codes=True)
sns.countplot(y="Embarked" , data = train_df)
sns.barplot(x="Embarked", y = "Survived", data = train_df)
sns.barplot(x="Sex", y= "Survived", hue="Pclass", data = train_df)<jupyter_output><empty_output><jupyter_text>All the missing data has been filled. Whats left now is to encode categorical values and feature scaling.Lets start with encoding.<jupyter_code>train_df = pd.get_dummies(train_df, columns = ["Sex", "Pclass","Embarked"])
test_df = pd.get_dummies(test_df, columns = ["Sex", "Pclass","Embarked"])
train_df.head(n=5)
train_df.count()<jupyter_output><empty_output><jupyter_text>The dataset is now complete. On to feature scaling and data splitting.<jupyter_code>from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

train, validate = train_test_split(train_df, test_size = 0.25, random_state = seed)
x_train = train.iloc[:,1:]
y_train = train.iloc[:,0]
x_val = validate.iloc[:,1:]
y_val = validate.iloc[:,0]
x_full = train_df.iloc[:,1:]
y_full = train_df.iloc[:,0]

scaler = StandardScaler()
x_train = scaler.fit_transform(x_train)
x_val = scaler.transform(x_val)
x_test = scaler.transform(test_df)
x_full = scaler.transform(x_full)
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
parameters_test1 = [{'C': [1, 10, 100, 1000, 2000], 'kernel': ['linear']},
              {'C': [1, 10, 100, 1000], 'kernel': ['rbf'], 'gamma': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]}]
grsearch1 = GridSearchCV(SVC(),
                        param_grid = parameters_test1, scoring = 'roc_auc', cv = 3, n_jobs = 4, return_train_score = True )
grsearch1 = grsearch1.fit(x_train, y_train)
pd.DataFrame(grsearch1.cv_results_)
grsearch1.best_score_, grsearch1.best_params_
measures = 10
result= []
Sum = 0
for i in range(measures):
    import time
    start = time.time()
    clf = SVC(kernel = 'rbf', gamma = 0.2, C = 1)
    clf.fit(x_train, y_train)
    finish = time.time()
    result = finish-start
    Sum = Sum+result
  
mean = Sum/measures
print('Total time elapsed: {time}' .format(time = Sum))
print('Average time elapsed per measure: {time}'.format(time = mean))
clf = SVC(kernel = 'rbf', gamma = 0.2, C = 1)
clf.fit(x_train, y_train)
y_pred = clf.predict(x_val)
y_pred
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_val, y_pred)
cm
1-(cm[0,1] + cm[1,0])/(cm[0,0]+cm[1,1])<jupyter_output><empty_output><jupyter_text>It seems we have about 76% accuracy on our test set with the current model.Time to make the real predictions and see what that gives us.<jupyter_code>predictions = clf.predict(x_test)
predictions

inputdf = pd.read_csv("test.csv")
data_to_submit = pd.DataFrame({
    'PassengerId':inputdf.iloc[:,0],
    'Survived':predictions
})
data_to_submit.to_csv("svm_results.csv", index=False)<jupyter_output><empty_output><jupyter_text>results on kaggle are now at 77,9% accuracy The grid search made above is quite heavy. What results would svm achieve when using default parameters?<jupyter_code>clf = SVC(kernel = 'rbf')
clf.fit(x_train, y_train)
val_predictions = clf.predict(x_val)
cm = confusion_matrix(val_predictions, y_val)
cm
1-(cm[0,1] + cm[1,0])/(cm[0,0]+cm[1,1])<jupyter_output><empty_output><jupyter_text>With the default parameters it seems we achieve 74% accuracy on our test set. Lets look a bit closer at our optimizations and see if there is more that can be done.<jupyter_code>parameters_test2 = {'C': [0.5, 0.8, 0.9, 1, 1.1, 1.2, 1.5, 2], 'kernel': ['rbf'], 'gamma': [0.1, 0.15, 0.2, 0.25, 0.3, 0.5, 0.9]}
grsearch2 = GridSearchCV(SVC(random_state = 7),
                        param_grid = parameters_test2, scoring = 'roc_auc', cv = 3, n_jobs = 4, return_train_score = True )
grsearch2 = grsearch2.fit(x_train, y_train)
pd.DataFrame(grsearch2.cv_results_)
grsearch2.best_score_, grsearch2.best_params_
parameters_test2 = {'C': [0.2, 0.3, 0.5, 0.8, 0.9, 1, 1.1, 1.2, 1.5, 2], 'kernel': ['rbf'], 'gamma': [0.01, 0.05, 0.08, 0.1, 0.15, 0.2, 0.25, 0.3, 0.5, 0.9]}
grsearch2 = GridSearchCV(SVC(random_state = 7),
                        param_grid = parameters_test2, scoring = 'roc_auc', cv = 3, n_jobs = 4, return_train_score = True )
grsearch2 = grsearch2.fit(x_train, y_train)
pd.DataFrame(grsearch2.cv_results_)
grsearch2.best_score_, grsearch2.best_params_
parameters_test2 = {'C': [0.3, 0.5, 0.8, 1], 'kernel': ['rbf'], 'gamma': [0.001, 0.005, 0.008, 0.01, 0.05, 0.08, 0.1, 0.15, 0.2, 0.25, 0.3, 0.5]}
grsearch2 = GridSearchCV(SVC(random_state = 7),
                        param_grid = parameters_test2, scoring = 'roc_auc', cv = 3, n_jobs = 4, return_train_score = True )
grsearch2 = grsearch2.fit(x_train, y_train)
pd.DataFrame(grsearch2.cv_results_)
grsearch2.best_score_, grsearch2.best_params_
clf = SVC(kernel = 'rbf', C = 1, gamma = 0.008)
clf.fit(x_train, y_train)
val_predictions = clf.predict(x_val)
cm = confusion_matrix(val_predictions, y_val)
cm
1-(cm[0,1] + cm[1,0])/(cm[0,0]+cm[1,1])<jupyter_output><empty_output><jupyter_text>Examining the test results it seems we have slighthly balanced the number of false positives and negatives (decreased variance) but increased the bias.We did get a slight increase in AUC. Lets see what result we get on kaggle.<jupyter_code>predictions = clf.predict(x_test)
inputdf = pd.read_csv("test.csv")
data_to_submit = pd.DataFrame({
    'PassengerId':inputdf.iloc[:,0],
    'Survived':predictions
})
data_to_submit.to_csv("svm_results.csv", index=False)<jupyter_output><empty_output><jupyter_text>Ahd the result is still 77,03%<jupyter_code>clf = SVC(kernel = 'rbf', C = 1, gamma = 0.008)
clf.fit(x_full, y_full)
predictions = clf.predict(x_test)
predictions = clf.predict(x_test)
inputdf = pd.read_csv("test.csv")
data_to_submit = pd.DataFrame({
    'PassengerId':inputdf.iloc[:,0],
    'Survived':predictions
})
data_to_submit.to_csv("full_svm_results.csv", index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15620980420180078, 'fraction_numerical': 0.04130341574960698, 'mean_word_length': 5.338768115942029, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '274045', 'license': 'Apache-2.0', 'n_tokens_mistral': 2875, 'n_tokens_neox': 2720, 'n_words': 610, 'path': 'Titanic_SVM.ipynb', 'repo_name': 'RickVM/Kaggle-Titanic', 'size': '6997'}