<jupyter_start><jupyter_text>Check age distribution for the raw data<jupyter_code>g = sns.FacetGrid(train, col='Survived')
g.map(plt.hist, 'Age', bins=20)<jupyter_output><empty_output><jupyter_text>Check age distribution for the processed data, where we've substituted missing values<jupyter_code>g = sns.FacetGrid(pd.concat([df, pd.DataFrame(y, columns=['Survived'])], axis=1), col='Survived')
g.map(plt.hist, 'Age', bins=20)
kmeans = KMeans(n_clusters=2)
kmeans.fit(df)
def predictScore(df, y):
    correct = 0
    for index, row in df.iterrows():
        predict_me = np.array(row.astype(float))
        predict_me = predict_me.reshape(-1, len(predict_me))
        prediction = kmeans.predict(predict_me)
        if prediction[0] == y[index]:
            correct += 1

    print(correct/len(df))<jupyter_output><empty_output><jupyter_text>In a real clustering problem, of course, we don't have the labels. This example is just for illustration.Your score may fluctuate between 2 percentages, depending on how the clusters were chosen.I.e. cluster0 == 'Survived' & cluster1=='Dead' vs. cluster1 == 'Survived' & cluster0=='Dead'This means a low or high score is good. Around 50% is not a good separation.<jupyter_code>predictScore(df, y)<jupyter_output>0.5230078563411896<jupyter_text>Our score is not that good. Which is probably because we did not scale our data yet.<jupyter_code>scaler = MinMaxScaler()
X_scaled = scaler.fit_transform(df)
X_scaled = pd.DataFrame(X_scaled, columns=[df.columns])
display_all(df.describe(include='all').T)
display_all(X_scaled.describe(include='all').T)
y_means = kmeans.fit(X_scaled)<jupyter_output><empty_output><jupyter_text>Your score may fluctuate between 2 percentages, depending on how the clusters were chosen.I.e. cluster0 == 'Survived' & cluster1=='Dead' vs. cluster1 == 'Survived' & cluster0=='Dead'This means a low or high score is good. Around 50% is not a good separation.<jupyter_code>predictScore(X_scaled, y)<jupyter_output>0.2132435465768799<jupyter_text>Much better.<jupyter_code>from sklearn.manifold import TSNE
tsne = TSNE(n_components=2)
tsne_fit = tsne.fit_transform(X_scaled)
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode, iplot
group_pred = kmeans.fit_predict(X_scaled)
tsne_fit
name = df.pop('Name')
init_notebook_mode(connected=True)
trace = go.Scatter(
    x=tsne_fit.T[0], 
    y=tsne_fit.T[1],
    mode='markers',
    name='Lines, Markers and Text',
    text=name,
    textposition='top left',
    marker=dict(
        color = group_pred, #set color equal to a variable
        colorscale='Portland',
        showscale=True
    )
)

data = [trace]
layout = go.Layout(
    showlegend=False
)
fig = go.Figure(data=data, layout=layout)
iplot(fig)
from sklearn.decomposition import PCA
pca = PCA(n_components=12)
pca.fit(X_scaled)
X_pca = pca.transform(X_scaled)
features = range(pca.n_components_)
plt.bar(features, pca.explained_variance_ratio_, color='black')
plt.xlabel('PCA features')
plt.ylabel('variance %')
plt.xticks(features)
init_notebook_mode(connected=True)
trace = go.Scatter(
    x=X_pca.T[0], 
    y=X_pca.T[1],
    mode='markers',
    name='Lines, Markers and Text',
    text=name,
    textposition='top left',
    marker=dict(
        color = group_pred, #set color equal to a variable
        colorscale='Portland',
        showscale=True
    )
)

data = [trace]
layout = go.Layout(
    showlegend=False
)
fig = go.Figure(data=data, layout=layout)
iplot(fig)
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure(1, figsize=(4, 3))
plt.clf()
ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=48, azim=134)

for name, label in [('Not survived', 0), ('Survived', 1)]:
    ax.text3D(X_pca[y == label, 0].mean(),
              X_pca[y == label, 1].mean(),
              0, name,
              horizontalalignment='center',
              bbox=dict(alpha=.5, edgecolor='w', facecolor='w'))
# Reorder the labels to have colors matching the cluster results
cy = np.choose(y, [1, 2, 0]).astype(np.float)
ax.scatter(X_pca[:, 0], X_pca[:, 1], X_pca[:, 2], c=cy, cmap=plt.cm.nipy_spectral,
           edgecolor='k')

ax.w_xaxis.set_ticklabels([])
ax.w_yaxis.set_ticklabels([])
ax.w_zaxis.set_ticklabels([])

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13686690223792697, 'fraction_numerical': 0.022143698468786808, 'mean_word_length': 4.7456021650879565, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '331118', 'license': 'MIT', 'n_tokens_mistral': 1636, 'n_tokens_neox': 1537, 'n_words': 344, 'path': 'PXL_DIGITAL_JAAR_2/AI & Robotics/Week 12/notebooks/Clustering_Titanic.ipynb', 'repo_name': 'Limoentaart/PXL_IT_JAAR_1', 'size': '4245'}