<jupyter_start><jupyter_text>Imbalanced Labels and Athletes Similar to Olympic Medalists<jupyter_code>#reload scripts when we edit them
%load_ext autoreload
%autoreload 2

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import qgrid
df = pd.read_csv('data/olympic/athlete_events.csv', encoding='utf8')
print(df.shape)
qgrid.show_grid(df)
#we're only interested in medalist or not
df['Medal'] = df.Medal.map(lambda x: 1 if x==x else 0)
df.Medal.mean()<jupyter_output><empty_output><jupyter_text>Coding Assignment Using only `Sex`, `Age`, `Height`, `Weight`, `Sport`, `City`, `Season` and `Team`, predict whether an athelete is a medalist or not.<jupyter_code>df = df[['Sex','Age','Height','Weight','Sport','City','Season','Team','Medal']].copy()
train = df.sample(int(df.shape[0]*0.8))
test = df.drop(train.index, errors="ignore")
train.shape, test.shape
import utils
for col in ['Sex','Sport','City','Season','Team']:
    #use same dict for train and test
    d = utils.other_dict(train,col,0.03)
    train = utils.otherify(train,col,d)
    test = utils.otherify(test,col,d)
train = utils.missingify(train).drop('Medal_missing',1)
test = utils.missingify(test).drop('Medal_missing',1)
test.head()
#y
y_train = np.array(train.Medal)
y_test = np.array(test.Medal)
train = train.drop('Medal',1)
test = test.drop('Medal',1)
y_train.shape,y_test.shape
#X_cat
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(categories=[list(train[col].unique())+['xxna'] for col in train.columns[:5]])

feature_cols = list(train.columns[:5])
X_train_cat = train[feature_cols].values
enc_fit = enc.fit(X_train_cat)
X_train_cat = enc_fit.transform(X_train_cat).toarray()
X_test_cat = enc_fit.transform(test.iloc[:,:5]).toarray()
X_train_cat.shape,X_test_cat.shape
#X
X_train = np.concatenate([X_train_cat,train.iloc[:,5:]],1)
X_test = np.concatenate([X_test_cat,test.iloc[:,5:]],1)
X_train.shape,X_test.shape
#standardize
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler_fit = scaler.fit(X_train)
X_train = scaler_fit.transform(X_train)
X_test = scaler_fit.transform(X_test)
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(n_estimators=10)
clf = clf.fit(X_train, y_train)
print(f'Accuracy: {clf.score(X_test,y_test)}')<jupyter_output>Accuracy: 0.8395728828562998<jupyter_text>**Conceptual Assignment** Is this a good performance? Why or why not?<jupyter_code>print(f'Baseline accuracy: {1 - y_test.mean()}')
from sklearn.metrics import confusion_matrix
import seaborn as sns
preds = clf.predict(X_test)
cm = confusion_matrix(y_test,preds)
sns.heatmap(cm/1000, annot=True,annot_kws={"size": 16})<jupyter_output><empty_output><jupyter_text>The Lookalike Approach Instead of using a model to predict if an athelete is a medalist or not we can reshape the problem as **which atheletes look like medalists the most**. This is especially useful for use cases with imbalanced data in real life. For instance, a retailer might have only 1% of its customer buying both online and offline. If we use a supervised model, we will end up with poor performance like above but that is not because the model is bad but maybe because those customers are still untapped and can be converted.We can either use supervised model to target them or find the similarity based on our available features. The upside of the lookalike approach is that it is very easy to explain intuitively. We are just finding those who looks like the people we know we want. **Conceptual Assignment** The following function takes an input of two feature vectors and return [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity). For each line, 1. What linear algebra operations are being done?2. What are the dimensions of input and output?Tensorflow 2 is quite fast at matrix multiplication so we advise you try using it. See [how-to](https://www.tensorflow.org/install/).<jupyter_code>import tensorflow as tf

def get_sim(f0,f1): 
    dot_product =  f0 @ tf.transpose(f1)
    f0_norm = tf.norm(f0,axis=1)
    f1_norm = tf.norm(f1,axis=1)
    return tf.reduce_mean(dot_product / (f0_norm[:,None]  * f1_norm[None,:]),1).numpy()

feat_tensor = tf.convert_to_tensor(X_train, dtype='float32')
feat_1 = tf.convert_to_tensor(X_train[y_train==1], dtype='float32')
ds = tf.data.Dataset.from_tensor_slices(feat_tensor).batch(3000)
feat_tensor.shape, feat_1.shape
import tqdm
sims = []
for f0 in tqdm.tqdm(ds):
    sims.append(get_sim(f0,feat_1))
s = np.concatenate(sims)
s.max(),s.min(),(s>0).sum()
plt.hist(s)
plt.hist(s[y_train==1])
#accuracy
((s>0).astype(int)==y_train).mean()
#confusion matrix
preds = (s>0).astype(int)
cm = confusion_matrix(y_train,preds)
sns.heatmap(cm/1000, annot=True,annot_kws={"size": 16})<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12844036697247707, 'fraction_numerical': 0.01834862385321101, 'mean_word_length': 6.402777777777778, 'pattern_counts': {'":': 2, '<': 16, '<?xml version=': 0, '>': 19, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '376674', 'license': 'Apache-2.0', 'n_tokens_mistral': 1748, 'n_tokens_neox': 1685, 'n_words': 438, 'path': 'imbalanced.ipynb', 'repo_name': 'witchapong/wangchan-analytica', 'size': '4796'}