<jupyter_start><jupyter_text>Determine the right NB of dimensions<jupyter_code># Standardise data
X_std = StandardScaler().fit_transform(features_train)

num_vars = len(loadImages(image_folder))

U, S, V = np.linalg.svd(X_std) 
eigvals = S**2 / np.sum(S**2)  # NOTE (@amoeba): These are not PCA eigenvalues. 
                               # This question is about SVD.

fig = plt.figure(figsize=(8,5))
sing_vals = np.arange(num_vars) + 1
plt.plot(sing_vals, eigvals, 'ro-', linewidth=1, c='blue')
plt.xlim(right = 50)
plt.title('Scree Plot')
plt.xlabel('Principal Component')
plt.ylabel('Eigenvalue')
#I don't like the default legend so I typically make mine like below, e.g.
#with smaller fonts and a bit transparent so I do not cover up data, and make
#it moveable by the viewer in case upper-right is a bad place for it 
leg = plt.legend(['Eigenvalues from SVD'], loc='best', borderpad=0.3, 
                 shadow=False, prop=matplotlib.font_manager.FontProperties(size='small'),
                 markerscale=0.4)
leg.get_frame().set_alpha(0.4)
#leg.draggable(state=True)
plt.show()<jupyter_output><empty_output><jupyter_text>From the above scree, we observe a plateau starting from the 25th to 30th components, implying that we could reduce the data to 25-30 dimensions arbitrarily. But considering that the differences between consecutive eigenvalues are relatively small to draw an optimal conclusion, another more quantitative way to determine number of principle components is to apply the "90% Rule" on eigenvalues of standardised data.<jupyter_code># Standardise data
X_std = StandardScaler().fit_transform(features_train)

# Covariance Matrix and Eigendecomposition from standardised data
mean_vec = np.mean(X_std, axis=0)
cov_mat = (X_std - mean_vec).T.dot((X_std - mean_vec)) / (X_std.shape[0]-1)
print('Covariance matrix \n%s' %cov_mat)
#Perform eigendecomposition on covariance matrix
# Each eigenvector has a corresponding eigenvalue, and the sum of the eigenvalues represents all of the variance within the entire dataset.
cov_mat = np.cov(X_std.T)
eig_vals, eig_vecs = np.linalg.eig(cov_mat)
print('Eigenvectors \n%s' %eig_vecs)
print('\nEigenvalues \n%s' %eig_vals)
# Select principle components
# Visually confirm that the list is correctly sorted by decreasing eigenvalues
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]
print('Eigenvalues in descending order:')
for i in eig_pairs:
    print(i[0])
#Explained variance
pca = PCA().fit(X_std)
plt.plot(np.cumsum(pca.explained_variance_ratio_),'ro-', linewidth=1, c='purple')
plt.xlabel('number of components')
plt.ylabel('cumulative explained variance')
plt.xlim(right=50)
plt.show()<jupyter_output><empty_output><jupyter_text>Having performed a standardised eigenvalue cumulative plot, we observe that the first 29 or 30 elements describe most of the variation (information) within the data, meaning a major dimension reduction from the initial 2,048 features.  Apply PCA and LDA<jupyter_code># Standardise data
scaler = StandardScaler()

# Fit on training set only.
scaler.fit(features_train)

# Apply transform to both the training set and the test set.
train_X = scaler.transform(features_train)
test_X = scaler.transform(features_test)
# instantiate PCA with determined dimension
pca = PCA(n_components=35)

# fit train data to PCA
pca.fit(train_X)

#  the mapping (transform) to both the training set and the test set
train_X = pca.transform(train_X)
test_X = pca.transform(test_X)
# Use the average resnet features to predict the labels

# number of runs for cross-validation
num_runs = 5
# number of splits for cross-validation
num_splits = 20

# Multiple cross validations on the training set
aucs = []
for seed in range(num_runs):
    # Use linear Discriminant Analysis
    model = LinearDiscriminantAnalysis()

    cv = sklearn.model_selection.StratifiedKFold(n_splits=num_splits, shuffle=True, random_state=seed)

    # Cross validation on the training set
    auc = sklearn.model_selection.cross_val_score(model, X=train_X, y=labels_train,
                                                  cv=cv, scoring="roc_auc", verbose=0)
    
    aucs.append(auc)

aucs = np.array(aucs)

print("Predicting weak labels by mean resnet")
print("AUC: mean {}, std {}".format(aucs.mean(), aucs.std()))
# Prediction on the test set

# Train a final model on the full training set
estimator = LinearDiscriminantAnalysis()
estimator.fit(features_train, labels_train)

preds_test = estimator.predict_proba(features_test)[:, 1]

# Check that predictions are in [0, 1]
assert np.max(preds_test) <= 1.0
assert np.min(preds_test) >= 0.0

# -------------------------------------------------------------------------
# Write the predictions in a csv file, to export them in the suitable format
# to the data challenge platform
ids_number_test = [i.split("ID_")[1] for i in ids_test]
test_output = pd.DataFrame({"ID": ids_number_test, "Target": preds_test})
test_output.set_index("ID", inplace=True)
test_output.to_csv("predictions/preds_PCA_lda.csv")<jupyter_output><empty_output><jupyter_text>Tune LDA hyperparameters with grid search<jupyter_code># Use the average resnet features to predict the labels

# number of runs for cross-validation
num_runs = 5
# number of splits for cross-validation
num_splits = 20

# Multiple cross validations on the training set

# Use linear Discriminant Analysis
model = LinearDiscriminantAnalysis()

cv = sklearn.model_selection.RepeatedStratifiedKFold(n_splits=num_splits, n_repeats=num_runs, random_state=1)

# define grid
grid = dict()
grid['solver'] = ['svd', 'lsqr', 'eigen']
# define search
search = sklearn.model_selection.GridSearchCV(model, grid, scoring='roc_auc', cv=cv, n_jobs=-1)
# perform the search
results = search.fit(X=train_X, y=labels_train)
    
# summarize
print('Mean Accuracy: %.3f' % results.best_score_)
print('Config: %s' % results.best_params_)<jupyter_output>Mean Accuracy: 0.714
Config: {'solver': 'svd'}
{'fraction_non_alphanumeric': 0.113868857957404, 'fraction_numerical': 0.010565151769243669, 'mean_word_length': 4.82421875, 'pattern_counts': {'":': 2, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '116629', 'license': 'MIT', 'n_tokens_mistral': 2011, 'n_tokens_neox': 1896, 'n_words': 582, 'path': 'LDA_PCA.ipynb', 'repo_name': 'Peischlili/BreastCancer_Metastases_Detection', 'size': '5963'}