<jupyter_start><jupyter_text>WhiteningThe goal of whitening is for the dataset to have zero mean, and identity covariance matrix. Let's first see if the existing whitening in the code gives that. Below we apply the whitening, and test the mean. If it is far away from zero, then there is a problem with the whitening.<jupyter_code>X = np.loadtxt(fname = "3PhData/DataTst.txt")
X_w = ngca_theoretical.whiten(X)
np.mean(X_w, axis=0)<jupyter_output><empty_output><jupyter_text>Obviously there is a problem with the whitening. Let's try to define a correct whitening.<jupyter_code>def center(X):
    c = np.mean(X, axis = 0)
    Xw = X - c
    return Xw, c

def whiten(X):
    X_c, c = center(X)
    U, s, VT = np.linalg.svd(X_c, full_matrices=False)
    W = VT.T * (math.sqrt(X.shape[0]) / s) 
    X_w = U * math.sqrt(X.shape[0]) 
    return X_w, c, W

X_w, c, W = whiten(X)<jupyter_output><empty_output><jupyter_text>Notice that the whitening procedure also returns the centering shift, and the transformation $W$ that made the covariance $I_n$. After the whitenning we should have:- The whitenned data $X_W$ should have zero mean, and identity covariance.- X_w  should be equal to applying W to the right after on X after centering (i.e. $X_w = (X - c)W$).Let's test these.<jupyter_code>np.mean(X_w, axis=0), np.linalg.norm(X_w.T @ X_w / X_w.shape[0] - np.eye(12)), np.linalg.norm(X_w - (X - c) @ W)<jupyter_output><empty_output><jupyter_text>NGCA algorithmNow that we whitenned the data, let's apply the NGCA algorithm.<jupyter_code>params = {'alpha1': 0.6754445940381727, 'alpha2': 0.29744739800298886, 
          'beta1': 0.3403472323546272, 'beta2': 0.6441926407645018}
X1 = X[:500]
X2 = X[500:]
approx_NG_subspace = ngca_theoretical.run_ngca_algorithm(X1, X2,
                                                         params['alpha1'], params['alpha2'], 
                                                         params['beta1'], params['beta2'])<jupyter_output><empty_output><jupyter_text>**The subspace should be a subspace of $\mathbb{R}^{12}$, so how can the basis below be 500-by-4???**<jupyter_code>approx_NG_subspace.shape<jupyter_output><empty_output><jupyter_text>**The problem seems to be in computing the Phi and Psi matrices. They should be 12-by-12, not 500-by-500**.<jupyter_code>matrix_phi = ngca_theoretical.compute_matrix_phi(X1, X2, params['alpha1'])
matrix_phi.shape<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11956971452213488, 'fraction_numerical': 0.04757964418700869, 'mean_word_length': 3.9346938775510205, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '327915', 'license': 'MIT', 'n_tokens_mistral': 953, 'n_tokens_neox': 854, 'n_words': 239, 'path': 'NGCA_test.ipynb', 'repo_name': 'shanys8/ngca', 'size': '2417'}