<jupyter_start><jupyter_text>Make scatter matricesBetween classes scatter matrix measures the distance between each class centers and data center.$$S_B = \sum_{c} N_c (\mu_c - \bar{x})(\mu_c - \bar{x})^\top$$Within classes scatter matrix measures the distance between data of a class and that class' center.$$S_W = \sum_{c} \sum_{i \in c} (x_i - \mu_c)(x_i - \mu_c)^\top$$<jupyter_code>y_tr = y[idx_tr]
X_tr_bar = np.mean(X_tr, axis=0)
S_B = np.zeros((1433, 1433))
S_W = np.zeros((1433, 1433))
for c in range(6):
    class_c_idx = np.where(y_tr == c)[0]
    N_c = len(class_c_idx)
    mu_c = np.mean(X_tr[class_c_idx], axis=0)
    S_B += N_c * (mu_c - X_tr_bar).reshape(1433, 1) @ (mu_c - X_tr_bar).reshape(1, 1433) 
for c in range(6):
    class_c_idx = np.where(y_tr == c)[0]
    mu_c = np.mean(X_tr[class_c_idx], axis=0)
    for i in class_c_idx:
        S_W += (X_tr[i] - mu_c).reshape(1433, 1) @ (X_tr[i] - mu_c).reshape(1, 1433) 
l, v = np.linalg.eigh(S_B)
l[l < 0] = 0  # S_B is not exactly symmetric due to numerical errors
S_B_p_half = v @ np.diag(l**0.5) @ v.T
S_W_inv = np.linalg.inv(S_W + 0.01 * np.eye(1433))
T = S_B_p_half @ S_W_inv @ S_B_p_half
lT, vT = np.linalg.eigh(T)
 
g = nx.from_numpy_array(adj.int().to_dense().numpy(), create_using=nx.Graph())
g.number_of_edges(), g.number_of_nodes()
for i in nx.connected_components(g):
    print(len(i))
    break
L = nx.laplacian_matrix(g)
L = L.todense()
lL, vL = np.linalg.eigh(L)
max(vL[:,1])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.17306397306397306, 'fraction_numerical': 0.037037037037037035, 'mean_word_length': 4.089041095890411, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '481881', 'license': 'MIT', 'n_tokens_mistral': 730, 'n_tokens_neox': 692, 'n_words': 143, 'path': 'notebooks/01-LDA-with-raw-features.ipynb', 'repo_name': 'gear/gfnn', 'size': '1485'}