<jupyter_start><jupyter_text>We find number of unique value for each column, to have an idea about which variables are continuous, which variables are binary, category. It depends on data, however it is likely that nu = 2 --> binary; nu = 3 or 4: --> category, n > 4: continuous. Of course, we have to see data in detail as well.<jupyter_code>X = Xy[:,:-1]
l,n = X.shape
nu = np.array([len(np.unique(X[:,i])) for i in range(n)])
print('number of uniques of each variable:')
print(nu)
# We then define variable type, 1: continuous, 2: binary, 3: category.
variable_type  = np.ones(n) # continuous
variable_type[5:8] = 2 # binary
variable_type[13:] = 2 # binary

print(variable_type)
def convert_continuous_to_binary(x,variable_type):
    """
    convert continuous to binary by subtracting them with mean values.
    """
    
    # create 2 initial columns
    x_new = np.zeros((x.shape[0],2))

    for i,i_type in enumerate(variable_type):
        if i_type == 1: # continuous            
            x1 = np.sign(x[:,i] - x[:,i].mean())            
        x_new = np.hstack((x_new,x1[:,np.newaxis]))
    # drop the 2 initial column
    x_new = x_new[:,2:]
    
    return x_new.astype(float)
X_new = convert_continuous_to_binary(X,variable_type)
nu = np.array([len(np.unique(X_new[:,i])) for i in range(n)])
print('number of uniques of each variable:')
print(nu)
print(X_new)
for i in range(n):
    print(i,np.unique(X_new[:,i],return_counts=True))
# disease vs non-disease
# select non-disease
t = Xy[:,-1] == 0
X1 = X_new[t]
X2 = X_new[~t]
for i in range(n):
    print(i,np.unique(X1[:,i],return_counts=True),np.unique(X2[:,i],return_counts=True))
np.savetxt('data_binary_nondisease.txt',X1,fmt='%i')
np.savetxt('data_binary_disease.txt',X2,fmt='%i')
np.savetxt('data_binary.txt',X_new,fmt='%i')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15472936030617823, 'fraction_numerical': 0.016402405686167305, 'mean_word_length': 4.213675213675214, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '283505', 'license': 'MIT', 'n_tokens_mistral': 692, 'n_tokens_neox': 672, 'n_words': 172, 'path': 'Ref/fig4_smoking_data_train30_test70/remove_conseved_variables/0convert_to_binary.ipynb', 'repo_name': 'danhtaihoang/e-machine', 'size': '1829'}