<jupyter_start><jupyter_text>Read data<jupyter_code>feature_name = pd.read_csv('labels.txt',header=None,sep="/").values
feature_name = feature_name.reshape(-1)
print(feature_name.shape)
print(feature_name)<jupyter_output>(535,)
['Subject index (1-40)' 'ECG_original_mean' 'ECG_original_std'
 'ECG_original_trimmean25' 'ECG_original_median' 'ECG_original_skewness'
 'ECG_original_kurtosis' 'ECG_original_max' 'ECG_original_min'
 'ECG_original_prctile25' 'ECG_original_prctile75'
 'ECG_original_geomean(abs)' 'ECG_original_harmmean' 'ECG_original_mad'
 'ECG_original_baseline' 'ECG_RR_window_mean' 'ECG_RR_window_std'
 'ECG_RR_window_trimmean25' 'ECG_RR_window_median'
 'ECG_RR_window_skewness' 'ECG_RR_window_kurtosis' 'ECG_RR_window_max'
 'ECG_RR_window_min' 'ECG_RR_window_prctile25' 'ECG_RR_window_prctile75'
 'ECG_RR_window_geomean(abs)' 'ECG_RR_window_harmmean' 'ECG_RR_window_mad'
 'ECG_RR_window_baseline' 'ECG_amplitude_RR_mean' 'ECG_amplitude_RR_std'
 'ECG_amplitude_RR_trimmean25' 'ECG_amplitude_RR_median'
 'ECG_amplitude_RR_skewness' 'ECG_amplitude_RR_kurtosis'
 'ECG_amplitude_RR_max' 'ECG_amplitude_RR_min'
 'ECG_amplitude_RR_prctile25' 'ECG_amplitude_RR_prctile75'
 'ECG_amplitude_RR_geomean(abs)' 'ECG_ampl[...]<jupyter_text>When I try to use ```data = pd.read_csv('data.txt', header=None, names = feature_name)```it returns errors, and says the names are duplicated<jupyter_code>data = pd.read_csv('data.txt', header=None)
print(data.shape)
print(data.values)
data_arr = data.values
X = data_arr[:, 1:-1]
y = data_arr[:,-1]
print(X.shape, y.shape)
# split data into train and test dataset

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y, shuffle=True)

print(X_train.shape, X_test.shape)<jupyter_output>(3360, 533) (1120, 533)<jupyter_text>Too many features, it's necessary to use some methods to reduce the dimensions.  Preprocessing  Scale data For some methods, it's necessary to scale the data<jupyter_code>scaler = StandardScaler()
scaler.fit(X_train)
X_train_scale = scaler.transform(X_train)
X_test_scale = scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>Feature selections If we use some traditional ML methods, it's important to reduce the dimension by removing most nuisance variables  PCA The simplest way is to apply PCA to select features<jupyter_code>pca = PCA(n_components=100,svd_solver='randomized', whiten=True)
pca.fit(X_train_scale)

X_train_pca = pca.transform(X_train_scale)
X_test_pca = pca.transform(X_test_scale)
print(np.sum(pca.explained_variance_ratio_))<jupyter_output>0.9894863674773985<jupyter_text>Random Forest We can also use RF to select important features<jupyter_code>n_component = 200

model = RandomForestClassifier(n_jobs=8, random_state=0)
model.fit(X_train_scale, y_train.ravel())

feature_importance = model.feature_importances_

index = np.argsort(feature_importance)[-1:-n_component-1:-1]

X_train_RF = X_train_scale[:, index]
X_test_RF = X_test_scale[:, index]

print(np.sum(feature_importance[index]))
print(feature_name[index])<jupyter_output>['IT_LF_harmmean' 'IT_Original_harmmean' 'ECG_hrv_std' 'ECG_hrv_min'
 'ECG_hrv_prctile25' 'ECG_HR_min_div_std' 'ECG_hrv_prctile75'
 'ECG_Entropyaprox' 'ECG_HR_min_div_trimmean25' 'ECG_RSAindex'
 'ECG_hrv_trimmean25' 'IT_Original_mean' 'IT_LF_kurtosis' 'IT_BRV_mad'
 'ECG_RR_window_prctile75' 'IT_LF_trimmean25' 'ECG_hrv_kurtosis'
 'IT_LF_mean' 'ECG_amplitude_RR_baseline' 'ECG_original_baseline'
 'IT_PSD_mad' 'IT_RF_harmmean' 'ECG_HR_min_div_min' 'EDA_processed_mad'
 'ECG_RR_window_trimmean25' 'IT_Original_kurtosis' 'IT_LF_prctile75'
 'IT_LF_mad' 'IT_VLF_prctile75' 'IT_VLF_baseline' 'IT_HF_mad' 'IT_MF_mad'
 'IT_MF_geomean(abs)' 'IT_LF_mad' 'IT_BRV_mean' 'IT_VLF_mad' 'IT_RF_mad'
 'IT_BRV_std' 'IT_p_Total_harmmean' 'IT_MF_prctile75' 'IT_VLF_std'
 'IT_Original_mad' 'IT_VLF_trimmean25' 'IT_LF_mean' 'IT_p_Total_mad'
 'IT_p_Total_kurtosis' 'IT_RF_prctile25' 'ECG_RR_window_prctile25'
 'ECG_RR_window_max' 'IT_RF_prctile75' 'IT_p_Total_geomean(abs)'
 'IT_PSD_baseline' 'IT_p_Total_mean' 'ECG_HR_min[...]<jupyter_text>Classifier  SVM<jupyter_code>p_grid_lsvm = {'C': [1e-3,1e-2,0.05,1e-1,0.5,1,2,5,1e1,1e2,15,20,40,60,80,120,140],
                'gamma': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1, 1,2,4,8,10,15,20,30,40,50,60], }
Lsvm = SVC(kernel='rbf')
grid_lsvm = GridSearchCV(estimator=Lsvm, param_grid=p_grid_lsvm, scoring='f1_macro', cv=5, n_jobs=8)<jupyter_output><empty_output><jupyter_text>PCA preprocessed data<jupyter_code>grid_lsvm.fit(X_train_pca, y_train.ravel())
test_score = grid_lsvm.score(X_test_pca, y_test.ravel())

print("Best training Score: {}".format(grid_lsvm.best_score_))
print("Best training params: {}".format(grid_lsvm.best_params_))
print("Test score: {}".format(test_score))<jupyter_output>Best training Score: 0.9218288217136322
Best training params: {'C': 100.0, 'gamma': 0.005}
Test score: 0.9173992595782752<jupyter_text>RF preprocessed data<jupyter_code>grid_lsvm.fit(X_train_RF, y_train.ravel())
test_score = grid_lsvm.score(X_test_RF, y_test.ravel())

print("Best training Score: {}".format(grid_lsvm.best_score_))
print("Best training params: {}".format(grid_lsvm.best_params_))
print("Test score: {}".format(test_score))<jupyter_output>Best training Score: 0.9573292439132775
Best training params: {'C': 140, 'gamma': 0.01}
Test score: 0.9580402509506486<jupyter_text>Comments The results of SVM seem very good, can achieve 0.958 f1 score.  Boost<jupyter_code>XGB = XGBClassifier()
p_grid_xgb = dict(
    max_depth = [4, 5, 6, 7],
    learning_rate = np.linspace(0.03, 0.3, 10),
    n_estimators = [100, 200]
)

grid_xgb = GridSearchCV(estimator=XGB, param_grid=p_grid_xgb, scoring='f1_macro', cv=5, n_jobs = 8)<jupyter_output><empty_output><jupyter_text>PCA preprocessed data<jupyter_code>grid_xgb.fit(X_train_pca, y_train.ravel())
test_score = grid_xgb.score(X_test_pca, y_test.ravel())

print("Best training Score: {}".format(grid_xgb.best_score_))
print("Best training params: {}".format(grid_xgb.best_params_))
print("Test score: {}".format(test_score))<jupyter_output>Best training Score: 0.9047633318624646
Best training params: {'learning_rate': 0.3, 'max_depth': 4, 'n_estimators': 200}
Test score: 0.9138744485163<jupyter_text>RF preprocessed data<jupyter_code>grid_xgb.fit(X_train_RF, y_train.ravel())
test_score = grid_xgb.score(X_test_RF, y_test.ravel())

print("Best training Score: {}".format(grid_xgb.best_score_))
print("Best training params: {}".format(grid_xgb.best_params_))
print("Test score: {}".format(test_score))<jupyter_output>Best training Score: 0.9958358824385221
Best training params: {'learning_rate': 0.3, 'max_depth': 5, 'n_estimators': 200}
Test score: 0.9982126770551818<jupyter_text>Comments Amazing!!!The best f1 score is 0.998 for boosting method, which is much better than SVM. Maybe this time, the candidate hyperparameter list is suitable, and then we can achieve a rather good result. From this result, we can find the advantage of ensemble method is that if we can find a good hyperparameter, the score will be close to 1The disadvantage is that it's too slow, I run about 1.5 hours for one cell.  MLP from sklearn<jupyter_code>MLP = MLPClassifier(activation='relu', alpha=1e-4, batch_size='auto', beta_1=0.9,
       beta_2=0.999, early_stopping=False, epsilon=1e-08,
       hidden_layer_sizes=(256,256,128), 
       learning_rate_init=0.01, max_iter=1000, momentum=0.9,
       nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,
       tol=0.0001, validation_fraction=0.1,
       warm_start=False, verbose=10)

p_grid_mlp = {'solver': ['adam', 'sgd'], 'learning_rate' : ['adaptive', 'constant']}
grid_mlp = GridSearchCV(estimator=MLP, param_grid=p_grid_mlp, scoring='f1_macro', cv=5, n_jobs = 8)<jupyter_output><empty_output><jupyter_text>Original dimension<jupyter_code>grid_mlp.fit(X_train_scale, y_train.ravel())
test_score = grid_mlp.score(X_test_scale, y_test.ravel())

print("Best training Score: {}".format(grid_mlp.best_score_))
print("Best training params: {}".format(grid_mlp.best_params_))
print("Test score: {}".format(test_score))<jupyter_output>Iteration 1, loss = 1.06711698
Iteration 2, loss = 0.59322218
Iteration 3, loss = 0.46943327
Iteration 4, loss = 0.40232163
Iteration 5, loss = 0.35621447
Iteration 6, loss = 0.31909582
Iteration 7, loss = 0.28996054
Iteration 8, loss = 0.25657780
Iteration 9, loss = 0.23495806
Iteration 10, loss = 0.21514567
Iteration 11, loss = 0.19332719
Iteration 12, loss = 0.17792871
Iteration 13, loss = 0.16293230
Iteration 14, loss = 0.15295348
Iteration 15, loss = 0.13975993
Iteration 16, loss = 0.13240095
Iteration 17, loss = 0.11866870
Iteration 18, loss = 0.11330361
Iteration 19, loss = 0.10457455
Iteration 20, loss = 0.09695808
Iteration 21, loss = 0.09003100
Iteration 22, loss = 0.08407912
Iteration 23, loss = 0.07867041
Iteration 24, loss = 0.08135036
Iteration 25, loss = 0.07039680
Iteration 26, loss = 0.07057068
Iteration 27, loss = 0.06347902
Iteration 28, loss = 0.05986975
Iteration 29, loss = 0.05586489
Iteration 30, loss = 0.05069077
Iteration 31, loss = 0.05151259
Iteration 32, los[...]<jupyter_text>PCA preprocessed data<jupyter_code>grid_mlp.fit(X_train_pca, y_train.ravel())
test_score = grid_mlp.score(X_test_pca, y_test.ravel())

print("Best training Score: {}".format(grid_mlp.best_score_))
print("Best training params: {}".format(grid_mlp.best_params_))
print("Test score: {}".format(test_score))<jupyter_output><empty_output><jupyter_text>RF preprocessed data<jupyter_code>grid_mlp.fit(X_train_RF, y_train.ravel())
test_score = grid_mlp.score(X_test_RF, y_test.ravel())

print("Best training Score: {}".format(grid_mlp.best_score_))
print("Best training params: {}".format(grid_mlp.best_params_))
print("Test score: {}".format(test_score))<jupyter_output>Iteration 1, loss = 1.04985960
Iteration 2, loss = 0.63727822
Iteration 3, loss = 0.52461790
Iteration 4, loss = 0.46089517
Iteration 5, loss = 0.41608143
Iteration 6, loss = 0.38129003
Iteration 7, loss = 0.35117329
Iteration 8, loss = 0.32573186
Iteration 9, loss = 0.30401093
Iteration 10, loss = 0.28400534
Iteration 11, loss = 0.26503061
Iteration 12, loss = 0.24958962
Iteration 13, loss = 0.23437216
Iteration 14, loss = 0.22313299
Iteration 15, loss = 0.20990740
Iteration 16, loss = 0.19822792
Iteration 17, loss = 0.19004263
Iteration 18, loss = 0.18048738
Iteration 19, loss = 0.17183089
Iteration 20, loss = 0.16583936
Iteration 21, loss = 0.15574333
Iteration 22, loss = 0.14756419
Iteration 23, loss = 0.14385961
Iteration 24, loss = 0.14065025
Iteration 25, loss = 0.13246616
Iteration 26, loss = 0.12535566
Iteration 27, loss = 0.12591611
Iteration 28, loss = 0.11796254
Iteration 29, loss = 0.11312187
Iteration 30, loss = 0.11272455
Iteration 31, loss = 0.10220925
Iteration 32, los[...]<jupyter_text>Comments This time, RF preprocessed data can have a better performance. I think one possible reason is that the original data has too many nuisance variables, which reduce the performance, but if we use RF to select the features, we can avoid this problem.  MLP from tensorflow Code is modified from the TP of our image course<jupyter_code>import tensorflow as tf

# import tensorflow models
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, Input, BatchNormalization
from tensorflow.keras.layers import Conv2D, MaxPooling2D
from tensorflow.keras import optimizers
print(tf.keras.__version__)
Y_train = tf.keras.utils.to_categorical(y_train)  # in order to convert y to a matrix with (num_examples, num_classes) (one-hot encoding)
Y_test = tf.keras.utils.to_categorical(y_test)  # in order to convert y to a matrix with (num_examples, num_classes) (one-hot encoding)
print(Y_train.shape, Y_test.shape)
Y_train = Y_train[:,1:]
Y_test = Y_test[:,1:]
print(Y_train.shape, Y_test.shape)
# Network Parameters
n_hidden_1 = 256 # 1st layer number of neurons
n_hidden_2 = 256 # 2nd layer number of neurons
n_hidden_3 = 128 # 2nd layer number of neurons

n_input = X_train.shape[1]
n_classes = 4
# TO CODE BY STUDENTS


model_mlp_multi_layer = Sequential()   # FILL IN STUDENTS
model_mlp_multi_layer.add(Dense(n_hidden_1, input_shape=(n_input,)))
model_mlp_multi_layer.add(BatchNormalization())
model_mlp_multi_layer.add(Activation('relu'))
model_mlp_multi_layer.add(Dense(n_hidden_2))
model_mlp_multi_layer.add(BatchNormalization())
model_mlp_multi_layer.add(Activation('relu'))
model_mlp_multi_layer.add(Dense(n_hidden_3))
model_mlp_multi_layer.add(BatchNormalization())
model_mlp_multi_layer.add(Activation('relu'))
model_mlp_multi_layer.add(Dense(n_classes, activation='softmax'))

# create the loss and optimiser, use 'categorical_crossentropy' in loss
learning_rate = 0.01
model_mlp_multi_layer.compile(loss="categorical_crossentropy", optimizer=optimizers.Adam(lr=learning_rate),metrics=["accuracy"])

# Run optimisation algorithm
n_epochs = 100
batch_size = 64

print('Training')
model_mlp_multi_layer.fit(X_train_scale, Y_train, epochs=n_epochs,batch_size=batch_size) # TO FILL IN

print('Testing')
model_mlp_multi_layer.evaluate(X_test_scale,  Y_test, verbose=2) # TO FILL IN<jupyter_output>Training
Epoch 1/100
53/53 [==============================] - 2s 6ms/step - loss: 0.8110 - accuracy: 0.6513
Epoch 2/100
53/53 [==============================] - 0s 6ms/step - loss: 0.4319 - accuracy: 0.8000
Epoch 3/100
53/53 [==============================] - 0s 6ms/step - loss: 0.3631 - accuracy: 0.8413
Epoch 4/100
53/53 [==============================] - 0s 6ms/step - loss: 0.3584 - accuracy: 0.8405
Epoch 5/100
53/53 [==============================] - 0s 6ms/step - loss: 0.3495 - accuracy: 0.8472
Epoch 6/100
53/53 [==============================] - 0s 6ms/step - loss: 0.3069 - accuracy: 0.8657
Epoch 7/100
53/53 [==============================] - 0s 6ms/step - loss: 0.2820 - accuracy: 0.8664
Epoch 8/100
53/53 [==============================] - 0s 6ms/step - loss: 0.2789 - accuracy: 0.8777
Epoch 9/100
53/53 [==============================] - 0s 6ms/step - loss: 0.2571 - accuracy: 0.8845
Epoch 10/100
53/53 [==============================] - 0s 6ms/step - loss: 0.2281 - accuracy: 0.9015
[...]
{'fraction_non_alphanumeric': 0.13967226949855827, 'fraction_numerical': 0.09290386103101483, 'mean_word_length': 6.935267857142857, 'pattern_counts': {'":': 0, '<': 54, '<?xml version=': 0, '>': 54, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '120267', 'license': 'MIT', 'n_tokens_mistral': 6466, 'n_tokens_neox': 5501, 'n_words': 1137, 'path': 'SD210_activity_recognition_Gr5.ipynb', 'repo_name': 'Mael-zys/SD210-Project', 'size': '14219'}