<jupyter_start><jupyter_text>Description of the problem and solutionThe task1 was to predict a person's age from the brain image data: a standard regression problem. The original dataset included 832 features as well as a lot of NaN values and a few outliers. A good preprocessing stage was necessary in order to have a well defined dataset that could be used in our regression model. First step was the imputation of the dataset. Filling each NaN value with the median of each feature column. The use of the median instead of other value (e.g. mean) is justified since a lot of outliers are included in the dataset. (e.g. 1 2 _ 5 20 median: 3 mean: 7). Next step was the feature extraction. By using the "autofeat" library (paper: https://arxiv.org/pdf/1901.07329.pdf), we extracted the 21 most important features. The way the algorithm works is going through a loop of correlation of features with target, select promising features, train Lasso regression model with promising features, filter the good features keeping the ones with non-zero regression weights. We updated the datasets by keeping only the 21 most important features. Finally, we used these updated datasets for the training of our final regression model. A lot of outlier detection techniques were used but we decided to keep the outliers and use a tree-based method for our final model. Tree-methods have been proved to be robust to outliers and we avoid risking excluded important features / points from the dataset. The "ExtraTreesRegressor" model from the "sklearn" package was used and fine tuned based on the R2 score performance in our validation set. The final model had a score >0.6 in the validation sets using cross-validation and in the submission leaderboard of ETH scored 0.6812 while the hard baseline was set to 0.65 by the Advanced Machine Learning Task1 team.  Include all the necessary packages<jupyter_code>!pip install autofeat

from sklearn.metrics import r2_score

try:
  %tensorflow_version 2.x
except Exception:
  pass
import tensorflow as tf

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

from autofeat import FeatureSelector

from sklearn.model_selection import train_test_split
from sklearn.ensemble import ExtraTreesRegressor<jupyter_output>Requirement already satisfied: autofeat in /usr/local/lib/python3.6/dist-packages (0.2.5)
Requirement already satisfied: pint in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.9)
Requirement already satisfied: pandas>=0.24.0 in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.25.2)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.21.3)
Requirement already satisfied: sympy in /usr/local/lib/python3.6/dist-packages (from autofeat) (1.1.1)
Requirement already satisfied: joblib in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.14.0)
Requirement already satisfied: numpy in /tensorflow-2.0.0/python3.6 (from autofeat) (1.17.3)
Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from autofeat) (0.16.0)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.24.0->autofeat) (2018.9)
Requirement already satisfied: python-dateutil>=2.6.1[...]<jupyter_text>Load the data from the CSV files<jupyter_code>column_names_x = ['id']
for i in range(832):
  column_names_x.append('x'+str(i))

raw_dataset_x = pd.read_csv('/content/X_train.csv', names=column_names_x,
                      na_values = "?", comment='\t',
                      sep=",", skipinitialspace=True, skiprows=True)

dataset_x = raw_dataset_x.copy()
dataset_x.tail()
column_names_y = ['id','y']
raw_dataset_y = pd.read_csv('/content/y_train.csv', names=column_names_y,
                      na_values = "?", comment='\t',
                      sep=",", skipinitialspace=True, skiprows=True)

dataset_y = raw_dataset_y.copy()
dataset_y.tail()<jupyter_output><empty_output><jupyter_text>Print the missing values<jupyter_code>missing_values = dataset_x.isna().sum()
print (missing_values)<jupyter_output>id        0
x0       81
x1      103
x2       92
x3       91
       ... 
x827     83
x828     78
x829     98
x830     84
x831     92
Length: 833, dtype: int64<jupyter_text>Split the data into training and test data<jupyter_code># Split using sklearn.model_selection
x_train, x_test, y_train, y_test = train_test_split(dataset_x, dataset_y, test_size=0.2, random_state = 100)
train_stats = x_train.describe()
train_stats.pop("id")
train_stats = train_stats.transpose()
train_stats<jupyter_output><empty_output><jupyter_text>Fill the NaN in the training data set with the median values of each column<jupyter_code>x_train = x_train.fillna(x_train.median())
x_test = x_test.fillna(x_test.median())
missing_values = x_train.isna().sum()
print (missing_values)<jupyter_output>id      0
x0      0
x1      0
x2      0
x3      0
       ..
x827    0
x828    0
x829    0
x830    0
x831    0
Length: 833, dtype: int64<jupyter_text>Remove the unnecessary "id" label<jupyter_code>y_train.pop("id")
y_test.pop("id")<jupyter_output><empty_output><jupyter_text>Feature Extraction using autofeat<jupyter_code>fsel = FeatureSelector(featsel_runs=4,
        max_it=150,
        w_thr=1e-6,
        keep=None,
        n_jobs=1,
        verbose=1)

new_X = fsel.fit_transform(pd.DataFrame(x_train, columns=column_names_x), y_train)
print(new_X.columns) 

df_train = pd.DataFrame(x_train, columns=column_names_x)
df_test = pd.DataFrame(x_test, columns=column_names_x)<jupyter_output>/usr/local/lib/python3.6/dist-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)<jupyter_text>Keep only the necessary features<jupyter_code>dataset_selected_x = x_train.copy()
x_test_selected= x_test.copy()

#accepted=[400, 757, 635, 516, 132, 15, 809, 116, 214,
       #556, 617, 93, 346, 596, 309, 252, 292, 474,
       #593, 614]

accepted=[400, 635, 757, 516, 809, 214, 556, 617, 93,
       346, 596, 255, 309, 252, 292, 738, 537, 593,
       474, 614, 502]

for j in range(832):
        if (j in accepted):
          print (j)
        else:
          del dataset_selected_x['x'+str(j)]
          del x_test_selected['x'+str(j)]
      
train_stats = dataset_selected_x.describe()
train_stats.pop("id")
train_stats = train_stats.transpose()
train_stats<jupyter_output>93
214
252
255
292
309
346
400
474
502
516
537
556
593
596
614
617
635
738
757
809<jupyter_text>Regression model<jupyter_code>rfr = ExtraTreesRegressor(n_jobs=1, max_depth=None, n_estimators=180, random_state=0, min_samples_split=3, max_features=None)

rfr.fit(dataset_selected_x, np.ravel(y_train))

y_pred = rfr.predict(x_test_selected) 

score = r2_score(y_test, y_pred)
print(score)<jupyter_output>0.6037280104368132<jupyter_text>Export the file with our predictions for submission<jupyter_code>#accepted=[400, 757, 635, 516, 132, 15, 809, 116, 214,
       #556, 617, 93, 346, 596, 309, 252, 292, 474,
       #593, 614]

accepted=[400, 635, 757, 516, 809, 214, 556, 617, 93,
       346, 596, 255, 309, 252, 292, 738, 537, 593,
       474, 614, 502]

column_names_x = ['id']
for i in range(832):
  column_names_x.append('x'+str(i))

raw_dataset_x_test = pd.read_csv('/content/X_test.csv', names=column_names_x,
                      na_values = "?", comment='\t',
                      sep=",", skipinitialspace=True, skiprows=True)

dataset_x_test = raw_dataset_x_test.copy()
dataset_x_test.tail()

dataset_x_test = dataset_x_test.fillna(dataset_x_test.median())

dataset_selected_x_test = dataset_x_test.copy()

for j in range(832):
        if (j in accepted):
          print (j)
        else:
          del dataset_selected_x_test['x'+str(j)]

predictions = rfr.predict(dataset_selected_x_test)

index = 0.0
with open('predictions.txt', 'w') as f:
    f.write("%s\n" % "id,y")
    for predict in predictions:
        writing_str = str(index)+','+str(predict.item(0))
        f.write("%s\n" % writing_str)
        index = index + 1<jupyter_output>93
214
252
255
292
309
346
400
474
502
516
537
556
593
596
614
617
635
738
757
809
{'fraction_non_alphanumeric': 0.10097087378640776, 'fraction_numerical': 0.07402912621359223, 'mean_word_length': 4.170012547051443, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 40, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '622998', 'license': 'MIT', 'n_tokens_mistral': 3240, 'n_tokens_neox': 2773, 'n_words': 840, 'path': 'Task 1/Task_1_AML.ipynb', 'repo_name': 'KonstantinosBarmpas/Advanced-Machine-Learning-Projects', 'size': '8240'}