<jupyter_start><jupyter_text>Preparing data<jupyter_code># Comment out if running in Colab
# work_dir = os.getcwd()
data_dir ="../../for_students/data_v2"
train_dir = os.path.join(data_dir, 'training')
train_str = os.path.join(train_dir, 'training_{}.csv')
# Uncomment below if running in Colab
# work_dir = os.getcwd()
# train_str = os.path.join(work_dir, 'training_{}.csv')<jupyter_output><empty_output><jupyter_text>Testing on a single datasetUse `roll_time_series(...., max_timeshift=100, min_timeshift=10)`https://tsfresh.readthedocs.io/en/v0.7.1/text/rolling.html<jupyter_code>from tsfresh.utilities.dataframe_functions import roll_time_series, make_forecasting_frame
test_2 = pd.read_csv(train_str.format(2))

# tsfresh needs (id, time info (for sorting), and features) to extract features
test_2["id"] = 1 # This number does not matter in our case. Keep them constant across dataset

test_2_rolled = roll_time_series(test_2, column_id="id", column_sort="timestamp", max_timeshift=100, min_timeshift=10)

print(test_2_rolled.shape)
print(test_2_rolled.head(15))<jupyter_output><empty_output><jupyter_text>Testing on all datasets (extracting toooooooo slow)<jupyter_code># Extracting features from ALL raw dataset do not work! 

# train_df = pd.DataFrame()
# for i in [1,2,3,4,5,100]:
#     train_df_i = pd.read_csv(train_str.format(i))
#     # each training dataset is assigned an id
#     train_df_i["id"] = i
# #     print("dataset {}: {}".format(i, train_df_i.shape))
#     train_df = train_df.append(train_df_i)
# print(train_df.shape)
# tsfresh needs (id, time info (for sorting), and features) to extract features
# train_df = train_df.drop("request_count", axis=1)
# train_df
# y = train_df.pop("anomaly_label")
# X_raw = train_df # preparing dataframe with (id, time, feature(s)) for tsfresh
# print(X_raw)
# print(y)<jupyter_output>timestamp  kpi_value   id
0                2014-04-01 00:00:00  18.622185    1
1                2014-04-01 00:05:00   8.163417    1
2                2014-04-01 00:10:00  13.292383    1
3                2014-04-01 00:15:00  11.667046    1
4                2014-04-01 00:20:00  12.940358    1
...                              ...        ...  ...
20154  2020-08-28T01:56:00.000+02:00   0.998149  100
20155  2020-08-28T01:57:00.000+02:00   0.998340  100
20156  2020-08-28T01:58:00.000+02:00   0.998364  100
20157  2020-08-28T01:59:00.000+02:00   0.998428  100
20158  2020-08-28T02:00:00.000+02:00   0.997407  100

[54337 rows x 3 columns]
0        0
1        0
2        0
3        0
4        0
        ..
20154    0
20155    0
20156    0
20157    0
20158    0
Name: anomaly_label, Length: 54337, dtype: int64<jupyter_text>Feature extraction<jupyter_code># Using rolling time-series as input with windows ranging from 10-100, see `roll_time_series()`
test_2_x = test_2_rolled.drop("anomaly_label", axis=1)
print(test_2_x.shape)
# https://tsfresh.readthedocs.io/en/latest/text/feature_extraction_settings.html#for-the-advanced-how-do-i-set-the-parameters-for-all-kind-of-time-series

extraction_settings = ComprehensiveFCParameters()

X_2 = extract_features(test_2_x, column_id="id", column_sort="timestamp", 
                     default_fc_parameters=extraction_settings, impute_function=impute)
extraction_settings = ComprehensiveFCParameters()

# the return of X_1 has the dimension of raw feature (unrolled) - 10 (the smallest window)
print(X_2.shape)
X_1.head() # All features extracted<jupyter_output><empty_output><jupyter_text>Feature selection<jupyter_code># Using unrolled y, drop the first 10 rows (as the first 10 rows of feature collapsed as a feature previously)
test_2_y = test_2["anomaly_label"].loc[10:]
print(test_2_y.head(5))
print(test_2_y.shape)
print(type(test_2_y))
X_filtered = select_features(X_2, test_2_y) # features will be selected based on y
print(X_filtered.shape)
X_filtered.head()<jupyter_output><empty_output><jupyter_text>Alternatively "Extract & Filter" in One Step(*I'm not sure if this works with rolling)<jupyter_code>X_filtered_2 = extract_relevant_features(df, y, column_id='id', column_sort='time',
                                         default_fc_parameters=extraction_settings)
(X_filtered.columns == X_filtered_2.columns).all()<jupyter_output><empty_output><jupyter_text>Evaluation<jupyter_code># Evaluate how these features perform in classifiers such as random forest, DTree, KNN ect.
# https://github.com/blue-yonder/tsfresh/tree/main/notebooks/examples<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11726602635693545, 'fraction_numerical': 0.09314272950636587, 'mean_word_length': 4.237426900584795, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '216738', 'license': 'MIT', 'n_tokens_mistral': 1868, 'n_tokens_neox': 1607, 'n_words': 343, 'path': 'notebooks/tsfresh.ipynb', 'repo_name': 'Pheobe-Sun/anomaly-detection-challenge-2020', 'size': '4477'}