<jupyter_start><jupyter_text>Resampling column description* **P_mean**: média da noite toda (dropped before classification)* **P_var**: variancia da noite toda* **P_amplitude**: max - min das dos dados agrupados por 1h (>0 para oscilações de baixa frequencia)* **Atividade_noturna**: (média das 00:00 as 05:00)/(média das 7:30 as 11:30)* **Fator de carga**: max/medio (a partir das médias de 5 minutos)<jupyter_code>hourly = result0.resample('H').mean()

resampled0 = result0.between_time('00:00:00', '05:00:00').resample('B').agg({
    'P':['mean', 'std']
})
resampled0['P_amplitude'] = hourly.between_time('00:00:00', '05:00:00').resample('B').max() - hourly.between_time('00:00:00', '05:00:00').resample('B').min()
resampled0.columns = resampled0.columns.map('_'.join)

resampled0 ['fator_de_carga'] = result0.between_time('00:00:00', '05:00:00').resample('B').max()/result0.between_time('00:00:00', '05:00:00').resample('B').mean()
resampled0 ['atividade_noturna']= hourly.between_time('00:00:00', '05:00:00').resample('B').mean()/hourly.between_time('07:30:00', '11:30:00').resample('D').mean()
resampled0 = resampled0.dropna()

# Transform standant deviation and amplitude to Coeficient of Variation
resampled0 ['P_std'] = resampled0 ['P_std'] / resampled0['P_mean']
resampled0 ['P_amplitude_'] = resampled0 ['P_amplitude_'] / resampled0['P_mean']

# The mean will be useless in the final analysis, drop it
resampled0 = resampled0.drop ('P_mean', axis=1)
resampled0['class']=0
resampled0
hourly = result1.resample('H').mean()

resampled0 = result1.between_time('00:00:00', '05:00:00').resample('B').agg({
    'P':['mean', 'std']
})
resampled0['P_amplitude'] = hourly.between_time('00:00:00', '05:00:00').resample('B').max() - hourly.between_time('00:00:00', '05:00:00').resample('B').min()
resampled0.columns = resampled0.columns.map('_'.join)

resampled0 ['fator_de_carga'] = result1.between_time('00:00:00', '05:00:00').resample('B').max()/result1.between_time('00:00:00', '05:00:00').resample('B').mean()
resampled0 ['atividade_noturna']= hourly.between_time('00:00:00', '05:00:00').resample('B').mean()/hourly.between_time('07:30:00', '11:30:00').resample('B').mean()
resampled0 = resampled0.dropna()

# Transform standant deviation and amplitude to Coeficient of Variation
resampled0 ['P_std'] = resampled0 ['P_std']*100 / resampled0['P_mean']
resampled0 ['P_amplitude_'] = resampled0 ['P_amplitude_']*100 / resampled0['P_mean']

# The mean will be useless in the final analysis, drop it
resampled0 = resampled0.drop ('P_mean', axis=1)
resampled0['class']=0
resampled0
merged = pd.concat([resampled0, resampled1], axis=0)
# minMax normalization 
normalized=(merged-merged.min())/(merged.max()-merged.min())
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(normalized.drop('class',axis=1), 
                                                    normalized['class'], test_size=0.3)

logmodel = LogisticRegression()
logmodel.fit(X_train,y_train)
## Model evaluation
from sklearn.metrics import classification_report, confusion_matrix

predictions = logmodel.predict(X_test)

## Classification report
# Precision = TP/(TP+FP)
# Recall (or sensibility) = TP/(TP+FN) 
print('Classification report:\n', classification_report(y_test,predictions))

conf_mat = confusion_matrix(y_true=y_test, y_pred=predictions)
print('\n\nConfusion matrix:\n', conf_mat)

labels = ['Class 0', 'Class 1']
fig = plt.figure()
ax = fig.add_subplot(111)
cax = ax.matshow(conf_mat, cmap=plt.cm.Blues)
fig.colorbar(cax)
ax.set_xticklabels([''] + labels)
ax.set_yticklabels([''] + labels)
plt.xlabel('Predicted')
plt.ylabel('Expected')
plt.show()<jupyter_output>Classification report:
               precision    recall  f1-score   support

         0.0       1.00      0.75      0.86         4
         1.0       0.89      1.00      0.94         8

   micro avg       0.92      0.92      0.92        12
   macro avg       0.94      0.88      0.90        12
weighted avg       0.93      0.92      0.91        12

Confusion matrix:
 [[3 1]
 [0 8]]
{'fraction_non_alphanumeric': 0.17429193899782136, 'fraction_numerical': 0.07576857903655289, 'mean_word_length': 4.919770773638969, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '541537', 'license': 'MIT', 'n_tokens_mistral': 1692, 'n_tokens_neox': 1537, 'n_words': 285, 'path': 'artificial_intelligence/03 - NightAnomalies (classification)/Logistic regression classifier.ipynb', 'repo_name': 'LeonardoSanBenitez/LorisWeb', 'size': '4131'}