<jupyter_start><jupyter_text>Parte 4O objetivo dessa parte seria focado na reducao de parametros para a criacao de um modelo adequado, identificandoquais metricas mais se correlacionam ao numero de quadros na saida. 1. Construct a training set and a test set from the trace as above.2. Method 1: Build all subsets of the feature set X that contain either one or two features (i.e., device statistics). Compute the models for each of these sets for linear regression over the training set. Plot a histogram of the error values (NMAE) of all the models for the test set. Identify the feature set that produces the model with the smallest error and give the device statistic(s) in this set.3. Method 2: Linear univariate feature selection. Take each feature of $X$ and compute the sample correlation of the feature with the corresponding $Y$ value over the training set. For observations $x_i, y_i$, the sample correlation is computed as $\frac{1}{m}\sum_{i=1}^m\frac{(x_i - \bar{x})(y_i - \bar{y}}{(\sigma_X * \sigma_Y)}$ whereby $\bar{x}$ and $\bar{y}$ are sample means and $m$ is the size of the training set; $\sigma_X$ is the standard deviation $\sqrt{(\frac{1}{m}\sum_{i=1}^m(x_i - \bar{x})^2)}$ and likewise for $\sigma_Y$. The correlation values fall into the interval [−1, +1]. Rank the features according to the square of the correlation values; the top feature has the highest value. Build nine feature sets composed of the top $k$ features, $k = 1..9$. Compute the model for each of these nine sets for linear regression over the training set and compute the error (NMAE) of these models over the test set. Produce a plot that shows the error value in function of the set $k$.4. Describe your observations and conclusions.<jupyter_code>from itertools import chain, combinations
import pandas as pd
import matplotlib.pyplot as pp

class Subset:
    def __init__(self, x_df, y_df, size, columns):
        self.columns = columns
        self.data = x_df[columns]
        x_train, x_test, y_train, y_test = \
            self.data[:size], self.data[size:], y_df[:size], y_df[size:]
        self.x_train = x_train
        self.x_test = x_test
        self.y_train = y_train
        self.y_test = y_test
        
    def data_width(self):
        return len(self.columns)

def superset(values):
    return map(lambda t: list(t), chain.from_iterable(combinations(values, r) for r in range(len(values) + 1)))
    
def create_subsets(x, y):
    subset_columns = list(filter(lambda t: len(t) > 0, superset(x.columns)))
    return [Subset(x, y, 2520, columns) for columns in list(subset_columns)]


x_data = pd.read_csv('./data/X.csv')
y_output = pd.read_csv('./data/Y.csv')

subsets = create_subsets(x_data, y_output)
print(f"Criado {len(subsets)} sub-sets de dados")
from sklearn import linear_model

def absolute_errors(expected, found):
    return [abs(y1 - y2) for (y1, y2) in zip(expected, found)]

def mean_errors(expected, found):
    return sum(absolute_errors(expected, found)) / len(expected)

def normalized_mean_absolute_error(expected, found):
    return mean_errors(expected, found) / expected.mean()

class TrainedModel:
    def __init__(self, data):
        self.data = data
        self.model = linear_model.LinearRegression()
        self.predicted = []
        
    def train(self):
        self.model.fit(self.data.x_train, self.data.y_train['DispFrames'])
        
    def predict(self):
        self.predicted = self.model.predict(self.data.x_test)
        
    def calculate_nmae(self):
        return normalized_mean_absolute_error(self.data.y_test['DispFrames'], self.predicted)
    
filtered_subsets = list(filter(lambda subset: subset.data_width() <= 2, subsets))

def create_and_train_model(data):
    model = TrainedModel(data)
    model.train()
    model.predict()
    return model, model.calculate_nmae()

models = [create_and_train_model(subset) for subset in filtered_subsets]

minimum_model = min(models, key = lambda t: t[1])

print(f"O modelo com as features {minimum_model[0].data.columns} possui NMAE: {minimum_model[1]}")

gathered_data = [[nmae] for (_, nmae) in models]

pd.DataFrame(gathered_data, columns=['NMAE']).plot(kind='hist', legend=True)
import numpy as np

def get_sample_correlation(x_data, y_data):
    x_mean = x_data.mean()
    y_mean = y_data.mean()
    x_std = np.std(x_data)
    y_std = np.std(y_data)

    correlation = sum([((x_data[idx] - x_mean) * (y_data[idx] - y_mean)) / (x_std * y_std) for idx, _ in enumerate(x_data)]) / x_data.size
    return correlation**2

up_to = 2520
x_train, x_test, y_train, y_test = \
    x_data[:up_to], x_data[up_to:], y_output[:up_to], y_output[up_to:]

x_column_names = ['TimeStamp', 
                'all_..idle', 
                'X..memused',
                'proc.s',
                'cswch.s',
                'file.nr',
                'sum_intr.s',
                'ldavg.1',
                'tcpsck',
                'pgfree.s']

results = []
for name in x_column_names:
    results.append(tuple((name, get_sample_correlation(x_train[name].to_numpy(), y_train['DispFrames'].to_numpy()))))

topNine = sorted(results, key=lambda tup: tup[1], reverse=True)[:9]
print(topNine)

nmaeList = []
for featureName, score in topNine:
    model = linear_model.LinearRegression()
    model.fit(x_train[[featureName]], y_train['DispFrames'])
    y_pred = model.predict(x_test[[featureName]])
    nmae = normalized_mean_absolute_error(y_test['DispFrames'], y_pred)
    nmaeList.append(tuple((featureName, nmae)))
    print("\nFeature name:", featureName)
    print("\nScore:", score)
    print("NMAE error:", nmae)
    
topNineNmae = sorted(nmaeList, key=lambda tup: tup[1], reverse=False)
pd.DataFrame(topNineNmae, columns =['Feature', 'NMAE']).plot(kind='bar', legend=True, x = 'Feature')<jupyter_output>[('ldavg.1', 0.6605156933263323), ('file.nr', 0.5431396904578119), ('tcpsck', 0.5414481371192515), ('cswch.s', 0.46536594437966994), ('TimeStamp', 0.23320617730219892), ('X..memused', 0.18179581598782335), ('all_..idle', 0.1559551275906424), ('sum_intr.s', 0.08398481584153435), ('proc.s', 0.02632612296338617)]

Feature name: ldavg.1

Score: 0.6605156933263323
NMAE error: 0.12273037543079203

Feature name: file.nr

Score: 0.5431396904578119
NMAE error: 0.10630719327916613

Feature name: tcpsck

Score: 0.5414481371192515
NMAE error: 0.08940690660169943

Feature name: cswch.s

Score: 0.46536594437966994
NMAE error: 0.17993333009828547

Feature name: TimeStamp

Score: 0.23320617730219892
NMAE error: 0.11388149027109429

Feature name: X..memused

Score: 0.18179581598782335
NMAE error: 0.18376875247916757

Feature name: all_..idle

Score: 0.1559551275906424
NMAE error: 0.19474520596136627

Feature name: sum_intr.s

Score: 0.08398481584153435
NMAE error: 0.17499600939990528

Feature name: pro[...]
{'fraction_non_alphanumeric': 0.12161764705882352, 'fraction_numerical': 0.07102941176470588, 'mean_word_length': 3.997060984570169, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '63287', 'license': 'Apache-2.0', 'n_tokens_mistral': 2671, 'n_tokens_neox': 2344, 'n_words': 641, 'path': 'task4.ipynb', 'repo_name': 'jabolina/pgc308a', 'size': '6800'}