<jupyter_start><jupyter_text>Uso Avançado de RNN> **3 Técnicas** para melhorar o desempenho e o poder de generalização das RNN:- *Recurrent dropout*: para cobater overfitting- *Stacking recurrent layers*: para aumentar o poder representacional da rede- *Bidirectional recurrent layers*: apresentar a mesma informação para redes diferentes, aumentando a acurácia e mitigando problemas. Case: Previsão de temperaturaDiferente dos casos anteriores, em que analisamos sequência de texto, iremos trabalhar agora com séries temporais, outra aplicação de deep learning para sequências.<jupyter_code>from google.colab import drive
drive.mount('/content/drive')
import os
data_dir = '/content/drive/My Drive/Deep_Learning/'
fname = os.path.join(data_dir, 'jena_climate_2009_2016.csv')
# Abrindo o arquivo
f = open(fname)
data = f.read()
f.close()

lines = data.split('\n')
header = lines[0].split(',')
lines = lines[1:]

print(header)
print(len(lines))<jupyter_output>['"Date Time"', '"p (mbar)"', '"T (degC)"', '"Tpot (K)"', '"Tdew (degC)"', '"rh (%)"', '"VPmax (mbar)"', '"VPact (mbar)"', '"VPdef (mbar)"', '"sh (g/kg)"', '"H2OC (mmol/mol)"', '"rho (g/m**3)"', '"wv (m/s)"', '"max. wv (m/s)"', '"wd (deg)"']
420551<jupyter_text>Precisamos converter o arquivo em um array do Numpy<jupyter_code>import numpy as np

#criando uma matriz de zeros
float_data = np.zeros((len(lines), len(header) - 1))

#preenchendo com os dados do dataset
for i, line in enumerate(lines):
    values = [float(x) for x in line.split(',')[1:]]
    float_data[i, :] = values<jupyter_output><empty_output><jupyter_text>Plotando a temperatura ao longo do tempo (*timeseries*)<jupyter_code>from matplotlib import pyplot as plt

temp = float_data[:, 1]
plt.plot(range(len(temp)), temp)<jupyter_output><empty_output><jupyter_text>Visualizando apenas os 10 primeiros dias:<jupyter_code>plt.plot(range(1440), temp[:1440])<jupyter_output><empty_output><jupyter_text>- As temperaturas possuem boa periodicidade quando olhamos anualmente.- Entretanto, a coisa fica mais caótica quando olhamos diariamente, sendo mais complexo prever a temperatura de um dia do que de um mês. Preparação dos dados> Settings:1. lookback = 720—Observations will go back 5 days.2. steps = 6—Observations will be sampled at one data point per hour.3. delay = 144—Targets will be 24 hours in the future.Precisamos:- normalizar os dados: $\frac{x-\bar{x}}{\sigma_x}$- fazer batches dos dados<jupyter_code># Preparando os dados
# usando apenas os 200000 primeiros dados para a média e o desvio padrão

mean = float_data[:200000].mean(axis=0)
float_data -= mean

std = float_data[:200000].std(axis=0)
float_data /= std<jupyter_output><empty_output><jupyter_text>Precisamos fazer uma *feature engineering* e obter as seguintes variáveis:- **data** —The original array of floating-point data, which you normalized in listing 6.32.- **lookback** —How many timesteps back the input data should go.- **delay** —How many timesteps in the future the target should be.- **min_index** and **max_index** —Indices in the data array that delimit which timestepsto draw from. This is useful for keeping a segment of the data for validationand another for testing.- **shuffle** —Whether to shuffle the samples or draw them in chronological order.- **batch_size** —The number of samples per batch.- **step** —The period, in timesteps, at which you sample data. You’ll set it to 6 inorder to draw one data point every hour.<jupyter_code>def generator(data, lookback, delay, min_index, max_index,
              shuffle=False, batch_size=128, step=6):
    if max_index is None:
        max_index = len(data) - delay - 1
    i = min_index + lookback
    while 1:
        if shuffle:
            rows = np.random.randint(
                min_index + lookback, max_index, size=batch_size)
        else:
            if i + batch_size >= max_index:
                i = min_index + lookback
            rows = np.arange(i, min(i + batch_size, max_index))
            i += len(rows)

        samples = np.zeros((len(rows),
                           lookback // step,
                           data.shape[-1]))
        targets = np.zeros((len(rows),))
        for j, row in enumerate(rows):
            indices = range(rows[j] - lookback, rows[j], step)
            samples[j] = data[indices]
            targets[j] = data[rows[j] + delay][1]
        yield samples, targets<jupyter_output><empty_output><jupyter_text>Com a função *generator* iremos selecionar as amostras de treino, validação e teste. As seleções de validação e teste serão *out of time*, ou seja, em períodos posteriores.<jupyter_code>#setting das constantes
lookback = 1440
step = 6
delay = 144
batch_size = 128

train_gen = generator(float_data,
                      lookback=lookback,
                      delay=delay,
                      min_index=0,
                      max_index=200000,
                      shuffle=True,
                      step=step, 
                      batch_size=batch_size)
val_gen = generator(float_data,
                    lookback=lookback,
                    delay=delay,
                    min_index=200001,
                    max_index=300000,
                    step=step,
                    batch_size=batch_size)
test_gen = generator(float_data,
                     lookback=lookback,
                     delay=delay,
                     min_index=300001,
                     max_index=None,
                     step=step,
                     batch_size=batch_size)

# Garantindo que a amostra de validação seja feita no batch subsequente à amostra de treino
val_steps = (300000 - 200001 - lookback) // batch_size

# Garantindo que a amostra de teste seja feita no batch subsequente à amostra de validação
test_steps = (len(float_data) - 300001 - lookback) // batch_size<jupyter_output><empty_output><jupyter_text>Baseline e Sanity CheckÉ sempre importante termos uma baseline para compararmos os resultados e fazermos um *sanity check*.No caso da previsão de tempo, podemos assumir que:- as temperaturas são contínuas.- a temperatura em 24h será a mesma que agora.Para avaliar esse baseline, usamos o erro absoluto médio (MAE).<jupyter_code>def evaluate_naive_method():
    batch_maes = []
    for step in range(val_steps):
        samples, targets = next(val_gen)
        preds = samples[:, -1, 1]
        mae = np.mean(np.abs(preds - targets))
        batch_maes.append(mae)
    print(np.mean(batch_maes))
    
evaluate_naive_method()<jupyter_output>0.2897359729905486<jupyter_text>> Considerando que fizemos a normalização dos dados, temos:$0.29 \times \sigma_{Temperature} = 2.57^o C$Temos que fazer uma previsão cujo MAE seja menor!<jupyter_code>from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

model = Sequential()
model.add(layers.Flatten(input_shape=(lookback // step, float_data.shape[-1])))
model.add(layers.Dense(32, activation='relu'))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen,
                              steps_per_epoch=500,
                              epochs=20,
                              validation_data=val_gen,
                              validation_steps=val_steps)
evaluate_naive_method()
# Visualização dos resultados
import matplotlib.pyplot as plt

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(loss))

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>- Nesse treino, utilizamos a sequência de dados como um todo, sem considerar como cada medida T em um tempo t influência o a temperatura T+1.- Da mesma maneira que com a sequência de palavras no review, a ordem e a causalidade importante!> Uilizaremos uma **GRU** (***gated recurrent unit***, "unidade recorrente fechada"), que são similares a LSTM, mas são mais baratas computacionalmente.  Baseline recorrente - uma primeira tentativa<jupyter_code>from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

model = Sequential()
model.add(layers.GRU(32, input_shape=(None, float_data.shape[-1])))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen,
                              steps_per_epoch=500,
                              epochs=20,
                              validation_data=val_gen,
                              validation_steps=val_steps)
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(loss))

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>Dropout para evitar overfittingPara evitar o **overfitting** usar o dropout é uma boa opção, uma vez que ele 'quebra' algumas correlações entre os dados de treinamento, evitando o superajuste.> Fazer isso em redes recorrentes não é trivial, pois necessitamos das informações prévias para o treinamento recorrente.**<NAME>**: a máscara de dropout deve ser *a mesma em todos os timesteps*. Uma máscara de dropout constante deve ser aplicada às ativações internas da camada recorrente, numa espécie de "***dropout recorrente***".No Keras, cada camada recorrente pode ter dois argumentos de dropout:- dropout, um valor decimal que diz a taxa de dropout para as unidades de input da camada- recurrent_dropout, que especifica a taxa de dropout das unidades recorrentes.<jupyter_code>from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

model = Sequential()
model.add(layers.GRU(32,
                     dropout=0.2,
                     recurrent_dropout=0.2,
                     input_shape=(None, float_data.shape[-1])))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen,
                              steps_per_epoch=500,
                              epochs=40,
                              validation_data=val_gen,
                              validation_steps=val_steps)
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(loss))

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>Empilhando Camadas Recorrentesbla bla<jupyter_code>from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

model = Sequential()
model.add(layers.GRU(32,
                     dropout=0.1,
                     recurrent_dropout=0.5,
                     return_sequences=True,
                     input_shape=(None, float_data.shape[-1])))
model.add(layers.GRU(64, activation='relu',
                     dropout=0.1, 
                     recurrent_dropout=0.5))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen,
                              steps_per_epoch=500,
                              epochs=40,
                              validation_data=val_gen,
                              validation_steps=val_steps)
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(loss))

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>Camadas Bidirecionaisbla bla bla<jupyter_code>def reverse_order_generator(data, lookback, delay, min_index, max_index,
                            shuffle=False, batch_size=128, step=6):
    if max_index is None:
        max_index = len(data) - delay - 1
    i = min_index + lookback
    while 1:
        if shuffle:
            rows = np.random.randint(
                min_index + lookback, max_index, size=batch_size)
        else:
            if i + batch_size >= max_index:
                i = min_index + lookback
            rows = np.arange(i, min(i + batch_size, max_index))
            i += len(rows)

        samples = np.zeros((len(rows),
                           lookback // step,
                           data.shape[-1]))
        targets = np.zeros((len(rows),))
        for j, row in enumerate(rows):
            indices = range(rows[j] - lookback, rows[j], step)
            samples[j] = data[indices]
            targets[j] = data[rows[j] + delay][1]
        yield samples[:, ::-1, :], targets
        
train_gen_reverse = reverse_order_generator(
    float_data,
    lookback=lookback,
    delay=delay,
    min_index=0,
    max_index=200000,
    shuffle=True,
    step=step, 
    batch_size=batch_size)
val_gen_reverse = reverse_order_generator(
    float_data,
    lookback=lookback,
    delay=delay,
    min_index=200001,
    max_index=300000,
    step=step,
    batch_size=batch_size)
model = Sequential()
model.add(layers.GRU(32, input_shape=(None, float_data.shape[-1])))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen_reverse,
                              steps_per_epoch=500,
                              epochs=20,
                              validation_data=val_gen_reverse,
                              validation_steps=val_steps)
loss = history.history['loss']
val_loss = history.history['val_loss']

epochs = range(len(loss))

plt.figure()

plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()

plt.show()<jupyter_output><empty_output><jupyter_text>bla bla bla<jupyter_code>from keras.datasets import imdb
from keras.preprocessing import sequence
from keras import layers
from keras.models import Sequential

# Number of words to consider as features
max_features = 10000
# Cut texts after this number of words (among top max_features most common words)
maxlen = 500

# Load data
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)

# Reverse sequences
x_train = [x[::-1] for x in x_train]
x_test = [x[::-1] for x in x_test]

# Pad sequences
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)

model = Sequential()
model.add(layers.Embedding(max_features, 128))
model.add(layers.LSTM(32))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['acc'])
history = model.fit(x_train, y_train,
                    epochs=10,
                    batch_size=128,
                    validation_split=0.2)<jupyter_output><empty_output><jupyter_text>blasmsdklfsdkjmfkdsjfndzkndfmzdx<jupyter_code>from keras import backend as K
K.clear_session()
model = Sequential()
model.add(layers.Embedding(max_features, 32))
model.add(layers.Bidirectional(layers.LSTM(32)))
model.add(layers.Dense(1, activation='sigmoid'))

model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])
history = model.fit(x_train, y_train, epochs=10, batch_size=128, validation_split=0.2)<jupyter_output><empty_output><jupyter_text>ndsklfdszdskjfjsd,zkjdjsfnds<jupyter_code>from keras.models import Sequential
from keras import layers
from keras.optimizers import RMSprop

model = Sequential()
model.add(layers.Bidirectional(
    layers.GRU(32), input_shape=(None, float_data.shape[-1])))
model.add(layers.Dense(1))

model.compile(optimizer=RMSprop(), loss='mae')
history = model.fit_generator(train_gen,
                              steps_per_epoch=500,
                              epochs=40,
                              validation_data=val_gen,
                              validation_steps=val_steps)<jupyter_output><empty_output><jupyter_text>mais blasescritose descritos<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10791676926865304, 'fraction_numerical': 0.01785274562915538, 'mean_word_length': 3.239300626304802, 'pattern_counts': {'":': 0, '<': 68, '<?xml version=': 0, '>': 75, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '443331', 'license': 'MIT', 'n_tokens_mistral': 5460, 'n_tokens_neox': 5152, 'n_words': 1317, 'path': 'assets/uploads/chollet_6_3.ipynb', 'repo_name': 'iagml/iagml.github.io', 'size': '16247'}