<jupyter_start><jupyter_text>Learning Rate Scheduler -----  0. 기존 작업<jupyter_code>import random
import numpy as np
import os
import tensorflow as tf

def seed_everything(seed: int = 42):
    random.seed(seed)
    np.random.seed(seed)
    os.environ["PYTHONHASHSEED"] = str(seed)
    tf.random.set_seed(seed)
    
seed_everything()
from tensorflow.keras.datasets import fashion_mnist

(x, y), (x_test, y_test) = fashion_mnist.load_data()


from tensorflow.keras.utils import to_categorical

x = x.astype('float32') 
x_test = x_test.astype('float32')

x /= 255
x_test /= 255

y = to_categorical(y, 10)
y_test = to_categorical(y_test, 10)


from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten
from tensorflow.keras.optimizers import SGD<jupyter_output><empty_output><jupyter_text>-----<jupyter_code>lr = 0.01
momentum = 0.9<jupyter_output><empty_output><jupyter_text>---<jupyter_code>def create_model(learning_rate=lr, momentum = momentum):
    model = Sequential()
    model.add(Flatten(input_shape=(28, 28)))
    model.add(Dense(units=32, activation='relu'))
    model.add(Dense(64, 'relu'))
    model.add(Dense(128, 'relu'))
    model.add(Dense(units=10, activation='softmax'))
    
    sgd = SGD(learning_rate=learning_rate, momentum=momentum, nesterov=False)
    model.compile(optimizer=sgd, 
                  loss='categorical_crossentropy',
                 metrics=['accuracy'])

    return model

model = create_model()
model.summary()<jupyter_output>Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
flatten_4 (Flatten)          (None, 784)               0         
_________________________________________________________________
dense_16 (Dense)             (None, 32)                25120     
_________________________________________________________________
dense_17 (Dense)             (None, 64)                2112      
_________________________________________________________________
dense_18 (Dense)             (None, 128)               8320      
_________________________________________________________________
dense_19 (Dense)             (None, 10)                1290      
=================================================================
Total params: 36,842
Trainable params: 36,842
Non-trainable params: 0
__________________________________________________[...]<jupyter_text>----  1. schedule 함수 만들기epoch의 값에 따라 learning_rate가 변하는 함수를 만들면 됩니다.<jupyter_code>def my_schedule(epoch, learning_rate=lr):
    if epoch < 5:
        return lr
    else:
        return float(lr * tf.math.exp(0.1 * (5- epoch)))
from tensorflow.keras.callbacks import LearningRateScheduler
lr_schedule_custom = LearningRateScheduler(my_schedule, verbose=1)
model = create_model()

model.fit(x, y,  epochs = 20, validation_split = 1/6, callbacks = [lr_schedule_custom], batch_size=512, shuffle=False)<jupyter_output>Epoch 1/20

Epoch 00001: LearningRateScheduler reducing learning rate to 0.01.
98/98 [==============================] - 2s 12ms/step - loss: 1.8717 - accuracy: 0.2971 - val_loss: 0.7189 - val_accuracy: 0.7420
Epoch 2/20

Epoch 00002: LearningRateScheduler reducing learning rate to 0.01.
98/98 [==============================] - 1s 7ms/step - loss: 0.6734 - accuracy: 0.7600 - val_loss: 0.5721 - val_accuracy: 0.7964
Epoch 3/20

Epoch 00003: LearningRateScheduler reducing learning rate to 0.01.
98/98 [==============================] - 1s 6ms/step - loss: 0.5441 - accuracy: 0.8099 - val_loss: 0.5153 - val_accuracy: 0.8174
Epoch 4/20

Epoch 00004: LearningRateScheduler reducing learning rate to 0.01.
98/98 [==============================] - 1s 7ms/step - loss: 0.4881 - accuracy: 0.8296 - val_loss: 0.4873 - val_accuracy: 0.8266
Epoch 5/20

Epoch 00005: LearningRateScheduler reducing learning rate to 0.01.
98/98 [==============================] - 1s 7ms/step - loss: 0.4589 - accuracy: 0.8405 -[...]<jupyter_text>2. tensorflow의 scheduler 사용https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/schedules **def** decayed_learning_rate(step):  > return initial_learning_rate * decay_rate ^ (step / decay_steps)    `decay_steps` 마다 `decay_rate`의 비율로 감소<jupyter_code>from tensorflow.keras.optimizers.schedules import ExponentialDecay
lr_scheduler_exp = ExponentialDecay(lr, decay_steps=10000, decay_rate=0.96, staircase=False, name=None)
def exp_model(learning_rate=lr, momentum = momentum):
    model = Sequential()
    model.add(Flatten(input_shape=(28, 28)))
    model.add(Dense(units=32, activation='relu'))
    model.add(Dense(64, 'relu'))
    model.add(Dense(128, 'relu'))
    model.add(Dense(units=10, activation='softmax'))

    return model
model = exp_model()
sgd = SGD(learning_rate=lr_scheduler_exp, momentum=momentum, nesterov=False)
model.compile(optimizer=sgd, 
                  loss='categorical_crossentropy',
                 metrics=['accuracy'])


model.fit(x, y,  epochs = 20, validation_split = 1/6, batch_size=512)
model.optimizer.lr(100000) # 0.96^10<jupyter_output><empty_output><jupyter_text>3. ReduceLRonPlateau - monitor에 'val_loss'를 입력하면 val_loss가 더이상 감소되지 않을 경우 ReduceLROnPlateau을 적용합니다.- factor은 Learning rate를 얼마나 감소시킬 지 정하는 인자값입니다.새로운 learning rate는 기존 learning rate * factor입니다.- patience는 patience는 3이고, 30에폭에 정확도가 99%였을 때,만약 31번째에 정확도 98%, 32번째에 98.5%, 33번째에 98%라면 모델의 개선이 (patience=3)동안 개선이 없었기에,  ReduceLROnPlateau 콜백함수를 실행합니다.- verbose는 화면에 적용되었다고 나타냅니다.<jupyter_code>from tensorflow.keras.callbacks import ReduceLROnPlateau
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=2, verbose=1)
model = create_model()

model.fit(x, y,  epochs = 30, validation_split = 1/6, callbacks = [reduce_lr], batch_size=512, shuffle=False)<jupyter_output>Epoch 1/30
98/98 [==============================] - 1s 7ms/step - loss: 1.7674 - accuracy: 0.3914 - val_loss: 0.7005 - val_accuracy: 0.7457
Epoch 2/30
98/98 [==============================] - 1s 6ms/step - loss: 0.6573 - accuracy: 0.7665 - val_loss: 0.5628 - val_accuracy: 0.8001
Epoch 3/30
98/98 [==============================] - 1s 6ms/step - loss: 0.5363 - accuracy: 0.8126 - val_loss: 0.5209 - val_accuracy: 0.8119
Epoch 4/30
98/98 [==============================] - 1s 6ms/step - loss: 0.4878 - accuracy: 0.8296 - val_loss: 0.4969 - val_accuracy: 0.8195
Epoch 5/30
98/98 [==============================] - 1s 6ms/step - loss: 0.4600 - accuracy: 0.8400 - val_loss: 0.4751 - val_accuracy: 0.8280
Epoch 6/30
98/98 [==============================] - 1s 7ms/step - loss: 0.4397 - accuracy: 0.8466 - val_loss: 0.4625 - val_accuracy: 0.8350
Epoch 7/30
98/98 [==============================] - 1s 6ms/step - loss: 0.4239 - accuracy: 0.8515 - val_loss: 0.4564 - val_accuracy: 0.8367
Epoch 8/30
98/98 [==[...]
{'fraction_non_alphanumeric': 0.18157933148445482, 'fraction_numerical': 0.08115603561523865, 'mean_word_length': 4.254601226993865, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '373045', 'license': 'MIT', 'n_tokens_mistral': 2805, 'n_tokens_neox': 2468, 'n_words': 506, 'path': 'Learning_Rate_Decay.ipynb', 'repo_name': 'jiwonlee-0218/colab', 'size': '6851'}