<jupyter_start><jupyter_text>실전 문제 해결 (부족한 데이터셋)  BorderlineSMOTE 설치<jupyter_code>!pip install imblearn
import tensorflow as tf
from imblearn.over_sampling import BorderlineSMOTE
import numpy as np
import random<jupyter_output><empty_output><jupyter_text>하이퍼 파라미터<jupyter_code>EPOCHS = 100<jupyter_output><empty_output><jupyter_text>모델 정의<jupyter_code>class MyModel(tf.keras.Model):
    def __init__(self):
        super(MyModel, self).__init__()
        self.flatten = tf.keras.layers.Flatten()
        self.dense1 = tf.keras.layers.Dense(1024, activation='relu')
        self.dense2 = tf.keras.layers.Dense(1, activation='sigmoid')

    def call(self, x, training=False, mask=None):
        x = self.flatten(x)
        x = self.dense1(x)
        return self.dense2(x)<jupyter_output><empty_output><jupyter_text>데이터셋 준비 (불균형한 데이터셋)<jupyter_code>cifar10 = tf.keras.datasets.cifar10 # 32x32x3

(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

# TODO: 학습 데이터를 Imbalanced small dataset으로 변형하기
x_train_small = list()
y_train_small = list()
for x, y in zip(x_train, y_train):
    if (y == 0 and random.randint(0, 100) < 10) or y == 1:
        x_train_small.append(x[:])
        y_train_small.append(y)

x_test_small = list()
y_test_small = list()
for x, y in zip(x_test, y_test):
    if y == 0 or y == 1:
        x_test_small.append(x[:])
        y_test_small.append(y)

x_train = np.stack(x_train_small, axis=0)
y_train = np.stack(y_train_small, axis=0)

x_test = np.stack(x_test_small, axis=0)
y_test = np.stack(y_test_small, axis=0)

train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train)).shuffle(10000).batch(32).prefetch(2048)
test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32).prefetch(2048)<jupyter_output><empty_output><jupyter_text>Keras API 모델 학습 (불균형한 데이터셋)<jupyter_code>model = MyModel()
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy',
                       tf.keras.metrics.Precision(name='precision'),
                       tf.keras.metrics.Recall(name='recall')])
model.fit(train_ds, validation_data=test_ds, epochs=EPOCHS)<jupyter_output>Epoch 1/100<jupyter_text>데이터셋 준비 (BorderlineSMOTE)<jupyter_code># TODO: BorderlineSMOTE 적용하기
x_train = x_train.reshape((x_train.shape[0], x_train.shape[1] * x_train.shape[2] * x_train.shape[3])).astype(np.float32)
x_test = x_test.reshape((x_test.shape[0], x_test.shape[1] * x_test.shape[2] * x_test.shape[3])).astype(np.float32)

smote = BorderlineSMOTE()
x_train, y_train = smote.fit_resample(x_train, y_train)

train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train)).shuffle(10000).batch(32).prefetch(2048)
test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32).prefetch(2048)<jupyter_output>/home/jeyong/.conda/envs/tf2/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().
  y = column_or_1d(y, warn=True)<jupyter_text>Keras API 모델 학습 (BorderlineSMOTE)<jupyter_code>model = MyModel()
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy',
                       tf.keras.metrics.Precision(name='precision'),
                       tf.keras.metrics.Recall(name='recall')])
model.fit(train_ds, validation_data=test_ds, epochs=EPOCHS)<jupyter_output>Epoch 1/100
313/313 [==============================] - 1s 5ms/step - loss: 0.5627 - accuracy: 0.6914 - precision: 0.7013 - recall: 0.6732 - val_loss: 0.4311 - val_accuracy: 0.8125 - val_precision: 0.8743 - val_recall: 0.7300
Epoch 2/100
313/313 [==============================] - 1s 3ms/step - loss: 0.3694 - accuracy: 0.8338 - precision: 0.8558 - recall: 0.8048 - val_loss: 0.4201 - val_accuracy: 0.8195 - val_precision: 0.8374 - val_recall: 0.7930
Epoch 3/100
313/313 [==============================] - 1s 3ms/step - loss: 0.3111 - accuracy: 0.8660 - precision: 0.8933 - recall: 0.8322 - val_loss: 0.4148 - val_accuracy: 0.8210 - val_precision: 0.8282 - val_recall: 0.8100
Epoch 4/100
313/313 [==============================] - 1s 3ms/step - loss: 0.2869 - accuracy: 0.8781 - precision: 0.9018 - recall: 0.8495 - val_loss: 0.4350 - val_accuracy: 0.8285 - val_precision: 0.8033 - val_recall: 0.8700
Epoch 5/100
313/313 [==============================] - 1s 3ms/step - loss: 0.2345 - accuracy: 0.9062[...]
{'fraction_non_alphanumeric': 0.17040159751497672, 'fraction_numerical': 0.07388506767250944, 'mean_word_length': 4.278688524590164, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '406289', 'license': 'MIT', 'n_tokens_mistral': 2060, 'n_tokens_neox': 1872, 'n_words': 305, 'path': 'RNN_TF20/13. Deep Learning Heuristic with Imbalance Dataset-Antonio.ipynb', 'repo_name': '46gom/DLND_Exercises', 'size': '4507'}