<jupyter_start><jupyter_text>To complete this exercise, you should modify this notebook to work with the Stack Overflow dataset by making the following modifications:At the top of your notebook, update the code that downloads the IMDB dataset with code to download the Stack Overflow dataset that has been prepreared. As the Stack Overflow dataset has a similar directory structure, you will not need to make many modifications.Modify the last layer of your model to read Dense(4), as there are now four output classes.When compiling the model, change the loss to tf.keras.losses.SparseCategoricalCrossentropy. This is the correct loss function to use for a multi-class classification problem, when the labels for each class are integers (in this case, they can be 0, 1, 2, or 3). In addition, change the metrics to metrics=['accuracy'], since this is a multi-class classification problem (tf.metrics.BinaryAccuracy is only used for binary classifiers).When plotting accuracy over time, change binary_accuracy and val_binary_accuracy to accuracy and val_accuracy, respectively.Once these changes are complete, you will be able to train a multi-class classifier.  Download Stackoverflow dataset<jupyter_code>ds_dir="datasets/stack_overflow/"
os.makedirs(ds_dir)
url = "http://storage.googleapis.com/download.tensorflow.org/data/stack_overflow_16k.tar.gz"
dataset = tf.keras.utils.get_file("stack_overflow_16k",url,untar=True, cache_dir=ds_dir, cache_subdir='')
dataset_dir  = os.path.join(os.path.dirname(dataset),'stack_overflow')
print (os.path.dirname(dataset))
dataset_dir = os.path.dirname(dataset)
os.listdir(dataset_dir)
train_dir = os.path.join(dataset_dir, 'train')
os.listdir(train_dir)
sample_file = os.path.join(train_dir, 'python/1655.txt')
with open(sample_file) as f:
  print(f.read())<jupyter_output>"program runs in pycharm but will not run in the console this is my first programming language class, so please be patient. the program runs fine in pycharm. but when i try to run the program in c:windowspy.exe it will populate the first question of my program. when i answer the question it closes suddenly. ..pounds = float(input('please enter the value of pounds you wish to convert to kilograms: ')).kilograms = pounds * 0.45359237.print(""the conversion to kilograms is: "", format(kilograms, '.2f'))"<jupyter_text>load the dataset<jupyter_code>batch_size = 32
seed = 42

raw_train_ds = tf.keras.preprocessing.text_dataset_from_directory(
    ds_dir+'/train', 
    batch_size=batch_size, 
    validation_split=0.2, 
    subset='training', 
    seed=seed)
for text_batch, label_batch in raw_train_ds.take(1):
  for i in range(3):
    print("Review", text_batch.numpy()[i])
    print("Label", label_batch.numpy()[i])
print("Label 0 corresponds to", raw_train_ds.class_names[0])
print("Label 1 corresponds to", raw_train_ds.class_names[1])
print("Label 2 corresponds to", raw_train_ds.class_names[2])
print("Label 3 corresponds to", raw_train_ds.class_names[3])
# now validation set

raw_val_ds = tf.keras.preprocessing.text_dataset_from_directory(
    ds_dir+ 'train', 
    batch_size=batch_size, 
    validation_split=0.2, 
    subset='validation', 
    seed=seed)
# now test set
raw_test_ds = tf.keras.preprocessing.text_dataset_from_directory(
    ds_dir+'test', 
    batch_size=batch_size)<jupyter_output>Found 8000 files belonging to 4 classes.<jupyter_text>preprocessing To deal with programming language, we need to process the puctuation. we will try two different versions of preprocess to test this idea.<jupyter_code>def custom_standardization(input_data):
  lowercase = tf.strings.lower(input_data)
  stripped_html = tf.strings.regex_replace(lowercase, '<br />', ' ')
  return stripped_html
  #return tf.strings.regex_replace(stripped_html,
  #                                '[%s]' % re.escape(string.punctuation),
  #                                '')
max_features = 10000
sequence_length = 250

vectorize_layer = TextVectorization(
    standardize=custom_standardization,
    max_tokens=max_features,
    output_mode='int',
    output_sequence_length=sequence_length)<jupyter_output><empty_output><jupyter_text>raw_train_ds is in a form of tf.data.Dataset> dataset = tf.data.Dataset.from_tensor_slices([1, 2, 3]) \> dataset = dataset.map(lambda x: x*2) \> list(dataset.as_numpy_iterator()) \> [2, 4, 6]<jupyter_code># Make a text-only dataset (without labels), then call adapt
# If desired, the user can call this layer's adapt() method on a dataset. When this layer is adapted,
# it will analyze the dataset, determine the frequency of individual string values,
# and create a 'vocabulary' from them. This vocabulary can have unlimited size or be capped,
# depending on the configuration options for this layer; if there are more unique values
# in the input than the maximum vocabulary size, the most frequent terms will be used to create the vocabulary.
train_text = raw_train_ds.map(lambda x, y: x)
vectorize_layer.adapt(train_text)
def vectorize_text(text, label):
  text = tf.expand_dims(text, -1)
  return vectorize_layer(text), label
# retrieve a batch (of 32 reviews and labels) from the dataset
text_batch, label_batch = next(iter(raw_train_ds))
first_review, first_label = text_batch[0], label_batch[0]
print("Review", first_review)
print("Label", raw_train_ds.class_names[first_label])
print("Vectorized review", vectorize_text(first_review, first_label))
print("1287 ---> ",vectorize_layer.get_vocabulary()[1287])
print(" 313 ---> ",vectorize_layer.get_vocabulary()[313])
print('Vocabulary size: {}'.format(len(vectorize_layer.get_vocabulary())))
train_ds = raw_train_ds.map(vectorize_text)
val_ds = raw_val_ds.map(vectorize_text)
test_ds = raw_test_ds.map(vectorize_text)

AUTOTUNE = tf.data.AUTOTUNE

train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)
test_ds = test_ds.cache().prefetch(buffer_size=AUTOTUNE)<jupyter_output><empty_output><jupyter_text>model<jupyter_code>embedding_dim = 16

model = tf.keras.Sequential([
  layers.Embedding(max_features + 1, embedding_dim),
  layers.Dropout(0.2),
  layers.GlobalAveragePooling1D(),
  layers.Dropout(0.2),
  layers.Dense(4)])

model.summary()
model.compile(loss=losses.SparseCategoricalCrossentropy(from_logits=True),
              optimizer='adam',
              metrics=['accuracy'])

#epochs = 30
epochs = 50
history = model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=epochs)
loss, accuracy = model.evaluate(test_ds)

print("Loss: ", loss)
print("Accuracy: ", accuracy)
history_dict = history.history
history_dict.keys()

acc = history_dict['accuracy']
val_acc = history_dict['val_accuracy']
loss = history_dict['loss']
val_loss = history_dict['val_loss']

epochs = range(1, len(acc) + 1)

# "bo" is for "blue dot"
plt.plot(epochs, loss, 'bo', label='Training loss')
# b is for "solid blue line"
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()

plt.show()
plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend(loc='lower right')

plt.show()
export_model = tf.keras.Sequential([
  vectorize_layer,
  model,
  layers.Activation('softmax')
])

export_model.compile(
    loss=losses.SparseCategoricalCrossentropy(from_logits=False), optimizer="adam", metrics=['accuracy']
)

# Test it with `raw_test_ds`, which yields raw strings
loss, accuracy = export_model.evaluate(raw_test_ds)
print(accuracy)<jupyter_output>250/250 [==============================] - 1s 5ms/step - loss: 0.5208 - accuracy: 0.8006
0.8006250262260437<jupyter_text>strip all the punctuation (30 epoch 250/250 [==============================] - 2s 6ms/step - loss: 0.5595 - accuracy: 0.79790.7978749871253967 remaining all the punctuation 250/250 [==============================] - 1s 5ms/step - loss: 0.5208 - accuracy: 0.80060.8006250262260437<jupyter_code>examples = [
  "Csharp",
  "virtual machine, java package",
  "Javascript often abbreviated as JS, is a programming language that conforms to the ECMAScript specification",
  "Python is a great script language!!"]

export_model.predict(examples)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11964478579143166, 'fraction_numerical': 0.02424096963878555, 'mean_word_length': 5.624801271860095, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '270455', 'license': 'Apache-2.0', 'n_tokens_mistral': 2837, 'n_tokens_neox': 2638, 'n_words': 725, 'path': '01_ML_basics_with_Keras/02 basic test classification_stackoverflow.ipynb', 'repo_name': 'snippetfactory/TF_tutorials', 'size': '8333'}