<jupyter_start><jupyter_text>IntroducciónEste ejemplo muestra cómo hacer una clasificación de texto a partir de texto sin formato (como un conjunto de archivos de texto en el disco). Demostramos el flujo de trabajo en el conjunto de datos de clasificación de sentimientos de IMDB (versión sin procesar). Usamos la capa **TextVectorization** para dividir e indexar palabras.https://www.imdb.com/<jupyter_code># importamos las librerías
import tensorflow as tf
import numpy as np
!curl -O https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
!tar -xf aclImdb_v1.tar.gz
# Buscamos un ejemplo
!cat /content/aclImdb/train/pos/6248_7.txt<jupyter_output>Being an Austrian myself this has been a straight knock in my face. Fortunately I don't live nowhere near the place where this movie takes place but unfortunately it portrays everything that the rest of Austria hates about Viennese people (or people close to that region). And it is very easy to read that this is exactly the directors intention: to let your head sink into your hands and say "Oh my god, how can THAT be possible!". No, not with me, the (in my opinion) totally exaggerated uncensored swinger club scene is not necessary, I watch porn, sure, but in this context I was rather disgusted than put in the right context.<br /><br />This movie tells a story about how misled people who suffer from lack of education or bad company try to survive and live in a world of redundancy and boring horizons. A girl who is treated like a whore by her super-jealous boyfriend (and still keeps coming back), a female teacher who discovers her masochism by putting the life of her super-cruel "lover" [...]<jupyter_text>Realizamos el paso de preprocesado`tf.keras.preprocessing.text_dataset_from_directory`<jupyter_code># Creamos el flujo de train
batch_size = 32
raw_train_ds = tf.keras.preprocessing.text_dataset_from_directory(
    "/content/aclImdb/train",
    batch_size=batch_size,
    validation_split=0.2,
    subset="training",
    seed=2021,
)
raw_val_ds = tf.keras.preprocessing.text_dataset_from_directory(
    "/content/aclImdb/train",
    batch_size=batch_size,
    validation_split=0.2,
    subset="validation",
    seed=2021,
)
raw_test_ds = tf.keras.preprocessing.text_dataset_from_directory(
    "/content/aclImdb/test",
    batch_size=batch_size
)

print(f"Número de batch en raw_train es {raw_train_ds.cardinality()}")
print(f"Número de batch en raw_validation es {raw_val_ds.cardinality()}")
print(f"Número de batch en raw_test es {raw_test_ds.cardinality()}")
# Observamos con un ejemplo realizando un batch
for text_batch, label_batch in raw_train_ds.take(1):
  for i in range(5):
    print(text_batch.numpy()[i])
    print(label_batch.numpy()[i])
# Parte de preprocesado y limpieza <br />
import re
import string
from tensorflow.keras.layers import TextVectorization
def custom_standarization(input_data):
  lowercase = tf.strings.lower(input_data)
  stripped_html = tf.strings.regex_replace(lowercase, "(<br />){1,}", " ")
  return tf.strings.regex_replace(
      stripped_html, f"[{re.escape(string.punctuation)}]", ""
  )
# Realizamos el paso de vectorización
# definimos unas constantes
max_features = 20000
embedding_dim = 128
sequence_length = 500

# Creamos nuestra capa de vectorización
vectorize_layer = TextVectorization(
    standardize=custom_standarization,
    max_tokens=max_features,
    output_mode="int",
    output_sequence_length=sequence_length,
)

# Aplicamos al solo texto
text_ds = raw_train_ds.map(lambda x, y: x)
vectorize_layer.adapt(text_ds)
vectorize_layer.output_shape()
# Aplicamos la vectorización a nustros datos
# opción 1

# text_input = tf.keras.Input(shape=(1,), dtype=tf.string, name='text')
# x = vectorize_layer(text_input)
# y = layers.Embedding(max_features + 1, embedding_dim)(x)
# Opción 2 en GPU / CPU
def vectorize_text(text, label):
    text = tf.expand_dims(text, -1)
    return vectorize_layer(text), label


# Vectorize the data.
train_ds = raw_train_ds.map(vectorize_text)
val_ds = raw_val_ds.map(vectorize_text)
test_ds = raw_test_ds.map(vectorize_text)

# Do async prefetching / buffering of the data for best performance on GPU.
train_ds = train_ds.cache().prefetch(buffer_size=10)
val_ds = val_ds.cache().prefetch(buffer_size=10)
test_ds = test_ds.cache().prefetch(buffer_size=10)<jupyter_output><empty_output><jupyter_text>Construimos el modeloCreamos un modelo simple de 1D convnet con una capa (layer Embedding).<jupyter_code>from tensorflow.keras import layers

# A integer input for vocab indices.
inputs = tf.keras.Input(shape=(None,), dtype="int64")

# Next, we add a layer to map those vocab indices into a space of dimensionality
# 'embedding_dim'.
x = layers.Embedding(max_features, embedding_dim)(inputs)
x = layers.Dropout(0.5)(x)

# Conv1D + global max pooling
x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.GlobalMaxPooling1D()(x)

# We add a vanilla hidden layer:
x = layers.Dense(128, activation="relu")(x)
x = layers.Dropout(0.5)(x)

# We project onto a single unit output layer, and squash it with a sigmoid:
predictions = layers.Dense(1, activation="sigmoid", name="predictions")(x)

model = tf.keras.Model(inputs, predictions)

# Compile the model with binary crossentropy loss and an adam optimizer.
model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
# Entrenamos el modelo con epoch = 1
epoch = 1

# Ajustamos el modelo a train y test dataset
model.fit(train_ds, validation_data=val_ds, epochs=epoch)
# Evaluando el modelo con los datos de test
model.evaluate(test_ds )
# Creación del modelo final
# A string input
inputs = tf.keras.Input(shape=(1,), dtype="string")
# Turn strings into vocab indices
indices = vectorize_layer(inputs)
# Turn vocab indices into predictions
outputs = model(indices)

# Our end to end model
end_to_end_model = tf.keras.Model(inputs, outputs)
end_to_end_model.compile(
    loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"]
)

# Test it with `raw_test_ds`, which yields raw strings
end_to_end_model.evaluate(raw_test_ds)<jupyter_output>782/782 [==============================] - 12s 15ms/step - loss: 1846527787008.0000 - accuracy: 0.5000
{'fraction_non_alphanumeric': 0.1019239942757195, 'fraction_numerical': 0.016854825886468437, 'mean_word_length': 5.47119341563786, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '537121', 'license': 'Apache-2.0', 'n_tokens_mistral': 2144, 'n_tokens_neox': 2062, 'n_words': 630, 'path': '05_DeepLearning/03_Text_Classification.ipynb', 'repo_name': 'MaryMP11/The_Bridge_School_DataScience_PT', 'size': '6289'}