<jupyter_start><jupyter_text>MLP para la clasificación de productos de Mercado Libre<jupyter_code>!pip install PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
mkdir -p data
# Meli Data
downloaded = drive.CreateFile({'id':"1TpzCRwb4hLMkPwlv1jNy1clpBBg0ajvL"})   # replace the id with id of file you want to access
downloaded.GetContentFile('./data/meli-challenge-2019.tar.bz2')        # replace the file name with your file

# Diccionary
downloaded = drive.CreateFile({'id':"11_zHF4AjPRUjkW_x9CFKJGiPBm6lCKO2"})   # replace the id with id of file you want to access
downloaded.GetContentFile('./data/SBW-vectors-300-min5.txt.bz2')        # replace the file name with your file

!nvcc --version
!pip install torch==1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html
!pip install --upgrade gensim mlflow tqdm
%%bash

#mkdir -p data
#curl -L https://cs.famaf.unc.edu.ar/\~ccardellino/resources/diplodatos/meli-challenge-2019.tar.bz2 -o ./data/meli-challenge-2019.tar.bz2
#curl -L http://cs.famaf.unc.edu.ar/~ccardellino/SBWCE/SBW-vectors-300-min5.txt.bz2 -o ./data/SBW-vectors-300-min5.txt.bz2
tar jxvf ./data/meli-challenge-2019.tar.bz2 -C ./data/
bunzip2 ./data/SBW-vectors-300-min5.txt.bz2
# para ahicar el dataset de train 
# Acordate de corregir en las funciones de prepocesamiento

%%bash

zcat ./data/meli-challenge-2019/spanish-train.csv.gz | head -n 1000000 | gzip > ./data/meli-challenge-2019/spanish-train-debug.csv.gz<jupyter_output>gzip: ./data/meli-challenge-2019/spanish-train.csv.gz: No such file or directory<jupyter_text>MLflow y un experimento sencillo  Librerías<jupyter_code>import gzip
import mlflow
import pandas as pd
import tempfile
import torch
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F

from gensim import corpora
from gensim.models import KeyedVectors
from gensim.parsing import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.metrics import balanced_accuracy_score
from torch.utils.data import IterableDataset, DataLoader
from tqdm.notebook import tqdm, trange<jupyter_output><empty_output><jupyter_text>DatasetNos basaremos en el dataset que creamos para el [notebook 2](./2_datasets.ipynb), con la diferencia de que pasaremos el dataframe de manera directa (esto es para poder hacer split en train/test).<jupyter_code>!zcat ./data/meli-challenge-2019/spanish.train.csv.gz | wc -l
import numpy as np

train_index = np.random.rand(6119101) <= 0.8

with gzip.open("./data/meli-challenge-2019/spanish.train.csv.gz", "rt") as spanish_corpus,\
    gzip.open("./data/meli-challenge-2019/spanish-train.csv.gz", "wt") as spanish_train,\
    gzip.open("./data/meli-challenge-2019/spanish-validation.csv.gz", "wt") as spanish_validation:
  for idx, line in enumerate(tqdm(spanish_corpus, total=6119101)):
    if train_index[idx]:
      spanish_train.write(line)
    else:
      spanish_validation.write(line)
!zcat ./data/meli-challenge-2019/spanish-train.csv.gz | wc -l
!zcat ./data/meli-challenge-2019/spanish-validation.csv.gz | wc -l
!zcat ./data/meli-challenge-2019/spanish.test.csv.gz | wc -l
!zcat ./data/meli-challenge-2019/spanish-validation.csv.gz | head
import csv
import random

class MeliChallengeDataset(IterableDataset):
    def __init__(self,
                 dataset_path,
                 dataset_size,
                 random_buffer_size=2048,
                 transform=None):
        assert random_buffer_size > 0
        self.dataset_path = dataset_path
        self.dataset_size = dataset_size
        self.random_buffer_size = random_buffer_size
        self.transform = transform
    
    def __len__(self):
        return self.dataset_size

    def __iter__(self):
        try:
            with gzip.open(self.dataset_path, "rt") as dataset:
                shuffle_buffer = []
                csv_reader = csv.reader(dataset)
                next(csv_reader)

                for line in csv_reader:
                    _, label_quality, title, category = line

                    if self.random_buffer_size == 1:
                        item = {
                            "title": f"{label_quality} {title}",
                            "category": category
                        }
                        if self.transform:
                            item = self.transform(item)
                        yield item
                    else:
                        shuffle_buffer.append({
                            "title": f"{label_quality} {title}",
                            "category": category
                        })

                        if len(shuffle_buffer) == self.random_buffer_size:
                            random.shuffle(shuffle_buffer)
                            for item in shuffle_buffer:
                                if self.transform:
                                    item = self.transform(item)
                                yield item
                            shuffle_buffer = []

            if len(shuffle_buffer) > 0:
                random.shuffle(shuffle_buffer)
                for item in shuffle_buffer:
                    if self.transform:
                        item = self.transform(item)
                    yield item
        except GeneratorExit:
            return<jupyter_output><empty_output><jupyter_text>PreprocesamientoEn este caso vamos a utilizar un sólo módulo para transformar los datos de MeLi. Este se encargará de preprocesar el texto (i.e. normalizarlo) y transformará las palabras en índices de un diccionario para luego poder pasar una secuencia de palabras para buscar en la matriz de embeddings y así permitir mayor manipulación de los embeddings (en lugar de utilizar embeddings fijos).<jupyter_code>class RawDataProcessor:
    def __init__(self, 
                 dataset_path,
                 dataset_size,
                 ignore_header=True, 
                 filters=None, 
                 vocab_size=50000):
        if filters:
            self.filters = filters
        else:
            self.filters = [
                lambda s: s.lower(),
                preprocessing.strip_tags,
                preprocessing.strip_punctuation,
                preprocessing.strip_multiple_whitespaces,
                preprocessing.strip_numeric,
                preprocessing.remove_stopwords,
                preprocessing.strip_short,
            ]
        
        # Create dictionary based on all the reviews (with corresponding preprocessing)
        self.dictionary = corpora.Dictionary()
        labels = set()

        with gzip.open(dataset_path, "rt") as dataset:
            csv_reader = csv.reader(dataset)
            if ignore_header:
              next(csv_reader)
            for line in tqdm(csv_reader, total=dataset_size):
                _, label_quality, title, category = line
                self.dictionary.add_documents(
                    [self._preprocess_string(f"{label_quality} {title}")]
                )
                labels.add(category)
        # Filter the dictionary and compactify it (make the indices continous)
        self.dictionary.filter_extremes(no_below=2, no_above=1, keep_n=vocab_size)
        self.dictionary.compactify()
        # Add a couple of special tokens
        self.dictionary.patch_with_special_tokens({
            "[PAD]": 0,
            "[UNK]": 1
        })
        self.idx_to_target = sorted(labels)
        self.target_to_idx = {t: i for i, t in enumerate(self.idx_to_target, start=1)}

    def _preprocess_string(self, string):
        return preprocessing.preprocess_string(string, filters=self.filters)

    def _sentence_to_indices(self, sentence):
        return self.dictionary.doc2idx(sentence, unknown_word_index=1)
    
    def encode_data(self, data):
        return self._sentence_to_indices(self._preprocess_string(data))
    
    def encode_target(self, target):
        return self.target_to_idx.get(target, 0)
    
    def __call__(self, item):
        if isinstance(item, list):
          data = [self.encode_data(i["title"]) for i in item]
          target = [self.encode_target(i["category"]) for i in item]
        else:
          data = self.encode_data(item["title"])
          target = self.encode_target(item["category"])
        
        return {
            "data": data,
            "target": target
        }<jupyter_output><empty_output><jupyter_text>Lectura de datosEn esta ocasión, leeremos los datos de MeLi y lo dividiremos en subconjuntos de entrenamiento y evaluación.<jupyter_code>preprocess = RawDataProcessor(
    "./data/meli-challenge-2019/spanish-train-debug.csv.gz",
    10000
#    preprocessing_buffer_size=100000
)
train_dataset = MeliChallengeDataset(
    "./data/meli-challenge-2019/spanish-train-debug.csv.gz",
    10000,
    random_buffer_size=2048,
    transform=preprocess
)
validation_dataset = MeliChallengeDataset(
    "./data/meli-challenge-2019/spanish-validation.csv.gz",
    1224416,
    random_buffer_size=1,
    transform=preprocess
)

test_dataset = MeliChallengeDataset(
    "./data/meli-challenge-2019/spanish.test.csv.gz",
    63681,
    random_buffer_size=1,
    transform=preprocess
)

print(f"Datasets loaded with {len(train_dataset)} training elements and {len(validation_dataset)} validation elements")
print(f"Sample train element:\n{next(iter(train_dataset))}")<jupyter_output>Datasets loaded with 10000 training elements and 1224416 validation elements
Sample train element:
{'data': [4, 436, 437, 1, 299], 'target': 342}<jupyter_text>Collation functionComo en este caso trabajamos con secuencias de palabras (representadas por sus índices en un vocabulario), cuando queremos buscar un *batch* de datos, el `DataLoader` de PyTorch espera que los datos del *batch* tengan la misma dimensión (para poder llevarlos todos a un tensor de dimensión fija). Esto lo podemos lograr mediante el parámetro de `collate_fn`. En particular, esta función se encarga de tomar varios elementos de un `Dataset` y combinarlos de manera que puedan ser devueltos como un tensor de PyTorch. Muchas veces la `collate_fn` que viene por defecto en `DataLoader` sirve (como se vio en el notbook 2), pero este no es el caso. Se define un módulo `PadSequences` que toma un valor mínimo, opcionalmente un valor máximo y un valor de relleno (*pad*) y dada una lista de secuencias, devuelve un tensor con *padding* sobre dichas secuencias.<jupyter_code>class PadSequences:
    def __init__(self, pad_value=0, max_length=None, min_length=1):
        assert max_length is None or min_length <= max_length
        self.pad_value = pad_value
        self.max_length = max_length
        self.min_length = min_length

    def __call__(self, items):
        data, target = list(zip(*[(item["data"], item["target"]) for item in items]))
        seq_lengths = [len(d) for d in data]

        if self.max_length:
            max_length = self.max_length
            seq_lengths = [min(self.max_length, l) for l in seq_lengths]
        else:
            max_length = max(self.min_length, max(seq_lengths))

        data = [d[:l] + [self.pad_value] * (max_length - l)
                for d, l in zip(data, seq_lengths)]
            
        return {
            "data": torch.LongTensor(data),
            "target": torch.LongTensor(target)
        }<jupyter_output><empty_output><jupyter_text>DataLoadersYa habiendo definido nuestros conjuntos de datos y nuestra `collation_fn`, podemos definir nuestros `DataLoader`s, uno para entrenamiento y otro para evaluación. Ver que la diferencia fundamental está en `shuffle`, no queremos mezclar los valores de evaluación cada vez que evaluamos porque al evaluar mediante *mini-batchs* nos puede generar inconsistencias.<jupyter_code># Batch size modificado. Original 128

pad_sequences = PadSequences()
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=False,
                          collate_fn=pad_sequences, drop_last=False)
validation_loader = DataLoader(validation_dataset, batch_size=64, shuffle=False,
                               collate_fn=pad_sequences, drop_last=False)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False,
                               collate_fn=pad_sequences, drop_last=False)
next(iter(train_loader))
next(iter(validation_loader))<jupyter_output><empty_output><jupyter_text>El modelo de clasificaciónPara clasificación utilizaremos un perceptrón multicapa de dos capas ocultas. Claramente este modelo es naive y prácticamente todo lo que está *hardcodeado* (e.g. los tamaños de las capas o la cantidad de capas) podría ser parte de los parámetros del modelo. En particular, tenemos la capa de `Embeddings` que es rellenada con los valores de embeddings preentrenados.<jupyter_code>!head ./data/SBW-vectors-300-min5.txt<jupyter_output><empty_output><jupyter_text>Multi-layer perceptron<jupyter_code>class MeliChallengeClassifier(nn.Module):
    def __init__(self, 
                 pretrained_embeddings_path, 
                 dictionary,
                 n_labels,
                 vector_size,
                 freeze_embedings):
        super().__init__()
        embeddings_matrix = torch.randn(len(dictionary), vector_size)
        embeddings_matrix[0] = torch.zeros(vector_size)
        with open(pretrained_embeddings_path, "rt") as fh:
            next(fh)
            for line in fh:
                word, vector = line.strip().split(None, 1)
                if word in dictionary.token2id:
                    embeddings_matrix[dictionary.token2id[word]] =\
                        torch.FloatTensor([float(n) for n in vector.split()])
        self.embeddings = nn.Embedding.from_pretrained(embeddings_matrix,
                                                       freeze=freeze_embedings,
                                                       padding_idx=0)
        self.hidden1 = nn.Linear(vector_size, 128)
        self.hidden2 = nn.Linear(128, 128)
        self.output = nn.Linear(128, n_labels)
        self.vector_size = vector_size
    
    def forward(self, x):
        x = self.embeddings(x)
        x = torch.mean(x, dim=1)
        x = F.relu(self.hidden1(x))
        x = F.relu(self.hidden2(x))
        x = self.output(x)
        return x
hidden_layers = [256, 128, 64, 32]
list(zip(hidden_layers[:-1], hidden_layers[1:]))
model = MeliChallengeClassifier("./data/SBW-vectors-300-min5.txt", 
                                preprocess.dictionary, 
                                len(preprocess.target_to_idx) + 1,
                                300, True)
print(model)<jupyter_output>MeliChallengeClassifier(
  (embeddings): Embedding(5634, 300, padding_idx=0)
  (hidden1): Linear(in_features=300, out_features=128, bias=True)
  (hidden2): Linear(in_features=128, out_features=128, bias=True)
  (output): Linear(in_features=128, out_features=605, bias=True)
)<jupyter_text>Experimento de MLflowPor último, ya tenemos todos los bloques para construir nuestro experimento de MLflow. Anotamos un par de parámetros  (estos pueden ser todos los que se consideren necesarios) y lanzamos a correr nuestro experimento. Cada vez que finaliza un epoch guardamos algunas métricas. Al finalizar todos los epochs corremos algunas métricas extras de evaluación y guardamos algunos datos extra que nos servirán para calcular otras métricas a futuro.<jupyter_code>EPOCHS = 150

mlflow.set_experiment("multilayer perceptron first attempt")

with mlflow.start_run():
    mlflow.log_param("model_name", "mlp")
    mlflow.log_param("freeze_embedding", True)
    mlflow.log_params({
        "embedding_size": 50,
        "hidden1_size": 128,
        "hidden2_size": 128
    })
    model = MeliChallengeClassifier("./data/SBW-vectors-300-min5.txt", 
                                    preprocess.dictionary, 
                                    len(preprocess.target_to_idx) + 1,
                                    300, True)
    device = torch.device("cuda")
    model = model.to(device)
    loss = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-5)
    for epoch in trange(EPOCHS):
        model.train()
        running_loss = []
        for idx, batch in enumerate(tqdm(train_loader)):
            optimizer.zero_grad()
            data = batch["data"].to(device)
            target = batch["target"].type(torch.LongTensor).to(device)
            output = model(data)
            loss_value = loss(output, target)
            loss_value.backward()
            optimizer.step()
            running_loss.append(loss_value.item())        
        mlflow.log_metric("train_loss", sum(running_loss) / len(running_loss), epoch)
        
        # Evaluación con los datos de validacion
        model.eval()
        running_loss = []
        targets = []
        predictions = []
        with torch.no_grad():
          for batch in tqdm(validation_loader):
            data = batch["data"].to(device)
            target = batch["target"].to(device)
            output = model(data)
            running_loss.append(
                loss(output, target).item()
            )
            targets.extend(batch["target"].numpy())
            predictions.extend(output.argmax(axis=1).detach().cpu().numpy())
        mlflow.log_metric("validation_loss", sum(running_loss) / len(running_loss), epoch)
        mlflow.log_metric("validation_bacc", balanced_accuracy_score(targets, predictions), epoch)

        # Evaluación con los datos de test
        model.eval()
        running_loss = []
        targets = []
        predictions = []
        with torch.no_grad():
          for batch in tqdm(test_loader):
            data = batch["data"].to(device)
            target = batch["target"].to(device)
            output = model(data)
            running_loss.append(
                loss(output, target).item()
            )
            targets.extend(batch["target"].numpy())
            predictions.extend(output.argmax(axis=1).detach().cpu().numpy())
        mlflow.log_metric("test_loss", sum(running_loss) / len(running_loss), epoch)
        mlflow.log_metric("test_bacc", balanced_accuracy_score(targets, predictions), epoch)


    
    with tempfile.TemporaryDirectory() as tmpdirname:
        targets = []
        predictions = []
        for batch in tqdm(test_loader):
            data = batch["data"].to(device)
            target = batch["target"].type(torch.LongTensor).to(device)
            output = model(data)
            targets.extend(target.cpu().detach().numpy())
            predictions.extend(output.squeeze().cpu().detach().numpy())
        pd.DataFrame({"prediction": predictions, "target": targets}).to_csv(
            f"{tmpdirname}/predictions.csv.gz", index=False
        )
        mlflow.log_artifact(f"{tmpdirname}/predictions.csv.gz")<jupyter_output>INFO: 'multilayer perceptron first attempt' does not exist. Creating a new experiment<jupyter_text>A pesar del error actual el código funciona. Se agotó el tiempo en Colab para correr con GPU.<jupyter_code>!tar zcvf mlruns.tar.gz mlruns
%%bash

# Una vez que lo descargan, lo descomprimen en su máquina con el siguiente comando
tar xvf mlruns.tar.gz

# Luego ejecutan en el lugar que lo descomprimieron lo siguiente
mlflow ui<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09468489770741235, 'fraction_numerical': 0.017322915595764366, 'mean_word_length': 2.6041126343090033, 'pattern_counts': {'":': 15, '<': 43, '<?xml version=': 0, '>': 44, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '142785', 'license': 'MIT', 'n_tokens_mistral': 6195, 'n_tokens_neox': 5815, 'n_words': 1395, 'path': 'docs/experiments/mlp_experiment/2/Practico DL.ipynb', 'repo_name': 'alessiobocco/AprendizajeProfundo', 'size': '19454'}