<jupyter_start><jupyter_text>T5 text summarizer - ExperimentoEste componente utiliza modelo T5 para sumarizar textos em inglêsEste notebook apresenta:- como usar o [SDK](https://platiagro.github.io/sdk/) para carregar datasets, salvar modelos e outros artefatos.- como declarar parâmetros e usá-los para criar componentes reutilizáveis.  Declare parâmetros e hiperparâmetros para o modeloOs componentes podem declarar (e usar) estes parâmetros como padrão:- dataset- targetUse estes parâmetros para carregar/salvar conjutos de dados, modelos, métricas e figuras com a ajuda do [SDK da PlatIAgro](https://platiagro.github.io/sdk/). É possível também declarar parâmetros personalizados para serem definidos ao executar um experimento. Selecione os hiperparâmetros e seus respectivos valores para serem usados ao treinar o modelo:- languageEstes parâmetros são alguns dos oferecidos pela classe do modelo, você também pode utilizar outros existentes. Dê uma olhada nos [parâmetros do modelo](https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.htmlsklearn-impute-simpleimputer) para mais informações.<jupyter_code># parâmetros
dataset = "/tmp/data/news_summary.csv" #@param {type:"string"}
target = "text" #@param {type:"string", label:"Atributo alvo", description:"Seu modelo será treinado para prever os valores do alvo."}
language = "english" #@param ["portuguese", "english"] {type:"string", label:"Linguagem", description:"Linguagem da qual os stopwords pertencem. Deve ser a mesma utilizada no dataset."}

#Hyperparâametros
model_name = 't5-base'  #@param ["t5-base","t5-small","unicamp-dl/ptt5-base-portuguese-vocab","unicamp-dl/ptt5-large-portuguese-vocab","unicamp-dl/ptt5-small-portuguese-vocab","unicamp-dl/ptt5-small-t5-vocab","unicamp-dl/ptt5-base-t5-vocab","unicamp-dl/ptt5-large-t5-vocab"] {type:"string"}
batch_size =  2#@param {type:"integer"}
max_epochs = 1 #@param {type:"integer"}
accumulate_grad_batches = 16  #@param {type:"integer"}
max_length = 512  #@param {type:"integer"}
summary_len = 150  #@param {type:"integer"}
learning_rate = 1e-4  #@param {type:"number"}
num_beams=2 #@param {type:"integer"}
repetition_penalty=2.5 #@param {type:"number"}
length_penalty=1.0 #@param {type:"number"}
seed = 7 #@param {type:"integer"}

# selected features to perform the model
filter_type = "incluir" #@param ['incluir','remover']  {type:"string",label:"Modo de seleção das features", description:"Se deseja informar quais features deseja incluir no modelo, selecione a opção [incluir]. Caso deseje informar as features que não devem ser utilizadas, selecione [remover]. "}
model_features = "ctext" #@param {type:"string",multiple:true,label:"Features para incluir/remover no modelo",description:"Seu modelo será feito considerando apenas as features selecionadas. Caso nada seja especificado, todas as features serão utilizadas"}

# preprocessamento
case = 'Lower' #@param ['Lower','Upper','NotApply']  {type:"string",label:"Aplicação de casing", description:"Caixa baixa, caixa alta ou não aplicação de caixa"}
remove_stop_words = False #@param [True,False]  {type:"boolean",label:" Remoção de Stop Words", description:"Remoção de palavras, conjunções, artigos e outros"}
remove_top_words = False #@param [True,False]  {type:"boolean",label:" Remoção de Top Words", description:"Remoção dea porcentagem palavras mais frequentes no texto"}
top_words_percentage = 0.01 #@param {type:"number",label:" Porcentagem de Top Words", description:"Porcentagem das palavras mais frequentes no texto"}
stemming = False #@param [True,False]  {type:"boolean",label:"Stemming"}
lemmatization = False #@param [True,False]  {type:"boolean",label:"Lemmatization"}
remove_punctuation = False #@param [True,False]  {type:"boolean",label:"Remoção de pontuação"}
remove_line_braks = True #@param [True,False]  {type:"boolean",label:"Remoção de quebras de lina",description:"Remoção de quebras de linha por \n e \r"}
remove_accents= False #@param [True,False]  {type:"boolean",label:"Remoção de acentos"}
remove_html = True #@param [True,False]  {type:"boolean",label:"Remoção de HTML"}
remove_css = True #@param [True,False]  {type:"boolean",label:"Remoção de CSS"}<jupyter_output><empty_output><jupyter_text>Configurações Gerais Definindo GPU como padrão e verificando status de hardware<jupyter_code>from multiprocessing import cpu_count
import torch
dev = "cuda:0" if torch.cuda.is_available() else "cpu"
device = torch.device(dev)
if dev == "cpu":
    print(f"number of CPU cores: {cpu_count()}")
else:
    print(f"GPU: {torch.cuda.get_device_name(0)}, number of CPU cores: {cpu_count()}")<jupyter_output>number of CPU cores: 4<jupyter_text>Impedir excesso de logs<jupyter_code>import logging
logging.getLogger("transformers.configuration_utils").setLevel(logging.WARNING)
logging.getLogger("transformers.modeling_utils").setLevel(logging.WARNING)
logging.getLogger("lightning").setLevel(logging.WARNING)<jupyter_output><empty_output><jupyter_text>Fixar semente de pesos aleatporios para replicabilidade<jupyter_code>import random
random.seed(seed)
torch.random.manual_seed(seed)
torch.cuda.manual_seed(seed)<jupyter_output><empty_output><jupyter_text>Instanciando Tokenizador<jupyter_code>from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained(model_name)<jupyter_output><empty_output><jupyter_text>Acesso ao conjunto de dadosO conjunto de dados utilizado nesta etapa será o mesmo carregado através da plataforma.O tipo da variável retornada depende do arquivo de origem:- [pandas.DataFrame](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) para CSV e compressed CSV: .csv .csv.zip .csv.gz .csv.bz2 .csv.xz- [Binary IO stream](https://docs.python.org/3/library/io.htmlbinary-i-o) para outros tipos de arquivo: .jpg .wav .zip .h5 .parquet etc<jupyter_code>import pandas as pd
df = pd.read_csv(dataset,encoding='latin-1')
df<jupyter_output><empty_output><jupyter_text>Trabalhando os dados<jupyter_code>df = df[[model_features,target]]
df.ctext = 'summarize: ' + df.ctext
df.head()<jupyter_output>/opt/conda/lib/python3.7/site-packages/pandas/core/generic.py:5208: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self[name] = value<jupyter_text>Acesso aos metadados do conjunto de dadosUtiliza a função `stat_dataset` do [SDK da PlatIAgro](https://platiagro.github.io/sdk/) para carregar metadados. Por exemplo, arquivos CSV possuem `metadata['featuretypes']` para cada coluna no conjunto de dados (ex: categorical, numerical, or datetime).<jupyter_code>import numpy as np
from platiagro import stat_dataset

#metadata = stat_dataset(name=dataset)
#featuretypes = metadata["featuretypes"]

columns = df.columns.to_numpy()
#featuretypes = np.array(featuretypes)
target_index = np.argwhere(columns == target)
columns = np.delete(columns, target_index)
#featuretypes = np.delete(featuretypes, target_index)<jupyter_output><empty_output><jupyter_text>Remoção de linhas com valores faltantes Caso haja linhas em que o atributo alvo contenha valores faltantes, é feita a remoção dos casos faltantes.<jupyter_code>df.dropna(subset = [target],inplace=True)
df.dropna(subset = [model_features],inplace=True)
y = df[target].to_numpy()<jupyter_output>/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.
/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy<jupyter_text>Filtragem das features Seleciona apenas as features que foram declaradas no parâmetro model_features. Se nenhuma feature for especificada, todo o conjunto de dados será utilizado para a modelagem.<jupyter_code>if filter_type == 'incluir':
    if len(model_features) >= 1:
        columns_index = (np.where(np.isin(columns,model_features)))[0]
        columns_index.sort()
        columns_to_filter = columns[columns_index]
        #featuretypes = featuretypes[columns_index]
    else:
        columns_to_filter = columns
else:
    if len(model_features) >= 1:
        columns_index = (np.where(np.isin(columns,model_features)))[0]
        columns_index.sort()
        columns_to_filter = np.delete(columns,columns_index)
        #featuretypes = np.delete(featuretypes,columns_index)
    else:
        columns_to_filter = columns

# keep the features selected
df_model = df[columns_to_filter]
X = df_model.to_numpy()<jupyter_output><empty_output><jupyter_text>Divisão do datset em subconjuntos de treino e testeSubconjunto de Treino: amostras de dados usado para treinar o modelo (``fit``). Subconjunto de Teste: a amostra de dados usada para fornecer uma avaliação imparcial de um modelo adequado ao conjunto de dados de treinamento.<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_, y_train, y_  = train_test_split(X, y,  train_size=0.8,random_state=seed)
X_valid, X_test, y_valid, y_test = train_test_split(X_, y_,  train_size=0.5)<jupyter_output><empty_output><jupyter_text>Busca por stopwordsStopwords (ou palavras de parada) são palavras que geralmente se referem às mais comuns em um idioma ou em um corpus. Elas podem ser ignoradas com segurança sem sacrificar o significado da frase, pois são palarvas que não agregram muito significado.<jupyter_code>import nltk
if remove_stop_words:
    # Download stopwords from nltk
    nltk.download('stopwords')

    # Get a list of stopwords for the defined language
    stopwords = nltk.corpus.stopwords.words(language)

if lemmatization:
    nltk.download('wordnet')<jupyter_output><empty_output><jupyter_text>Processamento do textoFunções auxiliares para processamento dos dados.<jupyter_code>from re import sub
from collections import defaultdict
from functools import reduce
import string  
from nltk.stem import PorterStemmer
from nltk.stem import WordNetLemmatizer 
from nltk.tokenize import word_tokenize
import unidecode
from ftfy import fix_text


def tokenize_text(text_list: list = None):
    """Tokenize Text without the hyperparâmeters defined.

    Args:
        text_list (list): a list of texts to be used.

    Returns:
        A list of tokenized text without punctuation.
    """
    
    tokenize_list = list()
    for text in text_list:
        text = text[0]
        text = fix_text(text)
        text = sub("<.*?>", ' ', text) if remove_html else text
        text = sub("{.*?}", ' ', text) if remove_css else text
        text = unidecode.unidecode(text) if remove_accents else text
        text = sub("/\r\n|\n|\r|", "",text) if remove_line_braks else text
        text = sub('['+string.punctuation+']', '', text) if remove_punctuation else text
        text = sub(' +', ' ',text) #only to avoid multiple spaces
        text = text.split(' ')
        
        tokenize_list.append(text)
    return tokenize_list


def top_tokens_stopwords(sentence_list: list, percentage: float = 0.01):
    """Selects the most relevant stops words of the tokerized texts.

    Args:
        sentence_list (list): list of tokens.
        percentage (float): percentage threshold.
    """
    percentage = top_words_percentage
    vocabulary = defaultdict(int)

    for sample in sentence_list:
        for token in sample:
            vocabulary[token] += 1

    all_tokens = sorted(vocabulary.items(), key=lambda token: token[1], reverse=True)
    top_tokens = all_tokens[:int(len(all_tokens) * percentage)]

    return [token[0] for token in top_tokens]


def remove_specific_tokens(sentence_list: list, tokens_to_be_removed: list = None):
    """Removes specific tokens from a token list.

    Args:
        sentence_list (list): list of tokens from which other tokens will be removed.
        tokens_to_be_removed (list): list of tokens that need to be removed.
    """
    sentence_list_ = list()
    sentence_list_ = [x for x in sentence_list if x not in tokens_to_be_removed]

    return sentence_list_

def apply_stemming(sentence_list: list):
    ps = PorterStemmer()
    sentence_list = [[ps.stem(word) for word in token_list ] for token_list in sentence_list]
    return sentence_list
    
def apply_lemmatization(sentence_list: list):
    lemmatizer = WordNetLemmatizer() 
    sentence_list = [[lemmatizer.lemmatize(word) for word in token_list ] for token_list in sentence_list]
    return sentence_list
    
def apply_casing(sentence_list: list,case:str):
    if case == 'Lower': 
        sentence_list = [[word.lower() for word in token_list ] for token_list in sentence_list]
    elif case == 'Upper':
        sentence_list = [[word.upper() for word in token_list ] for token_list in sentence_list]
    else:
        pass
    return sentence_list
   
def token_restructuring(sentence_list: list):
    """Reduce a nested list of tokens to a single list (1D).
    
    Args:
        sentence_list (list): list to be work on.
    """
    return reduce(lambda x, y: x + y, sentence_list)
def preprocess(X):
    vocab = tokenize_text(X)
    top_tokens = top_tokens_stopwords(vocab) if remove_top_words else None
    vocab = remove_specific_tokens(vocab,top_tokens) if remove_top_words else vocab
    vocab = remove_specific_tokens(vocab,stopwords) if remove_stop_words else vocab
    vocab = apply_stemming(vocab) if stemming else vocab
    vocab = apply_lemmatization(vocab) if lemmatization else vocab
    vocab = apply_casing(vocab,case)
    text = [' '.join(tokens) for tokens in vocab]
    return text

X_train  = preprocess(X_train)
X_valid  = preprocess(X_valid)
X_test = preprocess(X_test)

print(X_train[0])<jupyter_output>summarize: hit-list of security forces anil bhatt jammu, july 14 (pti) security forces and jammu and kashmir police have killed 102 terrorists till now this year, making it the highest number of killings in the january-july period in seven years, a police official said today. the security forces also have prepared a hit-list of several other terrorists belonging to lashkar-e-taiba (let), jaish-e-mohammad and hizbul mujahideen. the slain terrorists included lashkar-e-taiba (let) commander bashir lashkari who was involved in the killing of six policemen in south kashmir and top hizbul mujahideen militant sabzar ahmad bhat, he noted. the security forces and police are going after terrorists as part of the operation "hunt down" and on the basis of a "hit-list" of terrorists, said the official who refused to be identified. the killing of 102 terrorists till july 12 makes it the highest number in the past seven years, the official said. earlier in 2010, the highest number of 156 militants we[...]<jupyter_text>Criaçãod do Dataset Visualizando formato do tokenizador<jupyter_code>text= 'fwegeb is a big city!'
encoder_plus = tokenizer.encode(text)
encoder_plus
%%writefile Dataset.py
from typing import List
from torch.utils.data import Dataset
import torch

class MyDataset(Dataset):

    def __init__(self, X,target, tokenizer, source_len, summ_len):
        self.tokenizer = tokenizer
        self.source_len = source_len
        self.summ_len = summ_len
        self.text = target
        self.ctext = X

    def __len__(self):
        return len(self.text)

    def __getitem__(self, index,step = "Experiment"):
        
        ctext = str(self.ctext[index])
        ctext = ' '.join(ctext.split())
        source = self.tokenizer.batch_encode_plus([ctext], max_length= self.source_len, pad_to_max_length=True,truncation = True,return_tensors='pt')
        source_ids = source['input_ids'].squeeze()
        source_mask = source['attention_mask'].squeeze()
        
        if step == "Experiment":
            text = str(self.text[index])
            text = ' '.join(text.split())
            target = self.tokenizer.batch_encode_plus([text], max_length= self.summ_len, pad_to_max_length=True,truncation = True,return_tensors='pt')
            target_ids = target['input_ids'].squeeze()
            target_mask = target['attention_mask'].squeeze()

            return {
                'source_ids': source_ids.to(dtype=torch.long), 
                'source_mask': source_mask.to(dtype=torch.long), 
                'target_ids': target_ids.to(dtype=torch.long),
                'target_ids_y': target_ids.to(dtype=torch.long)
            }
        
        if step == "Deployment":
            return {
                'source_ids': source_ids.to(dtype=torch.long), 
                'source_mask': source_mask.to(dtype=torch.long)
            }
%run Dataset.py<jupyter_output><empty_output><jupyter_text>Criando e Testando Datasets<jupyter_code>from torch.utils.data import DataLoader

ctexts = X_train[:10]
texts = y_train[:10]

dataset_debug = MyDataset(
    X = ctexts,
    target = texts,
    source_len=max_length, 
    summ_len=summary_len,
    tokenizer=tokenizer)

print(dataset_debug[0])<jupyter_output>{'source_ids': tensor([21603,    10,  1560,    18,  3350,    13,  1034,  3859,    46,   173,
            3,   115,   547,    17,  7620,    51,    76,     6,     3,  2047,
          120,   968,    41,   102,    17,    23,    61,  1034,  3859,    11,
         7620,    51,    76,    11,     3,   157,  3198,  5884,  2095,    43,
         4792,     3, 14388, 10287,     7,  6501,   230,    48,   215,     6,
          492,    34,     8,  2030,   381,    13,  9357,     7,    16,     8,
            3,  7066,    76,  1208,    18,  2047,   120,  1059,    16,  2391,
          203,     6,     3,     9,  2095,  2314,   243,   469,     5,     8,
         1034,  3859,    92,    43,  2657,     3,     9,  1560,    18,  3350,
           13,   633,   119, 10287,     7, 12770,    12,     3,  8058,  4031,
           18,    15,    18,    17, 18071,    41,  1655,   201,  2662,  1273,
           18,    15,    18,    51,    32,  1483, 11374,    11,  7102,   172,
         6724,  4035,  1191, 11740,    15,    35,[...]<jupyter_text>Criando Dataloader Criando Dataloaders<jupyter_code>from torch.utils.data import DataLoader
dataloader_debug = DataLoader(dataset_debug, batch_size=batch_size, shuffle=True, num_workers=cpu_count())<jupyter_output><empty_output><jupyter_text>Testando Dataloaders<jupyter_code>print("------->Testando Dataloader <-------")
next(iter(dataloader_debug))<jupyter_output>------->Testando Dataloader <-------<jupyter_text>Visualizando o funcionamento da métrica RougeMuitos são os pacotes python que calculam a métrica rouge. Escolhemos o **rouge-score** visto que o mesmo do google e também utilizado pelo hugging faces. Vamos utilizar ele diretamente, sem ser pelo hugging faces, para calcular diretamnte os valores desejados.As bibliotecas encontradas foram:*   [sumeval](https://github.com/chakki-works/sumeval)*   [pythonrouge](https://github.com/chakki-works/sumeval) que necessita do  [procedimento](https://github.com/tagucci/pythonrouge/issues/31issuecomment-452270772) para funcionar*   [hugging_faces_rouge](https://huggingface.co/metrics/rouge) o qual utiliza o [rouge-score](https://github.com/google-research/google-research/tree/master/rouge)<jupyter_code>from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(['rouge1','rouge2', 'rougeL'], use_stemmer=True)
scores = scorer.score('The quick brown fox jumps over the lazy dog',
                      'The quick brown dog jumps on the log.')
scores
scores['rouge1'].precision<jupyter_output><empty_output><jupyter_text>Recuperando os pesos<jupyter_code>import os
cwd = os.getcwd()
cwd
# ! gsutil cp gs://platiagro/Glove_Bag_Of_Embeddings_For_Sentence_Classification/epoch=199.ckpt  /home/jovyan/tasks/GloVe_Bag_Of_Sentence_Classification<jupyter_output><empty_output><jupyter_text>Modelo Pytorch Lightning<jupyter_code>%%writefile Model_Lightning.py
from transformers import T5ForConditionalGeneration
from torch.utils.data.sampler import WeightedRandomSampler
from sklearn.metrics import accuracy_score
from torch.utils.data import DataLoader
from multiprocessing import cpu_count
import pytorch_lightning as pl
import torch
import functools
import traceback
import psutil
import pandas as pd


class T5SummarizerFinetuner(pl.LightningModule):

    def __init__(self, 
                 hyperparams,
                 model_parameters,
                 dataset_infos,
                 extra_infos):

        super(T5SummarizerFinetuner, self).__init__()

        #---------- hyperparams
        self.learning_rate = hyperparams['learning_rate']
        self.batch_size = hyperparams['batch_size']
        self.source_len = hyperparams['max_length']
        self.summary_len = hyperparams['summary_len']
        self.num_beams = hyperparams['num_beams']
        self.repetition_penalty = hyperparams['repetition_penalty']
        self.length_penalty = hyperparams['length_penalty']
        
        #---------- model_parameters
        self.model = T5ForConditionalGeneration.from_pretrained(model_parameters['model_name'])
        self.tokenizer = model_parameters['tokenizer']
        
        #---------- dataset_infos
        self.all_data = dataset_infos['all_data']
        self.CustomDataset = dataset_infos['CustomDataset']
        
        #---------- extra_infos
        self.overfit = extra_infos['overfit']
        self.sampler = extra_infos['sampler']
        
         #---------- other_infos
        self.predict_proba  =  torch.nn.Softmax(dim=1)
        self.rouge_metric = rouge_scorer.RougeScorer(['rouge1','rouge2', 'rougeL'], use_stemmer=True)
        self.sdk_step = "Experiment"

        #---------- Dados para gráfico de Acurácia e Loss
        self.df_performance_train_batch = pd.DataFrame(columns=['train_batch_loss','train_batch_rouge1_fmeasure','train_batch_rouge2_fmeasure','train_batch_rougeL_fmeasure'])
        self.df_performance_train_epoch = pd.DataFrame(columns=['train_epoch_loss','train_epoch_rouge1_fmeasure','train_epoch_rouge2_fmeasure','train_epoch_rougeL_fmeasure'])
        self.df_performance_valid_batch = pd.DataFrame(columns=['valid_batch_loss','valid_batch_rouge1_fmeasure','valid_batch_rouge2_fmeasure','valid_batch_rougeL_fmeasure'])
        self.df_performance_valid_epoch = pd.DataFrame(columns=['valid_epoch_loss','valid_epoch_rouge1_fmeasure','valid_epoch_rouge2_fmeasure','valid_epoch_rougeL_fmeasure'])
        self.df_performance_test_batch = pd.DataFrame(columns=['test_batch_rouge1_fmeasure','test_batch_rouge2_fmeasure','test_batch_rougeL_fmeasure'])
        self.df_performance_test_epoch = pd.DataFrame(columns=['test_epoch_rouge1_fmeasure','test_epoch_rouge2_fmeasure','test_epoch_rougeL_fmeasure'])


        #---------- Carregamento datasets
        self.targets_sampler = self.all_data[1]
        if overfit:
            self.train_dataset = self.CustomDataset(self.all_data[0],self.all_data[1],self.tokenizer, self.source_len, self.summary_len)
            self.valid_dataset = self.CustomDataset(self.all_data[0],self.all_data[1],self.tokenizer, self.source_len, self.summary_len)
            self.test_dataset =   self.CustomDataset(self.all_data[0],self.all_data[1],self.tokenizer, self.source_len, self.summary_len)
        else:
            self.train_dataset = self.CustomDataset(self.all_data[0],self.all_data[1],self.tokenizer, self.source_len, self.summary_len)
            self.valid_dataset = self.CustomDataset(self.all_data[2],self.all_data[3],self.tokenizer, self.source_len, self.summary_len)
            self.test_dataset = self.CustomDataset(self.all_data[4],self.all_data[5], self.tokenizer, self.source_len, self.summary_len)


        #---------- Datafame de comparação para o teste
        self.df_valid = pd.DataFrame(columns=['text','ctext','predicted_ctext',
                                              'rouge1_precision','rouge1_recall','rouge1_fmeasure',
                                              'rouge2_precision','rouge2_recall','rouge2_fmeasure',
                                              'rougeL_precision','rougeL_recall','rougeL_fmeasure'])
        self.df_test = pd.DataFrame(columns=['text','ctext','predicted_ctext',
                                             'rouge1_precision','rouge1_recall','rouge1_fmeasure',
                                             'rouge2_precision','rouge2_recall','rouge2_fmeasure',
                                             'rougeL_precision','rougeL_recall','rougeL_fmeasure'])

        
     
    def predict(self,text):
        self.sdk_step = "Deployment"
        inference_dataset = self.CustomDataset(X_inference_glove_ids, X_inference_glove_words,step = self.sdkstep)
        dataloader = DataLoader(inference_dataset, batch_size=self.batch_size,shuffle=False, num_workers=cpu_count(),collate_fn=self.my_collate)    
        
        
        data = self.tokenizer.encode_plus(text, return_tensors='pt')
        data_device = {'source_ids':data['source_ids'].to(device),
                       'source_mask':data['source_mask'].to(device),
                       'target_ids':data['target_ids'].to(device),
                       'target_ids_y':data['target_ids_y'].to(device)}
        generated_ids = self.forward(data_device,step='inference')
        preds = [self.tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids]
        return preds

    def forward(self, data,info_requested='loss'):
        if step=='loss':
            y = data['target_ids']
            y_ids = y[:, :-1].contiguous()
            lm_labels = y[:, 1:].clone().detach()
            lm_labels[y[:, 1:] == self.tokenizer.pad_token_id] = -100
            ids = data['source_ids']
            mask = data['source_mask']
            outputs =  self.model(input_ids = ids, attention_mask = mask, decoder_input_ids=y_ids, lm_labels=lm_labels)
            loss = outputs[0]
            retorno = loss
       
        elif info_requested=='generate_ids':
            ids = data['source_ids']
            mask = data['source_mask']

            generated_ids = self.model.generate(
                  input_ids = ids,
                  attention_mask = mask, 
                  max_length=self.summary_len, 
                  num_beams=self.num_beams,
                  repetition_penalty=self.repetition_penalty, 
                  length_penalty= self.length_penalty, 
                  early_stopping=True)

            retorno = generated_ids

        return retorno

    def training_step(self, batch, batch_nb):
        # batch
        data = batch
        
        # loss
        loss = self.forward(data,'loss')

        # metrics
        generated_ids = self.forward(data,'generate_ids')
        preds = [self.tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids]
        targets = [self.tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=True)for t in data['target_ids']]
        sources = [self.tokenizer.decode(s, skip_special_tokens=True, clean_up_tokenization_spaces=True)for s in data['source_ids']]
        mean_batch_rouge_metrics = self.calc_batch_rouge(sources,targets, preds)
        
        #final_dict
        final_dict = merge_dicts({'loss':loss},mean_batch_rouge_metrics)
        
        self.df_performance_train_batch = self.df_performance_train_batch.append(pd.Series([loss.item(),mean_batch_rouge_metrics['rouge1_fmeasure_batch'],mean_batch_rouge_metrics['rouge2_fmeasure_batch'],mean_batch_rouge_metrics['rougeL_fmeasure_batch']], index=self.df_performance_train_batch.columns ), ignore_index=True)

        return final_dict
     
    def training_epoch_end(self, outputs):
        if not outputs: return {}
        
        epoch_mean = self.calc_epoch_mean(outputs)

        self.df_performance_train_epoch = self.df_performance_train_epoch.append(pd.Series([epoch_mean['avg_loss'],epoch_mean['avg_rouge1_fmeasure'],epoch_mean['avg_rouge2_fmeasure'],epoch_mean['avg_rougeL_fmeasure']], index=self.df_performance_train_epoch.columns ), ignore_index=True)
        
        return epoch_mean

    def validation_step(self, batch, batch_nb):
        # batch
        data = batch
        
         # loss
        loss = self.forward(data,'loss')
        
        # fwd
        generated_ids = self.forward(data,'generate_ids')
        preds = [self.tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids]
        targets = [self.tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=True)for t in data['target_ids']]
        sources = [self.tokenizer.decode(s, skip_special_tokens=True, clean_up_tokenization_spaces=True)for s in data['source_ids']]
        mean_batch_rouge_metrics = self.calc_batch_rouge(sources,targets, preds)
        
        #final_dict
        final_dict = merge_dicts({'loss':loss},mean_batch_rouge_metrics)
        
        self.df_performance_valid_batch = self.df_performance_valid_batch.append(pd.Series([loss.item(),mean_batch_rouge_metrics['rouge1_fmeasure_batch'],mean_batch_rouge_metrics['rouge2_fmeasure_batch'],mean_batch_rouge_metrics['rougeL_fmeasure_batch']], index=self.df_performance_train_batch.columns ), ignore_index=True)
        
            
        return final_dict


    def validation_epoch_end(self, outputs):
        if not outputs: return {}
        
        epoch_mean = self.calc_epoch_mean(outputs)
        
        self.df_performance_valid_epoch = self.df_performance_valid_epoch.append(pd.Series([epoch_mean['avg_loss'],epoch_mean['avg_rouge1_fmeasure'],epoch_mean['avg_rouge2_fmeasure'],epoch_mean['avg_rougeL_fmeasure']], index=self.df_performance_valid_epoch.columns ), ignore_index=True)

        return epoch_mean


    def test_step(self, batch, batch_nb):
        # batch
        if self.sdk_step == "Experiment":
            data = batch
        
            # loss
            loss = self.forward(data,'loss')

            # metrics
            generated_ids = self.forward(data,'generate_ids')
            preds = [self.tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids]
            targets = [self.tokenizer.decode(t, skip_special_tokens=True, clean_up_tokenization_spaces=True)for t in data['target_ids']]
            sources = [self.tokenizer.decode(s, skip_special_tokens=True, clean_up_tokenization_spaces=True)for s in data['source_ids']]
            mean_batch_rouge_metrics = self.calc_batch_rouge(sources,targets, preds)
            self.df_performance_test_batch = self.df_performance_test_batch.append(pd.Series([mean_batch_rouge_metrics['rouge1_fmeasure_batch'],mean_batch_rouge_metrics['rouge2_fmeasure_batch'],mean_batch_rouge_metrics['rougeL_fmeasure_batch']], index=self.df_performance_test_batch.columns ), ignore_index=True)



            return mean_batch_rouge_metrics
            
            
        if self.sdk_step == "Deployment":
            data = batch
        
            # loss
            loss = self.forward(data,'loss')

            # metrics
            generated_ids = self.forward(data,'generate_ids')
            preds = [self.tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True) for g in generated_ids]
            not_apply_list = ['N/A'] * len(preds)
            sources = [self.tokenizer.decode(s, skip_special_tokens=True, clean_up_tokenization_spaces=True)for s in data['source_ids']]
            mean_batch_rouge_metrics = self.calc_batch_rouge(sources,not_apply_list, preds)
            
            return None
            
        return retorno

    def test_epoch_end(self, outputs):
        if not outputs: return {}
        
        epoch_mean = self.calc_epoch_mean(outputs)
        
        self.df_performance_test_epoch = self.df_performance_test_epoch.append(pd.Series([epoch_mean['avg_rouge1_fmeasure'],epoch_mean['avg_rouge2_fmeasure'],epoch_mean['avg_rougeL_fmeasure']], index=self.df_performance_test_epoch.columns ), ignore_index=True)

        return epoch_mean

    def calc_batch_rouge(self,sources,targets, preds,inference_step=None):
        #Rouge Metric Lists
        r1_p_list = []
        r1_r_list = []
        r1_f_list = []
        r2_p_list = []
        r2_r_list = []
        r2_f_list = []
        rL_p_list = []
        rL_r_list = []
        rL_f_list = []


        #Contructiong dataframe 
        for s,t,p in zip(sources,targets,preds):
            
            if self.sdk_step == "Experiment":
                rouge_metrics = self.rouge_metric.score(t,p)
                #rouge1
                r1_p = rouge_metrics['rouge1'].precision
                r1_p_list.append(r1_p)
                r1_r = rouge_metrics['rouge1'].recall
                r1_r_list.append(r1_r)
                r1_f = rouge_metrics['rouge1'].fmeasure
                r1_f_list.append(r1_f)
                #rouge2
                r2_p = rouge_metrics['rouge2'].precision
                r2_p_list.append(r2_p)
                r2_r = rouge_metrics['rouge2'].recall
                r2_r_list.append(r2_r)
                r2_f = rouge_metrics['rouge2'].fmeasure
                r2_f_list.append(r2_f)
                #rougeL
                rL_p = rouge_metrics['rougeL'].precision
                rL_p_list.append(rL_p)
                rL_r = rouge_metrics['rougeL'].recall
                rL_r_list.append(rL_r)
                rL_f = rouge_metrics['rougeL'].fmeasure
                rL_f_list.append(rL_f)
            
                #append dataframe
                if inference_step == 'valid':
                    self.df_valid = self.df_valid.append(pd.Series([s,t,p,r1_p,r1_r,r1_f,r2_p,r2_r,r2_f,rL_p,rL_r,rL_f], index=self.df_valid.columns), ignore_index=True)

                if inference_step == 'test':
                    self.df_test = self.df_test.append(pd.Series([s,t,p,r1_p,r1_r,r1_f,r2_p,r2_r,r2_f,rL_p,rL_r,rL_f], index=self.df_test.columns), ignore_index=True)
        
            if self.sdk_step == "Deployment":
                #append dataframe
                if inference_step == 'valid':
                    self.df_valid = self.df_valid.append(pd.Series([s,t,p,t,t,t,t,t,t,t,t,t], index=self.df_valid.columns), ignore_index=True)

                if inference_step == 'test':
                    self.df_test = self.df_test.append(pd.Series([s,t,p,t,t,t,t,t,t,t,t,t], index=self.df_test.columns), ignore_index=True)

                
        if self.sdk_step == "Experiment":                
            retorno = {'rouge1_precision_batch': np.mean(r1_p_list),
                                        'rouge1_recall_batch': np.mean(r1_r_list),
                                        'rouge1_fmeasure_batch': np.mean(r1_f_list),
                                        'rouge2_precision_batch': np.mean(r2_p_list),
                                        'rouge2_recall_batch': np.mean(r2_r_list),
                                        'rouge2_fmeasure_batch': np.mean(r2_f_list),
                                        'rougeL_precision_batch': np.mean(rL_p_list),
                                        'rougeL_recall_batch': np.mean(rL_r_list),
                                        'rougeL_fmeasure_batch': np.mean(rL_f_list)}
        
        if self.sdk_step == "Deployment":
            retorno = None

        return retorno
    
    def calc_epoch_mean(self, outputs):
        #loss
        avg_loss = torch.stack([x['loss'] for x in outputs]).mean().item()
        #rouge1
        avg_rouge1_precision = np.mean([x['rouge1_precision_batch'] for x in outputs])
        avg_rouge1_recall = np.mean([x['rouge1_recall_batch'] for x in outputs])
        avg_rouge1_fmeasure = np.mean([x['rouge1_fmeasure_batch'] for x in outputs])
        #rouge2
        avg_rouge2_precision = np.mean([x['rouge2_precision_batch'] for x in outputs])
        avg_rouge2_recall = np.mean([x['rouge2_recall_batch'] for x in outputs])
        avg_rouge2_fmeasure = np.mean([x['rouge2_fmeasure_batch'] for x in outputs])
        #rouge3
        avg_rougeL_precision = np.mean([x['rougeL_precision_batch'] for x in outputs])
        avg_rougeL_recall = np.mean([x['rougeL_recall_batch'] for x in outputs])
        avg_rougeL_fmeasure = np.mean([x['rougeL_fmeasure_batch'] for x in outputs])


        epoch_mean = {  'avg_loss':avg_loss,
                        'avg_rouge1_precision': avg_rouge1_precision,
                        'avg_rouge1_recall': avg_rouge1_recall,
                        'avg_rouge1_fmeasure': avg_rouge1_fmeasure,
                        'avg_rouge2_precision': avg_rouge2_precision,
                        'avg_rouge2_recall': avg_rouge2_recall,
                        'avg_rouge2_fmeasure': avg_rouge2_fmeasure,
                        'avg_rougeL_precision': avg_rougeL_precision,
                        'avg_rougeL_recall': avg_rougeL_recall,
                        'avg_rougeL_fmeasure': avg_rougeL_fmeasure
                    }
          
        return epoch_mean
    
    def configure_optimizers(self):
        return torch.optim.Adam(params = self.parameters(),lr=self.learning_rate)

    def merge_dicts(dict1, dict2): 
        res = {**dict1, **dict2} 
        return res
    
    def gpu_mem_restore(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            try:
                return func(*args, **kwargs)
            except:
                type, val, tb = sys.exc_info()
                traceback.clear_frames(tb)
                raise type(val).with_traceback(tb) from None
        return wrapper
    @gpu_mem_restore
    def train_dataloader(self):
        shuffle = False if self.overfit else True
        return DataLoader(self.train_dataset, batch_size=self.batch_size, shuffle=shuffle,num_workers=cpu_count())
    @gpu_mem_restore
    def val_dataloader(self):
        return DataLoader(self.valid_dataset, batch_size=self.batch_size,shuffle=False, num_workers=cpu_count())    

    @gpu_mem_restore
    def test_dataloader(self):
        return DataLoader(self.test_dataset, batch_size=self.batch_size,shuffle=False, num_workers=cpu_count())
%run Model_Lightning.py<jupyter_output><empty_output><jupyter_text>Parâmetros do Modelo<jupyter_code>all_data = [X_train,y_train,X_valid,y_valid,X_test,y_test]
hyperparams = {'learning_rate':learning_rate,'batch_size':batch_size,'max_epochs':max_epochs,'accumulate_grad_batches':accumulate_grad_batches,'max_length':max_length,'summary_len':summary_len,'num_beams':num_beams,'repetition_penalty':repetition_penalty,'length_penalty':length_penalty} 
model_parameters = {'model_name':model_name,'tokenizer':tokenizer}
dataset_infos = {'all_data':all_data,'CustomDataset':MyDataset}
extra_infos = {'overfit':False,'sampler':False}
model = T5SummarizerFinetuner(hyperparams=hyperparams,
                       model_parameters=model_parameters,
                       dataset_infos=dataset_infos,
                       extra_infos = extra_infos)  
sum([torch.tensor(x.size()).prod() for x in model.parameters() if x.requires_grad]) # trainable parameters
trainer = pl.Trainer(gpus=0, 
                     checkpoint_callback=False,  # Disable checkpoint saving.
                     fast_dev_run=True)
trainer.fit(model)
trainer.test(model)<jupyter_output><empty_output><jupyter_text>Recuperando ou treinando o modelo<jupyter_code>from pytorch_lightning.callbacks import ModelCheckpoint
import os
checkpoint_path = cwd + '/epoch=199.ckpt'
checkpoint_dir = os.path.dirname(os.path.abspath(checkpoint_path))
print(f'Files in {checkpoint_dir}: {os.listdir(checkpoint_dir)}')
print(f'Saving checkpoints to {checkpoint_dir}')
checkpoint_callback = ModelCheckpoint(filepath=checkpoint_dir, save_top_k=-1,monitor="val_acc")  # Keeps all checkpoints.

resume_from_checkpoint = None
if os.path.exists(checkpoint_path):
    print(f'Restoring checkpoint: {checkpoint_path}')
    resume_from_checkpoint = checkpoint_path

trainer = pl.Trainer(gpus=0,
                     max_epochs=max_epochs,
                     check_val_every_n_epoch=100,
                     profiler=True,
                     checkpoint_callback=checkpoint_callback,
                     progress_bar_refresh_rate=100,
                     resume_from_checkpoint=resume_from_checkpoint)

model = GloveFinetuner(hyperparams=hyperparams,
                       model_parameters=model_parameters,
                       dataset_infos=dataset_infos,
                       extra_infos = extra_infos) 

trainer.fit(model)
trainer.test(model)<jupyter_output><empty_output><jupyter_text>Visualizando resultados<jupyter_code>model.df_valid
model.df_test<jupyter_output><empty_output><jupyter_text>Visualização do desempenhoA [**Matriz de Confusão**](https://en.wikipedia.org/wiki/Confusion_matrix) (Confusion Matrix) é uma tabela que nos permite a visualização do desempenho de um algoritmo de classificação. É extremamente útil para mensurar [Accuracy](https://en.wikipedia.org/wiki/Accuracy_and_precisionIn_binary_classification), [Recall, Precision, and F-measure](https://en.wikipedia.org/wiki/Precision_and_recall).<jupyter_code>from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_recall_fscore_support
from sklearn.metrics import accuracy_score

# uses the model to make predictions on the Test Dataset
y_pred = list(model.df_test['PREDICTED_CODE']) 
y_prob =  [[a,b] for a,b in zip(list(model.df_test['POS_PROBA']),list(model.df_test['NEG_PROBA']))]

# computes confusion matrix
labels = np.unique(y)
data = confusion_matrix(y_test, y_pred, labels=labels)

# computes precision, recall, f1-score, support (for multiclass classification problem) and accuracy
if len(labels) > 2:
    # multiclass classification
    p, r, f1, s = precision_recall_fscore_support(y_test, y_pred,
                                                  labels=labels,
                                                  average=None)
    
    commom_metrics = pd.DataFrame(data=zip(p, r, f1, s),columns=['Precision','Recall','F1-Score','Support']) 
    
    average_options = ('micro', 'macro', 'weighted')
    for average in average_options:
        if average.startswith('micro'):
            line_heading = 'accuracy'
        else:
            line_heading = average + ' avg'

        # compute averages with specified averaging method
        avg_p, avg_r, avg_f1, _ = precision_recall_fscore_support(
            y_test, y_pred, labels=labels,
            average=average)
        avg = pd.Series({'Precision':avg_p,  'Recall':avg_r,  'F1-Score':avg_f1,  'Support':np.sum(s)},name=line_heading)
        commom_metrics = commom_metrics.append(avg)
else:
    # binary classification
    p, r, f1, _ = precision_recall_fscore_support(y_test, y_pred,
                                                  average='binary')
    accuracy=accuracy_score(y_test, y_pred)
    commom_metrics = pd.DataFrame(data={'Precision':p,'Recall':r,'F1-Score':f1,'Accuracy':accuracy},index=[1])

# puts matrix in pandas.DataFrame for better format
labels = label_encoder.inverse_transform(labels)
confusion_matrix = pd.DataFrame(data, columns=labels, index=labels)

# add correct index labels to commom_metrics DataFrame (for multiclass classification)
if len(labels)>2:
    as_list = commom_metrics.index.tolist()
    as_list[0:len(labels)] = labels
    commom_metrics.index = as_list
confusion_matrix
commom_metrics<jupyter_output><empty_output><jupyter_text>Salva métricasUtiliza a função `save_metrics` do [SDK da PlatIAgro](https://platiagro.github.io/sdk/) para salvar métricas. Por exemplo: `accuracy`, `precision`, `r2_score`, `custom_score` etc.<jupyter_code>from platiagro import save_metrics

save_metrics(confusion_matrix=confusion_matrix,commom_metrics=commom_metrics)<jupyter_output><empty_output><jupyter_text>Salva figurasUtiliza a função `save_figures` do [SDK da PlatIAgro](https://platiagro.github.io/sdk/) para salvar figuras do [matplotlib](https://matplotlib.org/3.2.1/gallery/index.html). A avaliação do desempenho do modelo pode ser feita por meio da análise da [Curva ROC (ROC)](https://pt.wikipedia.org/wiki/Caracter%C3%ADstica_de_Opera%C3%A7%C3%A3o_do_Receptor).  Esse gráfico permite avaliar a performance de um classificador binário para diferentes pontos de cortes. A métrica [AUC (Area under curve)](https://en.wikipedia.org/wiki/Receiver_operating_characteristicArea_under_the_curve) também é calculada e indicada na legenda do gráfico.Se a variável resposta tiver mais de duas categorias, o cálculo da curva ROC e AUC é feito utilizando o algoritmo [one-vs-rest](https://scikit-learn.org/stable/modules/model_evaluation.htmlroc-metrics), ou seja, calcula-se a curva ROC e AUC de cada classe em relação ao restante.<jupyter_code>from matplotlib.pyplot import cm
#from platiagro import save_figure
#from platiagro import list_figures
from sklearn.metrics import roc_curve, auc
from sklearn import preprocessing
import matplotlib.pyplot as plt

y_pred = list(model.df_test['PREDICTED_CODE']) 
y_prob =  [[a,b] for a,b in zip(list(model.df_test['INC_PROBA']),list(model.df_test['REQ_PROBA']))]

def plot_roc_curve(y_test,y_prob,labels):
    n_classes = len(labels)
    y_test = list(y_test)
    y_prob=np.array(y_prob)

    if n_classes == 2:
        # Compute ROC curve 
        fpr, tpr, _ = roc_curve(y_test, y_prob[:, 1])
        roc_auc = auc(fpr, tpr)  
        
        # Plot ROC Curve
        plt.figure()
        lw = 2
        plt.plot(fpr, tpr, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)' % roc_auc)
        plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
        plt.xlim([-0.01, 1.0])
        plt.ylim([0.0, 1.05])
        plt.xlabel('False Positive Rate')
        plt.ylabel('True Positive Rate')
        plt.title('ROC Curve')
        plt.legend(loc="lower right")
        plt.show()
        
    else:  
        # Binarize the output
        lb = preprocessing.LabelBinarizer()
        y_test_bin = lb.fit_transform(y_test)

        # Compute ROC curve for each class
        fpr = dict()
        tpr = dict()
        roc_auc = dict()  

        for i in range(n_classes):
            fpr[i], tpr[i], _ = roc_curve(y_test_bin[:, i], y_prob[:, i])
            roc_auc[i] = auc(fpr[i], tpr[i])
        
        color=cm.rainbow(np.linspace(0,1,n_classes+1))
        plt.figure()
        lw = 2
        plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
        plt.xlim([-0.01, 1.0])
        plt.ylim([0.0, 1.05])
        plt.xlabel('False Positive Rate')
        plt.ylabel('True Positive Rate')
        
        for i,c in zip(range(n_classes),color):                   
            plt.plot(fpr[i], tpr[i], color=c,
             lw=lw, label='ROC curve - Class %s (area = %0.2f)' % (labels[i] ,roc_auc[i]))
            plt.title('ROC Curve One-vs-Rest')
            plt.legend(loc="lower right")
        
        plt.show()

#funciona apenas quando se treina o modelo, quando recupera os pesos aparentemente não está
def performance_visualization(loss_list, acc_list,epoch_or_batch="Epoch",step = "Train"):

    x = range(len(loss_list))
    plt.title(step + ' Loss Performance')
    
    # Loss plot
    plt.subplot(1, 2, 1)
    plt.xlabel(epoch_or_batch)
    plt.ylabel("Loss")
    plt.plot(x, loss_list, '-')
    plt.title(step + ' Loss Performance')

    # Accuracy plot
    plt.subplot(1, 2, 2)
    plt.xlabel(epoch_or_batch)
    plt.ylabel("Accuracy")
    plt.plot(x, acc_list, '-')
    plt.title(step + '  Acc Performance')

    plt.subplots_adjust(wspace=0.4)
    plt.suptitle(epoch_or_batch + ' - ' + step)
    plt.show()


plot_roc_curve(y_test,y_prob,labels)
performance_visualization(model.df_performance_train_batch['train_batch_loss'].to_numpy(), model.df_performance_train_batch['train_batch_acc'].to_numpy(),epoch_or_batch="Batch",step = "Train")
performance_visualization(model.df_performance_train_epoch['train_epoch_loss'].to_numpy(), model.df_performance_train_epoch['train_epoch_acc'].to_numpy(),epoch_or_batch="Epoch",step = "Train")
performance_visualization(model.df_performance_valid_batch['valid_batch_loss'].to_numpy(), model.df_performance_valid_batch['valid_batch_acc'].to_numpy(),epoch_or_batch="Batch",step = "Valid")
performance_visualization(model.df_performance_valid_epoch['valid_epoch_loss'].to_numpy(), model.df_performance_valid_epoch['valid_epoch_acc'].to_numpy(),epoch_or_batch="Epoch",step = "Valid")<jupyter_output>DEBUG:matplotlib.font_manager:findfont: Matching sans\-serif:style=normal:variant=normal:weight=normal:stretch=normal:size=12.0.
DEBUG:matplotlib.font_manager:findfont: score(<Font 'cmss10' (cmss10.ttf) normal normal 400 normal>) = 10.05
DEBUG:matplotlib.font_manager:findfont: score(<Font 'STIXNonUnicode' (STIXNonUni.ttf) normal normal 400 normal>) = 10.05
DEBUG:matplotlib.font_manager:findfont: score(<Font 'STIXGeneral' (STIXGeneralBol.ttf) normal normal 700 normal>) = 10.335
DEBUG:matplotlib.font_manager:findfont: score(<Font 'STIXNonUnicode' (STIXNonUniBol.ttf) normal normal 700 normal>) = 10.335
DEBUG:matplotlib.font_manager:findfont: score(<Font 'STIXGeneral' (STIXGeneral.ttf) normal normal 400 normal>) = 10.05
DEBUG:matplotlib.font_manager:findfont: score(<Font 'DejaVu Sans Display' (DejaVuSansDisplay.ttf) normal normal 400 normal>) = 10.05
DEBUG:matplotlib.font_manager:findfont: score(<Font 'STIXNonUnicode' (STIXNonUniIta.ttf) italic normal 400 normal>) = 11.05
DEBUG:matplotlib.[...]<jupyter_text>Salva modelo e outros artefatos Modelo Pytorch Lightining<jupyter_code>file_name = 'pytorch_model.pt'
torch.save(model.state_dict(), f'/tmp/data/{file_name}')<jupyter_output><empty_output><jupyter_text>Artefatos<jupyter_code>import pickle
dataset_infos = {'all_data':all_data}
deployment_infos = {'columns':columns,'X_test':X_test,'glove_vocab':glove_vocab}
artifacts = {'hyperparams':hyperparams,'model_parameters':model_parameters,'dataset_infos':dataset_infos,'extra_infos':extra_infos,'deployment_infos':deployment_infos}
file_name = 'artifacts.p'
pickle.dump(artifacts, open(f'/tmp/data/{file_name}', "wb" ))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10830095932767779, 'fraction_numerical': 0.01496471894077539, 'mean_word_length': 3.5379564669904657, 'pattern_counts': {'":': 9, '<': 116, '<?xml version=': 0, '>': 120, 'https://': 20, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '136852', 'license': 'Apache-2.0', 'n_tokens_mistral': 17222, 'n_tokens_neox': 16569, 'n_words': 3228, 'path': 'samples/nlp-t5-summarizer/Experiment.ipynb', 'repo_name': 'Matheus158257/projects', 'size': '50452'}