<jupyter_start><jupyter_text>Utilizing Trained GPT word Embeddings for classification. prerequisite1. [Training GPT](https://github.com/huggingface/transformers/tree/master/examples)2. [Training Sentencepiece](https://github.com/google/sentencepiece)<jupyter_code>import torch
import torchtext
import numpy as np
import sentencepiece as spm
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder
import pandas as pd
import sklearn
embeddings = torch.load("gpt_model.pt") # load the saved gpt trained model
embeddings =  embeddings["transformer.tokens_embed.weight"].cpu().numpy() # get the embedding layer
# load the sentence piece model since using sentence piece makes our model and problem language agnostic
sp = spm.SentencePieceProcessor()
sp.load("SentencePiece_Tokenizer.model")
def get_token_vector(sent):
    """This methods returns the embedding of a specific token by performing a lookup
    """
    return embeddings[sp.encode_as_ids(sent)]<jupyter_output><empty_output><jupyter_text>Suppose In a sentence we have 8 words so when use `get_token_vector` method we will get 8 vectors. To perform classification we need a fixed length sentence vector for that we use 3 different strategy to generate sentence vector.1. Mean - Take mean of all 8 word vectors2. Max - Take Max of all 8 word vectors3. Mean-Max - compute Mean and Max of 8 vectors and concatenate it.<jupyter_code>def get_sentence_vector(sent , pool_strategy="mean_max_concat"):
    """ pooling strategy
    mean,
    max,
    mean_max_concat
    """
    tokens  = get_token_vector(sent)
    if pool_strategy =="mean":
        return tokens.mean(axis=0)
    elif pool_strategy == "max":
        return tokens.max(axis=0)
    elif pool_strategy == "mean_max_concat":
        return np.concatenate([get_sentence_vector(sent,pool_s="mean"),get_sentence_vector(sent,pool_s="max")])
    
df = pd.read_csv("my_sentiment_data.csv") # load your sentiment csv data
# it should have two columns text, sentiment
# encode the target column.
encoder = LabelEncoder()
target = encoder.fit_transform(df.sentiment)
# get sentence vector for all text inputs
input = np.vstack(df.text.apply(get_sentence_vector).values)
# create train test split
X_train, X_test, y_train, y_test  = train_test_split(input,target,test_size=0.3,stratify=target)
clf = LogisticRegression(max_iter=500).fit(X_train, y_train)

prediction = clf.predict(X_test)

print(classification_report(y_test,prediction,target_names=encoder.classes_))

sklearn.metrics.plot_confusion_matrix(clf,X_test,y_test,display_labels=encoder.classes_,values_format="d")<jupyter_output>precision    recall  f1-score   support

    negative       0.59      0.64      0.62      1645
     neutral       0.57      0.09      0.15       930
    positive       0.55      0.73      0.63      1953

    accuracy                           0.57      4528
   macro avg       0.57      0.49      0.46      4528
weighted avg       0.57      0.57      0.53      4528
{'fraction_non_alphanumeric': 0.08193609617209743, 'fraction_numerical': 0.02847200253084467, 'mean_word_length': 4.083601286173633, 'pattern_counts': {'":': 3, '<': 8, '<?xml version=': 0, '>': 8, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '175747', 'license': 'CC0-1.0', 'n_tokens_mistral': 1006, 'n_tokens_neox': 945, 'n_words': 268, 'path': 'NLP/6_Utilizing_Pretrained_Embeddings/Classification_using_GPT_word_embeddings.ipynb', 'repo_name': 'entiretydotai/Meetup-Content', 'size': '3161'}