<jupyter_start><jupyter_text>Zero-shot classification using the embeddingsIn this notebook we will classify the sentiment of reviews using embeddings and zero labeled data! The dataset is created in the [Obtain_dataset Notebook](Obtain_dataset.ipynb).We'll define positive sentiment to be 4 and 5-star reviews, and negative sentiment to be 1 and 2-star reviews. 3-star reviews are considered neutral and we won't use them for this example.We will perform zero-shot classification by embedding descriptions of each class and then comparing new samples to those class embeddings.<jupyter_code>import pandas as pd
import numpy as np

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, accuracy_score

df = pd.read_csv('output/embedded_1k_reviews.csv')
df['babbage_similarity'] = df.babbage_similarity.apply(eval).apply(np.array)
df['babbage_search'] = df.babbage_search.apply(eval).apply(np.array)

df= df[df.Score!=3]
df['sentiment'] = df.Score.replace({1:'negative', 2:'negative', 4:'positive', 5:'positive'})<jupyter_output><empty_output><jupyter_text>Zero-Shot ClassificationTo perform zero shot classification, we want to predict labels for our samples without any training. To do this, we can simply embed short descriptions of each label, such as positive and negative, and then compare the cosine distance between embeddings of samples and label descriptions. The highest similarity label to the sample input is the predicted label. We can also define a prediction score to be the difference between the cosine distance to the positive and to the negative label. This score can be used for plotting a precision-recall curve, which can be used to select a different tradeoff between precision and recall, by selecting a different threshold.<jupyter_code>from openai.embeddings_utils import cosine_similarity, get_embedding
from sklearn.metrics import PrecisionRecallDisplay

def evaluate_emeddings_approach(
    labels = ['negative', 'positive'], 
    engine = 'babbage-similarity',
):
    label_embeddings = [get_embedding(label, engine=engine) for label in labels]

    def label_score(review_embedding, label_embeddings):
        return cosine_similarity(review_embedding, label_embeddings[1]) - cosine_similarity(review_embedding, label_embeddings[0])

    engine_col_name = engine.replace('-','_').replace('_query','')
    probas = df[engine_col_name].apply(lambda x: label_score(x, label_embeddings))
    preds = probas.apply(lambda x: 'positive' if x>0 else 'negative')

    report = classification_report(df.sentiment, preds)
    print(report)

    display = PrecisionRecallDisplay.from_predictions(df.sentiment, probas, pos_label='positive')
    _ = display.ax_.set_title("2-class Precision-Recall curve")

evaluate_emeddings_approach(labels=['negative', 'positive'], engine='babbage-similarity')<jupyter_output>precision    recall  f1-score   support

    negative       0.67      0.88      0.76       136
    positive       0.98      0.93      0.95       789

    accuracy                           0.92       925
   macro avg       0.82      0.90      0.86       925
weighted avg       0.93      0.92      0.92       925<jupyter_text>We can see that this classifier already performs extremely well. We used similarity embeddings, and the simplest possible label name. Let's try to improve on this by using more descriptive label names, and search embeddings.<jupyter_code>evaluate_emeddings_approach(labels=['An Amazon review with a negative sentiment.', 'An Amazon review with a positive sentiment.'], engine='babbage-similarity')<jupyter_output>precision    recall  f1-score   support

    negative       0.65      0.93      0.76       136
    positive       0.99      0.91      0.95       789

    accuracy                           0.92       925
   macro avg       0.82      0.92      0.86       925
weighted avg       0.94      0.92      0.92       925<jupyter_text>Using the search embeddings and descriptive names leads to an additional improvement in performance.<jupyter_code>evaluate_emeddings_approach(labels=['An Amazon review with a negative sentiment.', 'An Amazon review with a positive sentiment.'], engine='babbage-search-query')<jupyter_output>precision    recall  f1-score   support

    negative       0.77      0.79      0.78       136
    positive       0.96      0.96      0.96       789

    accuracy                           0.94       925
   macro avg       0.87      0.88      0.87       925
weighted avg       0.94      0.94      0.94       925
{'fraction_non_alphanumeric': 0.07268770402611534, 'fraction_numerical': 0.03917301414581066, 'mean_word_length': 3.3813155386081983, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '236147', 'license': 'MIT', 'n_tokens_mistral': 1383, 'n_tokens_neox': 1256, 'n_words': 438, 'path': 'examples/embeddings/Zero-shot_classification.ipynb', 'repo_name': 'filipeabperes/openai-python', 'size': '4595'}