Customizing embeddings

# Customizing embeddings

This notebook demonstrates one way to customize OpenAI embeddings to a particular task.

The input is training data in the form of [text_1, text_2, label] where label is +1 if the pairs are similar and -1 if the pairs are dissimilar.

The output is a matrix that you can use to multiply your embeddings. The product of this multiplication is a 'custom embedding' that will better emphasize aspects of the text relevant to your use case. In binary classification use cases, we've seen error rates drop by as much as 50%.

In the following example, I use 1,000 sentence pairs picked from the SNLI corpus. Each pair of sentences are logically entailed (i.e., one implies the other). These pairs are our positives (label = 1). We generate synthetic negatives by combining sentences from different pairs, which are presumed to not be logically entailed (label = -1).

For a clustering use case, you can generate positives by creating pairs from texts in the same clusters and generate negatives by creating pairs from sentences in different clusters.

With other data sets, we have seen decent improvement with as little as ~100 training examples. Of course, performance will be better with  more examples.

# 0. Imports

```python
# imports
from typing import List, Tuple  # for type hints

import numpy as np  # for manipulating arrays
import pandas as pd  # for manipulating data in dataframes
import pickle  # for saving the embeddings cache
import plotly.express as px  # for plots
import random  # for generating run IDs
from sklearn.model_selection import train_test_split  # for splitting train & test data
import torch  # for matrix optimization

from utils.embeddings_utils import get_embedding, cosine_similarity  # for embeddings

```

## 1. Inputs

Most inputs are here. The key things to change are where to load your datset from, where to save a cache of embeddings to, and which embedding engine you want to use.

Depending on how your data is formatted, you'll want to rewrite the process_input_data function.

```python
# input parameters
embedding_cache_path = "data/snli_embedding_cache.pkl"  # embeddings will be saved/loaded here
default_embedding_engine = "text-embedding-3-small"
num_pairs_to_embed = 1000  # 1000 is arbitrary
local_dataset_path = "data/snli_1.0_train_2k.csv"  # download from: https://nlp.stanford.edu/projects/snli/

def process_input_data(df: pd.DataFrame) -> pd.DataFrame:
    # you can customize this to preprocess your own dataset
    # output should be a dataframe with 3 columns: text_1, text_2, label (1 for similar, -1 for dissimilar)
    df["label"] = df["gold_label"]
    df = df[df["label"].isin(["entailment"])]
    df["label"] = df["label"].apply(lambda x: {"entailment": 1, "contradiction": -1}[x])
    df = df.rename(columns={"sentence1": "text_1", "sentence2": "text_2"})
    df = df[["text_1", "text_2", "label"]]
    df = df.head(num_pairs_to_embed)
    return df

```

## 2. Load and process input data

```python
# load data
df = pd.read_csv(local_dataset_path)

# process input data
df = process_input_data(df)  # this demonstrates training data containing only positives

# view data
df.head()

```

    /var/folders/r4/x3kdvs816995fnnph2gdpwp40000gn/T/ipykernel_17509/1977422881.py:13: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      df["label"] = df["label"].apply(lambda x: {"entailment": 1, "contradiction": -1}[x])

      text_1
      text_2
      label

      2
      A person on a horse jumps over a broken down a...
      A person is outdoors, on a horse.
      1

      4
      Children smiling and waving at camera
      There are children present
      1

      7
      A boy is jumping on skateboard in the middle o...
      The boy does a skateboarding trick.
      1

      14
      Two blond women are hugging one another.
      There are women showing affection.
      1

      17
      A few people in a restaurant setting, one of t...
      The diners are at a restaurant.
      1

## 3. Split data into training test sets

Note that it's important to split data into training and test sets *before* generating synethetic negatives or positives. You don't want any text strings in the training data to show up in the test data. If there's contamination, the test metrics will look better than they'll actually be in production.

```python
# split data into train and test sets
test_fraction = 0.5  # 0.5 is fairly arbitrary
random_seed = 123  # random seed is arbitrary, but is helpful in reproducibility
train_df, test_df = train_test_split(
    df, test_size=test_fraction, stratify=df["label"], random_state=random_seed
)
train_df.loc[:, "dataset"] = "train"
test_df.loc[:, "dataset"] = "test"

```

## 4. Generate synthetic negatives

This is another piece of the code that you will need to modify to match your use case.

If you have data with positives and negatives, you can skip this section.

If you have data with only positives, you can mostly keep it as is, where it generates negatives only.

If you have multiclass data, you will want to generate both positives and negatives. The positives can be pairs of text that share labels, and the negatives can be pairs of text that do not share labels.

The final output should be a dataframe with text pairs, where each pair is labeled -1 or 1.

```python
# generate negatives
def dataframe_of_negatives(dataframe_of_positives: pd.DataFrame) -> pd.DataFrame:
    """Return dataframe of negative pairs made by combining elements of positive pairs."""
    texts = set(dataframe_of_positives["text_1"].values) | set(
        dataframe_of_positives["text_2"].values
    )
    all_pairs = {(t1, t2) for t1 in texts for t2 in texts if t1 < t2}
    positive_pairs = set(
        tuple(text_pair)
        for text_pair in dataframe_of_positives[["text_1", "text_2"]].values
    )
    negative_pairs = all_pairs - positive_pairs
    df_of_negatives = pd.DataFrame(list(negative_pairs), columns=["text_1", "text_2"])
    df_of_negatives["label"] = -1
    return df_of_negatives

```

```python
negatives_per_positive = (
    1  # it will work at higher values too, but more data will be slower
)
# generate negatives for training dataset
train_df_negatives = dataframe_of_negatives(train_df)
train_df_negatives["dataset"] = "train"
# generate negatives for test dataset
test_df_negatives = dataframe_of_negatives(test_df)
test_df_negatives["dataset"] = "test"
# sample negatives and combine with positives
train_df = pd.concat(
    [
        train_df,
        train_df_negatives.sample(
            n=len(train_df) * negatives_per_positive, random_state=random_seed
        ),
    ]
)
test_df = pd.concat(
    [
        test_df,
        test_df_negatives.sample(
            n=len(test_df) * negatives_per_positive, random_state=random_seed
        ),
    ]
)

df = pd.concat([train_df, test_df])

```

## 5. Calculate embeddings and cosine similarities

Here, I create a cache to save the embeddings. This is handy so that you don't have to pay again if you want to run the code again.

```python
# establish a cache of embeddings to avoid recomputing
# cache is a dict of tuples (text, engine) -> embedding
try:
    with open(embedding_cache_path, "rb") as f:
        embedding_cache = pickle.load(f)
except FileNotFoundError:
    precomputed_embedding_cache_path = "https://cdn.openai.com/API/examples/data/snli_embedding_cache.pkl"
    embedding_cache = pd.read_pickle(precomputed_embedding_cache_path)

# this function will get embeddings from the cache and save them there afterward
def get_embedding_with_cache(
    text: str,
    engine: str = default_embedding_engine,
    embedding_cache: dict = embedding_cache,
    embedding_cache_path: str = embedding_cache_path,
) -> list:
    if (text, engine) not in embedding_cache.keys():
        # if not in cache, call API to get embedding
        embedding_cache[(text, engine)] = get_embedding(text, engine)
        # save embeddings cache to disk after each update
        with open(embedding_cache_path, "wb") as embedding_cache_file:
            pickle.dump(embedding_cache, embedding_cache_file)
    return embedding_cache[(text, engine)]

# create column of embeddings
for column in ["text_1", "text_2"]:
    df[f"{column}_embedding"] = df[column].apply(get_embedding_with_cache)

# create column of cosine similarity between embeddings
df["cosine_similarity"] = df.apply(
    lambda row: cosine_similarity(row["text_1_embedding"], row["text_2_embedding"]),
    axis=1,
)

```

## 6. Plot distribution of cosine similarity

Here we measure similarity of text using cosine similarity. In our experience, most distance functions (L1, L2, cosine similarity) all work about the same. Note that our embeddings are already normalized to length 1, so cosine similarity is equivalent to dot product.

The graphs show how much the overlap there is between the distribution of cosine similarities for similar and dissimilar pairs. If there is a high amount of overlap, that means there are some dissimilar pairs with greater cosine similarity than some similar pairs.

The accuracy I compute is the accuracy of a simple rule that predicts 'similar (1)' if the cosine similarity is above some threshold X and otherwise predicts 'dissimilar (0)'.

```python
# calculate accuracy (and its standard error) of predicting label=1 if similarity>x
# x is optimized by sweeping from -1 to 1 in steps of 0.01
def accuracy_and_se(cosine_similarity: float, labeled_similarity: int) -> Tuple[float]:
    accuracies = []
    for threshold_thousandths in range(-1000, 1000, 1):
        threshold = threshold_thousandths / 1000
        total = 0
        correct = 0
        for cs, ls in zip(cosine_similarity, labeled_similarity):
            total += 1
            if cs > threshold:
                prediction = 1
            else:
                prediction = -1
            if prediction == ls:
                correct += 1
        accuracy = correct / total
        accuracies.append(accuracy)
    a = max(accuracies)
    n = len(cosine_similarity)
    standard_error = (a * (1 - a) / n) ** 0.5  # standard error of binomial
    return a, standard_error

# check that training and test sets are balanced
px.histogram(
    df,
    x="cosine_similarity",
    color="label",
    barmode="overlay",
    width=500,
    facet_row="dataset",
).show()

for dataset in ["train", "test"]:
    data = df[df["dataset"] == dataset]
    a, se = accuracy_and_se(data["cosine_similarity"], data["label"])
    print(f"{dataset} accuracy: {a:0.1%} ± {1.96 * se:0.1%}")

```

    train accuracy: 89.1% ± 2.4%
    test accuracy: 88.8% ± 2.4%

## 7. Optimize the matrix using the training data provided

```python
def embedding_multiplied_by_matrix(
    embedding: List[float], matrix: torch.tensor
) -> np.array:
    embedding_tensor = torch.tensor(embedding).float()
    modified_embedding = embedding_tensor @ matrix
    modified_embedding = modified_embedding.detach().numpy()
    return modified_embedding

# compute custom embeddings and new cosine similarities
def apply_matrix_to_embeddings_dataframe(matrix: torch.tensor, df: pd.DataFrame):
    for column in ["text_1_embedding", "text_2_embedding"]:
        df[f"{column}_custom"] = df[column].apply(
            lambda x: embedding_multiplied_by_matrix(x, matrix)
        )
    df["cosine_similarity_custom"] = df.apply(
        lambda row: cosine_similarity(
            row["text_1_embedding_custom"], row["text_2_embedding_custom"]
        ),
        axis=1,
    )

```

```python
def optimize_matrix(
    modified_embedding_length: int = 2048,  # in my brief experimentation, bigger was better (2048 is length of babbage encoding)
    batch_size: int = 100,
    max_epochs: int = 100,
    learning_rate: float = 100.0,  # seemed to work best when similar to batch size - feel free to try a range of values
    dropout_fraction: float = 0.0,  # in my testing, dropout helped by a couple percentage points (definitely not necessary)
    df: pd.DataFrame = df,
    print_progress: bool = True,
    save_results: bool = True,
) -> torch.tensor:
    """Return matrix optimized to minimize loss on training data."""
    run_id = random.randint(0, 2 ** 31 - 1)  # (range is arbitrary)
    # convert from dataframe to torch tensors
    # e is for embedding, s for similarity label
    def tensors_from_dataframe(
        df: pd.DataFrame,
        embedding_column_1: str,
        embedding_column_2: str,
        similarity_label_column: str,
    ) -> Tuple[torch.tensor]:
        e1 = np.stack(np.array(df[embedding_column_1].values))
        e2 = np.stack(np.array(df[embedding_column_2].values))
        s = np.stack(np.array(df[similarity_label_column].astype("float").values))

        e1 = torch.from_numpy(e1).float()
        e2 = torch.from_numpy(e2).float()
        s = torch.from_numpy(s).float()

        return e1, e2, s

    e1_train, e2_train, s_train = tensors_from_dataframe(
        df[df["dataset"] == "train"], "text_1_embedding", "text_2_embedding", "label"
    )
    e1_test, e2_test, s_test = tensors_from_dataframe(
        df[df["dataset"] == "test"], "text_1_embedding", "text_2_embedding", "label"
    )

    # create dataset and loader
    dataset = torch.utils.data.TensorDataset(e1_train, e2_train, s_train)
    train_loader = torch.utils.data.DataLoader(
        dataset, batch_size=batch_size, shuffle=True
    )

    # define model (similarity of projected embeddings)
    def model(embedding_1, embedding_2, matrix, dropout_fraction=dropout_fraction):
        e1 = torch.nn.functional.dropout(embedding_1, p=dropout_fraction)
        e2 = torch.nn.functional.dropout(embedding_2, p=dropout_fraction)
        modified_embedding_1 = e1 @ matrix  # @ is matrix multiplication
        modified_embedding_2 = e2 @ matrix
        similarity = torch.nn.functional.cosine_similarity(
            modified_embedding_1, modified_embedding_2
        )
        return similarity

    # define loss function to minimize
    def mse_loss(predictions, targets):
        difference = predictions - targets
        return torch.sum(difference * difference) / difference.numel()

    # initialize projection matrix
    embedding_length = len(df["text_1_embedding"].values[0])
    matrix = torch.randn(
        embedding_length, modified_embedding_length, requires_grad=True
    )

    epochs, types, losses, accuracies, matrices = [], [], [], [], []
    for epoch in range(1, 1 + max_epochs):
        # iterate through training dataloader
        for a, b, actual_similarity in train_loader:
            # generate prediction
            predicted_similarity = model(a, b, matrix)
            # get loss and perform backpropagation
            loss = mse_loss(predicted_similarity, actual_similarity)
            loss.backward()
            # update the weights
            with torch.no_grad():
                matrix -= matrix.grad * learning_rate
                # set gradients to zero
                matrix.grad.zero_()
        # calculate test loss
        test_predictions = model(e1_test, e2_test, matrix)
        test_loss = mse_loss(test_predictions, s_test)

        # compute custom embeddings and new cosine similarities
        apply_matrix_to_embeddings_dataframe(matrix, df)

        # calculate test accuracy
        for dataset in ["train", "test"]:
            data = df[df["dataset"] == dataset]
            a, se = accuracy_and_se(data["cosine_similarity_custom"], data["label"])

            # record results of each epoch
            epochs.append(epoch)
            types.append(dataset)
            losses.append(loss.item() if dataset == "train" else test_loss.item())
            accuracies.append(a)
            matrices.append(matrix.detach().numpy())

            # optionally print accuracies
            if print_progress is True:
                print(
                    f"Epoch {epoch}/{max_epochs}: {dataset} accuracy: {a:0.1%} ± {1.96 * se:0.1%}"
                )

    data = pd.DataFrame(
        {"epoch": epochs, "type": types, "loss": losses, "accuracy": accuracies}
    )
    data["run_id"] = run_id
    data["modified_embedding_length"] = modified_embedding_length
    data["batch_size"] = batch_size
    data["max_epochs"] = max_epochs
    data["learning_rate"] = learning_rate
    data["dropout_fraction"] = dropout_fraction
    data[
        "matrix"
    ] = matrices  # saving every single matrix can get big; feel free to delete/change
    if save_results is True:
        data.to_csv(f"{run_id}_optimization_results.csv", index=False)

    return data

```

```python
# example hyperparameter search
# I recommend starting with max_epochs=10 while initially exploring
results = []
max_epochs = 30
dropout_fraction = 0.2
for batch_size, learning_rate in [(10, 10), (100, 100), (1000, 1000)]:
    result = optimize_matrix(
        batch_size=batch_size,
        learning_rate=learning_rate,
        max_epochs=max_epochs,
        dropout_fraction=dropout_fraction,
        save_results=False,
    )
    results.append(result)

```

    Epoch 1/30: train accuracy: 89.1% ± 2.4%
    Epoch 1/30: test accuracy: 88.4% ± 2.4%
    Epoch 2/30: train accuracy: 89.5% ± 2.3%
    Epoch 2/30: test accuracy: 88.8% ± 2.4%
    Epoch 3/30: train accuracy: 90.6% ± 2.2%
    Epoch 3/30: test accuracy: 89.3% ± 2.3%
    Epoch 4/30: train accuracy: 91.2% ± 2.2%
    Epoch 4/30: test accuracy: 89.7% ± 2.3%
    Epoch 5/30: train accuracy: 91.5% ± 2.1%
    Epoch 5/30: test accuracy: 90.0% ± 2.3%
    Epoch 6/30: train accuracy: 91.9% ± 2.1%
    Epoch 6/30: test accuracy: 90.4% ± 2.2%
    Epoch 7/30: train accuracy: 92.2% ± 2.0%
    Epoch 7/30: test accuracy: 90.7% ± 2.2%
    Epoch 8/30: train accuracy: 92.7% ± 2.0%
    Epoch 8/30: test accuracy: 90.9% ± 2.2%
    Epoch 9/30: train accuracy: 92.7% ± 2.0%
    Epoch 9/30: test accuracy: 91.0% ± 2.2%
    Epoch 10/30: train accuracy: 93.0% ± 1.9%
    Epoch 10/30: test accuracy: 91.6% ± 2.1%
    Epoch 11/30: train accuracy: 93.1% ± 1.9%
    Epoch 11/30: test accuracy: 91.8% ± 2.1%
    Epoch 12/30: train accuracy: 93.4% ± 1.9%
    Epoch 12/30: test accuracy: 92.1% ± 2.0%
    Epoch 13/30: train accuracy: 93.6% ± 1.9%
    Epoch 13/30: test accuracy: 92.4% ± 2.0%
    Epoch 14/30: train accuracy: 93.7% ± 1.8%
    Epoch 14/30: test accuracy: 92.7% ± 2.0%
    Epoch 15/30: train accuracy: 93.7% ± 1.8%
    Epoch 15/30: test accuracy: 92.7% ± 2.0%
    Epoch 16/30: train accuracy: 94.0% ± 1.8%
    Epoch 16/30: test accuracy: 93.0% ± 1.9%
    Epoch 17/30: train accuracy: 94.0% ± 1.8%
    Epoch 17/30: test accuracy: 93.0% ± 1.9%
    Epoch 18/30: train accuracy: 94.2% ± 1.8%
    Epoch 18/30: test accuracy: 93.1% ± 1.9%
    Epoch 19/30: train accuracy: 94.2% ± 1.8%
    Epoch 19/30: test accuracy: 93.1% ± 1.9%
    Epoch 20/30: train accuracy: 94.3% ± 1.8%
    Epoch 20/30: test accuracy: 93.0% ± 1.9%
    Epoch 21/30: train accuracy: 94.5% ± 1.7%
    Epoch 21/30: test accuracy: 93.1% ± 1.9%
    Epoch 22/30: train accuracy: 94.5% ± 1.7%
    Epoch 22/30: test accuracy: 93.3% ± 1.9%
    Epoch 23/30: train accuracy: 94.6% ± 1.7%
    Epoch 23/30: test accuracy: 93.3% ± 1.9%
    Epoch 24/30: train accuracy: 94.6% ± 1.7%
    Epoch 24/30: test accuracy: 93.3% ± 1.9%
    Epoch 25/30: train accuracy: 94.8% ± 1.7%
    Epoch 25/30: test accuracy: 93.3% ± 1.9%
    Epoch 26/30: train accuracy: 94.8% ± 1.7%
    Epoch 26/30: test accuracy: 93.4% ± 1.9%
    Epoch 27/30: train accuracy: 94.8% ± 1.7%
    Epoch 27/30: test accuracy: 93.4% ± 1.9%
    Epoch 28/30: train accuracy: 94.9% ± 1.7%
    Epoch 28/30: test accuracy: 93.4% ± 1.9%
    Epoch 29/30: train accuracy: 94.9% ± 1.7%
    Epoch 29/30: test accuracy: 93.4% ± 1.9%
    Epoch 30/30: train accuracy: 94.9% ± 1.7%
    Epoch 30/30: test accuracy: 93.3% ± 1.9%
    Epoch 1/30: train accuracy: 89.7% ± 2.3%
    Epoch 1/30: test accuracy: 89.1% ± 2.4%
    Epoch 2/30: train accuracy: 89.8% ± 2.3%
    Epoch 2/30: test accuracy: 89.9% ± 2.3%
    Epoch 3/30: train accuracy: 90.3% ± 2.2%
    Epoch 3/30: test accuracy: 90.0% ± 2.3%
    Epoch 4/30: train accuracy: 91.0% ± 2.2%
    Epoch 4/30: test accuracy: 90.3% ± 2.2%
    Epoch 5/30: train accuracy: 91.3% ± 2.1%
    Epoch 5/30: test accuracy: 90.3% ± 2.2%
    Epoch 6/30: train accuracy: 91.8% ± 2.1%
    Epoch 6/30: test accuracy: 90.4% ± 2.2%
    Epoch 7/30: train accuracy: 92.4% ± 2.0%
    Epoch 7/30: test accuracy: 91.0% ± 2.2%
    Epoch 8/30: train accuracy: 92.8% ± 2.0%
    Epoch 8/30: test accuracy: 91.3% ± 2.1%
    Epoch 9/30: train accuracy: 93.1% ± 1.9%
    Epoch 9/30: test accuracy: 91.6% ± 2.1%
    Epoch 10/30: train accuracy: 93.4% ± 1.9%
    Epoch 10/30: test accuracy: 91.9% ± 2.1%
    Epoch 11/30: train accuracy: 93.4% ± 1.9%
    Epoch 11/30: test accuracy: 91.8% ± 2.1%
    Epoch 12/30: train accuracy: 93.6% ± 1.9%
    Epoch 12/30: test accuracy: 92.1% ± 2.0%
    Epoch 13/30: train accuracy: 93.7% ± 1.8%
    Epoch 13/30: test accuracy: 92.4% ± 2.0%
    Epoch 14/30: train accuracy: 93.7% ± 1.8%
    Epoch 14/30: test accuracy: 92.5% ± 2.0%
    Epoch 15/30: train accuracy: 93.9% ± 1.8%
    Epoch 15/30: test accuracy: 92.8% ± 2.0%
    Epoch 16/30: train accuracy: 94.0% ± 1.8%
    Epoch 16/30: test accuracy: 92.8% ± 2.0%
    Epoch 17/30: train accuracy: 94.0% ± 1.8%
    Epoch 17/30: test accuracy: 92.8% ± 2.0%
    Epoch 18/30: train accuracy: 94.2% ± 1.8%
    Epoch 18/30: test accuracy: 92.8% ± 2.0%
    Epoch 19/30: train accuracy: 94.2% ± 1.8%
    Epoch 19/30: test accuracy: 92.8% ± 2.0%
    Epoch 20/30: train accuracy: 94.2% ± 1.8%
    Epoch 20/30: test accuracy: 93.1% ± 1.9%
    Epoch 21/30: train accuracy: 94.3% ± 1.8%
    Epoch 21/30: test accuracy: 93.3% ± 1.9%
    Epoch 22/30: train accuracy: 94.3% ± 1.8%
    Epoch 22/30: test accuracy: 93.3% ± 1.9%
    Epoch 23/30: train accuracy: 94.5% ± 1.7%
    Epoch 23/30: test accuracy: 93.3% ± 1.9%
    Epoch 24/30: train accuracy: 94.5% ± 1.7%
    Epoch 24/30: test accuracy: 93.3% ± 1.9%
    Epoch 25/30: train accuracy: 94.6% ± 1.7%
    Epoch 25/30: test accuracy: 93.4% ± 1.9%
    Epoch 26/30: train accuracy: 94.6% ± 1.7%
    Epoch 26/30: test accuracy: 93.3% ± 1.9%
    Epoch 27/30: train accuracy: 94.6% ± 1.7%
    Epoch 27/30: test accuracy: 93.4% ± 1.9%
    Epoch 28/30: train accuracy: 94.8% ± 1.7%
    Epoch 28/30: test accuracy: 93.4% ± 1.9%
    Epoch 29/30: train accuracy: 94.8% ± 1.7%
    Epoch 29/30: test accuracy: 93.3% ± 1.9%
    Epoch 30/30: train accuracy: 94.8% ± 1.7%
    Epoch 30/30: test accuracy: 93.4% ± 1.9%
    Epoch 1/30: train accuracy: 90.7% ± 2.2%
    Epoch 1/30: test accuracy: 89.9% ± 2.3%
    Epoch 2/30: train accuracy: 90.9% ± 2.2%
    Epoch 2/30: test accuracy: 90.3% ± 2.2%
    Epoch 3/30: train accuracy: 91.6% ± 2.1%
    Epoch 3/30: test accuracy: 90.3% ± 2.2%
    Epoch 4/30: train accuracy: 92.2% ± 2.0%
    Epoch 4/30: test accuracy: 90.7% ± 2.2%
    Epoch 5/30: train accuracy: 92.4% ± 2.0%
    Epoch 5/30: test accuracy: 91.3% ± 2.1%
    Epoch 6/30: train accuracy: 92.5% ± 2.0%
    Epoch 6/30: test accuracy: 91.8% ± 2.1%
    Epoch 7/30: train accuracy: 93.0% ± 1.9%
    Epoch 7/30: test accuracy: 92.2% ± 2.0%
    Epoch 8/30: train accuracy: 93.1% ± 1.9%
    Epoch 8/30: test accuracy: 92.7% ± 2.0%
    Epoch 9/30: train accuracy: 93.3% ± 1.9%
    Epoch 9/30: test accuracy: 92.5% ± 2.0%
    Epoch 10/30: train accuracy: 93.4% ± 1.9%
    Epoch 10/30: test accuracy: 92.7% ± 2.0%
    Epoch 11/30: train accuracy: 93.6% ± 1.9%
    Epoch 11/30: test accuracy: 92.8% ± 2.0%
    Epoch 12/30: train accuracy: 93.7% ± 1.8%
    Epoch 12/30: test accuracy: 92.8% ± 2.0%
    Epoch 13/30: train accuracy: 94.0% ± 1.8%
    Epoch 13/30: test accuracy: 93.0% ± 1.9%
    Epoch 14/30: train accuracy: 93.9% ± 1.8%
    Epoch 14/30: test accuracy: 93.0% ± 1.9%
    Epoch 15/30: train accuracy: 94.2% ± 1.8%
    Epoch 15/30: test accuracy: 93.0% ± 1.9%
    Epoch 16/30: train accuracy: 94.2% ± 1.8%
    Epoch 16/30: test accuracy: 93.0% ± 1.9%
    Epoch 17/30: train accuracy: 94.3% ± 1.8%
    Epoch 17/30: test accuracy: 93.0% ± 1.9%
    Epoch 18/30: train accuracy: 94.5% ± 1.7%
    Epoch 18/30: test accuracy: 93.1% ± 1.9%
    Epoch 19/30: train accuracy: 94.5% ± 1.7%
    Epoch 19/30: test accuracy: 93.1% ± 1.9%
    Epoch 20/30: train accuracy: 94.6% ± 1.7%
    Epoch 20/30: test accuracy: 93.3% ± 1.9%
    Epoch 21/30: train accuracy: 94.8% ± 1.7%
    Epoch 21/30: test accuracy: 93.3% ± 1.9%
    Epoch 22/30: train accuracy: 94.8% ± 1.7%
    Epoch 22/30: test accuracy: 93.4% ± 1.9%
    Epoch 23/30: train accuracy: 94.8% ± 1.7%
    Epoch 23/30: test accuracy: 93.4% ± 1.9%
    Epoch 24/30: train accuracy: 94.8% ± 1.7%
    Epoch 24/30: test accuracy: 93.4% ± 1.9%
    Epoch 25/30: train accuracy: 94.8% ± 1.7%
    Epoch 25/30: test accuracy: 93.4% ± 1.9%
    Epoch 26/30: train accuracy: 94.9% ± 1.7%
    Epoch 26/30: test accuracy: 93.6% ± 1.9%
    Epoch 27/30: train accuracy: 94.9% ± 1.7%
    Epoch 27/30: test accuracy: 93.6% ± 1.9%
    Epoch 28/30: train accuracy: 94.9% ± 1.7%
    Epoch 28/30: test accuracy: 93.6% ± 1.9%
    Epoch 29/30: train accuracy: 95.1% ± 1.6%
    Epoch 29/30: test accuracy: 93.6% ± 1.9%
    Epoch 30/30: train accuracy: 95.1% ± 1.6%
    Epoch 30/30: test accuracy: 93.6% ± 1.9%

```python
runs_df = pd.concat(results)

# plot training loss and test loss over time
px.line(
    runs_df,
    line_group="run_id",
    x="epoch",
    y="loss",
    color="type",
    hover_data=["batch_size", "learning_rate", "dropout_fraction"],
    facet_row="learning_rate",
    facet_col="batch_size",
    width=500,
).show()

# plot accuracy over time
px.line(
    runs_df,
    line_group="run_id",
    x="epoch",
    y="accuracy",
    color="type",
    hover_data=["batch_size", "learning_rate", "dropout_fraction"],
    facet_row="learning_rate",
    facet_col="batch_size",
    width=500,
).show()

```

## 8. Plot the before & after, showing the results of the best matrix found during training

The better the matrix is, the more cleanly it will separate the similar and dissimilar pairs.

```python
# apply result of best run to original data
best_run = runs_df.sort_values(by="accuracy", ascending=False).iloc[0]
best_matrix = best_run["matrix"]
apply_matrix_to_embeddings_dataframe(best_matrix, df)

```

```python
# plot similarity distribution BEFORE customization
px.histogram(
    df,
    x="cosine_similarity",
    color="label",
    barmode="overlay",
    width=500,
    facet_row="dataset",
).show()

test_df = df[df["dataset"] == "test"]
a, se = accuracy_and_se(test_df["cosine_similarity"], test_df["label"])
print(f"Test accuracy: {a:0.1%} ± {1.96 * se:0.1%}")

# plot similarity distribution AFTER customization
px.histogram(
    df,
    x="cosine_similarity_custom",
    color="label",
    barmode="overlay",
    width=500,
    facet_row="dataset",
).show()

a, se = accuracy_and_se(test_df["cosine_similarity_custom"], test_df["label"])
print(f"Test accuracy after customization: {a:0.1%} ± {1.96 * se:0.1%}")

```

    Test accuracy: 88.8% ± 2.4%

    Test accuracy after customization: 93.6% ± 1.9%

```python
best_matrix  # this is what you can multiply your embeddings by

```

    array([[-1.2566795e+00, -1.5297449e+00, -1.3271648e-01, ...,
            -1.2859761e+00, -5.3254390e-01,  4.8364732e-01],
           [-1.4826347e+00,  9.2656955e-02, -4.2437232e-01, ...,
             1.1872858e+00, -1.0831847e+00, -1.0683593e+00],
           [-2.2029283e+00, -1.9703420e+00,  3.1125939e-01, ...,
             2.2947595e+00,  5.5780332e-03, -6.0171342e-01],
           ...,
           [-1.1019799e-01,  1.3599515e+00, -4.7677776e-01, ...,
             6.5626711e-01,  7.2359240e-01,  3.0733588e+00],
           [ 1.6624762e-03,  4.2648423e-01, -1.1380885e+00, ...,
             8.7202555e-01,  9.3173909e-01, -1.6760436e+00],
           [ 7.7449006e-01,  4.9213606e-01,  3.5407653e-01, ...,
             1.3460466e+00, -1.9509128e-01,  7.7514690e-01]], dtype=float32)

```python

```


## User and product embeddings

We calculate user and product embeddings based on the training set, and evaluate the results on the unseen test set. We will evaluate the results by plotting the user and product similarity versus the review score. The dataset is created in the [Get_embeddings_from_dataset Notebook](Get_embeddings_from_dataset.ipynb).

### 1. Calculate user and product embeddings

We calculate these embeddings simply by averaging all the reviews about the same product or written by the same user within the training set.

```python
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from ast import literal_eval

df = pd.read_csv('data/fine_food_reviews_with_embeddings_1k.csv', index_col=0)  # note that you will need to generate this file to run the code below
df.head(2)
```

      ProductId
      UserId
      Score
      Summary
      Text
      combined
      n_tokens
      embedding

      0
      B003XPF9BO
      A3R7JR3FMEBXQB
      5
      where does one  start...and stop... with a tre...
      Wanted to save some to bring to my Chicago fam...
      Title: where does one  start...and stop... wit...
      52
      [0.03599238395690918, -0.02116263099014759, -0...

      297
      B003VXHGPK
      A21VWSCGW7UUAR
      4
      Good, but not Wolfgang Puck good
      Honestly, I have to admit that I expected a li...
      Title: Good, but not Wolfgang Puck good; Conte...
      178
      [-0.07042013108730316, -0.03175969794392586, -...

```python
df['babbage_similarity'] = df["embedding"].apply(literal_eval).apply(np.array)
X_train, X_test, y_train, y_test = train_test_split(df, df.Score, test_size = 0.2, random_state=42)

user_embeddings = X_train.groupby('UserId').babbage_similarity.apply(np.mean)
prod_embeddings = X_train.groupby('ProductId').babbage_similarity.apply(np.mean)
len(user_embeddings), len(prod_embeddings)

```

    (577, 706)

We can see that most of the users and products appear within the 50k examples only once.

### 2. Evaluate the embeddings

To evaluate the recommendations, we look at the similarity of the user and product embeddings amongst the reviews in the unseen test set. We calculate the cosine distance between the user and product embeddings, which gives us a similarity score between 0 and 1. We then normalize the scores to be evenly split between 0 and 1, by calculating the percentile of the similarity score amongst all predicted scores.

```python
from utils.embeddings_utils import cosine_similarity

# evaluate embeddings as recommendations on X_test
def evaluate_single_match(row):
    user_id = row.UserId
    product_id = row.ProductId
    try:
        user_embedding = user_embeddings[user_id]
        product_embedding = prod_embeddings[product_id]
        similarity = cosine_similarity(user_embedding, product_embedding)
        return similarity
    except Exception as e:
        return np.nan

X_test['cosine_similarity'] = X_test.apply(evaluate_single_match, axis=1)
X_test['percentile_cosine_similarity'] = X_test.cosine_similarity.rank(pct=True)

```

#### 2.1 Visualize cosine similarity by review score

We group the cosine similarity scores by the review score, and plot the distribution of cosine similarity scores for each review score.

```python
import matplotlib.pyplot as plt
import statsmodels.api as sm

correlation = X_test[['percentile_cosine_similarity', 'Score']].corr().values[0,1]
print('Correlation between user & vector similarity percentile metric and review number of stars (score): %.2f%%' % (100*correlation))

# boxplot of cosine similarity for each score
X_test.boxplot(column='percentile_cosine_similarity', by='Score')
plt.title('')
plt.show()
plt.close()

```

    Correlation between user & vector similarity percentile metric and review number of stars (score): 29.56%

![png](output_8_1.png)

We can observe a weak trend, showing that the higher the similarity score between the user and the product embedding, the higher the review score. Therefore, the user and product embeddings can weakly predict the review score - even before the user receives the product!

Because this signal works in a different way than the more commonly used collaborative filtering, it can act as an additional feature to slightly improve the performance on existing problems.


Kusto as a Vector database

# Kusto as a Vector database

[Azure Data Explorer aka Kusto](https://azure.microsoft.com/en-us/products/data-explorer) is a cloud-based data analytics service that enables users to perform advanced analytics on large datasets in real-time. It is particularly well-suited for handling large volumes of data, making it an excellent choice for storing and searching vectors.

Kusto supports a special data type called dynamic, which can store unstructured data such as arrays and properties bag. [Dynamic data type](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/scalar-data-types/dynamic) is perfect for storing vector values. You can further augment the vector value by storing metadata related to the original object as separate columns in your table.
Kusto also supports in-built function [series_cosine_similarity_fl](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/functions-library/series-cosine-similarity-fl) to perform vector similarity searches.

[Get started](https://aka.ms/kustofree) with Kusto for free.

![Kusto_Vector](./images/kusto_vector_db.png)

## Getting started with Kusto and Open AI embedding

### Demo Scenario

![Wiki_embeddings](./images/wiki_embeddings.png)

![semantic_search_flow](./images/semantic_search_user_flow.png)

If you’d like to try this demo, please follow the instructions in the [Notebook](Getting_started_with_kusto_and_openai_embeddings.ipynb).

It will allow you to -

1. Use precomputed embeddings created by OpenAI API.

2. Store the embeddings in Kusto.

3. Convert raw text query to an embedding with OpenAI API.

4. Use Kusto to perform cosine similarity search in the stored embeddings.


How to handle rate limits

# How to handle rate limits

When you call the OpenAI API repeatedly, you may encounter error messages that say `429: 'Too Many Requests'` or `RateLimitError`. These error messages come from exceeding the API's rate limits.

This guide shares tips for avoiding and handling rate limit errors.

To see an example script for throttling parallel requests to avoid rate limit errors, see [api_request_parallel_processor.py](https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py).

## Why rate limits exist

Rate limits are a common practice for APIs, and they're put in place for a few different reasons.

- First, they help protect against abuse or misuse of the API. For example, a malicious actor could flood the API with requests in an attempt to overload it or cause disruptions in service. By setting rate limits, OpenAI can prevent this kind of activity.
- Second, rate limits help ensure that everyone has fair access to the API. If one person or organization makes an excessive number of requests, it could bog down the API for everyone else. By throttling the number of requests that a single user can make, OpenAI ensures that everyone has an opportunity to use the API without experiencing slowdowns.
- Lastly, rate limits can help OpenAI manage the aggregate load on its infrastructure. If requests to the API increase dramatically, it could tax the servers and cause performance issues. By setting rate limits, OpenAI can help maintain a smooth and consistent experience for all users.

Although hitting rate limits can be frustrating, rate limits exist to protect the reliable operation of the API for its users.

## Default rate limits

Your rate limit and spending limit (quota) are automatically adjusted based on a number of factors. As your usage of the OpenAI API goes up and you successfully pay the bill, we automatically increase your usage tier. You can find specific information regarding rate limits using the resources below.

### Other rate limit resources

Read more about OpenAI's rate limits in these other resources:

- [Guide: Rate limits](https://platform.openai.com/docs/guides/rate-limits?context=tier-free)
- [Help Center: Is API usage subject to any rate limits?](https://help.openai.com/en/articles/5955598-is-api-usage-subject-to-any-rate-limits)
- [Help Center: How can I solve 429: 'Too Many Requests' errors?](https://help.openai.com/en/articles/5955604-how-can-i-solve-429-too-many-requests-errors)

### Requesting a rate limit increase

If you'd like your organization's rate limit increased, please visit your [Limits settings page](https://platform.openai.com/account/limits) to see how you can increase your usage tier

```python
import openai
import os

client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

## Example rate limit error

A rate limit error will occur when API requests are sent too quickly. If using the OpenAI Python library, they will look something like:

```
RateLimitError: Rate limit reached for default-codex in organization org-{id} on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues or if you’d like to request an increase.
```

Below is example code for triggering a rate limit error.

```python
# request a bunch of completions in a loop
for _ in range(100):
    client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": "Hello"}],
        max_tokens=10,
    )
```

## How to avoid rate limit errors

### Retrying with exponential backoff

One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached.

This approach has many benefits:

- Automatic retries means you can recover from rate limit errors without crashes or missing data
- Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail
- Adding random jitter to the delay helps retries from all hitting at the same time

Note that unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work.

Below are a few example solutions.

#### Example #1: Using the Tenacity library

[Tenacity](https://tenacity.readthedocs.io/en/latest/) is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.

To add exponential backoff to your requests, you can use the `tenacity.retry` [decorator](https://peps.python.org/pep-0318/). The following example uses the `tenacity.wait_random_exponential` function to add random exponential backoff to a request.

Note that the Tenacity library is a third-party tool, and OpenAI makes no guarantees about its reliability or security.

```python
from tenacity import (
    retry,
    stop_after_attempt,
    wait_random_exponential,
)  # for exponential backoff

@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
def completion_with_backoff(**kwargs):
    return client.chat.completions.create(**kwargs)

completion_with_backoff(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Once upon a time,"}])
```

    ChatCompletion(id='chatcmpl-8PAu6anX2JxQdYmJRzps38R8u0ZBC', choices=[Choice(finish_reason='stop', index=0, message=ChatCompletionMessage(content='in a small village nestled among green fields and rolling hills, there lived a kind-hearted and curious young girl named Lily. Lily was known for her bright smile and infectious laughter, bringing joy to everyone around her.\n\nOne sunny morning, as Lily played in the meadows, she stumbled upon a mysterious book tucked away beneath a tall oak tree. Intrigued, she picked it up and dusted off its weathered cover to reveal intricate golden patterns. Without hesitation, she opened it, discovering that its pages were filled with magical tales and enchanting adventures.\n\nAmong the stories she found, one particularly caught her attention—a tale of a long-lost treasure hidden deep within a mysterious forest. Legend had it that whoever found this hidden treasure would be granted one wish, no matter how big or small. Excited by the prospect of finding such treasure and fulfilling her wildest dreams, Lily decided to embark on a thrilling journey to the forest.\n\nGathering her courage, Lily told her parents about the magical book and her quest to find the hidden treasure. Though concerned for their daughter\'s safety, they couldn\'t help but admire her spirit and determination. They hugged her tightly and blessed her with love and luck, promising to await her return.\n\nEquipped with a map she found within the book, Lily ventured into the depths of the thick forest. The trees whispered tales of forgotten secrets, and the enchanted creatures hidden within watched her every step. But Lily remained undeterred, driven by her desire to discover what lay ahead.\n\nDays turned into weeks as Lily traversed through dense foliage, crossed swift rivers, and climbed treacherous mountains. She encountered mystical beings who offered guidance and protection along her perilous journey. With their help, she overcame countless obstacles and grew braver with each passing day.\n\nFinally, after what felt like an eternity, Lily reached the heart of the forest. There, beneath a jeweled waterfall, she found the long-lost treasure—a magnificent chest adorned with sparkling gemstones. Overwhelmed with excitement, she gently opened the chest to reveal a brilliant light that illuminated the forest.\n\nWithin the glow, a wise voice echoed, "You have proven your courage and pure heart, young Lily. Make your wish, and it shall be granted."\n\nLily thought deeply about her wish, realizing that her true treasure was the love and happiness she felt in her heart. Instead of making a wish for herself, she asked for the wellbeing and prosperity of her village, spreading joy and harmony to everyone living there.\n\nAs the light faded, Lily knew her quest was complete. She retraced her steps through the forest, returning home to find her village flourishing. Fields bloomed with vibrant flowers, and laughter filled the air.\n\nThe villagers greeted Lily with open arms, recognizing her selflessness and the magic she had brought into their lives. From that day forward, they told the tale of Lily\'s journey, celebrating her as a heroine who embodied the power of love, kindness, and the belief that true treasure lies within oneself.\n\nAnd so, the story of Lily became an everlasting legend, inspiring generations to follow their dreams, be selfless, and find the true treasures that lie within their hearts.', role='assistant', function_call=None, tool_calls=None))], created=1701010806, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=641, prompt_tokens=12, total_tokens=653))

#### Example #2: Using the backoff library

Another library that provides function decorators for backoff and retry is [backoff](https://pypi.org/project/backoff/).

Like Tenacity, the backoff library is a third-party tool, and OpenAI makes no guarantees about its reliability or security.

```python
import backoff  # for exponential backoff

@backoff.on_exception(backoff.expo, openai.RateLimitError)
def completions_with_backoff(**kwargs):
    return client.chat.completions.create(**kwargs)

completions_with_backoff(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Once upon a time,"}])

```

    ChatCompletion(id='chatcmpl-8PAwkg7Q9pPeAkvVuAZ8AyA108WhR', choices=[Choice(finish_reason='stop', index=0, message=ChatCompletionMessage(content="in a small village, there lived a young girl named Lily. She had fiery red hair, lively green eyes, and a spirit as wild as the rushing river nearby. Lily was known for her curious nature and her desire to explore the world beyond the village boundaries.\n\nOne day, while playing near the river, Lily spotted an injured bird nested on a branch. Its wing was broken, and it seemed unable to fly away. Lily's heart filled with sadness, and she knew she couldn't leave the bird alone.\n\nCarefully, she climbed up the tree and gently placed the bird inside her pocket. Lily brought it home and made a cozy bed for it in a small wooden box. She named the bird Ruby, after its shimmering red feathers.\n\nDays turned into weeks, and Ruby's wing slowly healed under Lily's constant care and attention. As they spent time together, a deep bond grew between them. Ruby would chirp happily whenever Lily approached, and she would spend hours talking to the bird, sharing stories of her adventures, dreams, and fears.\n\nOne evening, as Lily was about to go to bed, a peculiar thing happened. Ruby hopped out of his box and fluttered onto the windowsill. He turned to face Lily with his bright eyes and began to sing a beautiful melody.\n\nLily was astonished. Never before had she heard Ruby sing. The tune was so captivating that it filled the room and made the quiet night come alive. The magical music seemed to touch Lily's soul, awakening a deep sense of wonder and wanderlust within her.\n\nFilled with an undeniable urge to explore, Lily decided it was time to go on an adventure with her newfound friend, Ruby. She packed a small bag and bid farewell to her family and friends, promising to return one day.\n\nTogether, Lily and Ruby embarked on a grand journey, soaring across expansive skies, diving into lush forests, and exploring hidden caves. They encountered magnificent landscapes, unique creatures, and encountered kind-hearted individuals who shared their wisdom and stories.\n\nThroughout their journey, Ruby's song continued to inspire and guide them. It became a symbol of hope, reminding them to embrace bravery, follow their dreams, and always remain true to themselves.\n\nAs the years passed, Lily and Ruby traversed the world, weaving their stories into the tapestry of time. They became renowned for their extraordinary bond and the magic they shared with everyone they encountered.\n\nEventually, it was time for Lily to return to her village, a place eagerly awaiting her return. She had grown wise, learned many lessons, and gained a deeper understanding of herself and the world around her.\n\nWith Ruby perched on her shoulder, they descended upon the village like a ray of sunshine, bringing joy and wonder to every heart. Lily shared the wisdom she had acquired and inspired others to embrace their own adventures, no matter how big or small.\n\nAnd so, the tale of Lily and Ruby became legend, passed down from generation to generation. Their story reminded people to cherish the connections they make, to nurture their dreams, and to believe in the magic that lies within them.", role='assistant', function_call=None, tool_calls=None))], created=1701010970, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=621, prompt_tokens=12, total_tokens=633))

#### Example 3: Manual backoff implementation

If you don't want to use third-party libraries, you can implement your own backoff logic.

```python
# imports
import random
import time

# define a retry decorator
def retry_with_exponential_backoff(
    func,
    initial_delay: float = 1,
    exponential_base: float = 2,
    jitter: bool = True,
    max_retries: int = 10,
    errors: tuple = (openai.RateLimitError,),
):
    """Retry a function with exponential backoff."""

    def wrapper(*args, **kwargs):
        # Initialize variables
        num_retries = 0
        delay = initial_delay

        # Loop until a successful response or max_retries is hit or an exception is raised
        while True:
            try:
                return func(*args, **kwargs)

            # Retry on specified errors
            except errors as e:
                # Increment retries
                num_retries += 1

                # Check if max retries has been reached
                if num_retries > max_retries:
                    raise Exception(
                        f"Maximum number of retries ({max_retries}) exceeded."
                    )

                # Increment the delay
                delay *= exponential_base * (1 + jitter * random.random())

                # Sleep for the delay
                time.sleep(delay)

            # Raise exceptions for any errors not specified
            except Exception as e:
                raise e

    return wrapper

@retry_with_exponential_backoff
def completions_with_backoff(**kwargs):
    return client.chat.completions.create(**kwargs)

completions_with_backoff(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Once upon a time,"}])
```

    ChatCompletion(id='chatcmpl-8PAxGvV3GbLpnOoKSvJ00XCUdOglM', choices=[Choice(finish_reason='stop', index=0, message=ChatCompletionMessage(content="in a faraway kingdom, there lived a young princess named Aurora. She was known for her beauty, grace, and kind heart. Aurora's kingdom was filled with lush green meadows, towering mountains, and sparkling rivers. The princess loved spending time exploring the enchanting forests surrounding her castle.\n\nOne day, while Aurora was wandering through the woods, she stumbled upon a hidden clearing. At the center stood a majestic oak tree, its branches reaching towards the sky. Aurora approached the tree with curiosity, and as she got closer, she noticed a small door at its base.\n\nIntrigued, she gently pushed open the door and was amazed to find herself in a magical realm. The forest transformed into a breathtaking wonderland, with colorful flowers blooming in every direction and woodland creatures frolicking joyously. Aurora's eyes widened with wonder as she explored this extraordinary world.\n\nAs she explored further, Aurora came across a small cottage in the distance. Curiosity overcame her, and she cautiously approached the cottage. To her surprise, an elderly woman with twinkling eyes and a warm smile stood in the doorway, welcoming her inside.\n\nThe woman revealed herself to be a fairy named Luna. Luna informed Aurora that she had been chosen to undertake a quest that would bring harmony to both her kingdom and the mystical realm. Aurora, eager to help, listened intently as Luna explained that a powerful enchantress had cast a spell on the kingdom, causing darkness and despair to loom over the land.\n\nTo break the curse, Aurora had to embark on a journey to retrieve a magical crystal hidden deep within the heart of an ancient cave. Without hesitation, the princess agreed and bid farewell to Luna, promising to return victorious.\n\nWith newfound determination, Aurora set off on her quest. Along the way, she encountered numerous challenges and obstacles but never lost hope. She often drew strength from the enchanting woodland creatures who accompanied her on this journey, reminding her that she was not alone.\n\nAfter a long and arduous journey, Aurora reached the entrance of the ancient cave. Inside, she faced a series of tests that pushed her physical and emotional limits. With sheer determination and unwavering courage, she overcame each trial, paving her way to the crystal's resting place.\n\nAs Aurora held the crystal in her hands, its warmth spread through her body. The artifact contained unimaginable power that could shatter the enchantress's curse and restore light to her kingdom. Brimming with joy and newfound strength, she made her way back to Luna's cottage.\n\nUpon her return, Aurora and Luna performed a powerful ritual, using the crystal's magic to break the curse. Waves of light and color spread across the kingdom, banishing darkness and despair. The once-gray skies turned blue, and laughter filled the air once again. The kingdom rejoiced, thanking Princess Aurora for her bravery and selflessness.\n\nFrom that day forward, Aurora was hailed as a hero, not only in her kingdom but also in the mystical realm. She continued to be a beacon of hope and kindness, reminding everyone that true courage lies within, waiting to be awakened.\n\nAnd so, Princess Aurora's tale lived on as a timeless reminder that even in the darkest of times, there is always light and hope to be found.", role='assistant', function_call=None, tool_calls=None))], created=1701011002, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=657, prompt_tokens=12, total_tokens=669))

## How to maximize throughput of batch processing given rate limits

If you're processing real-time requests from users, backoff and retry is a great strategy to minimize latency while avoiding rate limit errors.

However, if you're processing large volumes of batch data, where throughput matters more than latency, there are a few other things you can do in addition to backoff and retry.

### Proactively adding delay between requests

If you are constantly hitting the rate limit, then backing off, then hitting the rate limit again, then backing off again, it's possible that a good fraction of your request budget will be 'wasted' on requests that need to be retried. This limits your processing throughput, given a fixed rate limit.

Here, one potential solution is to calculate your rate limit and add a delay equal to its reciprocal (e.g., if your rate limit 20 requests per minute, add a delay of 3–6 seconds to each request). This can help you operate near the rate limit ceiling without hitting it and incurring wasted requests.

#### Example of adding delay to a request

```python
# imports
import time

# Define a function that adds a delay to a Completion API call
def delayed_completion(delay_in_seconds: float = 1, **kwargs):
    """Delay a completion by a specified amount of time."""

    # Sleep for the delay
    time.sleep(delay_in_seconds)

    # Call the Completion API and return the result
    return client.chat.completions.create(**kwargs)

# Calculate the delay based on your rate limit
rate_limit_per_minute = 20
delay = 60.0 / rate_limit_per_minute

delayed_completion(
    delay_in_seconds=delay,
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Once upon a time,"}]
)

```

    ChatCompletion(id='chatcmpl-8PAyCR1axKsomV0e349XiCN1Z81pH', choices=[Choice(finish_reason='stop', index=0, message=ChatCompletionMessage(content="in a small village, there lived a young girl named Maya. Maya was known for her kindness and love for nature. She spent hours exploring the forests surrounding the village, admiring the vibrant flowers and talking to the animals.\n\nOne sunny day, as Maya was picking wildflowers, she stumbled upon a wounded blackbird with a broken wing. Feeling sorry for the bird, Maya gently picked it up and cradled it in her hands. She knew she had to help the bird, so she hurried back to her cottage.\n\nMaya set up a cozy nest for the blackbird and carefully splinted its wing. She fed it worms and berries, doing everything she could to nurse it back to health. Each day, she would sing lullabies and tell stories to keep the blackbird company. Slowly, the bird's wing healed, and before long, it was ready to fly again.\n\nOn a beautiful morning, Maya opened the window of her cottage and released the blackbird into the sky. As the bird soared into the air, Maya's heart filled with joy and gratitude. Little did she know, this act of kindness would change her life forever.\n\nThe following night, a mysterious glowing light illuminated Maya's room. Startled, she sat up and saw a magical creature standing before her. It was a fairy, tiny yet radiating warmth and light.\n\nThe fairy introduced herself as Luna, the Guardian of the Forest. She had witnessed Maya's kindness towards the blackbird and had been watching her ever since. Luna explained that she had come to reward Maya for her selflessness.\n\nWith a wave of her wand, Luna granted Maya the ability to communicate with animals. Maya's eyes widened with amazement as she realized she could now understand the language of nature. Birds chirped melodies, rabbits whispered secrets, and trees shared their ancient wisdom.\n\nOver time, Maya's ability made her beloved by both humans and animals. Farmers sought her advice on how to care for their crops, and children flocked to her for stories of her enchanting encounters with the forest creatures. Maya used her gift to teach others about the importance of living in harmony with nature.\n\nAs years passed, Maya became known as the Village Guardian. She dedicated herself to protecting the surrounding forests from harm and educating others on sustainable living. The village flourished under Maya's guidance, and animals and humans lived side by side peacefully.\n\nAnd so, Maya's story became a legend passed down through generations. Her kindness, love for nature, and her ability to communicate with animals inspired people to treat the world around them with compassion and care.", role='assistant', function_call=None, tool_calls=None))], created=1701011060, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=524, prompt_tokens=12, total_tokens=536))

### Batching requests

The OpenAI API has separate limits for requests per minute and tokens per minute.

If you're hitting the limit on requests per minute, but have headroom on tokens per minute, you can increase your throughput by batching multiple tasks into each request. This will allow you to process more tokens per minute, especially with the smaller models.

Sending in a batch of prompts works exactly the same as a normal API call, except that pass in a list of strings to `prompt` parameter instead of a single string.

**Warning:** the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the `index` field.

#### Example without batching

```python
num_stories = 10
content = "Once upon a time,"

# serial example, with one story completion per request
for _ in range(num_stories):
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": content}],
        max_tokens=20,
    )

    # print story
    print(content + response.choices[0].message.content)

```

    Once upon a time,in a small village nestled between rolling green hills, there lived a young girl named Lily. She had
    Once upon a time,in a small village nestled in the heart of a lush forest, lived a young girl named Evelyn.
    Once upon a time,in a faraway kingdom, there lived a young princess named Aurora. She was known for her kind
    Once upon a time,in a faraway kingdom called Enchantia, there lived a young girl named Ella. Ella was
    Once upon a time,in a small village nestled among the rolling hills, lived a young woman named Lucy. Lucy was known
    Once upon a time,in a small village nestled between rolling hills, there lived a young girl named Ava. Ava was a
    Once upon a time,in a faraway kingdom, there lived a wise and just king named Arthur. King Arthur ruled over
    Once upon a time,in a small village nestled among towering mountains, lived a young girl named Lily. She was known for
    Once upon a time,in a small village nestled in the heart of a lush forest, there lived a young girl named Lily
    Once upon a time,in a far-off kingdom, there lived a kind and beloved queen named Isabella. She ruled with

#### Example with batching

```python
num_stories = 10
prompts = ["Once upon a time,"] * num_stories

# batched example, with 10 stories completions per request
response = client.chat.completions.create(
    model="curie",
    prompt=prompts,
    max_tokens=20,
)

# match completions to prompts by index
stories = [""] * len(prompts)
for choice in response.choices:
    stories[choice.index] = prompts[choice.index] + choice.text

# print stories
for story in stories:
    print(story)

```

    Once upon a time, I lived in hope. I convinced myself I knew best, because, naive as it might sound,
    Once upon a time, Thierry Henry was invited to have a type of frosty exchange with English fans, in which
    Once upon a time, and a long time ago as well, PV was passively cooled because coils cooled by use of metal driving
    Once upon a time, there was a land called Texas. It was about the size of Wisconsin. It contained, however,
    Once upon a time, there was an old carpenter who had three sons. The locksmith never learned to read or write
    Once upon a time, there was a small farming town called Moonridge Village, far West across the great vast plains that lay
    Once upon a time, California’s shorelines, lakes, and valleys were host to expanses of untamed wilderness
    Once upon a time, she said. It started with a simple question: Why don’t we know any stories?
    Once upon a time, when I was a young woman, there was a movie named Wuthering Heights. Stand by alleges
    Once upon a time, a very long time I mean, in the year 1713, died a beautiful Duchess called the young

## Example parallel processing script

We've written an example script for parallel processing large quantities of API requests: [api_request_parallel_processor.py](https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py).

The script combines some handy features:
- Streams requests from file, to avoid running out of memory for giant jobs
- Makes requests concurrently, to maximize throughput
- Throttles both request and token usage, to stay under rate limits
- Retries failed requests, to avoid missing data
- Logs errors, to diagnose problems with requests

Feel free to use it as is or modify it to suit your needs.


# Using Tool Required for Customer Service

The `ChatCompletion` endpoint now includes the ability to specify whether a tool **must** be called every time, by adding `tool_choice='required'` as a parameter.

This adds an element of determinism to how you build your wrapping application, as you can count on a tool being provided with every call. We'll demonstrate here how this can be useful for a contained flow like customer service, where having the ability to define specific exit points gives more control.

The notebook concludes with a multi-turn evaluation, where we spin up a customer GPT to imitate our customer and test the LLM customer service agent we've set up.

```python
import json
from openai import OpenAI
import os

client = OpenAI()
GPT_MODEL = 'gpt-4-turbo'
```

## Config definition

We will define `tools` and `instructions` which our LLM customer service agent will use. It will source the right instructions for the problem the customer is facing, and use those to answer the customer's query.

As this is a demo example, we'll ask the model to make up values where it doesn't have external systems to source info.

```python
# The tools our customer service LLM will use to communicate
tools = [
{
  "type": "function",
  "function": {
    "name": "speak_to_user",
    "description": "Use this to speak to the user to give them information and to ask for anything required for their case.",
    "parameters": {
      "type": "object",
      "properties": {
        "message": {
          "type": "string",
          "description": "Text of message to send to user. Can cover multiple topics."
        }
      },
      "required": ["message"]
    }
  }
},
{
  "type": "function",
  "function": {
    "name": "get_instructions",
    "description": "Used to get instructions to deal with the user's problem.",
    "parameters": {
      "type": "object",
      "properties": {
        "problem": {
          "type": "string",
          "enum": ["fraud","refund","information"],
          "description": """The type of problem the customer has. Can be one of:
          - fraud: Required to report and resolve fraud.
          - refund: Required to submit a refund request.
          - information: Used for any other informational queries."""
        }
      },
      "required": [
        "problem"
      ]
    }
  }
}
]

# Example instructions that the customer service assistant can consult for relevant customer problems
INSTRUCTIONS = [ {"type": "fraud",
                  "instructions": """• Ask the customer to describe the fraudulent activity, including the the date and items involved in the suspected fraud.
• Offer the customer a refund.
• Report the fraud to the security team for further investigation.
• Thank the customer for contacting support and invite them to reach out with any future queries."""},
                {"type": "refund",
                 "instructions": """• Confirm the customer's purchase details and verify the transaction in the system.
• Check the company's refund policy to ensure the request meets the criteria.
• Ask the customer to provide a reason for the refund.
• Submit the refund request to the accounting department.
• Inform the customer of the expected time frame for the refund processing.
• Thank the customer for contacting support and invite them to reach out with any future queries."""},
                {"type": "information",
                 "instructions": """• Greet the customer and ask how you can assist them today.
• Listen carefully to the customer's query and clarify if necessary.
• Provide accurate and clear information based on the customer's questions.
• Offer to assist with any additional questions or provide further details if needed.
• Ensure the customer is satisfied with the information provided.
• Thank the customer for contacting support and invite them to reach out with any future queries.""" }]
```

```python
assistant_system_prompt = """You are a customer service assistant. Your role is to answer user questions politely and competently.
You should follow these instructions to solve the case:
- Understand their problem and get the relevant instructions.
- Follow the instructions to solve the customer's problem. Get their confirmation before performing a permanent operation like a refund or similar.
- Help them with any other problems or close the case.

Only call a tool once in a single message.
If you need to fetch a piece of information from a system or document that you don't have access to, give a clear, confident answer with some dummy values."""

def submit_user_message(user_query,conversation_messages=[]):
    """Message handling function which loops through tool calls until it reaches one that requires a response.
    Once it receives respond=True it returns the conversation_messages to the user."""

    # Initiate a respond object. This will be set to True by our functions when a response is required
    respond = False

    user_message = {"role":"user","content": user_query}
    conversation_messages.append(user_message)

    print(f"User: {user_query}")

    while respond is False:

        # Build a transient messages object to add the conversation messages to
        messages = [
            {
                "role": "system",
                "content": assistant_system_prompt
            }
        ]

        # Add the conversation messages to our messages call to the API
        [messages.append(x) for x in conversation_messages]

        # Make the ChatCompletion call with tool_choice='required' so we can guarantee tools will be used
        response = client.chat.completions.create(model=GPT_MODEL
                                                  ,messages=messages
                                                  ,temperature=0
                                                  ,tools=tools
                                                  ,tool_choice='required'
                                                 )

        conversation_messages.append(response.choices[0].message)

        # Execute the function and get an updated conversation_messages object back
        # If it doesn't require a response, it will ask the assistant again.
        # If not the results are returned to the user.
        respond, conversation_messages = execute_function(response.choices[0].message,conversation_messages)

    return conversation_messages

def execute_function(function_calls,messages):
    """Wrapper function to execute the tool calls"""

    for function_call in function_calls.tool_calls:

        function_id = function_call.id
        function_name = function_call.function.name
        print(f"Calling function {function_name}")
        function_arguments = json.loads(function_call.function.arguments)

        if function_name == 'get_instructions':

            respond = False

            instruction_name = function_arguments['problem']
            instructions = INSTRUCTIONS['type' == instruction_name]

            messages.append(
                                {
                                    "tool_call_id": function_id,
                                    "role": "tool",
                                    "name": function_name,
                                    "content": instructions['instructions'],
                                }
                            )

        elif function_name != 'get_instructions':

            respond = True

            messages.append(
                                {
                                    "tool_call_id": function_id,
                                    "role": "tool",
                                    "name": function_name,
                                    "content": function_arguments['message'],
                                }
                            )

            print(f"Assistant: {function_arguments['message']}")

    return (respond, messages)

```

## Example

To test this we will run an example for a customer who has experienced fraud, and see how the model handles it.

Play the role of the user and provide plausible next steps to keep the conversation going.

```python
messages = submit_user_message("Hi, I have had an item stolen that was supposed to be delivered to me yesterday.")
```

    User: Hi, I have had an item stolen that was supposed to be delivered to me yesterday.
    Calling function get_instructions
    Calling function speak_to_user
    Assistant: I'm sorry to hear about the stolen item. Could you please provide me with more details about the fraudulent activity, including the date and the items involved? This information will help us to investigate the issue further and proceed with the necessary actions, including offering you a refund.

```python
messages = submit_user_message("For sure, it was a shirt, it was supposed to be delivered yesterday but it never arrived.",messages)
```

    User: For sure, it was a shirt, it was supposed to be delivered yesterday but it never arrived.
    Calling function speak_to_user
    Assistant: Thank you for providing the details. I will now proceed to report this incident to our security team for further investigation and arrange a refund for the stolen shirt. Please confirm if you would like me to go ahead with the refund.
    Calling function speak_to_user
    Assistant: Thank you for contacting us about this issue. Please don't hesitate to reach out if you have any more questions or need further assistance in the future.

```python
messages = submit_user_message("Yes I would like to proceed with the refund.",messages)
```

    User: Yes I would like to proceed with the refund.
    Calling function get_instructions
    Calling function speak_to_user
    Assistant: Thank you for confirming. I have processed the refund for the stolen shirt. The amount should be reflected in your account within 5-7 business days. If you have any more questions or need further assistance, please feel free to contact us.

```python
messages = submit_user_message("Thanks very much.",messages)
```

    User: Thanks very much.
    Calling function speak_to_user
    Assistant: You're welcome! If you need any more help in the future, don't hesitate to reach out. Have a great day!

## Evaluation

Now we'll do a simple evaluation where a GPT will pretend to be our customer. The two will go back and forth until a resolution is reached.

We'll reuse the functions above, adding an `execute_conversation` function where the customer GPT will continue answering.

```python
customer_system_prompt = """You are a user calling in to customer service.
You will talk to the agent until you have a resolution to your query.
Your query is {query}.
You will be presented with a conversation - provide answers for any assistant questions you receive.
Here is the conversation - you are the "user" and you are speaking with the "assistant":
{chat_history}

If you don't know the details, respond with dummy values.
Once your query is resolved, respond with "DONE" """

# Initiate a bank of questions run through
questions = ['I want to get a refund for the suit I ordered last Friday.',
            'Can you tell me what your policy is for returning damaged goods?',
            'Please tell me what your complaint policy is']
```

```python
def execute_conversation(objective):

    conversation_messages = []

    done = False

    user_query = objective

    while done is False:

        conversation_messages = submit_user_message(user_query,conversation_messages)

        messages_string = ''
        for x in conversation_messages:
            if isinstance(x,dict):
                if x['role'] == 'user':
                    messages_string += 'User: ' + x['content'] + '\n'
                elif x['role'] == 'tool':
                    if x['name'] == 'speak_to_user':
                        messages_string += 'Assistant: ' + x['content'] + '\n'
            else:
                continue

        messages = [
            {
            "role": "system",
            "content": customer_system_prompt.format(query=objective,chat_history=messages_string)
            },
            {
            "role": "user",
            "content": "Continue the chat to solve your query. Remember, you are in the user in this exchange. Do not provide User: or Assistant: in your response"
            }
        ]

        user_response = client.chat.completions.create(model=GPT_MODEL,messages=messages,temperature=0.5)

        conversation_messages.append({
            "role": "user",
            "content": user_response.choices[0].message.content
            })

        if 'DONE' in user_response.choices[0].message.content:
            done = True
            print("Achieved objective, closing conversation\n\n")

        else:
            user_query = user_response.choices[0].message.content
```

```python
for x in questions:

    execute_conversation(x)
```

    User: I want to get a refund for the suit I ordered last Friday.
    Calling function get_instructions
    Calling function speak_to_user
    Assistant: I understand you'd like a refund for the suit you ordered last Friday. Could you please provide more details about the issue with the suit? This will help us process your refund request accurately.
    User: The suit I received is not the color I ordered. I ordered a navy blue suit, but the one I received is black.
    Calling function speak_to_user
    Assistant: Thank you for providing the details. I will proceed with the refund for the navy blue suit that was incorrectly sent as black. Please confirm if you would like me to go ahead with the refund.
    User: Yes, please go ahead with the refund.
    Calling function speak_to_user
    Assistant: The refund for the incorrectly colored suit has been processed. You should see the amount credited back to your original payment method within 5-7 business days. Thank you for contacting us, and if you have any more questions or need further assistance, please feel free to reach out.
    Achieved objective, closing conversation

    User: Can you tell me what your policy is for returning damaged goods?
    Calling function get_instructions
    Calling function speak_to_user
    Assistant: It seems there was a misunderstanding in my previous request. I'm looking for information on our policy for returning damaged goods. Could you please provide me with the details on how to handle returns for damaged items?
    User: Yes, I'd appreciate that. Could you please tell me the steps I need to follow to return a damaged item?
    Calling function get_instructions
    Calling function speak_to_user
    Assistant: I apologize for the confusion earlier. Here's the correct information regarding our policy for returning damaged goods:

    1. Please provide a description of the damage and the item involved.
    2. Include the date of purchase and your order number if available.
    3. You can choose to return the item by mail or in person at one of our stores. Please let us know which method you prefer, and we will provide the necessary details for the return process.
    4. Once we receive the item, we will inspect it and process a refund or exchange based on your preference and our return policy guidelines.

    Please let me know if you need further assistance with this process or if there's anything else I can help you with!
    User: I would like to return the item by mail. Could you please provide me with the details on how to do that?
    Calling function speak_to_user
    Assistant: Certainly! To return the damaged item by mail, please follow these steps:

    1. Pack the item securely in its original packaging, if possible.
    2. Include all original materials such as manuals, accessories, and receipts.
    3. Address the package to our returns department at the following address:
       [Your Company's Return Address]
    4. Use a trackable shipping service and keep the tracking number for your records.
    5. Once we receive and inspect the item, we will process your refund or exchange as per your request.

    Please let me know if you need a prepaid shipping label or any other assistance with your return!
    User: Yes, please, I would need a prepaid shipping label.
    Calling function speak_to_user
    Assistant: I will arrange for a prepaid shipping label to be sent to your email address. Please ensure that the email address we have on file is correct. You should receive the label shortly, which you can print and attach to your return package.

    Once you've shipped the item, please provide us with the tracking number so we can monitor the return's progress. If there's anything else you need help with, feel free to ask!
    User: That sounds great, thank you! My email address is correct on file. I'll look out for the prepaid shipping label and send the package back as soon as I receive it.
    Calling function speak_to_user
    Assistant: You're welcome! I'm glad I could assist you. Once you receive the prepaid shipping label, please proceed with the return as discussed. If you have any more questions or need further assistance in the future, don't hesitate to reach out. Have a great day!
    Achieved objective, closing conversation

    User: Please tell me what your complaint policy is
    Calling function get_instructions
    Calling function speak_to_user
    Assistant: It seems there was a misunderstanding. I will provide you with information about our complaint policy. Please give me a moment to gather the correct details for you.
    User: Sure, I'll wait. Thank you.
    Calling function speak_to_user
    Assistant: Our complaint policy is designed to ensure that any issues you encounter are addressed promptly and fairly. Here’s a brief overview:

    1. **Submission**: You can submit your complaint through our website, by email, or by calling our customer service hotline.
    2. **Acknowledgment**: We will acknowledge receipt of your complaint within 24 hours.
    3. **Investigation**: We will investigate your complaint thoroughly. This process typically takes 3-5 business days.
    4. **Resolution**: After the investigation, we will contact you with the outcome and any steps we will take to resolve the issue.
    5. **Follow-up**: If you are not satisfied with the resolution, you can request a review of the decision.

    Please let me know if you need more detailed information or if there's anything else I can assist you with!
    User: That covers everything I needed to know, thank you!
    Calling function speak_to_user
    Assistant: You're welcome! I'm glad I could help. If you have any more questions in the future or need further assistance, feel free to reach out. Have a great day!
    Achieved objective, closing conversation

## Conclusion

You can now control your LLM's behaviour explicitly by making tool use mandatory, as well as spin up GPT testers to challenge your LLM and to act as automated test cases.

We hope this has given you an appreciation for a great use case for tool use, and look forward to seeing what you build!


Long Document Content Extraction

# Long Document Content Extraction

GPT-3 can help us extract key figures, dates or other bits of important content from documents that are too big to fit into the context window. One approach for solving this is to chunk the document up and process each chunk separately, before combining into one list of answers.

In this notebook we'll run through this approach:
- Load in a long PDF and pull the text out
- Create a prompt to be used to extract key bits of information
- Chunk up our document and process each chunk to pull any answers out
- Combine them at the end
- This simple approach will then be extended to three more difficult questions

## Approach

- **Setup**: Take a PDF, a Formula 1 Financial Regulation document on Power Units, and extract the text from it for entity extraction. We'll use this to try to extract answers that are buried in the content.
- **Simple Entity Extraction**: Extract key bits of information from chunks of a document by:
    - Creating a template prompt with our questions and an example of the format it expects
    - Create a function to take a chunk of text as input, combine with the prompt and get a response
    - Run a script to chunk the text, extract answers and output them for parsing
- **Complex Entity Extraction**: Ask some more difficult questions which require tougher reasoning to work out

## Setup

```python
!pip install textract
!pip install tiktoken
```

```python
import textract
import os
import openai
import tiktoken

client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

# Extract the raw text from each PDF using textract
text = textract.process('data/fia_f1_power_unit_financial_regulations_issue_1_-_2022-08-16.pdf', method='pdfminer').decode('utf-8')
clean_text = text.replace("  ", " ").replace("\n", "; ").replace(';',' ')
```

## Simple Entity Extraction

```python
# Example prompt -
document = '<document>'
template_prompt=f'''Extract key pieces of information from this regulation document.
If a particular piece of information is not present, output \"Not specified\".
When you extract a key piece of information, include the closest page number.
Use the following format:\n0. Who is the author\n1. What is the amount of the "Power Unit Cost Cap" in USD, GBP and EUR\n2. What is the value of External Manufacturing Costs in USD\n3. What is the Capital Expenditure Limit in USD\n\nDocument: \"\"\"<document>\"\"\"\n\n0. Who is the author: Tom Anderson (Page 1)\n1.'''
print(template_prompt)
```

    Extract key pieces of information from this regulation document.
    If a particular piece of information is not present, output "Not specified".
    When you extract a key piece of information, include the closest page number.
    Use the following format:
    0. Who is the author
    1. What is the amount of the "Power Unit Cost Cap" in USD, GBP and EUR
    2. What is the value of External Manufacturing Costs in USD
    3. What is the Capital Expenditure Limit in USD

    Document: """<document>"""

    0. Who is the author: Tom Anderson (Page 1)
    1.

```python
# Split a text into smaller chunks of size n, preferably ending at the end of a sentence
def create_chunks(text, n, tokenizer):
    tokens = tokenizer.encode(text)
    """Yield successive n-sized chunks from text."""
    i = 0
    while i < len(tokens):
        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens
        j = min(i + int(1.5 * n), len(tokens))
        while j > i + int(0.5 * n):
            # Decode the tokens and check for full stop or newline
            chunk = tokenizer.decode(tokens[i:j])
            if chunk.endswith(".") or chunk.endswith("\n"):
                break
            j -= 1
        # If no end of sentence found, use n tokens as the chunk size
        if j == i + int(0.5 * n):
            j = min(i + n, len(tokens))
        yield tokens[i:j]
        i = j

def extract_chunk(document,template_prompt):
    prompt = template_prompt.replace('<document>',document)

    messages = [
            {"role": "system", "content": "You help extract information from documents."},
            {"role": "user", "content": prompt}
            ]

    response = client.chat.completions.create(
            model='gpt-4',
            messages=messages,
            temperature=0,
            max_tokens=1500,
            top_p=1,
            frequency_penalty=0,
            presence_penalty=0
        )
    return "1." + response.choices[0].message.content
```

```python
# Initialise tokenizer
tokenizer = tiktoken.get_encoding("cl100k_base")

results = []

chunks = create_chunks(clean_text,1000,tokenizer)
text_chunks = [tokenizer.decode(chunk) for chunk in chunks]

for chunk in text_chunks:
    results.append(extract_chunk(chunk,template_prompt))
    #print(chunk)
    print(results[-1])

```

```python
groups = [r.split('\n') for r in results]

# zip the groups together
zipped = list(zip(*groups))
zipped = [x for y in zipped for x in y if "Not specified" not in x and "__" not in x]
zipped
```

    ['1. What is the amount of the "Power Unit Cost Cap" in USD, GBP and EUR: USD 95,000,000 (Page 2); GBP 76,459,000 (Page 2); EUR 90,210,000 (Page 2)',
     '2. What is the value of External Manufacturing Costs in USD: US Dollars 20,000,000 in respect of each of the Full Year Reporting Periods ending on 31 December 2023, 31 December 2024 and 31 December 2025, adjusted for Indexation (Page 10)',
     '3. What is the Capital Expenditure Limit in USD: US Dollars 30,000,000 (Page 32)']

## Complex Entity Extraction

```python
# Example prompt -
template_prompt=f'''Extract key pieces of information from this regulation document.
If a particular piece of information is not present, output \"Not specified\".
When you extract a key piece of information, include the closest page number.
Use the following format:\n0. Who is the author\n1. How is a Minor Overspend Breach calculated\n2. How is a Major Overspend Breach calculated\n3. Which years do these financial regulations apply to\n\nDocument: \"\"\"<document>\"\"\"\n\n0. Who is the author: Tom Anderson (Page 1)\n1.'''
print(template_prompt)
```

    Extract key pieces of information from this regulation document.
    If a particular piece of information is not present, output "Not specified".
    When you extract a key piece of information, include the closest page number.
    Use the following format:
    0. Who is the author
    1. How is a Minor Overspend Breach calculated
    2. How is a Major Overspend Breach calculated
    3. Which years do these financial regulations apply to

    Document: """<document>"""

    0. Who is the author: Tom Anderson (Page 1)
    1.

```python
results = []

for chunk in text_chunks:
    results.append(extract_chunk(chunk,template_prompt))

groups = [r.split('\n') for r in results]

# zip the groups together
zipped = list(zip(*groups))
zipped = [x for y in zipped for x in y if "Not specified" not in x and "__" not in x]
zipped
```

    ['1. How is a Minor Overspend Breach calculated: A Minor Overspend Breach arises when a Power Unit Manufacturer submits its Full Year Reporting Documentation and Relevant Costs reported therein exceed the Power Unit Cost Cap by less than 5% (Page 24)',
     '2. How is a Major Overspend Breach calculated: A Material Overspend Breach arises when a Power Unit Manufacturer submits its Full Year Reporting Documentation and Relevant Costs reported therein exceed the Power Unit Cost Cap by 5% or more (Page 25)',
     '3. Which years do these financial regulations apply to: 2026 onwards (Page 1)',
     '3. Which years do these financial regulations apply to: 2023, 2024, 2025, 2026 and subsequent Full Year Reporting Periods (Page 2)',
     '3. Which years do these financial regulations apply to: 2022-2025 (Page 6)',
     '3. Which years do these financial regulations apply to: 2023, 2024, 2025, 2026 and subsequent Full Year Reporting Periods (Page 10)',
     '3. Which years do these financial regulations apply to: 2022 (Page 14)',
     '3. Which years do these financial regulations apply to: 2022 (Page 16)',
     '3. Which years do these financial regulations apply to: 2022 (Page 19)',
     '3. Which years do these financial regulations apply to: 2022 (Page 21)',
     '3. Which years do these financial regulations apply to: 2026 onwards (Page 26)',
     '3. Which years do these financial regulations apply to: 2026 (Page 2)',
     '3. Which years do these financial regulations apply to: 2022 (Page 30)',
     '3. Which years do these financial regulations apply to: 2022 (Page 32)',
     '3. Which years do these financial regulations apply to: 2023, 2024 and 2025 (Page 1)',
     '3. Which years do these financial regulations apply to: 2022 (Page 37)',
     '3. Which years do these financial regulations apply to: 2026 onwards (Page 40)',
     '3. Which years do these financial regulations apply to: 2022 (Page 1)',
     '3. Which years do these financial regulations apply to: 2026 to 2030 seasons (Page 46)',
     '3. Which years do these financial regulations apply to: 2022 (Page 47)',
     '3. Which years do these financial regulations apply to: 2022 (Page 1)',
     '3. Which years do these financial regulations apply to: 2022 (Page 1)',
     '3. Which years do these financial regulations apply to: 2022 (Page 56)',
     '3. Which years do these financial regulations apply to: 2022 (Page 1)',
     '3. Which years do these financial regulations apply to: 2022 (Page 16)',
     '3. Which years do these financial regulations apply to: 2022 (Page 16)']

## Consolidation

We've been able to extract the first two answers safely, while the third was confounded by the date that appeared on every page, though the correct answer is in there as well.

To tune this further you can consider experimenting with:
- A more descriptive or specific prompt
- If you have sufficient training data, fine-tuning a model to find a set of outputs very well
- The way you chunk your data - we have gone for 1000 tokens with no overlap, but more intelligent chunking that breaks info into sections, cuts by tokens or similar may get better results

However, with minimal tuning we have now answered 6 questions of varying difficulty using the contents of a long document, and have a reusable approach that we can apply to any long document requiring entity extraction. Look forward to seeing what you can do with this!