<jupyter_start><jupyter_text>Visualizing the embeddings in 2DWe will use t-SNE to reduce the dimensionality of the embeddings from 2048 to 2. Once the embeddings are reduced to two dimensions, we can plot them in a 2D scatter plot. The dataset is created in the [Obtain_dataset Notebook](Obtain_dataset.ipynb).  1. Reduce dimensionalityWe reduce the dimensionality to 2 dimensions using t-SNE decomposition.<jupyter_code>import pandas as pd
from sklearn.manifold import TSNE

# Load the embeddings
df = pd.read_csv('output/embedded_1k_reviews.csv')

# Convert to a list of lists of floats
matrix = df.babbage_similarity.apply(eval).to_list()

# Create a t-SNE model and transform the data
tsne = TSNE(n_components=2, perplexity=15, random_state=42, init='random', learning_rate=200)
vis_dims = tsne.fit_transform(matrix)
vis_dims.shape<jupyter_output><empty_output><jupyter_text>2. Plotting the embeddingsWe colour each review by its star rating, ranging from red to green. We can observe a decent data separation even in the reduced 2 dimensions. There seems to be a cluster of mostly negative reviews.<jupyter_code>import matplotlib.pyplot as plt
import matplotlib
import numpy as np

colors = ["red", "darkorange", "gold", "turquoise", "darkgreen"]
x = [x for x,y in vis_dims]
y = [y for x,y in vis_dims]
color_indices = df.Score.values - 1

colormap = matplotlib.colors.ListedColormap(colors)
plt.scatter(x, y, c=color_indices, cmap=colormap, alpha=0.3)
for score in [0,1,2,3,4]:
    avg_x = np.array(x)[df.Score-1==score].mean()
    avg_y = np.array(y)[df.Score-1==score].mean()
    color = colors[score]
    plt.scatter(avg_x, avg_y, marker='x', color=color, s=100)
plt.title("Amazon ratings visualized in language using t-SNE")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09977324263038549, 'fraction_numerical': 0.01870748299319728, 'mean_word_length': 5.512915129151292, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '351542', 'license': 'MIT', 'n_tokens_mistral': 618, 'n_tokens_neox': 587, 'n_words': 192, 'path': 'examples/embeddings/Visualize_in_2d.ipynb', 'repo_name': 'BastianZim/openai-python', 'size': '1764'}