<jupyter_start><jupyter_text>Cats and DogsCan the AI identify and differentiate between cats and dogs with only a few examples?Note that this notebook requires umap and umap.plot; you can install both via `pip3 install umap-learn umap-learn[plot]`This notebook also requires installing `datasets`.<jupyter_code>from imgbeddings import imgbeddings
from PIL import Image
import logging
import numpy as np
import datasets
from numpy.random import default_rng
import umap
import umap.plot


logger = logging.getLogger()
logger.setLevel(logging.INFO)<jupyter_output><empty_output><jupyter_text>The next cell downloads the famous [Cats vs. Dogs dataset](https://huggingface.co/datasets/cats_vs_dogs) which includes about 23 thousand images; the download is about 800MB.<jupyter_code>df = datasets.load_dataset("cats_vs_dogs", split="train", ignore_verifications=True)
df[0:3]<jupyter_output>Using custom data configuration default
Reusing dataset cats_vs_dogs (/Users/maxwoolf/.cache/huggingface/datasets/cats_vs_dogs/default/0.0.0/de304955b4952383ceadee4ab96ba2b291f67986bb213efb2255963db39d9ed8)<jupyter_text>Select 1000 each of random cats (label `0`) and random dogs (label `1`) for a perfectly balanced dataset, as all datasets should be.<jupyter_code>n = 1000

rng = default_rng(42)
cat_idx = rng.choice(np.where(np.array(df["labels"]) == 0)[0], n, replace=False)
dog_idx = rng.choice(np.where(np.array(df["labels"]) == 1)[0], n, replace=False)
idx = np.hstack((cat_idx, dog_idx))
rng.shuffle(idx)

idx[0:5]
img_source = [df[int(x)]["image_file_path"] for x in idx]
images = [df[int(x)]["image"] for x in idx]
labels = [df[int(x)]["labels"] for x in idx]<jupyter_output><empty_output><jupyter_text>Preview the images we selected:<jupyter_code># Show some of the loaded images in a grid:
# https://stackoverflow.com/a/65583584


def image_grid(imgs, rows, cols):
    assert len(imgs) == rows*cols

    w, h = imgs[0].size
    grid = Image.new('RGB', size=(cols*w, rows*h))
    
    for i, img in enumerate(imgs):
        grid.paste(img, box=(i%cols*w, i//cols*h))
    return grid

n_rows = 4
n_cols = 8
image_grid([x.resize((64,64)) for x in images[0:(n_rows*n_cols)]], n_rows, n_cols)
print(labels[0:(n_rows*n_cols)])<jupyter_output>[1, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0]<jupyter_text>Now all the images can be converted to embeddings!<jupyter_code>ibed = imgbeddings()
embeddings = ibed.to_embeddings(images)<jupyter_output>INFO:root:Creating image embeddings in batches of 64.
100%|██████████| 2000/2000 [00:44<00:00, 45.11it/s]<jupyter_text>UMAP unsupervised clusteringumap is a tool that can cluster embeddings (better/more efficient than the popular tSNE) and then visualize the results.Using cosine similarity works best here.<jupyter_code>umap_catsdogs = umap.UMAP(
    n_neighbors=10, n_components=2, metric="cosine", random_state=42
).fit(embeddings)

umap.plot.points(umap_catsdogs, labels=np.array(labels))<jupyter_output><empty_output><jupyter_text>The two groups are noticeably distinct, although there are several outliers in both cases.  Cat/Dog Classifier with a simple Logistic RegressionHere's a simple `scikit-learn` binary classifier using the generated embeddings to determine whether an image is a cat or a dog, which is just as simple as any other generic online tutorial.However, `max_iter` is increased due to the nature of providing raw embeddings vs. tabular data.<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

classifier = LogisticRegression(max_iter=500)
test_size = 0.2

x_train, x_test, y_train, y_test = train_test_split(embeddings, labels, test_size=test_size, random_state=42, stratify=labels)
classifier.fit(x_train, y_train)
classifier.score(x_test, y_test)<jupyter_output><empty_output><jupyter_text>As expected. For comparison, the [Kaggle Competiton](https://www.kaggle.com/competitions/dogs-vs-cats/leaderboard) for this dataset had a best score of 0.989, although with a different/larger validation set, and without the power of modern AI frameworks.Let's look at misclassified images by finding the indices of where the predictions were incorrect.<jupyter_code>predicted = classifier.predict(x_test)
idx_mismatch = np.where(y_test != predicted)
idx_mismatch<jupyter_output><empty_output><jupyter_text>Map those indicies back to the original 2000 image dataset.<jupyter_code># https://stackoverflow.com/a/51352806
misclass_idx = np.where((embeddings[:, None] == x_test[idx_mismatch]).all(-1).any(-1))
misclass_idx
n_rows = 1
n_cols = len(list(misclass_idx[0]))
image_grid([images[x].resize((128,128)) for x in list(misclass_idx[0])], n_rows, n_cols)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11471269206482845, 'fraction_numerical': 0.03809724268575037, 'mean_word_length': 6.507109004739337, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 34, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '234381', 'license': 'MIT', 'n_tokens_mistral': 1804, 'n_tokens_neox': 1631, 'n_words': 452, 'path': 'examples/cats_dogs.ipynb', 'repo_name': 'minimaxir/imgbeddings', 'size': '4751'}