<jupyter_start><jupyter_text>Near duplicate image detection_This notebook is part of a tutorial series on [txtai](https://github.com/neuml/txtai), an AI-powered semantic search platform._[txtai](https://github.com/neuml/txtai) executes machine-learning workflows to transform data and build AI-powered semantic search applications.This notebook will give an overview of how perceptual image hashes can be used to detect duplicate and near duplicate images.  Install dependenciesInstall `txtai` and all dependencies.<jupyter_code>%%capture
!pip install git+https://github.com/neuml/txtai#egg=txtai[pipeline] textdistance

# Get test data
!wget -N https://github.com/neuml/txtai/releases/download/v3.5.0/tests.tar.gz
!tar -xvzf tests.tar.gz<jupyter_output><empty_output><jupyter_text>Generate hashesThe example below generates perceptual image hashes for a list of images.<jupyter_code>import glob

from PIL import Image

from txtai.pipeline import ImageHash

def show(image):
  width, height = image.size
  return image.resize((int(width / 2.25), int((width / 2.25) * height / width)))

# Get and scale images
images = [Image.open(image) for image in glob.glob('txtai/*jpg')]

# Create image pipeline
ihash = ImageHash()

# Generate hashes
hashes = ihash(images)
hashes<jupyter_output><empty_output><jupyter_text>Hash searchNext we'll generate a search hash to use to find similar near-duplicate images. This logic takes a section of an image and generates a hash for that<jupyter_code># Select portion of image
width, height = images[0].size

# Get dimensions for middle of image
left = (width - width/3)/2
top = (height - height/1.35)/2
right = (width + width/3)/2
bottom = (height + height/1.35)/2

# Crop image
search = images[0].crop((left, top, right, bottom))
show(search)<jupyter_output><empty_output><jupyter_text>Now let's compare the hash to all the image hashes using [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance). We'll use the [textdistance](https://github.com/life4/textdistance) library for that.<jupyter_code>import textdistance

# Find closest image hash using textdistance
shash = ihash(search)

# Calculate distances for search hash
distances = [int(textdistance.levenshtein.distance(h, shash)) for h in hashes]

# Show closest image hash
low = min(distances)
show(images[distances.index(low)])<jupyter_output><empty_output><jupyter_text>And as expected, the closest match is the original full image!  Generate hashes with Embeddings indexesNext we'll add a custom field with a perceptual image hash and a custom SQL function to calculate Levenshtein distance. An index of images is built and then a search query run using the distance from the same search hash.<jupyter_code>from txtai.embeddings import Embeddings

def distance(a, b):
  if a and not b:
    return len(a)
  if not a and b:
    return len(b)
  if not a and not b:
    return 0
  
  return int(textdistance.levenshtein.distance(a, b))

# Create embeddings index with content enabled. The default behavior is to only store indexed vectors.
embeddings = Embeddings({"path": "sentence-transformers/nli-mpnet-base-v2", "content": True, "objects": "image", "functions": [distance]})

# Create an index for the list of text
embeddings.index([(uid, {"object": image, "text": ihash(image)}, None) for uid, image in enumerate(images)])

# Find image that is closest to hash
show(embeddings.search(f"select object from txtai order by distance(text, '{shash}')")[0]["object"])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10230834995725278, 'fraction_numerical': 0.007694499857509262, 'mean_word_length': 5.405109489051095, 'pattern_counts': {'":': 6, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '317079', 'license': 'Apache-2.0', 'n_tokens_mistral': 1096, 'n_tokens_neox': 1057, 'n_words': 372, 'path': 'examples/31_Near_duplicate_image_detection.ipynb', 'repo_name': 'kokizzu/txtai', 'size': '3509'}