library_name: transformers
tags: []
inference: false
SuperGlue
The SuperGlue model was proposed in SuperGlue: Learning Feature Matching with Graph Neural Networks by Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz and Andrew Rabinovich.
This model consists of matching two sets of interest points detected in an image. Paired with the SuperPoint model, it can be used to match two images and estimate the pose between them. This model is useful for tasks such as image matching, homography estimation, etc.
The abstract from the paper is the following:
This paper introduces SuperGlue, a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. Assignments are estimated by solving a differentiable optimal transport problem, whose costs are predicted by a graph neural network. We introduce a flexible context aggregation mechanism based on attention, enabling SuperGlue to reason about the underlying 3D scene and feature assignments jointly. Compared to traditional, hand-designed heuristics, our technique learns priors over geometric transformations and regularities of the 3D world through end-to-end training from image pairs. SuperGlue outperforms other learned approaches and achieves state-of-the-art results on the task of pose estimation in challenging real-world indoor and outdoor environments. The proposed method performs matching in real-time on a modern GPU and can be readily integrated into modern SfM or SLAM systems. The code and trained weights are publicly available at this URL.
This model was contributed by stevenbucaille. The original code can be found here.
Model Details
Model Description
SuperGlue is a neural network that matches two sets of local features by jointly finding correspondences and rejecting non-matchable points. It introduces a flexible context aggregation mechanism based on attention, enabling it to reason about the underlying 3D scene and feature assignments. The architecture consists of two main components: the Attentional Graph Neural Network and the Optimal Matching Layer.
The Attentional Graph Neural Network uses a Keypoint Encoder to map keypoint positions and visual descriptors. It employs self- and cross-attention layers to create powerful representations. The Optimal Matching Layer creates a score matrix, augments it with dustbins, and finds the optimal partial assignment using the Sinkhorn algorithm.
- Developed by: MagicLeap
- Model type: Image Matching
- License: ACADEMIC OR NON-PROFIT ORGANIZATION NONCOMMERCIAL RESEARCH USE ONLY
Model Sources [optional]
- Repository: https://github.com/magicleap/SuperGluePretrainedNetwork
- Paper: https://arxiv.org/pdf/1911.11763
- Demo: https://psarlin.com/superglue/
Uses
Direct Use
SuperGlue is designed for feature matching and pose estimation tasks in computer vision. It can be applied to a variety of multiple-view geometry problems and can handle challenging real-world indoor and outdoor environments. However, it may not perform well on tasks that require different types of visual understanding, such as object detection or image classification.
How to Get Started with the Model
Here is a quick example of using the model. Since this model is an image matching model, it requires pairs of images to be matched:
from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image
import requests
url = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/phototourism_sample_images/london_bridge_78916675_4568141288.jpg?raw=true"
im1 = Image.open(requests.get(url, stream=True).raw)
url = "https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/assets/phototourism_sample_images/london_bridge_19481797_2295892421.jpg?raw=true"
im2 = Image.open(requests.get(url, stream=True).raw)
images = [im1, im2]
processor = AutoImageProcessor.from_pretrained("stevenbucaille/superglue_indoor")
model = AutoModel.from_pretrained("stevenbucaille/superglue_indoor")
inputs = processor(images, return_tensors="pt")
outputs = model(**inputs)
The outputs contain the list of keypoints detected by the keypoint detector as well as the list of matches with their corresponding matching scores. Due to the nature of SuperGlue, to output a dynamic number of matches, you will need to use the mask attribute to retrieve the respective information:
from transformers import AutoImageProcessor, AutoModel
import torch
from PIL import Image
import requests
url_image_1 = "https://github.com/cvg/LightGlue/blob/main/assets/sacre_coeur1.jpg?raw=true"
image_1 = Image.open(requests.get(url_image_1, stream=True).raw)
url_image_2 = "https://github.com/cvg/LightGlue/blob/main/assets/sacre_coeur2.jpg?raw=true"
image_2 = Image.open(requests.get(url_image_2, stream=True).raw)
images = [image_1, image_2]
processor = AutoImageProcessor.from_pretrained("stevenbucaille/superglue_indoor")
model = AutoModel.from_pretrained("stevenbucaille/superglue_indoor")
inputs = processor(images, return_tensors="pt")
with torch.no_grad():
outputs = model(**inputs)
# Get the respective image masks
image0_mask, image1_mask = outputs_mask[0]
image0_indices = torch.nonzero(image0_mask).squeeze()
image1_indices = torch.nonzero(image1_mask).squeeze()
image0_matches = outputs.matches[0, 0][image0_indices]
image1_matches = outputs.matches[0, 1][image1_indices]
image0_matching_scores = outputs.matching_scores[0, 0][image0_indices]
image1_matching_scores = outputs.matching_scores[0, 1][image1_indices]
You can then print the matched keypoints on a side-by-side image to visualize the result :
import cv2
import numpy as np
# Create side by side image
input_data = inputs['pixel_values']
height, width = input_data.shape[-2:]
matched_image = np.zeros((height, width * 2, 3))
matched_image[:, :width] = input_data.squeeze()[0].permute(1, 2, 0).cpu().numpy()
matched_image[:, width:] = input_data.squeeze()[1].permute(1, 2, 0).cpu().numpy()
matched_image = (matched_image * 255).astype(np.uint8)
# Retrieve matches by looking at which keypoints in image0 actually matched with keypoints in image1
image0_mask = outputs.mask[0, 0]
image0_indices = torch.nonzero(image0_mask).squeeze()
image0_matches_indices = torch.nonzero(outputs.matches[0, 0][image0_indices] != -1).squeeze()
image0_keypoints = outputs.keypoints[0, 0][image0_matches_indices]
image0_matches = outputs.matches[0, 0][image0_matches_indices]
image0_matching_scores = outputs.matching_scores[0, 0][image0_matches_indices]
# Retrieve matches from image1
image1_mask = outputs.mask[0, 1]
image1_indices = torch.nonzero(image1_mask).squeeze()
image1_keypoints = outputs.keypoints[0, 1][image0_matches]
# Draw matches
for keypoint0, keypoint1, score in zip(image0_keypoints, image1_keypoints, image0_matching_scores):
keypoint0_x, keypoint0_y = int(keypoint0[0]), int(keypoint0[1])
keypoint1_x, keypoint1_y = int(keypoint1[0] + width), int(keypoint1[1])
color = [0, 1, 0, 0.5] # Set color based on score
plt.plot([keypoint0_x, keypoint1_x], [keypoint0_y, keypoint1_y], color=color, linewidth=1)
# Save the image
plt.savefig("matched_image.png", dpi=300, bbox_inches='tight')
plt.close()
Training Details
Training Data
SuperGlue is trained on large annotated datasets for pose estimation, enabling it to learn priors for pose estimation and reason about the 3D scene. The training data consists of image pairs with ground truth correspondences and unmatched keypoints derived from ground truth poses and depth maps.
Training Procedure
SuperGlue is trained in a supervised manner using ground truth matches and unmatched keypoints. The loss function maximizes the negative log-likelihood of the assignment matrix, aiming to simultaneously maximize precision and recall.
Training Hyperparameters
- Training regime: fp32
Speeds, Sizes, Times
SuperGlue is designed to be efficient and runs in real-time on a modern GPU. A forward pass takes approximately 69 milliseconds (15 FPS) for an indoor image pair. The model has 12 million parameters, making it relatively compact compared to some other deep learning models. The inference speed of SuperGlue is suitable for real-time applications and can be readily integrated into modern Simultaneous Localization and Mapping (SLAM) or Structure-from-Motion (SfM) systems.
Citation [optional]
BibTeX:
@inproceedings{sarlin2020superglue,
title={Superglue: Learning feature matching with graph neural networks},
author={Sarlin, Paul-Edouard and DeTone, Daniel and Malisiewicz, Tomasz and Rabinovich, Andrew},
booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
pages={4938--4947},
year={2020}
}