<jupyter_start><jupyter_text>Demo of MS SVConv for point cloud registrationThe goal of this notebook is to quickly show how you can use MS-SVConv on your projects.  InstallationFirst we install necessary packages<jupyter_code>!pip install torch==1.8.1
!pip uninstall -y torch-scatter
!pip uninstall -y torch-sparse
!pip uninstall -y torch-cluster
!pip install torch-scatter -f https://data.pyg.org/whl/torch-1.8.1+cu102.html
!pip install torch-sparse -f https://data.pyg.org/whl/torch-1.8.1+cu102.html
!pip install torch-cluster -f https://data.pyg.org/whl/torch-1.8.1+cu102.html
!pip install torch-spline-conv -f https://data.pyg.org/whl/torch-1.8.1+cu102.html
!pip install torch-geometric
!pip install pyvista
!pip install --upgrade jsonschema

!pip install git+https://github.com/nicolas-chaulet/torch-points3d.git@e090530eab5e3e5798c5abc764088d6a1f9827c3
!apt-get update
!apt-get install -qq xvfb libgl1-mesa-glx
!pip install pyvistaqt
!apt-get install build-essential python3-dev libopenblas-dev
!pip install -U git+https://github.com/NVIDIA/MinkowskiEngine@9<PASSWORD>cd08ee4f<PASSWORD>cf<PASSWORD> --no-deps<jupyter_output>Collecting pyvistaqt
  Downloading pyvistaqt-0.5.0-py3-none-any.whl (162 kB)
[K     |████████████████████████████████| 162 kB 2.3 MB/s 
[?25hRequirement already satisfied: QtPy>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from pyvistaqt) (1.11.2)
Requirement already satisfied: pyvista>=0.25.0 in /usr/local/lib/python3.7/dist-packages (from pyvistaqt) (0.32.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pyvista>=0.25.0->pyvistaqt) (1.19.5)
Requirement already satisfied: meshio<5.0,>=4.0.3 in /usr/local/lib/python3.7/dist-packages (from pyvista>=0.25.0->pyvistaqt) (4.4.6)
Requirement already satisfied: appdirs in /usr/local/lib/python3.7/dist-packages (from pyvista>=0.25.0->pyvistaqt) (1.4.4)
Requirement already satisfied: vtk in /usr/local/lib/python3.7/dist-packages (from pyvista>=0.25.0->pyvistaqt) (9.0.3)
Requirement already satisfied: scooby>=0.5.1 in /usr/local/lib/python3.7/dist-packages (from pyvista>=0.25.0->pyvistaqt) (0.5.7)
Requ[...]<jupyter_text>We also install torchsparse<jupyter_code>!apt-get install libsparsehash-dev
!pip install --upgrade git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0<jupyter_output>Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  libsparsehash-dev
0 upgraded, 1 newly installed, 0 to remove and 80 not upgraded.
Need to get 72.4 kB of archives.
After this operation, 612 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libsparsehash-dev all 2.0.2-1 [72.4 kB]
Fetched 72.4 kB in 1s (115 kB/s)
Selecting previously unselected package libsparsehash-dev.
(Reading database ... 155026 files and directories currently installed.)
Preparing to unpack .../libsparsehash-dev_2.0.2-1_all.deb ...
Unpacking libsparsehash-dev (2.0.2-1) ...
Setting up libsparsehash-dev (2.0.2-1) ...
Collecting git+https://github.com/mit-han-lab/torchsparse.git@v1.4.0
  Cloning https://github.com/mit-han-lab/torchsparse.git (to revision v1.4.0) to /tmp/pip-req-build-sfuk9gia
  Running command git clone -q https://github.com/mit-han-lab/torchsparse.git /tmp[...]<jupyter_text>RegistrationWe will use pyvista for the visualization. But it is not interactive...<jupyter_code>import numpy as np
import open3d as o3d
import os
import os.path as osp
import pathlib
import requests
import torch
from torch_points3d.applications.pretrained_api import PretainedRegistry
from torch_points3d.core.data_transform import GridSampling3D, AddFeatByKey, AddOnes, Random3AxisRotation
from torch_points3d.datasets.registration.pair import Pair
from torch_points3d.utils.registration import get_matches, fast_global_registration
from torch_geometric.data import Data
from torch_geometric.transforms import Compose
from torch_points3d.metrics.registration_metrics import compute_hit_ratio
from zipfile import ZipFile

import pyvista as pv
import panel as pn

pn.extension('vtk')
os.system('/usr/bin/Xvfb :99 -screen 0 1024x768x24 &')
os.environ['DISPLAY'] = ':99'
os.environ['PYVISTA_OFF_SCREEN'] = 'True'
os.environ['PYVISTA_USE_PANEL'] = 'True'<jupyter_output><empty_output><jupyter_text>We need to download the datasets, and the models<jupyter_code>MODEL = {"MS_SVCONV_2cm_X2_3head_3dm.pt": "https://cloud.mines-paristech.fr/index.php/s/hRc6y2YIFtYsGAI/download",
         "MS_SVCONV_4cm_X2_3head_eth.pt": "https://cloud.mines-paristech.fr/index.php/s/pUmGPtHUG2ASxlJ/download"}
DATA = {"gazebo_winter_12.pcd": "https://cloud.mines-paristech.fr/index.php/s/zgO88hYFeogTj2s/download",
        "gazebo_winter_11.pcd": "https://cloud.mines-paristech.fr/index.php/s/NpsabVL7bz5qFEe/download",
        "kitchen_0.ply": "https://cloud.mines-paristech.fr/index.php/s/lArxiaV0DPo4bBU/download",
        "kitchen_10.ply": "https://cloud.mines-paristech.fr/index.php/s/357BXcA2qcrw2Uy/download"}

def download(url, out, name):
  """
  download a file and extract the zip file
  """
  req = requests.get(url)
  pathlib.Path(out).mkdir(exist_ok=True)
  with open(osp.join(out, name), "wb") as archive:
    archive.write(req.content)
def extract(out, name):
  with ZipFile(osp.join(out, name+".zip"), "r") as zip_obj:
    zip_obj.extractall(osp.join(out, name))
# Download Models and data for the demo
download(MODEL["MS_SVCONV_2cm_X2_3head_3dm.pt"], "models", "MS_SVCONV_2cm_X2_3head_3dm.pt")
download(MODEL["MS_SVCONV_4cm_X2_3head_eth.pt"], "models", "MS_SVCONV_4cm_X2_3head_eth.pt")
download(DATA["gazebo_winter_12.pcd"], "data", "gazebo_winter_12.pcd")
download(DATA["gazebo_winter_11.pcd"], "data", "gazebo_winter_11.pcd")
download(DATA["kitchen_0.ply"], "data", "kitchen_0.ply")
download(DATA["kitchen_10.ply"], "data", "kitchen_10.ply")



def read_pcd(path):
  pcd = o3d.io.read_point_cloud(path)
  data = Pair(pos=torch.from_numpy(np.asarray(pcd.points)).float(), batch=torch.zeros(len(pcd.points)).long())
  return data<jupyter_output><empty_output><jupyter_text>modify the variable `choice_data` and `choice_model` if you want to change the dataset and the model. For the transformation, we perform random rotation beforehand just to show that MS-SVConv is rotation invariant.<jupyter_code>#@title Data and Model
#@markdown Please choose the data and the model you want
choice_data = "3dm"  #@param ['eth', '3dm']
choice_model = "3dm"  #@param ['eth', '3dm']



# DATA
pcd_path = {
    "eth": ["data/gazebo_winter_12.pcd", "data/gazebo_winter_11.pcd"],
    "3dm": ["data/kitchen_0.ply", "data/kitchen_10.ply"] 
}
# Model
pcd_model = {
    "eth": "models/MS_SVCONV_4cm_X2_3head_eth.pt",
    "3dm": "models/MS_SVCONV_2cm_X2_3head_3dm.pt"
}

# Data augmentation
transfo_3dm = Compose([Random3AxisRotation(rot_x=180, rot_y=180, rot_z=180), GridSampling3D(size=0.02, quantize_coords=True, mode='last'), AddOnes(), AddFeatByKey(add_to_x=True, feat_name="ones")])
transfo_eth = Compose([Random3AxisRotation(rot_x=180, rot_y=180, rot_z=180), GridSampling3D(size=0.04, quantize_coords=True, mode='last'), AddOnes(), AddFeatByKey(add_to_x=True, feat_name="ones")])
pcd_DA = {
    "eth": transfo_eth,
    "3dm": transfo_3dm
}

model = PretainedRegistry.from_file(pcd_model[choice_model], mock_property={})

data_s = pcd_DA[choice_model](read_pcd(pcd_path[choice_data][0]))
data_t = pcd_DA[choice_model](read_pcd(pcd_path[choice_data][1]))


#pl1 = pvqt.BackgroundPlotter()
pl1 = pv.Plotter(notebook=True)
pl1.add_points(data_s.pos.numpy(),color=[0.9, 0.7, 0.1])
pl1.add_points(data_t.pos.numpy(),color=[0.1, 0.7, 0.9])

pl1.enable_eye_dome_lighting()
pl1.show()
def ransac(pos_s, pos_t, feat_s, feat_t, distance_threshold=0.1):
  pcd_s = o3d.geometry.PointCloud()
  pcd_s.points = o3d.utility.Vector3dVector(pos_s.numpy())
  pcd_t = o3d.geometry.PointCloud()
  pcd_t.points = o3d.utility.Vector3dVector(pos_t.numpy())

  f_s = o3d.pipelines.registration.Feature()
  f_s.data = feat_s.T.numpy()
  f_t = o3d.pipelines.registration.Feature()
  f_t.data = feat_t.T.numpy()
  result = o3d.pipelines.registration.registration_ransac_based_on_feature_matching(
      source=pcd_s, target=pcd_t, source_feature=f_s, target_feature=f_t, 
      mutual_filter=True,
      max_correspondence_distance=distance_threshold,
      estimation_method=o3d.pipelines.registration.TransformationEstimationPointToPoint(False),
      ransac_n=4,
      checkers=[o3d.pipelines.registration.CorrespondenceCheckerBasedOnEdgeLength(0.9),
       o3d.pipelines.registration.CorrespondenceCheckerBasedOnDistance(distance_threshold)],
      criteria=o3d.pipelines.registration.RANSACConvergenceCriteria(4000000, 500))
  return torch.from_numpy(result.transformation).float()<jupyter_output><empty_output><jupyter_text>To registrate, we use RANSAC of Open3D, but it's possible to use an other estimator (such as Fast Global Registration or TEASER)<jupyter_code># Let us registrate !!

# Compute the matches
with torch.no_grad():
  model.set_input(data_s, "cuda")
  feat_s = model.forward()
  model.set_input(data_t, "cuda")
  feat_t = model.forward()
# select random points and compute transformation using FGR
rand_s = torch.randint(0, len(feat_s), (5000, ))
rand_t = torch.randint(0, len(feat_t), (5000, ))
# matches = get_matches(feat_s[rand_s], feat_t[rand_t], sym=True)
T_est = ransac(data_s.pos[rand_s], data_t.pos[rand_t], feat_s[rand_s], feat_t[rand_t])
print(T_est)
pl2 = pv.Plotter(notebook=True)
transformed = data_s.pos @ T_est[:3, :3].T + T_est[:3, 3]
pl2.add_points(np.asarray(transformed.cpu().numpy()),color=[0.9, 0.7, 0.1])
pl2.add_points(np.asarray(data_t.pos.cpu().numpy()),color=[0.1, 0.7, 0.9])
pl2.enable_eye_dome_lighting()
pl2.show()<jupyter_output><empty_output><jupyter_text>That's it ! Now, you know how to use MS-SVConv ! Don't hesitate using it in your projects. And if you like it, please cite our work :```@inproceedings{horache2021mssvconv,      title={3D Point Cloud Registration with Multi-Scale Architecture and Unsupervised Transfer Learning},       author={<NAME> and <NAME> and <NAME>},      year={2021},      journal={arXiv preprint arXiv:2103.14533}}```<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14033203125, 'fraction_numerical': 0.04228515625, 'mean_word_length': 5.585852090032154, 'pattern_counts': {'":': 12, '<': 34, '<?xml version=': 0, '>': 47, 'https://': 16, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '507230', 'license': 'MIT', 'n_tokens_mistral': 4204, 'n_tokens_neox': 3961, 'n_words': 734, 'path': 'notebook/demo_MSSVConv.ipynb', 'repo_name': 'humanpose1/MS-SVConv', 'size': '10275'}