After time synchronization, we calibrate the multi-camera system.
The Record3D app provides camera parameters and poses at each time step, but the poses only relate to each other within each capture sequence.
In fact, each camera pose is recorded as relative pose to the first frame in each sequence, with the first pose being identity.
We therefore need to solve the relative \(SE(3)\)  transforms between the first frame in each test sequence with respect to the first training frame.
This problem can be formulated as a Perspective-n-Point (PnP) problem where, given a set of 3D points and their corresponding 2D pixels in two sequences, we aim to solve the camera pose.
In practice, given a training RGBD frame and a testing RGB frame, we compute a set of 2D correspondences by SIFT feature matching [1]} and obtain their 3D point positions (in the training sequence's world space) by back-projecting the 2D keypoints with the training frame depth map.
This process is repeated for all time steps.
We exploit our problem structure by constraining the camera poses within each test sequence to be the same, i.e., static camera.
We use the RANSAC PnP solver [2]} in OpenCV [3]}.
