The task of pairwise relationship prediction may be viewed as the finest grain of categorization, where the number of unique pairwise relationship labels is linear \(O(N)\)  in the number of data \(N\) , e.g. 434 families from N=1010 images, as opposed to typical classification where all data are associated with a small number of labels (e.g. male, female). Pairwise prediction thus represents a challenge for ubiquitous deep neural network methods, which generally require large numbers of training data per category, and training for specific modalities and body parts. In contrast, generic 3D SIFT keypoints may be used as-is with arbitrary imaging modalities and contexts with no training. In future work, generic keypoint correspondences here could potentially be used to train domain-specific keypoint models [1]}, [2]}. Auto-encoders used for anomaly detection could potentially be adapted to generate subject-specific codes between pairs of images [3]}. Analysis beyond pairs of subjects could be generalized via graph theory to model the clique structure of families [4]}. All code required to reproduce our results may be obtained at https://github.com/3dsift-rank.
