In this paper we propose a novel, highly specific distance measure between volumetric images represented as invariant keypoints. Our measure generalizes the Jaccard index to account for soft equivalence between keypoints, and a novel kernel estimator is proposed to model keypoint geometry in terms of location and scale within normalized image space. The soft Jaccard index is used as a distance measure to predict pairwise relationships between MRI brain scans of 1010 subjects from 434 families, including siblings and twins, and significantly improves upon previous work based on binary equivalence and appearance descriptors [1]}, [2]}. We report the first results for predicting relationships from medical images, a new task, where monozygotic twins can be identified with virtually perfect accuracy. A minor modification allows the Jaccard distance to predict group labels such as sex with high accuracy. Our geometry kernel requires spatial normalization, however trials involving various linear subject-to-atlas alignment solutions including robust keypoint-based alignment [3]} show that the prediction performance to be insensitive to the specific alignment solution used. The improved prediction afforded by our geometry kernel thus suggests that family members tend to align in a consistent manner regardless of the solution used.
