paragraph40.75ex plus1ex minus.2ex-1emLeopart fine-tuning loss.
To train the ViT's spatial tokens, we first randomly crop the image \(V\) -times into \(v_g\)  global views and \(v_l\)  local views.
When sampling the views we compute their pairwise intersection in bounding box format and store it in a matrix \(B\) .
We denote the transformed version of the image as \(x_{t_j}, j\in \lbrace 1, \dots , V\rbrace \) .
Then, we forward the spatial tokens through a MLP projection head \(g\)  with a L2-normalization bottleneck to get spatial features for each crop:
\(g(f(x_{t_j})) = Z_{t_j}\ \in \mathbb {R}^{D \times N}\) .
To create prediction targets, we next find an optimal soft cluster assignment \(Q_{t_j}\)  of all spatial token's feature vector \(Z_{t_j}\)  to \(K\)  prototype vectors \([c_1, \dots , c_K] = C \in \mathbb {R}^{D \times K}\) .
For that, we follow the online optimization objective of SwAV [1]} that works on the whole image batch \(b\) .
\(Q\)  is optimized such that the similarity between all feature vectors in the batch and the prototypes is maximized, while at the same time being regularized towards assigning equal probability mass to each prototype vector.
This can be cast to an optimal transport problem and is solved efficiently with the Sinkhorn-Knopp algorithm [2]}, [3]}.
Instead of optimizing over \(|b|\)  feature vectors, we instead optimize over \(N \! \cdot \! |b|\)  spatial feature vectors as we have \(N\)  spatial tokens for each image.
As our batchsizes are small, we utilize a small queue that keeps the past 8192 features, as is done in SwAV.
