However, using FPN presents a problem because ViT produces feature maps at a single scale (, \(1/16\) th), in contrast to the multi-scale feature maps produced by typical CNNs.We view the natural 2D spatial arrangement of intermediate ViT patch embeddings as a standard 2D feature map. To address this discrepancy, we employ a simple technique from [1]} (used for the single-scale XCiT backbone) to either upsample or downsample intermediate ViT feature maps by placing four resolution-modifying modules at equally spaced intervals of \(d/4\)  transformer blocks, where \(d\)  is the total number of blocks. See Figure REF  (green blocks).
