Dual-Stream Transformer with Cross-Attention on Whole-Slide Image Pyramids for Cancer Prognosis
The cancer prognosis on gigapixel Whole-Slide Images (WSIs) has always been a
challenging task. Most existing approaches focus solely on single-resolution
images. The multi-resolution schemes, utilizing image pyramids to enhance WSI
visual representations, have not yet been paid enough attention to. In order to
explore a multi-resolution solution for improving cancer prognosis accuracy,
this paper proposes a dual-stream architecture to model WSIs by an image
pyramid strategy. This architecture consists of two sub-streams: one for
low-resolution WSIs, and the other especially for high-resolution ones.
Compared to other approaches, our scheme has three highlights: (i) there exists
a one-to-one relation between stream and resolution; (ii) a square pooling
layer is added to align the patches from two resolution streams, largely
reducing computation cost and enabling a natural stream feature fusion; (iii) a
cross-attention-based method is proposed to pool high-resolution patches
spatially under the guidance of low-resolution ones. We validate our scheme on
three publicly-available datasets with a total number of 3,101 WSIs from 1,911
patients. Experimental results verify that (i) hierarchical dual-stream
representation is more effective than single-stream ones for cancer prognosis,
gaining an average C-Index rise of 5.0% and 1.8% on a single low-resolution and
high-resolution stream, respectively; (ii) our dual-stream scheme could
outperform current state-of-the-art ones, by an average C-Index improvement of
5.1%; (iii) the cancer diseases with observable survival differences could have
different preferences for model complexity. Our scheme could serve as an
alternative tool for further facilitating WSI prognosis research.