Hossain et al. have been instead the first ones to exploit attention models. In [1]}, they introduced a multi-column cnn-based architecture with scale-aware attentions for crowd density estimation and counting. The attention in their model plays a similar role to the "switch" (i.e., density classifier) in [2]}. However, in [2]}, the switch makes a hard decision by selecting a particular scale based on the output of the density classifier and only uses the features corresponding to that scale for the final prediction. The problem is that if the density classifier is not completely accurate, it might select the wrong scale and, in the end, lead to incorrect density estimation. In contrast, the attention model proposed in [1]} acts as a "soft switch." Instead of selecting a particular scale, the authors re-weight the features of a particular scale based on the attention score corresponding to that scale.
On the other hand, the authors of [5]} proposed RANet (Relational Attention Network), characterized by a self-attention mechanism for capturing interdependence of pixels. RANet improves the self-attention mechanism by introducing two attention modules, i.e., local self-attention (LSA) and global self-attention (GSA). To be more specific, LSA applies the self-attention to the original feature, only operating on spatially local neighbors which are closely correlated to the central position. Moreover, correlated pixels from long distances should also be taken into account for capturing inter-dependencies, while, to be efficient, GSA selects distinctive pixels from the whole map by max pooling.
