ZoomCount: A Zooming Mechanism for Crowd Counting in Static Images

This paper proposes a novel approach for crowd counting in low to high density scenarios in static images. Current approaches cannot handle huge crowd diversity well and thus perform poorly in extreme cases, where the crowd density in different regions of an image is either too low or too high, leading to crowd underestimation or overestimation. The proposed solution is based on the observation that detecting and handling such extreme cases in a specialized way leads to better crowd estimation. Additionally, existing methods find it hard to differentiate between the actual crowd and the cluttered background regions, resulting in further count overestimation. To address these issues, we propose a simple yet effective modular approach, where an input image is first subdivided into fixed-size patches and then fed to a four-way classification module labeling each image patch as low, medium, high-dense or no-crowd. This module also provides a count for each label, which is then analyzed via a specifically devised novel decision module to decide whether the image belongs to any of the two extreme cases (very low or very high density) or a normal case. Images, specified as high- or low-density extreme or a normal case, pass through dedicated zooming or normal patch-making blocks respectively before routing to the regressor in the form of fixed-size patches for crowd estimate. Extensive experimental evaluations demonstrate that the proposed approach outperforms the state-of-the-art methods on four benchmarks under most of the evaluation criteria.


I. INTRODUCTION
I N recent years, convolutional neural networks have attracted a lot of attention and been successfully applied to various computer vision problems, such as object detection [24], [29], [46], face recognition [8], depth estimation [16], [17], image classification [7], [50], image-to-image translation [44], [45], and crowd counting [34]. Crowd counting is an integral part of crowd analysis. It plays an important role in event management of huge gatherings like Hajj, sporting, and musical events or political rallies. Automated crowd count can lead to better and effective management of such events and prevent any unwanted incident [18]. Crowd counting is an active research problem due to different challenges pertaining to large perspective, huge variance in scale and image resolution, The work was supported in part by NSF NRI and USDA NIFA under the award no. 2019-67021-28996, and the Nvidia GPU grant.
U. Sajid [3] methods overestimate in case of crowd-like cluttered background patches as in (a) and (b), where there is no crowd at all. Similarly, these methods highly underestimate or overestimate in two extreme cases, where most crowd patches belong to either high or low-crowd count as in (c) and (d) respectively, as compared to the ground truth (GT).
severe occlusions and dense crowd-like cluttered background regions. Also, manual crowd counting subjects to very slow and inaccurate results due to the complex issues as mentioned above.
To obtain accurate, fast and automated results, CNN-based approaches have been proposed that achieve superior performance over traditional approaches [9], [10], [43]. CNNbased methods can be broadly classified into three categories; regression-based, detection-based, and density map estimation methods. Regression-based methods [41] directly regress the count from the input image. However, these CNN regressors alone cannot handle huge diversity in the crowd images varying from very low to very high. CNN detection-based methods [13], [32] first detect persons in the image and then sum all detection results to yield the final crowd count estimate. Detection-based methods perform well in low crowd images but could not be generalized well to high-density crowd images as detection fails miserably in such cases due to very few pixels per head or person. Density map estimation methods [22], [39], [40]   Left and right graphs compare ten cases each, belonging to high dense and low dense extreme respectively for Density map [3], DenseNet [20] based direct regression and our method. As shown, other models either highly underestimate or overestimate, whereas the proposed method remains the closest to the ground truth (GT) bar in most cases.
for each image pixel. The final estimate is then calculated by summing all density map values. These methods do not rely on localizing crowd but rather on estimating crowd density in each region of the crowd image. Density map estimation methods outperform other approaches and recent state-of-theart methods mostly belong to this category. However, density per pixel estimation remains a huge challenge as indicated in [31] due to large variations in the crowd density across different images. This naturally leads to a question: In which scenarios these methods may fail and why?
One key issue with regression and density map methods is that they only rely on direct count estimate and density map estimation per pixel for the input image respectively, thus, they may get subjected to large crowd count for cluttered background image patches. As shown in Fig. 1, models [22] based on these methods consider this 224 × 224 image patch as a crowd patch and make false estimates, making the system unreliable as similar patterns are bound to occur in many practical scenarios.
In addition, we observe that both types of methods perform well for images which contain most crowd patches with neither low nor high crowd density. Problem arises when images have most crowd patches with either high or low-density crowd. Due to the limitation in handling such practical diversity in crowd density, these methods may either highly underestimate or overestimate the crowd count in these two extreme cases, as shown in Fig. 1. To further explain this phenomenon, we analyze ten such cases for both extremes separately from very recent UCF-QNRF dataset [22] on the state-of-the-art density map method [3], [22] and direct regression-based method as shown in Fig. 2. It can be observed that, in both extreme cases, the crowd estimates are either highly overestimated or underestimated due to the limitations as discussed above.
To solve these fundamental problems, we propose a modular approach as shown in Fig. 3. It comprises of a Crowd Density Classifier (CDC), a novel Decision Module (DM), and a Count Regressor Module (CRM). The input image is first sub-divided into fixed-size patches (224×224) and fed to the CDC module that contains a deep CNN classifier to perform a four-way classification (low, medium, high-density, and no-crowd) on each patch. The classification module eliminates any crowd- Fig. 3. ZoomCount architecture. The test image X, divided into 224 × 224 patches, first passes through the Crowd Density Classifier (CDC) module, which discards no-crowd patches as classified by the robust 4-way DenseNet classifier. The accumulator stores patch count per class (P CC X ) of this image. Decision Module (DM), based on CDC module output and using either autonomous RFDB or heuristic-based RSE module, decides whether this image should be divided into all normal (N ormal) patches or make all either zoom-in (Z in ) or zoom-out (Zout) based patches before proceeding to the patch-based regressor (COU N T ER) for each patch crowd count. Image final crowd estimate is then obtained by summing all patches count. like background patches (no-crowd) from the test image and feeds the information about the number of patches belonging to each of the no-crowd, low, medium and high-density classes to the Decision Module (DM) using an accumulator. DM uses either the machine learning based RFDB module or heuristicbased Rule-Set Engine (RSE) module to determine if the image belongs to a case of low, normal, or high density. Based on the DM decision, this image is then divided into fixed-size patches using one of three independent image patch-making modules (Z in , N ormal, Z out ). The image that belongs to lowdensity extreme case is divided into patches using the zoomout (Z out ) patch-maker; the image that belongs to high-density extreme case is divided via zoom-in (Z in ) patch-making block; and the normal case image is split into patches using normal (N ormal) patch-maker. These patches are then routed one by one to the patch-wise count regressor (COU N T ER) for crowd estimate and the image total crowd count is obtained by summing all patches count.
The Z in block divides each input patch into four 112 × 112 patches, and then up-scales each patch by 2× before routing each patch to the count regressor. Intuitively, this module is further zooming-in into the image and looking in-detail all patches by using 1/2 input patch size instead of the original 224 × 224 patches. Similarly, zoom-out patch-maker divides the input image into 448 × 448 patches, and down-scales each patch by 2× as it is dealing with the image containing lowdensity crowd patches mostly. The normal case image directly employs the original 224 × 224 patch size with no up-scaling or down-scaling. The main contributions of this work include: • The paper reveals and analyzes the fact that extremely high and low dense crowd images greatly influence the performance of the state-of-the-art regression and density map based methods for crowd counting.
• A novel strategy is proposed to address the problem of counting in highly varying crowd density images by first classifying the images into either one of the extreme cases (of very low or very high density) or a normal case, and then feeding them to specifically designed patch-makers and crowd regressor for counting.
• A novel rule-set engine is developed to determine whether the image belongs to an extreme case. For images of extremely high density, a zoom-in strategy is developed to look into more details of the image; while for images of low-density extreme, a zoom-out based regression is employed to avoid overestimate.
• We created four new datasets, each from the corresponding crowd counting benchmark, for the training and testing of different machine learning algorithms to classify an image as normal, high or low-dense extreme case using its patches classification count. These manually verified datasets will facilitate the researchers in analyzing complex crowd diversity, which is at the core of the crowd analysis. The proposed ZoomCount scheme is thoroughly evaluated on four benchmarks: UCF-QNRF [22], ShanghaiTech [49], WorldExpo'10 [48], and AHU-Crowd [19]. The experimental results demonstrate the effectiveness and generality of the proposed strategy and rule-sets, which are never realized for crowd counting. The overall performance of the proposed model outperforms the state-of-the-art approaches on most of the evaluation criteria. The proposed models and source code, as well as the created datasets, will be available on the author's website.

II. RELATED WORK
Crowd counting is an active research area in computer vision with different challenges related to large perspective, occlusion, cluttered background regions and high variance in crowd density across different images. Earlier work [42], [43] focused on the head or full-body detection for counting using handcrafted features for detectors learning. These methods failed in case of high dense images, where it is hard to find such handcrafted features. The approaches were shifted towards regression based counting [9], [10], [33], where a mapping function was learned to directly regress count from local patches of an image. These methods improved the counting process, however, they could not handle huge crowd diversity and also lack awareness about crowd density across all parts of the image.
Recently, CNN-based approaches have been widely used [22], [26], [27], [41]. They are broadly categorized into three classes; Counting by detection, counting by direct regression, and counting using density map estimation. CNN-based object detectors [13], [32] detect each person in the image, and the final count is then calculated by summing all detections. These methods [25], [36] deteriorate in high density and severe occlusion cases, where each head only occupies a few pixels. Counting by direct regression methods [41] directly regress count by learning feature maps from the input image patch. Wang et al. [41] proposed an end-to-end AlexNet [23] based regressor for crowd count. These methods alone cannot handle huge diversity in different crowd images.
Density map estimation methods [6], [22], [26], [27], [39] learn to map crowd density per pixel of an image without localizing the counts. The final estimate is calculated by summing all density estimations. Zhang et al. [49] proposed a three-column CNN architecture (MCNN) to handle crowd diversity across images. Each column is designed to handle different scales using different receptive field sizes. Sindagi et al. [40] extended the idea of MCNN to incorporate contextual information for high-quality density maps generation. Recently, Sam. et al. [35] proposed SwitchCNN which routes each input patch to one of three independent CNN regressors using a switch CNN classifier. Based on the classification and regression idea, Sindagi et al. [39] designed a Cascaded-MTL that estimates count for the whole image by using cascaded 10-way classification prior and final density map estimation.
Crowd counting models, based on whole image estimation and training from the scratch, are subjected to over-fitting due to limited dataset availability (only a few hundred training images). Thus, patch-based models are widely used nowadays. The final sum is computed by adding up all patch count estimates. Liu et al. [27] proposed a hybrid approach by incorporating both regression and detection blocks using an attention-guided mechanism to handle low and high-density cases simultaneously. Li et al. [26] designed a CSRNet to get multi-scale contextual information by incorporating dilationbased convolutional layers. Idress et al. [22] proposed a composition loss based model for simultaneous crowd counting and localization. Existing methods perform worse in extreme cases where most crowd patches belong to either high density or low density. Moreover, these methods lack the ability to fully discard any cluttered background regions in the image, thus resulting in overestimate.

III. PROPOSED APPROACH
The proposed framework is shown in Fig. 3, which is composed of three modules namely Crowd Density Classifier (CDC), Decision Module (DM) and Count Regressor Module (CRM). The input image is first sub-divided into 224 × 224 size patches and each patch then passes through the CDC module for 4-way classification (low, medium, high-density or no-crowd). The accumulator gathers and feeds patch count per class information to the Decision Module. Based on accumulator information and utilizing either Random Forest based Decision Block (RFDB) or heuristic-based Rule-Set Engine (RSE), DM routes this image to one of three specialized patch-making blocks (Z in , N ormal, Z out ) of CRM where the input image is divided into corresponding patches, followed by the crowd estimate for each patch via crowd regressor (COU N T ER). Finally, the image crowd count is calculated by summing all patches count. Below we will discuss the details of each module, as well as the rules defined for the two possible extremes.

A. Crowd Density Classifier (CDC) Module
The CDC module is composed of a deep CNN 4-way classifier that specializes in making a distinction between nocrowd (NC), low-density (LC), medium-density crowd (MC), and high-density crowd (HC) for each input patch. Let X be a test image sub-divided into N patches [x 1 , x 2 , ...x N ], each with a size of 224 × 224. The accumulator gathers each patch classification result for the input image X as follows: for i = 1, 2, ....N and y belongs to either NC, LC, MC or HC class label. In the end, the accumulator passes the patch count per class (P CC X ) of this image to the decision module (DM) as: where P N C , P LC , P M C and P HC denote the total number of patches being classified as NC, LC, MC and HC respectively of the image X. Patches being classified as NC are discarded, and thus remaining {N − P N C } crowd patches are going to be used for final crowd estimate. As a result, the crowd-like cluttered background regions (such as the tree leaves shown in Fig. 1), which may result in overestimation otherwise, will be eliminated.
Definitions of NC, LC, MC and HC class labels. During experiments for each crowd counting benchmark dataset, we randomly extract patches from its training images for the CDC classifier training and assign a ground truth class label (NC,LC,MC,HC) to each extracted patch. Since these datasets also contain the localization of people, so we generate the ground truth class label for each patch using this information Blue blocks represent Dense Blocks 1, 2, 3 and 4 from left to right, followed by a fully connected layer and final softmax 4-way classification. 224 × 224 size input patch is expected. and the maximum people count possible in any image patch of the corresponding dataset. LC class label is assigned to a patch if the ground truth people count for that patch is less than or equal to 5% of the maximum possible count but greater than zero as zero crowd means NC class patch. Similarly, patches with ground truth people count between 5% to 20% of the maximum possible count are assigned the MC class label, while patches containing more than 20% of the maximum people count are labeled as HC category patches. In the end, a total of 90, 000 patches, with an equal amount per class label, are generated for the CDC classifier training in each benchmark setting. Example patches for each class label are shown in Fig. 4.
Classifier Details. We use DenseNet-201 [20] as our 4-way classifier, as shown in Fig. 5. It has four dense blocks with transition layers (convolution and pooling) in between them to adjust feature maps size accordingly. The DenseNet-201 has consecutive 1 × 1 and 3 × 3 convolutional layers in each dense block in {6, 12, 48, 32} sets respectively. At the end of the last dense block, a classification layer is composed of 7 × 7 global average pooling, followed by 1000 − D fully connected layer and the final 4-way softmax classification with cross-entropy loss.

B. Decision Module (DM)
The decision module, based on the CDC module output P CC X , decides if the test image should be treated as a normal image or a low or a high-density extreme case image. DM makes this decision by utilizing one of the two separate and independent decision-making blocks, namely Rule-Set Engine (RSE) and Random Forest based Decision Block (RFDB). RSE is a novel heuristic-based approach which employs the rule-sets to detect if the test image is either an extreme or a normal case, while RFDB is an automated decision-making block based on Random Forest algorithm that learns to map the test image features (P N C , P LC , P M C , P HC ) to the respective class label (Z in , N ormal, Z out ). We also create new RFDB training datasets, each from corresponding crowd counting benchmark, for the training of RFDB module as explained in Sec. III-B3.
1) Rule-Set Engine (RSE): The accumulated patch count per class (P CC X ) from CDC module is tested against two different rule-sets to determine if an input image is a case of low or a high density extreme or a normal one so that

Algorithm 1 Rule-Set Engine Algorithm
Input: P CC X (Patch Count per Class for Test Image X)= {P NC , P LC , P MC , P HC } Output: Normal or Z in or Z out Let P all = P N C + P LC + P M C + P HC if input patch count satisfies any of following rules then if P N C > 0 and P HC <= (P all * 0.05) end else if input patch count satisfies any of following rules then Output = Z in Rule 5: if P LC + P M C == 0 Rule 6: if P LC + P HC == 0 Rule 7: if P HC > (P all * 0.50) Rule 8: if P N C > 0 and P M C >= (P all * 0.33) and P HC >= (P all * 0.33) end else Output = N ormal it can be divided into patches using the most suitable patchmaking block (Z in , N ormal, Z out ). The overall goal of RSE is to encourage an image with more number of high-density patches to pass through zoom-in patch-making block (Z in ), whereas the image with more number of low-density patches goes through a zoom-out patch-making block (Z out ). If the image does not belong to any of the two extreme cases, it will be treated as a normal case that uses the normal patch-maker (N ormal).
Rules. The RSE module consists of two generalized rulesets, aiming to detect the images belonging to any of the two extreme cases: the low-density extreme (Rules 1-4) and the high-density extreme (Rules 5-8). As illustrated in Algorithm 1, if no rule applies to the test image X, it will use N ormal patch-maker, whereas the image satisfying any rule from (1 − 4) or (5 − 8) will generate its patches using Z out or Z in patch-making blocks respectively. Each rule is explained in detail in Table I. This table also shows the most affected images by a specific rule in terms of resolution. For example, Rule 7 is highly applicable on relatively lower resolution (LR) images, whereas Rule 2 can affect images of any resolution equally. It is important to note that these rule-sets are used consistently and evaluated across all four publicly available datasets in the experiments, thus demonstrating the generality and efficacy of such rule-sets. In addition, the current rule sets are extendable by adding more rules to refine the classification/decision process. Please note that all parameters in Table  I are chosen empirically. 2) Random Forest based Decision Block (RFDB): The scalable rule-sets based decision process yields promising results as demonstrated throughout the experiments in Sec. V. Nevertheless, there are many heuristics to handle and it requires manual input and special attention while inducting new rules. To address this issue, we propose an automated machine learning based approach that learns the decision process by mapping the four features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) to respective class label (Z in , N ormal or Z out ) for each image, where the features denote percentages instead of total image patches belonging to NC, LC, MC and HC classes respectively and labels represent zoom-in, normal and zoomout patch-making blocks required to generate the patches from the particular input image before proceeding to the count regressor. We employ percentages for features because of the huge variance in resolution across different images in a dataset, which directly influences the features and hence training quality. In addition, since there is no such dataset available for the crowd counting problem to-date that can help in learning this mapping, thus we generate a new RFDB training dataset from each corresponding benchmark as explained in detail in next subsection. To automate the decision block process, we explored different machine learning classification models and found the random forest-based model to be the most effective as demonstrated in the experiments in Sec. V. Thus, we choose the random forest algorithm and hence named this module as Random Forest based Decision Block.
Random Forest (RF), being a bootstrap aggregation or bagging based ensemble method, can be used both for classifi-cation and regression. We employ the RF algorithm to classify the four features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) to a class label of (Z in , Normal or Z out ) by building, training and tuning a large collection of de-correlated binary decision trees. Each tree then casts a vote for class prediction for the test sample. Finally, the class label with a majority vote is assigned to that test sample i.e., the input image.
Each RF decision tree t k is built using a bootstrap sample BS(t k ) which is generated from the training data. Such bootstrap sample is given as: for K = 0, 1, 2, ...N − 1, where N denotes the total number of RF trees. Each row represents one training sample for the tree t k with the class label as the last entry. We use N = 100, which is set empirically as no significant improvement has been observed in performance beyond this number. The trees are grown using the classification and regression (CART) algorithm, where the nodes get split until all leaves become unmixed or contain less than m min samples [1]. We use m min = 2 throughout our experiments, thus splitting nodes until they contain either only one sample or become pure. To quantify the quality of a tree node split, Gini Impurity has been used as: where L denotes the total unique class labels and F i denotes the frequency of class label i at node n. During testing, each RF tree gives its class prediction for test image X. Final class label is obtained by the majority vote criterion [14] as follows: where C k (X) represents the class prediction by the k th RF tree. Feature Importance (FI) depicts the role of each feature in determining the node split and eventually the quality of the RF decision trees building. Features with much lesser FI value can be easily discarded as they do not play any significant role in decreasing the node impurity. As shown in the left graph in Fig. 6, all four features have approximately the same FI values in each RFDB dataset. Thus, we keep and use all four available features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) in all four newly generated RFDB datasets.
3) Dataset generation for RFDB: The RFDB module learns to map the image extracted features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) to the respective class label (Z in , N ormal or Z out ) using training dataset with the corresponding mapping. No such dataset has been created to-date. Thus, for each benchmark (ShanghaiTech [49], UCF-QNRF [22], AHU [19]), we created a new respective RFDB dataset which contains this mapping. To create the new RFDB dataset, each training image's required features (P N C(%) , P LC(%) , P M C(%) , P HC(%) ) are extracted using ground truth crowd localization information and definitions of class labels (N C, LC, M C, HC) as stated in III-A, followed by manual verification and ground truth (GT) class label assignment. To ensure the quality of the generated dataset, each sample entry was then double checked for any inconsistency, duplicates, missing and erroneous cases. For the extracted features, we use percentages instead of the actual number of patches (P N C , P LC , P M C , P HC ) belonging to each category because of the huge resolution difference across the images within each benchmark dataset.
Statistics. For each of the four crowd counting benchmarks, we create the corresponding RFDB dataset using its corresponding training images. For instance, in the case of ShanghaiTech dataset (300 training images), we generate the new 300 samples RFDB dataset with each entry being created using one of the respective training image, followed by manual verification that also includes removal or modification of inconsistent entries. In total, 220, 2830 and 812 samples are finalized for the three RFDB datasets based on ShanghaiTech [49], WorldExpo'10 [48] and UCF-QNRF [22] benchmarks, respectively. For AHU [19] based RFDB dataset, 90 out of 96 available entries are kept on average with 5-fold crossvalidation. The lower graph in Fig. 6 shows the percentage of each class label in all four newly created RFDB datasets.

C. Count Regressor Module (CRM)
The CRM module comprises of three independent patch-making blocks and a deep CNN count regressor (COU N T ER). The decision module routes the test image to one of these patch-makers for dividing it into 224 × 224 patches after required up-scaling or down-scaling, followed by the crowd count for each image patch via the count regressor (COU N T ER). The regressor employs DenseNet-201 [20] inspired architecture with a single neuron after the fully connected layer to directly regress the crowd count. Mean squared error (MSE), as defined below, has been employed as the loss function for the count regressor c : where N is the number of training patches per batch, Y i is the ground truth crowd count for the input patch X i , and F is the function that maps the input patch X i to the crowd count with learnable parameters Θ. Zoom-in based Patch Maker (Z in ): Ideally, the decision module (DM ) routes the image, with most crowd patches being classified as high-density crowd, to this patch-maker. The image, using this patch-maker, is further sub-divided into equal 112 × 112 patches, and then up-scaled by 2× before proceeding to the count regressor for each patch crowd count. Intuitively, it looks into each patch in detail by estimating the count on smaller zoomed-in highly crowded patches. In this way, it greatly stabilizes and improves the count estimate for high-density images, where other methods may either underestimate or overestimate too much due to fixed patch sizes, as demonstrated in the experiments Sec. V.
Zoom-out based Patch Maker (Z out ): This block is responsible for handling the low-density extreme case images as detected and routed by the decision module. Z out takes 448 × 448 original patches of the test image X, down-scales them by 2 times, and feeds each resultant patch to the CDC classifier to eliminate any no-crowd patches, as shown in Fig.  7. The count estimate for each crowd patch is then computed through CRM count regressor (COU N T ER) followed by the image total count estimate, which is the sum of all patches crowd counts. In other words, it assists the count regressor by using larger area per input patch (448 × 448 down-scaled to 224 × 224) which alleviates the overestimation problem.
Normal case: In N ormal case, the images are divided into 224 × 224 size patches with no up-or down-scaling before patch-wise count regression. It is also worth mentioning that there is no need to explicitly look for and eliminate any nocrowd patches in case of N ormal and Z in case images as such background patches are automatically removed during the CDC module classification process, and thus we can also reuse the remaining CDC module crowd patches in both cases for crowd estimate.
The input image that is classified as N ormal case, may contain mixed crowd numbers in different regions. Empirically, it has been observed that the deep CNN based crowd counter (i.e. the CRM regressor) can directly handle such images effectively without any rescaling or classification process. Thus, this work only focuses on the images of extreme case (low-density or high-density) that contain most of the regions with same crowd level (low-or high-density), since these extremes have huge influence on crowd count, and a special attention to these cases will significantly mitigate the over-or under-estimation issue as discussed in the introduction.

D. ZoomCount and Switch-CNN Comparison
Switch-CNN [35] is one of the state-of-the-art crowd counting models, and it classifies the input image each patch into one of three crowd-density labels (low, medium, and high). However, our approach employs this classification process fundamentally different from Switch-CNN. The key differences and comparison are as follows: • Crowd-Density Classification Purpose and Usage.
Switch-CNN primarily uses the crowd-density classification to route the input patch to one of three specialized count regressors. Subsequently, crowd estimation for each patch is being done using one of these deep count regressors. On the other hand, we employ the crowddensity classification process to facilitate the Decision Module (DM) in detecting whether the input image is either one of the two extremes (low-or high-density) or a normal case image. Then, the patches are rescaled accordingly before proceeding to the only count regressor for the crowd estimation. Thus, the underlying usage and purpose of crowd classification completely differ in both methods. • Patch-based vs Image-based Decision. Based on the crowd-density classification as discussed in the above point, Switch-CNN selects the appropriate crowd count regressor individually for each input patch (patch-wise decision). While ZoomCount selects the most appropriate rescaling operation suitable for the whole image (imagebased decision) using this information. Thus, our method also takes the global context of the image into consideration during the decision-making process instead of relying solely on the local image patches.
In addition, Switch-CNN trains using multiple complex training steps including pretraining, differential training, switch training, and coupled training, while ZoomCount employs standard classifier and regressor training process. Switch-CNN uses three specialized count regressors, whereas the proposed approach simplifies this process by employing only a single count regressor. Moreover, our method outperforms the Switch-CNN under all evaluation metrics as demonstrated in the experiments section.

A. Training Details
The CDC classifier and the count regressor (COU N T ER) expect fixed size patch of 224 × 224 as the input. For both modules, we randomly extract 112 × 112, 224 × 224 and 448 × 448 patches from the training images. Around 90,000 such patches with mixed crowd numbers are generated for each of these modules. The count regressor is trained for 80 epochs with Adam optimizer and a batch size of 16 and starting learning rate of 0.001, decreased by half after every 20 epochs. The classifier employs the stochastic gradient descent (SGD) based optimization with multi-step learning rate starting at 0.1 and decreased by half after 25% and 50% epochs with 80 epochs in total. For each dataset, around 10% training data has been used for validation as recommended in the corresponding literature. For the random forest algorithm in RFDB, we utilize machine learning library scikit-learn for python programming. The Random Forest model was trained using 100 RF decision trees, where each RF tree is trained using the bootstrapped sample with Gini Imprity as node split quality criterion. 10% of the training data has been used for validation in case of each RFDB dataset.

B. Evaluation Details
In order to make a fair and consistent comparison with other methods, we employ three evaluation metrics namely Mean Absolute Error (M AE), Mean Normalized Absolute Error (M N AE) and Root Mean Squared Error (RM SE) defined as below: where N denotes the total number of test images, and Y i andŶ i are the ground truth and the estimated counts respectively for the test image i.

V. EXPERIMENTS
In this section, we demonstrate both quantitative and qualitative results from extensive four benchmark datasets: UCF-QNRF [22], ShanghaiTech [49], WorldExpo'10 [48], and AHU-Crowd [19]. These datasets contain images with huge crowd variance, different camera perspective, and complex cluttered background regions. Details about each benchmark are given in Table II. At the end of this section, we also discuss computational time analysis and compare that with state-ofthe-art methods.
Two different versions of the proposed model, the Rule-Set Engine (ZoomCount-RSE) based and the automated RFDB module (ZoomCount-RFDB) based version, are being compared separately with the state-of-the-art techniques throughout this section. Both ZoomCount versions give almost identical and much better performance under most of the evaluation criteria on the four benchmark datasets.

A. Experiments on UCF-QNRF Dataset
The dataset was recently published by Idrees et al. [22], which is a challenging and the first dataset of its kind. On one hand, it contains images with resolution as high as (6666 × 9999) and as low as (300 × 377); on the other hand, crowd count per image ranges from a maximum value of 12, 865 to a minimum count of 65. The total number of annotations in this dataset is 1, 251, 642, indicating the level of crowd complexity. It contains 1535 images in total, out of which 1201 and 334 images are used for training and testing respectively. We compare ZoomCount with the state-of-the-art methods and tabulate the results in Table III. It is evident that both versions of our method outperform all other approaches in terms of MAE and MNAE; while performing competitively closer to the best in terms of RMSE.
In order to evaluate the influence of different rules, we perform the ablation experiments, as shown in Table IV. We analyze the effect of all rules (R1 to R8) by removing them one at a time in ZoomCount-RSE version of the proposed method. As shown in the results, doing so greatly decreases the performance of our method, thus demonstrating the importance of those rules. We also analyze the effect of removing both or either of the zoom-in and zoom-out patch-makers in the experiments. From the results in Table IV, it is evident that both modules play an effective role in improving the overall performance of our method. The last three columns of Table  IV show the number of the test images passed through the zoom-in, normal and zoom-out patch-makers respectively. In the original setting, 75 (∼ 22%) images passed through the zoom-in patch-maker, whereas the zoom-out block handled 162 (∼ 48.5%) images and normal patch-maker was used only for 97 (∼ 29.5%) images, showcasing quantitative importance of these extreme case handlers, as shown in Fig. 8. We also compare the crowd estimate of ten test images each, for both extreme cases with DenseNet [20] direct regression and the state-of-the-art CL [22] density map method. Our method performs much better in both cases, as shown in Fig. 2.

B. Experiments on ShanghaiTech Dataset
The ShanghaiTech part A dataset contains a total of 482 images with 241,677 annotations, randomly collected from the internet, with a split of 300 and 182 images for training and testing respectively. We compare our method with the stateof-the-art methods as shown in Table V. The results show that our method outperforms all other methods on every evaluation  [40] 73.6 -106.4 CSRNet [26] 68.2 -115.0 IG-CNN [4] 72.5 -118.2 L2R [28] 72.0 -106.6 ICC [31] 68.5 -116.2 SA-Net [6] 67.0 -104.5 Deep-NCL [37] 73  Table VI, where we remove each rule one at a time. It is clear that the error increases by removing these rules. In the same table, We also analyze the effect of removing the zoom-in and zoom-out blocks separately and together. As expected, the performance plunges dramatically as error increases without using them. The last three columns show the number of test images passing through the zoom-in, normal and zoom-out patch-makers respectively. In the original setting, 21 (∼ 11.5%), 121 (∼ 66.5%) and 40 (∼ 22%) images are handled by the zoom-in, normal and Fig. 9. 182 test images are divided into ten groups with total crowd count in each group increasing from left to right. Each group contains 18 images except group number 10. Vertical axis indicates average count for each group. It is evident that ZoomCount remains closer to the ground truth (GT) bar in most cases as compared to the state-of-the-art methods. zoom-out blocks respectively, thus proving the quantitative importance of all of them and associated rules in ZoomCount-RSE, as shown in Fig. 8. In Fig. 9, we analyze the performance of our method on the average count across image groups with different total crowd counts. As compared with the state-ofthe-art methods, ZoomCount performs the best in most cases.

C. Experiments on WorldExpo'10 Dataset
The WorldExpo'10 [48] is a large dataset, composed of 1132 video sequences taken by 108 different cameras. The training set consists of 3380 images from 103 different scenes, whereas the testing set has 5 scenes with a total of 600 frames. This benchmark also consists of Region of Interest (RoI) and perspective maps. We only utilize the RoIs in the images during training and testing stages. The MAE based evaluation results on the five test scenes and the average MAE error are shown in Table VII. As shown, the proposed model achieves the lowest average MAE and the best performance on two scenes (S1 and S3). For the other three scenes, the proposed scheme yields reasonable and competitive results to the stateof-the-art. During the testing experiments on ZoomCount-RSE model, 168 (∼ 28%), 240 (∼ 40%) and 192 (∼ 32%) images are handled by the zoom-in, normal, and zoom-out blocks, respectively, as shown in Fig. 8. This demonstrates the quantitative importance of the associated rules in ZoomCount-RSE.

D. Experiments on AHU-Crowd Dataset
AHU-Crowd [19] dataset contains 107 images with 45, 807 human annotations. The crowd count ranges from 58 to 2201 per image. As per the standard being followed for this dataset [19], we performed 5-fold cross-validation and evaluated our method using the same three evaluation metrics. ZoomCount outperforms all the other methods as shown in Table VIII. It is worth-mentioning that ZoomCount decreases MAE and MNAE significantly by ∼ 7.5% (81 to 74.9) and ∼ 4.5% (0.199 to 0.190) respectively, whereas RMSE decreases drastically by ∼ 11.2% (124.9 to 111).

E. CDC Classifier and CRM Regressor Architecture Selection
Choosing an appropriate network architecture for the CDC classifier and the CRM regressor is essential for the effectiveness of the proposed scheme. In this section, we analyze different state-of-the-art architectures for this objective, including VGG-16 [38], ResNet-101 [15], and DenseNet-201 [20]. For the classifier choice evaluation, the final 1000-way classification layer in VGG-16 and ResNet-101 is replaced with a 4-way classification layer. Similarly, for the CRM regressor evaluation, the final FC layer in VGG-16 and ResNet-101 networks is followed by a single neuron to directly regress the crowd count with mean square error (MSE) loss function as given in Eq. 6. As per the standard practice, VGG-16 is configured with the batch-normalization layer being added inbetween each convolution and activation layer. DenseNet-201 classifier and regressor are configured as stated in sec. III-A, and all architectures are trained as discussed in section IV-A.
The results on the test images of ShanghaiTech Part-A dataset are shown in Table IX. We can see from the results that DenseNet-201 based classifier outperforms other state-ofthe-art architectures with the highest classification accuracy and ∼ 7% improvement, thus, we choose this model as the CDC classifier in the proposed method. For the choice of the CRM Regressor, we evaluated different architectures in terms of mean absolute error (MAE) in two separate experimental settings. In the first case, each regressor architecture has been evaluated separately by plugging-in into the proposed scheme as the CRM Regressor with the same architecture, while in the second experiment setting, each regressor architecture has been evaluated independently as a standalone crowd count regression model. The results of both experimental settings are shown in the second-rightmost and the rightmost columns respectively of Table IX. From the results, we can see that using DenseNet based architecture as the CDC classifier and CRM Regressor achieves the highest classification accuracy and the lowest MAE errors, respectively.

F. RSE and RFDB based Decision Modules Performance
The classification accuracy of the decision module directly affects the overall performance of the proposed method. In this section, we show the individual classification accuracy performance of the RSE and RFDB based DM modules on the four benchmarks. The accuracy results are shown in Table  X, from which we can see that both modules perform quite effectively. Also, the RFDB version of the DM module gives slightly better accuracy in most cases compared to the RSE based DM module. The overall system outperforms the stateof-the-art owing to the improved and reasonable classification accuracy of the DM module.

G. RFDB Algorithm Selection
In this paper, we adopt the Random Forest algorithm for the RFDB module. In practice, other machine learning-based classification algorithms can also be employed. In order to choose the best one for our system, we experimented with different classifiers to select the appropriate decision-making algorithm. The results based on the ShanghaiTech and UCF-QNRF datasets are shown in Table XI. We observe that ensemble based methods perform better on our relatively smaller and imbalanced RFDB datasets as they prevent over-fitting and high variance by combining several machine learning techniques. After evaluation, the Random Forest appears to be the best choice as the RFDB algorithm as shown in Table  XI, where the top five best results justify the selection of the Random Forest algorithm. For these experiments, we used machine learning library scikit-learn for python programming [2].

H. Qualitative Results
In Fig. 10, we show some good and bad case qualitative results from UCF-QNRF and ShanghaiTech datasets. We also compare our results with the ground truth (GT), DenseNet [20] Regression (DR) and the state-of-the-art density map methods. In each row, the first three cases demonstrate the good results followed by two bad estimates. The bad case results happen mostly due to the test image being detected as wrong extreme case type by the decision module (DM). We also show some crowd-density classification visual results to demonstrate the qualitative performance of the CDC classifier in Fig. 11.

I. Computational Time Analysis
In this experiment, we compare the proposed network with two state-of-the-art models (CSRNET [26],CP-CNN [40]) in terms of real computational time on the ShanghaiTech [14] dataset. The results are shown in Table XII, where T total and T avg represent the total and average time taken for the whole dataset test images respectively, whereas, T smallest and T largest represent the computation time taken for the smallest and the highest resolution test image respectively in the dataset. For the fairness of comparison, all networks have been evaluated on the same NVIDIA Titan Xp GPU.
As shown in the Table XII, our method takes a reasonable computation time (in between the two state-of-the-art networks with much closer to the one with the lesser time). However, the proposed method outperforms other models on all standard evaluation metrics. Since our method is modular, we have also shown the pipeline approach (ZoomCount − RSE pl , ZoomCount−RF DB pl ) on the same test images, where next image patches start using the CDC classifier once previous image patches are done using the classifier and now passing through the next modules ( 10. Some examples of the qualitative results. First and second rows show qualitative results of our method on the UCF-QNRF [22] and ShanghaiTech [49] datasets respectively. First three columns show good results, followed by two bad case images. Each result also shows the estimates of DenseNet [20] Direct Regression (DR) and the Density map method as a comparison.

Method
T total (secs) Tavg (secs) T smallest (secs) T largest (secs) MAE RMSE CSRNET [26] 60. regressor). We can see that this pipeline approach dramatically decreases the computational time further. This is useful in case of multiple test images. To the best of our knowledge, most of the crowd counting research papers do not provide time analysis, because the main objective for crowd counting in static images is to design a more effective model for counting with lesser error, and less consideration has been given towards the computational time. We believe that efficiency is also a major factor to compare and evaluate different crowd counting models.

VI. CONCLUSION
In this work, we have proposed a novel zoom-in and zoom-out based mechanism for effective and accurate crowd counting in highly diverse images. We propose to employ a decision module to detect the extreme high and low dense cases, where most state-of-the-art regression and density map based methods perform worse. The cluttered background regions are also discarded using the rigorous deep CNN 4-way classifier. Even without using any density maps, ZoomCount outperforms the state-of-the-art approaches on four benchmark datasets, thus proving the effectiveness of the proposed model.