A Cloud-based Deep Learning Framework for Early Detection of Pushing at Crowded Event Entrances

Abstract: Crowding at the entrances of large events may lead to critical and life-threatening situations, particularly when people start pushing each other to reach the event faster. Automatic and timely identification of pushing behavior would help organizers and security forces to intervene early and mitigate dangerous situations. In this paper, we propose a cloud-based deep learning framework for automatic early detection of pushing in crowded event entrances. The proposed framework initially modifies and trains the EfficientNetV2B0 Convolutional Neural Network model. Subsequently, it integrates the adapted model with an accurate and fast pre-trained deep optical flow model with the color wheel method to analyze video streams and identify pushing patches in real-time. Moreover, the framework uses live capturing technology and a cloud-based environment to collect video streams of crowds in real-time and provide early-stage results. A novel dataset is generated based on five real-world experiments and their associated ground truth data to train the adapted EfficientNetV2B0 model. The experimental setups simulated a crowded event entrance, while the ground truths for each video experiment was generated manually by social psychologists. Several experiments on the videos and the generated dataset are carried out to evaluate the accuracy and annotation delay time of the proposed framework. The experimental results show that the proposed framework identified pushing behaviors with an accuracy rate of 87% within a reasonable delay time.

\section*{Introduction}

Human safety and comfort are at stake in large-scale events, mainly where abnormal behavior may appear~\cite{farooq2022motion, alia2022exploitation}. Timely detection of such behaviors is essential, as it can provide valuable information to the organizers and the security team for better event management, thereby improving people’s safety and comfort~\cite{tyagi2022review}. To achieve that, surveillance cameras have recently been widely integrated with detection techniques. Some of these techniques rely on the manual analysis of incoming video streams from the cameras, to identify potential abnormal behaviors. However, this manual procedure is tedious, labor-intensive, and may lead to errors due to the limitations of human abilities~\cite{sreenu2019intelligent}. An alternative solution is automatically analyzing the video streams and detecting the abnormality among crowds in real-time. Several behaviors have been addressed automatically, such as people suddenly grouping or dispersing~\cite{george2018crowd}, walking in the wrong direction~\cite{roshtkhari2013line}, human falls~\cite{santos2019accelerometer}, high density crowds~\cite{xiang2022crowd}, and acts of violence ~\cite{kaur2022violence}. However, to the best of our knowledge, early pushing behavior detection at crowded event entrances has not received attention in the existing literature, despite its importance~\cite{usten2022pushing}.

The entrances of events are organized as bottlenecks for access control, ticket validation, or security check~\cite{adrian2020crowds}. In these scenarios, some pedestrians could start pushing each other to gain faster access to the event.
Indeed, such behavior increases the crowd's density over time~\cite{adrian2020crowds,haghani2019push}, resulting in a lack of comfort zones and, more importantly, can lead to dangerous situations~\cite{filingeri2017factors,johnson1987panic}. In such cases, early pushing detection could help the organizers and security team take some measures to alleviate dangerous situations, such as directing visitors to different areas and attempting to change behavior via acoustic information or other security measures. Moreover,  this detection could assist in ensuring a smoother flow at entrances and a higher level of well-being.
According to Lügering et al.~\cite{usten2022pushing}, pushing is defined as ``a behavior that can involve using arms, shoulders, or elbows; or simply the upper body, in which one person actively applies force to another person (or people) to overtake, while shifting their direction to the side or back, or force them to move forward more quickly.''
Additionally, Ref.~\cite{usten2022pushing} considers using gaps among crowds as a strategy of pushing because it is a form of overtaking~\cite{usten2022pushing}. Since manual pushing identification in real-time is complex, and the current automatic approaches in the abnormal behavior detection field are inefficient for early pushing detection; developing an automatic system for early pushing detection in live video streams is crucial.

Several automatic approaches for abnormal behavior detection in videos have been presented in the available literature. They generally can be classified into traditional-based and machine learning-based approaches~\cite{ammar2021deeprod,bahamid2022review}.
The main idea of the first group is to use some measures to distinguish between normal and abnormal behaviors.
For example, a high value in the entropy~\cite{zhang2019crowd} or the speed~\cite{fradi2015towards} may indicate the presence of abnormal behavior in crowds.
However, these approaches are inefficient for complex behaviors due to the difficulty of describing them accurately by rules~\cite{bahamid2022review}.
Furthermore, several traditional-based techniques need to calculate the features for the whole video or long live stream before applying the detection, which does not satisfy early detection requirements~\cite{ammar2021deeprod}.
To overcome these limitations, the approaches in the second group learn the behaviors from the data.
In particular, Convolutional Neural Network (CNN)-based approaches achieved remarkable success in various image or video object/behavior detection applications~\cite{wu2022research,kim2019cnn,ilyas2022effective}, including abnormal behavior detection~\cite{tay2019robust,wang2021comparative}.
The main advantage of CNN is that it can automatically learn the relevant features and classification from data without any human effort~\cite{yamashita2018convolutional}.
For instance, Tay et al.~\cite{tay2019robust} trained a customized CNN-based approach on normal and abnormal images to detect abnormal behaviors in low-dense crowds.
However, the lack of datasets containing the required abnormalities makes the learning of abnormal behaviors by CNN difficult.
Acknowledging this limitation, some CNN-based approaches~\cite{xu2019efficient, smeureanu2017deep} can learn from normal behaviors to detect abnormalities.
Preparing a dataset with only normal behaviors is easier than a dataset including normal and abnormal behaviors.
Nevertheless, these approaches are popular when abnormal behavior data is rare or not well-defined~\cite{khan2014one}.
In contrast, pushing behavior is well-defined and not rare, especially in high-density and competitive scenarios.
Moreover, all CNN-based approaches require a large dataset to build an accurate model~\cite{wang2021comparative, duman2019anomaly}.
Unfortunately, large datasets are generally unavailable in the abnormal behavior detection field.
In order to take advantage of CNN capabilities on small datasets, researchers recently combined a CNN with one or more handcrafted feature descriptors~\cite {ilyas2021hybrid,direkoglu2020abnormal}.
In these hybrid approaches, the descriptors extract valuable information from data. Then,  CNN learns the relevant features from the extracted information, where these features provide helpful information to the detection algorithms~\cite{alia2021enhanced, alia2017feature}.
For instance, Duman et al.~\cite{duman2019anomaly} employed  CNN and the classical Farnebäck optical flow to detect abnormal behaviors.
In a similar line of the hybrid approaches, Direkoglu et al.~\cite{direkoglu2020abnormal} combined Lucas–Kanade optical flow and CNN to detect escape and panic behaviors.

Hybrid-based approaches would be more appropriate for automatic pushing behavior detection because labeled data for this behavior is rare.
However, most available hybrid-based approaches are inefficient for pushing behavior detection since: (1) Their descriptors can only extract some required information from high-density crowds for pushing behavior representation.
(2) Some commonly used CNN architectures are ineffective at handling the increased variations in the pushing behavior (intra-class variance) and the high similarity between pushing and non-pushing behaviors (high inter-class similarity), can lead to misclassification.
In 2022, Alia et al.~\cite{alia2022hybrid} proposed a hybrid deep learning and visualization framework for pushing behavior detection in video recordings of crowded event entrances.
This framework used firstly CPU-based deep optical flow and visualization techniques to extract motion information. Then, it employed EfficientNetV1B0-based CNN with a false reduction algorithm for pushing patches annotation.
However, this framework does not cope with early detection requirements due to three reasons.
First, it can only handle offline-recorded videos. Second, the CPU-based deep optical flow technique employed in motion extraction is slow.
Third,  it needs to identify pushing for the whole video before producing the output.
Moreover, as reported by the authors, the accuracy of the EfficientNetV1B0 model decreases with complex scenarios of pushing.

To meet the requirements of early identification, a novel cloud-based deep learning system is proposed for pushing patches detection in the live video stream acquired from crowded event entrances. As shown in~\cref{fig:sys-arch}, the system comprises three main components. The first component is preprocessing, which aims to display the live camera stream on the web client in real-time and collects data about the entrance area (Region Of Interest (ROI)) from the stream every two seconds. Simultaneously, the second component, the motion descriptor, generates Motion Information Maps (MIMs)-patches from the collected data. Each patch represents the visual motion information for a specific region in ROI at a particular time. This component is based on pre-trained Recurrent All-pairs Field Transforms (RAFT) model~\cite{teed2020raft} and a color wheel method~\cite{flow_vis,baker2011database}, where RAFT is one of the promising deep optical flow models, and color wheel is widely used as a visualization optical flow method. In the last component, pushing detection and annotation, we aim to identify pushing MIM-patches and directly annotate them on the live stream on the web client.
Additionally, this component stores the annotated data in the cloud storage, which is crucial to assist planners and organizers
in evaluating their events and enhancing their future plans. To protect people’s privacy, the stored data is blurred.
The component
uses the trained adapted EffcientNetV2B0-based classifier~\cite{tan2021efficientnetv2} for the pushing patches identification. Furthermore, a new public dataset is built based on multiple real-world ground truths of pushing videos to train and evaluate the classification
model.

\begin{figure}[ht]
\centering
\includegraphics[width=1\linewidth]{./images/Architicture.pdf}

\caption{The proposed system architecture and data flow.
Firstly,  the preprocessing component launches the camera attached to the web client and shows a live stream on the client in real-time. At the same time, based on user-defined ROI coordinates, it collects a frame containing ROI only from the live stream every two seconds, named ROI keyframe, where ROI refers to the entrance area.
The motion descriptor component then constructs Motion Information Map (MIM)-patches from each pair of consecutive ROI keyframes. User-defined row ($n$) and column (m) are used to split MIM into  $n \times  m$ patches. After that, the pushing detection and annotation component identifies the pushing patches. Finally, this component directly annotates the pushing patches on the live stream on the web client and saves the blurred annotated ROI keyframes in the cloud storage.
}
\label{fig:sys-arch}
\end{figure}

\section*{Material and methods}
This section describes the employed methodology for developing the proposed system.
The following subsections discuss the system's architecture; dataset preparation; the training process and implementation details; and the utilized evaluation metrics.

\subsection*{The architecture of the proposed system}

\label{subsec:propsedsystem}
In this section, we describe the proposed system for early detection of pushing within the live camera stream of crowded event entrances, where the camera is fixed and top-view.
~\cref{fig:sys-arch} shows that the architecture of our system comprises three major components: preprocessing; motion descriptor; and pushing detection and annotation.

\subsubsection*{Preprocessing }

In order to reduce the computational time of the system without sacrificing performance, the preprocessing component directly displays the client camera stream on the web client. At the same time, it collects only the data required for detection purposes from the live stream.
Let  $\{f^t\}$ represents the live camera stream, where $t$ is the time of the frame $f$ in the stream. Firstly, this component displays the live stream on the web client in real-time without uploading it to the cloud. Then,  a frame $f^t$ is collected from the stream every two seconds, hereafter referred to as keyframe.
After that, this component utilizes the user-defined coordinates in pixel units to crop the entrance area $\bar{f^t}$ (ROI keyframe) from its corresponding keyframe $f^t$.  Finally, $\bar{f^t}$ is submitted as an input to the second component.
For the brevity, we name the ROI keyframe sequence $\bigl\{\bar{f}^t,\bar{f}^{t+ 2}, \bar{f}^{t+ 4},  \dots \bigr\}$ as
 $\{ \bar{f}_i \, | \, i=1,2,3, \dots   \}$, where $i$ is the order of the ROI keyframe in the stream, and $t$ is the time in seconds.
~\cref{fig:MIM-pipeline}a displays two examples of $\bar{f_i}$.

\begin{figure}[ht]
\centering
\includegraphics[width=1\linewidth]{./images/motionDescriptor.pdf}
\caption{Motion descriptor component pipeline. (a) A pair of ROI keyframes $\bar{f}_i$ and  $\bar{f}_{i+1}$.
(b) The pre-trained RAFT model aims to estimate the dense displacement field $d_i$ from the pair of ROI keyframes.
(c) Color wheel method is used to generate a Motion Information Map (MIM$_i$) from $d_i$.
(d) An example of MIM$_i$.
(e) The patching process divides MIM$_i$ into $n \times m$ MIM$_i$-patches $p_{i, k}$, where $p$ represents a patch, and $K$ is the order of $p$ in MIM$_i$.
(f) Examples of MIM$_i$-patches.}
\label{fig:MIM-pipeline}
\end{figure}
\subsubsection*{Motion descriptor}

Using this component, we aim to extract the crowd's motion characteristics at the patch level. More specifically, this component estimates the motion direction, magnitude, and associated spatio-temporal information from the crowds, and accordingly visualizes this information. The displayed information includes relevant features that are important for representing the pushing behavior.
As shown in~\cref{fig:MIM-pipeline}, the component uses the optical flow and color wheel methods to achieve its purpose.
Unlike the majority of the already used optical flow methods~\cite{farneback2003two,yin2019hierarchical},  a GPU-based pre-trained RAFT model performs well in terms of speed, accuracy, and generality for dense crowds~\cite{alia2022hybrid, teed2020raft}.
This model was created by training an ensemble of CNN and recurrent neural networks on the Sintel dataset to calculate the optical flow between two images~\cite{teed2020raft}.
Firstly, the component uses the pre-trained model to calculate the displacement of each pixel $\langle x,y  \rangle$ between each pair of  $\bar{f_i}$ and $\bar{f}_{i+1}$, generating the dense displacement field $d_i$. Each pixel location $\langle x,y \rangle$ in $d_i$ is presented by a vector, given by

\begin{equation}
\label{eq:OF_vector}
\langle u_{\langle x,y\rangle}, v_{\langle x,y\rangle} \rangle_{\bar{f_i}, \bar{f}_{i+1}}  =RAFT(\langle x,y\rangle_{\bar{f_i}, \bar{f}_{i+1}}  ),
\end{equation}
where $u$ and $v$ are horizontal and vertical displacements of a pixel at the $\langle x,y\rangle$ location between $\bar{f_i}$ and $ \bar{f}_{i+1}$,  respectively.
This implies that $d_i$ is a matrix of the vectors, as described in
\begin{equation}
\label{eq:dense_OF}
d_i=\bigg\{ \langle u_{\langle x,y\rangle}, v_{\langle x,y\rangle} \rangle_{\bar{f_i}, \bar{f}_{i+1}} \bigg\} _{(x, y)=(1,1)}^{(w,h)},
\end{equation}
where $w$ and $h$ are the $\bar{f_i}$ width and height, respectively.

After the estimation of $d_i$, the descriptor applies the color wheel method to deduce the visual motion information from $d_i$. It begins by calculating the direction $\theta$ and magnitude of each vector $\langle u_{\langle x,y\rangle}, v_{\langle x,y\rangle} \rangle$   in $d_i$ using~\cref{eq:angle} and the Euclidean distance ~\cite{cohen2004precalculus}, respectively. The color wheel then visualizes the magnitude and direction information to generate MIM$_i$  from the calculated information, where MIM$_i \in \mathbb{R}^{w \times h \times 3}$, and $3$ is the number of channels in MIM$_i$.
~\cref{fig:MIM-pipeline}c is the color wheel scheme,  and~\cref{fig:MIM-pipeline}d is an example of MIM$_i$ that is generated from the pair of  $\bar{f_{i}}$ and $\bar{f_{i+1}}$ (\cref{fig:MIM-pipeline}a).
According to the wheel schema, the color represents the motion direction, while the color intensity denotes the motion magnitude or speed.
\begin{equation}
\label{eq:angle}
\theta({\langle x,y \rangle})_{\bar{f_i}, \bar{f}_{i+1}} =\pi^{-1}\arctan(\frac{v_{\langle x,y \rangle}}{u_{\langle x,y \rangle}}).
\end{equation}

The motion descriptor component divides each MIM$_i$ into $n \times m$ MIM$_i$-patches to help the system localizing pushing in ROI.
The MIM$_i$-patches can be expressed as $\{p_{i,k} \in \mathbb{R}^{(w/m) \,  \times\, (h/n)\, \times \,3} \, |\; k=1, 2,\cdots, n\times m\}$,
where  $k$ is the order of the patch in MIM$_i$.
For more clarity,  MIM$_i$ (\cref{fig:MIM-pipeline}d) is divided into $2 \times 4$ MIM-patches (\cref{fig:MIM-pipeline}f).
It is worth noting that the patch should cover an area on the ground that can accommodate a group of pedestrians, as crowd characteristics are required for representing pushing behavior. To summarize, the MIM-patches represent the output of the motion descriptor component and the input of the next component.

\subsubsection*{Pushing detection and annotation }

The primary purpose of this component is to localize the pushing patches in the live stream, as well as blurring and storing the annotated ROI keyframes in the cloud storage. Labeling MIM-patches as pushing or non-pushing is the most important aspect of localizing pushing in the live stream. Therefore, we created an efficient binary classifier by adapting and training the EfficientnetV2B0 CNN architecture~\cite{tan2021efficientnetv2}  from scratch, which is then utilized to label the MIM-patches.

CNN is a class of deep learning inspired by the organization of the animal visual cortex and is the most widely used in vision-related applications~\cite{aerts2014decoding}. The main advantage of CNN is that it automatically and adaptively learns the relevant features and classification from data~\cite{gu2018recent}. Typical CNN architecture can be described as repeated blocks of convolution and pooling layers, followed by one or more fully connected layers. Convolution and pooling layers perform feature extraction, whereas the fully connected layers map the extracted features into the final layer, which is responsible for predicting the class of the input image~\cite{yamashita2018convolutional}. The CNN architecture is essential for enhancing the model performance, where several changes to the architecture have been made. In particular, the most efficient developments in CNN architectures have been made in building new blocks along with scaling the existing architectures' depth, width, or resolution~\cite{khan2020survey}.
InceptionResNet~\cite{szegedy2017inception}, DenseNet50~\cite{huang2017densely}, NASNetMobile~\cite{zoph2018learning}, VGG16~\cite{simonyan2014very}, VGG19~\cite{simonyan2014very}, ResNet~\cite{he2016deep}, Xception~\cite{chollet2017xception} and EfficientNets family~\cite{tan2019efficientnet, tan2021efficientnetv2} are popular CNN architectures.

The EfficientNet family includes several architectures that achieve and outperform state-of-the-art accuracy in different classification tasks. EfficienetNetV2B0 is one of the most powerful (simple and accurate) CNN architectures.
Since the fully connected layer in the original EfficientNetV2B0 classifies the images between 1,000 categories, this architecture is unsuitable for binary classification.
Yet, pushing detection requires labeling the input image into one of two possible classes.
As a result, we replaced the fully connected layer with a new one, which contains a neuron and a Sigmoid activation function to support the binary classification.
After adapting the architecture of EfficientNetV2B0, we trained it from scratch to build a model that can classify the MIM-patches into pushing and non-pushing labels.
In more detail, the trained EfficientNetV2B-based classifier first performs a  3$\times$3 convolution on   MIM-patch, which has dimensions of 224$\times$224$\times$3. Next, the following 22 blocks (5 Fused Mobile inverted bottleneck Convolutions (Fused-MBConv)~\cite{gupta2019efficientnet} and 17 Mobile inverted Bottleneck Convolutions (MBConv)~\cite{sandler2018mobilenetv2}) extract the feature maps from the input. The global average pooling2D layer then decreases the dimensions of the stacked feature maps to 1$\times$1$\times$1280 and assigns them to the fully connected layer.
Finally, the fully connected layer with a Sigmoid activation function gives the probability of pushing and non-pushing classes for MIM-patch, resulting in the final output label.

\begin{figure}[ht]
\centering
\includegraphics[width=1\linewidth]{./images/pushingDetection.pdf}
\caption{The pipeline of pushing detection and annotation component. (a) Examples of MIM$i$-patches. (b) A trained classifier is responsible for finding the label of each MIM$_i$-patch. (c) An annotation process aims to generate an annotated ROI keyframe and an annotation mask based on the labels of MIM$_i$-patch. (d) An example of an annotation mask. (e) An example of an annotated ROI keyframe. Red boxes mean pushing patches. (f) An example of a live camera stream. (g) Cloud storage. }
\label{fig:pushingDetection-pipeline}
\end{figure}

~\cref{fig:pushingDetection-pipeline} shows the pipeline of the pushing detection and annotation component. Firstly, the trained classifier (\cref{fig:pushingDetection-pipeline}b) labels   MIM-patches $p_{i,k}$ (\cref{fig:pushingDetection-pipeline}a) received from the previous component. Then, the current component displays an annotation mask of the pushing patches (\cref{fig:pushingDetection-pipeline}d) in the live stream on the web client  (\cref{fig:pushingDetection-pipeline}f). Simultaneously, it blurs and annotates the corresponding ROI keyframe $\bar{f_i}$  (\cref{fig:pushingDetection-pipeline}e) before saving it in the cloud storage  (\cref{fig:pushingDetection-pipeline}g). Notably, web clients can access this storage via an internet connection.

\subsection*{Dataset preparation} \label{sec:datasetpreparation}
In this section, we explain how we prepared the labeled dataset (training, validation, and test sets) for training and evaluating the adapted EfficientNetV2B0 as well as all models used in the evaluation. The dataset contains two classes of MIM-patches, which are pushing and non-pushing.

\subsubsection*{Ethical considerations}
The experiments used in the dataset were conducted according to the guidelines of the Declaration of Helsinki and approved by the ethics board at the University of Wuppertal, Germany. Informed consent was obtained from all subjects involved in the experiments.

\subsubsection*{Data collection}
Here, we discuss the data sources used to obtain our dataset. The sources are mainly based on video experiments of crowded event entrances, trajectory data, and ground truth data for pushing behavior.
Five video experiments with their trajectory data are chosen from the data archive hosted by Forschungszentrum Jülich under CC Attribution 4.0 International license~\cite{crowdqueue, entrance2}.
Static top-view cameras were used to record the videos with a frame rate of 25 frames per second.
It is worth mentioning that the selected experiments contain varied characteristics, which help to improve the generality of the dataset, as seen in~\cref{tab:characterstics}.
The last data source is the ground truth data constructed by the manual rating system ~\cite{usten2022pushing} from each video experiment. This data identifies whether the behavior of each pedestrian over every frame is pushing or non-pushing.

\begin{table}[ht]
\centering
\footnotesize
\begin{tabular}{|l|c|c|c|c|c|c|c|c|}
\hline
\textbf{Video}  &
  \textbf{Entrance type} &
 \textbf{Gates} &
  \textbf{Width (m)} &
 \textbf{ Pedestrians} &
 \textbf{Duration (s)} &
 \textbf{Resolution} &
 \textbf{ROI coordinates (pixel)} &
\textbf{n $\times$ m *} \\ \hline
110         & Straight             & 1 & 1.2 & 63  & 53  & 1920 $\times$  1440 & (374 , 548) , (1382 , 864)  & 1 $\times$ 3 \\ \hline
150         & Straight             & 1 & 5.6 & 57  & 57  & 1920 $\times$ 1440 & (364 , 200) ,  (1378 , 1250)    & 3 $\times$ 3 \\ \hline
270         & Straight             & 1 & 3.4 & 67  & 59  & 1920 $\times$ 1440 & (374 , 330) , (1390 , 1070) & 2 $\times$ 3 \\ \hline
280         & Straight             & 1 & 3.4 & 67  & 67  & 1920 $\times$ 1440 & (374 , 330) , (1390 , 1070) & 2 $\times$ 3 \\ \hline
Entrance\_2 & 90$^{\circ}$  Corner & 2 & 2   & 123 & 125 & 1920 $\times$ 1080 & (213 , 110) , (1337 , 540)  & 2 $\times$ 4\\
\hline

\end{tabular}
\caption{\label{tab:characterstics}Characteristics of the selected experiments. The video experiments' names are the same as reported in~\cite{crowdqueue, entrance2}. ROI coordinates: left–top and bottom–right coordinates of ROI in the pixel unit. $n \times m$: number of rows
and columns that are used to divide ROI into $n \times m$ regions, which are required for dividing MIM$_i$ into $n \times m$  MIM$_i$-patches. * These values ensure that the dimensions of each region on the ground are greater than one meter,  which is enough to accommodate a group of pedestrian~\cite{alia2022hybrid}. }
\end{table}

\subsubsection*{Dataset generation}

The methodology of the labeled dataset generation, as seen in~\cref{fig:datasetPreparationMethodology}, includes three steps: (1)  MIM-patches generation, (2)  MIM-patches labeling and (3) Labeled dataset generation.
\begin{figure}[ht]
\centering
\includegraphics[width=1\linewidth]{./images/datasetGeneration.pdf}
\caption{The methodology of the labeled dataset generation. }
\label{fig:datasetPreparationMethodology}
\end{figure}

In the MIM-patches generation step,   the motion descriptor component was employed  (\cref{fig:MIM-pipeline}) on the video experiments and their $n \times m$ patches (\cref{tab:characterstics}) to produce  MIM-patches.
To increase the number of patches, the component is applied four times for each video with a different commencement; half a second is the delay duration of each time compared to the previous time. According to~\cite{alia2022hybrid}, half a second delay helps to generate diverse MIM-patches, while less than this period may result in redundant samples.
Based on the trajectory and ground truth data, the second step labels the patches as pushing and non-pushing.
Patches are classified as pushing if it contains at least one pushing behavior, and non-pushing if no pedestrians engage in pushing behavior. On the other hand, the patches that only show a portion of one pedestrian pushing are discarded; because they do not offer complete information about pushing or non-pushing behavior.
According to the labels of the patches, the last step stores the patches in pushing and non-pushing directories to create the labeled dataset.
To generate the holdout data, the produced dataset is randomly divided into three sets:  70$\%$ for training,  15$\%$ for validation, and 15$\%$ for testing. This split ratio is one of the most commonly used splitting methods in the deep learning field~\cite{genc2019optimal}.
~\cref{tab:sets} shows the number of pushing and non-pushing samples in the training, validation, and test sets.

\begin{table}[ht]

\centering
\footnotesize
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|}
\hline
              & \multicolumn{2}{c|}{\textbf{Training set}} & \multicolumn{2}{c|}{\textbf{Validation set}} & \multicolumn{2}{c|}{\textbf{Test set}} & \multicolumn{3}{c|}{\textbf{Dataset}}  \\
\textbf{Experiment}    & \textbf{P}    & \textbf{NP}                        & \textbf{P}   &\textbf{ NP}                           & \textbf{P}   & \textbf{NP}                     & \textbf{P}    & \textbf{NP}   & \textbf{Total}          \\ \hline
110 & 122  & 72                        & 26  & 15                           & 26  & 15                     & 174  & 102  & 276            \\\hline
150 & 182  & 206                       & 38  & 44                           & 38  & 44                     & 258  & 294  & 552            \\\hline
270 & 215  & 197                       & 45  & 42                           & 45  & 42                     & 305  & 281  & 586            \\\hline
280 & 258  & 182                       & 55  & 38                           & 55  & 38                     & 368  & 258  & 626                      \\\hline
Entrance\_2     & 808  & 525                       & 172 & 112                          & 172 & 112                    & 1152 & 749  & 1901           \\\hline
All           & 1585 & 1182                      & 336 & 251                          & 336 & 251                    & 2257 & 1684 & 3941     \\ \hline
\end{tabular}
\caption{\label{tab:sets} A number of samples in training, validation, and test sets. `All' refers to all experiments. P means pushing. NP is non-pushing. Total represents the number of pushing and non-pushing samples.}
\end{table}

\subsection*{Training process implementation}

We utilized the training and evaluation sets to learn the adapted EfficentNetV2B0-based classifier and the deep learning models used in the evaluation. The Adam optimizer with a binary cross-entropy loss function was applied for the training process. In addition, the learning rate, batch size, and epochs were set to 0.001, 32, and 100, respectively. Furthermore, if the validation accuracy did not improve after 20 epochs, we immediately stop the training. The implementations and all experiments in our article were carried out on Google Colaboratory Pro with GPU~\cite{colab} using JavaScript and Python 3 with Keras and OpenCV libraries.

\subsection*{Evaluation metrics}

We evaluated the performance of the trained classifiers using the test set.
Accuracy, and macro F1-score were used as evaluation metrics. In addition to those two metrics, computational time is an essential metric for measuring the system's performance. The metrics are explained as follows.

\textbf{Accuracy:} the ratio of successfully classified  MIM-patches to the total number of samples in the test set, and mathematically can be defined as
\begin{equation}
\label{eq:accuracy}
accuracy=\frac{TP+TN}{TP+FP+TN+FN},
\end{equation}
where TP and TN denote correctly classified pushing and non-pushing patches, respectively. FP and FN represent incorrectly predicted pushing and non-pushing samples.
Accuracy is not enough to evaluate the classifier's performance over an imbalanced dataset, such as our used dataset. Therefore, we used the macro F1-score metric, which is valuable for evaluating imbalanced classification problems.

\textbf{Macro F1-score:} the mean of class-wise F1-scores as described in the formula below

\begin{equation}
Macro \; F1-score=\frac{F1-score(Pushing) + F1-score(Non-pushing)}{2},
\end{equation}
where F1-score is the harmonic average of precision and recall as described in

\begin{equation}
F1-score=\frac{2 \times precision \times recall}{precision + recall},
\end{equation}
 where recall of pushing class is the ratio of correctly classified pushing  MIM-patches to all pushing samples, while precision of pushing class is the ratio of correctly classified pushing patches out of all the samples labeled as pushing by the classifier.  Recall and precision are defined in~\cref{eq:recall} and~\cref{eq:precision}, respectively.

 \begin{equation}
 \label{eq:recall}
recall=\frac{TP}{TP+FN},
\end{equation}

 \begin{equation}
 \label{eq:precision}
precision=\frac{TP}{TP+FP}.
\end{equation}

\textbf{ Computational time:} this metric was employed to calculate how long the proposed system takes to read, analyze and annotate every input, which is two seconds of stream. In other words, computational time determines whether our system can detect pushing patches within a reasonable time or not.

\section*{Results and discussion}

\subsection*{Performance evaluation of the proposed classifier}
In order to evaluate the performance of the adapted EfficientnetV2B0-based classifier, we conducted two comparative empirical experiments. In the first experiment,  we compared the proposed classifier against twelve of the most popular CNN-based classifiers.
\cref{tab:popularCNNresults} shows the popular classifiers as well as the results of the comparison based on the accuracy and F1-score metrics.
It is clear that the adapted version of the EfficientNetV2B0 classifier outperformed the rest of the exploited classifiers. In particular, the proposed classifier achieved  87$\%$ accuracy and 86$\%$ F1-score, whereas the second top model in this comparison, DenseNet169, produced  83$\%$ level of accuracy and F1-score.  This finding is primarily attributable to EfficientNetV2B0's superior parameter efficiency compared to earlier CNN architectures. The main reason for the parameter efficiency is the combination of  MBConv and Fused-MBConv blocks used in EFficientNetV2B0.

\begin{table}[htb]
\footnotesize
\centering
\begin{tabular}{|l|l|c|c|}
\hline
\textbf{\#} & \textbf{CNN-based classifier} & \textbf{Accuracy  $\%$ } & \textbf{F1-score  $\%$ }\\ \hline
1  & Xception~\cite{chollet2017xception}             & 81          & 81          \\ \hline
2  & VGG16~\cite{simonyan2014very}                & 57          & 36          \\ \hline
3  & VGG19~\cite{simonyan2014very}                & 61          & 61          \\ \hline
4  & ResNet50~\cite{he2016deep}             & 80          & 79         \\ \hline
5  & ResNet50V2~\cite{he2016identity}           & 77          & 76          \\ \hline
6  & ResNet101~\cite{he2016deep}            & 72          & 70          \\ \hline
7  & ResNet101V2~\cite{he2016identity}          & 72          & 72          \\ \hline
8  & ResNet152V2~\cite{he2016identity}           & 74          & 73          \\ \hline
9  & InceptionResNetV2~\cite{szegedy2017inception}    & 82          & 82          \\ \hline
10 & DenseNet121~\cite{huang2017densely}          & 79          & 79          \\ \hline
11 & DenseNet169~\cite{huang2017densely}          & 83          & 83          \\ \hline
12 & NASNetMobile~\cite{zoph2018learning}         & 57          & 56
\\ \hline
13 & \textbf{Our classifier }        & \textbf{87}          & \textbf{86}  \\ \hline
\end{tabular}
\caption{\label{tab:popularCNNresults} Comparison results to the well-known CNN-based classifiers.}
\end{table}

In order to evaluate some existing customized and adapted CNN architectures in related works for pushing detection, the second experiment selected  CNN-1~\cite{direkoglu2020abnormal}, CNN-2~\cite{tay2019robust}, and adapted EfficientNetV1B0~\cite{alia2022hybrid}, which are of the most recent CNN architectures in the abnormal human behavior detection field.
The architectures of CNN-1 and CNN-2 are simple; CNN-1 employed $75  \times  75$ pixels as an input image. Furthermore, three convolutional layers, batch normalization, and max pooling operations were used for feature extraction. The developers of this model utilized a fully connected layer with a softmax activation function for classification.
The second architecture, CNN-2, downsized the input images to   $32  \times  32$  pixels before employing three convolutional layers with three max-pooling layers. For classification, it used two fully connected layers, with the first layer based on a ReLU activation function and the second layer employing a softmax activation function. In the adapted EfficientNetV1B0, the authors utilized the EfficintNetV1B0 for feature extraction. Additionally,   a fully connected layer with a ReLU activation function comes before the final fully connected layer with a Sigmoid activation function for labeling.
The results in~\cref{fig:comparisontocustomizedCNN} show that our classifier surpassed the three classifiers in accuracy and F1-score. Furthermore, due to the high complexity of pushing detection in crowds, the simple CNN-1 and CNN-2 architectures failed to identify pushing MIM-patches. Additionally, our classifier outperformed the adapted EfficientNetV1B0 by 4$\%$. This superiority is due to the power of MBConv and Fused-MBConv blocks used in EfficientNetV2B0.

\begin{figure}[htb]
\centering
\includegraphics[width=0.6\linewidth]{./images/Comparisons-to-customized-CNN.pdf}
\caption{Comparison between our classifier and three CNN-based classifiers in related works. }
\label{fig:comparisontocustomizedCNN}
\end{figure}

\subsection*{Performance evaluation of the proposed system}
To evaluate the quality of the proposed system, we measured its accuracy and F1-score based on a qualitative methodology.
In addition, the computational time was calculated for each component of the system.
The evaluation methodology comprised several steps as follows:  (1) To simulate acquiring the actual inputs, we created a live video stream of crowded event entrances using video recordings of entrances (\cref{tab:characterstics}) and a virtual camera on a web client. In this context, we changed the camera's input to the video recordings. Moreover, we down-scaled the dimensions of each video to half their original resolution to reduce the computational time of the system.
(2) We executed the cloud-based system to display the live camera stream on the web client, detect pushing patches, record the predicted labels for the patches in a file, and store the gathered frames in a directory.
(3) We visualized the predicted labels of the test patches on the collected frames, see~\cref{fig:evaluationexample}a.
(4) Based on the visualized data, the social psychologists who created the corresponding manual rating system labeled the test patches manually.
(4) From the actual and predicted labels, we counted the number of true pushing, false pushing, true non-pushing, and false non-pushing.
(5) We computed the accuracy and F1-score metrics.
From the confusion matrix of our system, as shown in \cref{fig:evaluationexample}b, the proposed system can achieve 89$\%$ accuracy and F1-score.

\begin{figure}[htb]
\centering
\includegraphics[width=1\linewidth]{./images/manualevaluationexample.pdf}
\caption{ (a) An example of the visualized two seconds of stream in the qualitative methodology.
Red boxes are the predicted pushing patches. Green box means the predicted non-pushing patch. White boxes denote the patches not used in the evaluation because they were utilized in training, while the red and green boxes are for testing. FNP is false non-pushing. TP is true pushing. (b) Confusion matrix for the proposed system. }
\label{fig:evaluationexample}
\end{figure}

In order to evaluate the overall computational time of the proposed system, we computed the required time for each component in the system.
Then, we compared the results against the corresponding parts in the baseline system~\cite{alia2022hybrid}, and to the best of our knowledge, it is the only published work for pushing detection.
After running both the proposed and baseline systems in the same environment using twenty inputs, where each input is a two-second video stream, we calculated the average run-time of all runs.
As mentioned in the previous paragraph, our system read the videos using a live camera, while the baseline system read the same videos directly. \cref{fig:timeOurModel} depicts the time of each component in the proposed system over every experiment.
As indicated by the produced results, the preprocessing component took more than 50$\%$ of the overall time to collect and process keyframes from the client camera stream. The resolution of the keyframes primarily determines the time required for this component. For example, experiments 110, 150, 270, and 280 took roughly the same time because they have the same resolution, while experiment entrance\_2  needed less time because its resolution is lower.
In contrast, the motion descriptor component took the least time compared to others, where the ROI resolution plays the most critical role in this component speed. While in the pushing detection and annotation component, the number of patches affects the computational time of this component because each patch requires one classification process.
\cref{fig:frameAndROIAreas} and~\cref{tab:characterstics} display the ROI resolution and the number of patches in each experiment, respectively.

In general, the computation time increases as the number of patches, frames resolution, and ROI size increase. \cref{fig:timeOurModel} shows that our system needed less than two seconds to collect, process, detect and annotate each input from the live stream camera.
This means that our system can annotate the live camera stream within 4 seconds; two seconds for the input duration and lower than two seconds for identifying the pushing patches.

\begin{figure}[htb]
\centering
\includegraphics[width=0.8\linewidth]{./images/timeOurModel.pdf}
\caption{Computational time of each component of the proposed system for annotating two seconds of stream. }
\label{fig:timeOurModel}
\end{figure}

\begin{figure}[htb]
\centering
\includegraphics[width=0.6\linewidth]{./images/area.pdf}
\caption{Comparison between frame and ROI resolutions for each experiment.}
\label{fig:frameAndROIAreas}
\end{figure}

\begin{table}[!ht]
\footnotesize
\centering
\begin{tabular}{|l|c|c|c|c|} \hline
            & \multicolumn{2}{c|}{\textbf{Baseline system}} & \multicolumn{2}{c|}{\textbf{Our system}} \\
\textbf{Experiment} & \multicolumn{1}{l|}{\textbf{Motion information extraction (s)}} & \textbf{Detection (s)} & \textbf{Motion descriptor (s)} & \textbf{Detection and annotation (s)} \\  \hline
110         & 19.42   & 0.15                       & 0.10           & 0.19          \\  \hline
150         & 19.86     &0.47   & 0.20           & 0.53          \\  \hline
270         & 19.38   & 0.32                       & 0.16           & 0.35          \\  \hline
280         & 19.30  & 0.30                     & 0.16            & 0.36          \\  \hline
Entrance\_2 & 13.51   & 0.41                     & 0.11           & 0.45 \\  \hline
\end{tabular}
\caption{\label{tab:baselinesystemtime} The computational time of motion descriptor and detection components in our and the baseline systems.}
\end{table}

The results in~\cref{tab:baselinesystemtime} show the comparisons between the motion descriptor, and pushing detection and annotation components in our system with the corresponding parts in the baseline system. The motion information extraction part in the baseline system is similar to the motion descriptor component in our system, whereas the motion information extraction is slow; it needs more than 13.5 seconds to generate  MIM-patches from two seconds of the video stream. The main reason for this slowness is that it employed CPU-based RAFT to estimate the optical flow vectors for all pixels in the frame. To address this problem, the motion descriptor in our system utilized GPU-based RAFT to calculate the optical flow vectors for each pixel in ROIs instead of all pixels in the frame. As shown in~\cref{fig:frameAndROIAreas}, the number of pixels in ROIs is lesser than 40$\%$ of the total pixels in the corresponding frames. As a result, the new component took 0.2 seconds or less to produce  MIM-patches from the two seconds of the live stream.
On the other hand, the baseline system's detection part is slightly faster than the detection and annotation component in the proposed system.
For example, the previous and new components required 0.47 and 0.53 seconds to work with one input from experiment 150, respectively.
It is important to highlight that the detection part in the baseline system only finds the labels of the patches, whereas the component in our system labels, annotates, blurs, and stores the inputs.

In summary, the proposed cloud-based system can annotate the pushing patches in the live camera stream within four seconds and an accuracy rate of 89$\%$.

\section*{Conclusion, limitations, and future research}
This paper proposed a novel automatic system for the early detection of pushing patches in crowded event entrances.
The proposed system is based on live camera streaming technology, cloud environment, visualization method, and deep learning algorithms.
The system first displays the live camera stream of the entrances on the web client in real-time.
Then, it relies on the color wheel method and pre-trained RAFT model to extract the visual motion information from the live stream.
After that, the EfficientNetV2B0-based classifier is adapted and trained to identify pushing patches from the extracted information.
Finally, the system annotates the pushing patches in the live stream on the web client.
Additionally, it stores the annotated data in the cloud storage, where the stored data is blurred to protect people’s privacy.
In order to train and evaluate the classifier, a new dataset was generated using five real-world video experiments and their associated ground truth data.
The experimental results show that the system identified pushing patches from the live camera stream with  89$\%$ accuracy rate within a reasonable time delay.

One of the current limitations of the proposed system is that it is only compatible with a fixed and top-view camera. Another limitation is that the classification model was trained on a small dataset, which could reduce the model's generality.

In future, the plan is to develop a new pushing data representation method for machine learning. This method aims to generate dynamic patches based on temporal, spatial, and size dimensions, focusing on one pedestrian for labeling each patch.
This could potentially help to generate a large dataset with a more efficient sample representation.

\section*{Data and code availability}
All videos and trajectory data used in generating the patch-based dataset were obtained from the data archive hosted by the Forschungszentrum Jülich under CC Attribution 4.0 International license~\cite{entrance2,crowdqueue}. The undistorted video experiments, implementation of the proposed system, as well as codes used for building and training the models, are publicly available at: \url{https://github.com/PedestrianDynamics/CloudFast-DL4PuDe} (accessed on 15 Jan 2023). The generated patch-based dataset is available from the corresponding authors upon request.

\section*{Acknowledgements}
The authors are thankful to Anna Sieben, Helena Lügering, and Ezel Üsten for the valuable discussions, manual annotation of the pushing behavior in the video experiments, and for revising the proposed system's output.

\section*{Author contributions statement}

\noindent
Conceptualization, A.A., M.M., M.C. and A.S.;
methodology, A.A., M.M., M.C. and A.S.;
software, A.A.;
validation, A.A.;
formal analysis, A.A.;
investigation, A.A., M.M., M.C. and A.S.;
data curation, A.A.;
writing—original draft preparation, A.A.;
writing—review and editing, A.A., M.M., M.C., A.T. and A.S.;
visualization, A.A. and A.T.;
supervision, M.M., M.C. and A.S.
All authors have read and agreed to the published version of the manuscript.

\section*{Funding}
This work was funded by the German Federal Ministry of Education and Research (BMBF:
funding number 01DH16027) within the Palestinian-German Science Bridge project framework, and by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—491111487.

\section*{Additional information}

\textbf{Correspondence} and requests for materials should be addressed to A.A., M.M.or A.S.

\input{main.bbl}

\end{document}

Machine Learning (cs.LG); Computer Vision and Pattern Recognition (cs.CV)