AdaEnlight: Energy-aware Low-light Video Stream Enhancement on Mobile Devices

Abstract: The ubiquity of camera-embedded devices and the advances in deep learning have stimulated various intelligent mobile video applications. These applications often demand on-device processing of video streams to deliver real-time, high-quality services for privacy and robustness concerns. However, the performance of these applications is constrained by the raw video streams, which tend to be taken with small-aperture cameras of ubiquitous mobile platforms in dim light. Despite extensive low-light video enhancement solutions, they are unfit for deployment to mobile devices due to their complex models and and ignorance of system dynamics like energy budgets. In this paper, we propose AdaEnlight, an energy-aware low-light video stream enhancement system on mobile devices. It achieves real-time video enhancement with competitive visual quality while allowing runtime behavior adaptation to the platform-imposed dynamic energy budgets. We report extensive experiments on diverse datasets, scenarios, and platforms and demonstrate the superiority of AdaEnlight compared with state-of-the-art low-light image and video enhancement solutions.

\section{Fast Low-light Video Enhancement}
\label{sec:algorithm}
This section presents {\sf UbiVE}\xspace's fast low-light video enhancement model.
It is built upon prior curve Based image enhancement solutions (\secref{subsec:algo:primer}).
To satisfy our special requirements, \emph{i.e.},\xspace near \textit{real-time} processing and \textit{video} enhancement, we exploit Gamma correction based curve to accelerate the enhancement speed (\secref{subsec:algo:gamma}) and extend the image-specific solution to videos by incorporating temporal consistency between frames (\secref{subsec:algo:temporal}).
The detailed implementations are in \secref{subsec:algo:together}.

\subsection{Primer on Curve Based Image Enhancement}
\label{subsec:algo:primer}
Low-light image enhancement takes a low-light image as input, and outputs a normal-light, high-quality image \cite{bib:TPAMI21:Li}.
We base our design upon Zero-DCE \cite{bib:CVPR20:Guo}, a recent zero-shot learning low-light image enhancement scheme via deep curvature estimation.
The reasons are as follows.
\textit{(i)}. We choose the \textit{zero-shot learning} based solution to eliminate the need for paired training data, \emph{i.e.},\xspace videos of the same scene taken in dim and normal illumination. This is desired because access to such paired videos is limited in real-world mobile applications with high variability.

\textit{(ii)}. We select the \textit{curve-based model} for it yields competitive image enhancement performance despite its lightweight architecture \cite{bib:TPAMI21:Li2}, which holds potential for execution on resource-constrained mobile devices.
As next, we provide a brief review on Zero-DCE \cite{bib:CVPR20:Guo} and discuss its limitations for fast video enhancement.

\subsubsection{Principles of Zero-DCE}
The design of Zero-DCE \cite{bib:CVPR20:Guo} follows the curve adjustment in photo editing software.
It derives an \textit{image-specific curve} that maps a low-light image to its enhanced version.
The curve is defined as a high-order pixel-wise function learned by a deep neural network, where its loss function is designed as a set of differentiable non-reference losses to enable zero-shot learning.
The curve function is defined as:
\begin{equation}\label{equ:dce:curve}
    E_{n}(\mathbf{x}) = E_{n-1}(\mathbf{x}) + A_{n}(\mathbf{x})\cdot E_{n-1}(\mathbf{x}) \cdot \left(1 - E_{n-1}(\mathbf{x})\right)
\end{equation}
where $\mathbf{x}$ is the pixel coordinates of the input image $I(\mathbf{x})$, $E_{n}(\mathbf{x})$ is the enhanced output image (note that $E_{0}(\mathbf{x})=I(\mathbf{x})$), $n$ is the number of iteration to control the curvature, $A_{n}(\mathbf{x})$ is a parameter map of the same size as $I(\mathbf{x})$, where each element in $A_{n}(\mathbf{x})$ is a trainable curve parameter in $[-1,1]$.
Each pixel in $I(\mathbf{x})$ is normalized to $[0,1]$ and the enhancement is performed separately to three RGB channels.
The curve function $E_{n}(\mathbf{x})$ is learned via a neural network with a loss function that consists of spatial consistency, exposure control, color constancy, and illumination smoothness.
\rev{The spatial consistency loss $L_{spa}$ encourages spatial coherence of the enhanced image.
The exposure loss $L_{exp}$ controls the exposure level of enhanced image.
The color loss $L_{col}$ corrects the potential color deviations.
The illumination smoothness loss $L_{tvA}$ preserves monotonicity relations between neighboring pixels.
More details can be found in \cite{bib:CVPR20:Guo}.}

\subsubsection{Limitations of Zero-DCE}
\label{subsubsec:limitation}
Despite being \rev{one of the most \textit{lightweight image enhancement}} scheme, Zero-DCE fails to support \textit{fast video enhancement} on resource-constrained mobile devices for the following reasons.

\textit{(i)}, the iterative procedure in \emph{i.e.},\xspace \equref{equ:dce:curve}, for image enhancement,  incurs \textit{notable delay} that prohibits real-time video enhancement.
For example, Zero-DCE takes an average of $11.74s$ to enhance a $270 \times 480$ RGB image on a Raspberry Pi 4B platform (detailed setups in \secref{sec_exp_setup}).
The follow-up Zero-DCE++ \cite{bib:TPAMI21:Li2} accelerates Zero-DCE by adopting depth-wise separable convolutions and reusing the curve parameter maps $A_{n}(\mathbf{x})$.
However, Zero-DCE++ still induces $0.19s$  delay, \emph{i.e.},\xspace only $5$ frames per second.
The main bottleneck lies in the iterative procedure, where $n$ is empirically optimized to $8$ in both Zero-DCE and Zero-DCE++.

\textit{(ii)}, directly applying image enhancement schemes to videos leads to \textit{flicking} between frames \cite{bib:ICCV19:Chen, bib:CVPR21:Zhang}.
\figref{fig:flicking} shows the difference between two consecutive images enhanced by Zero-DCE and Zero-DCE++.
There is notable differences between adjacent frames (up to 9.192) and thereby results in flicking in videos.
These limitations motivate us to rethink the design for the fast video enhancement in two aspects, as will be explained in \secref{subsec:algo:gamma} and \secref{subsec:algo:temporal}.

\begin{figure*}[t]
  \centering
   \subfloat[Zero-DCE]{
   \includegraphics[height=0.1\textwidth]{image/dce.pdf}}
   \hspace{15pt}

   \subfloat[Zero-DCE++]{
   \includegraphics[height=0.1\textwidth]{image/dce_plus.pdf}}

  \hspace{15pt}
  \subfloat[{\sf UbiVE}\xspace]{
  \includegraphics[height=0.1\textwidth]{image/ours.pdf}}
\caption{Illustration of the flicking problems when applying low-light image enhancement schemes, \emph{e.g.},\xspace (a) Zero-DCE \cite{bib:CVPR20:Guo} and (b) Zero-DCE++ \cite{bib:TPAMI21:Li2} to videos. The figures show the difference between two consecutive frames. The greater the difference, the more serious the flicking problem is. In comparison, (c) shows the low frame difference of {\sf UbiVE}\xspace.}
\label{fig:flicking}
\vspace{-5mm}
\end{figure*}

\subsection{Non-iterative Frame Enhancement with Gamma Correction-based Curve}
\label{subsec:algo:gamma}
As previously mentioned (\secref{subsubsec:limitation}), the latency bottleneck of curve-based image enhancement~\cite{bib:CVPR20:Guo, bib:TPAMI21:Li2}
lies in the iterative enhancement process.
We observe that the iterative procedure may be eliminated if a non-iterative curve function \rev{with higher non-linearity} is adopted.
Therefore, we propose to replace the traditional \rev{quadratic} curve in \equref{equ:dce:curve} by an \rev{exponential} curve to induce higher-order non-linearity and smaller latency.
Specifically, we propose the following non-iterative image enhancement scheme.
\begin{equation}\label{equ:ada:curve}
    E(\mathbf{x}) = I(\mathbf{x})^{\Gamma(\mathbf{x})}
\end{equation}
where $I(\mathbf{x})$ and $E(\mathbf{x})$ denote the input and enhanced image, respectively.
$\Gamma(\mathbf{x})$ are pixel-wise trainable parameters learned from a neural network.
The detailed training methodology of $\Gamma(\mathbf{x})$ are deferred to \secref{subsec:algo:together}.
We make two detail notes on our design of \equref{equ:ada:curve}.
\begin{itemize}
    \item
    The exponential curve function in \equref{equ:ada:curve} is inspired by the Gamma correction, which is typically used to flexibly adjust the luminance or tristimulus in images \cite{bib:JIVP16:Rahman}.
    Our novelty is to apply pixel-wise parameters $\Gamma(\mathbf{x})$ for fine-grained adjustment, which is learned from a neural network.
    In contrast, conventional Gamma correction adopts the same coefficient $\Gamma$ for the entire image, which tends to introduce artifacts and color deviations to the enhanced images \cite{bib:TPAMI21:Li}.

    \item
    One may want to design an iterative enhancement process by rewriting \equref{equ:ada:curve} as
    $E_n(\mathbf{x}) = E_{n-1}(\mathbf{x})^{\Gamma_n(\mathbf{x})}$, where $E_{0}(\mathbf{x})=I(\mathbf{x})$.
    We exclude this option because our experiments show that the enhanced visual quality does not notably increase with more iterations (see \secref{subsec:exp_ite}).

\end{itemize}
To summarize, the learned pixel-wise Gamma correction-based curve in \equref{equ:ada:curve} tends to deliver high-quality visual enhancement while avoiding the long latency.
We defer the quantitative comparisons with the traditional iterative enhancement scheme to \secref{subsec:exp:bench}.

\subsection{Incorporating Temporal Consistency Loss for Video Enhancement}
\label{subsec:algo:temporal}
As mentioned in \secref{subsec:algo:primer}, naively adopting image enhancement schemes to videos \rev{collected by mobile devices} causes flicking.
A remedy is to account for temporal consistency between frames when designing the loss function to train the enhancement curve \emph{i.e.},\xspace \equref{equ:ada:curve}.
Given two enhanced frames $E_{t+1}(\mathbf{x})$ and $E_{t}(\mathbf{x})$ at timestamp $t+1$ and $t$, we consider their temporal consistency with the following losses.
\begin{itemize}
    \item \textit{Exposure Consistency Loss}:
    We define the exposure consistency loss between two enhanced frames as
    \rev{
        \begin{equation}\label{equ:ada:exposure}
            \mathcal{L}_{exp, temporal} = \sum_{x}\left|\sum_{i=r,g,b} E_{t+1}^{i}(\mathbf{x}) - \sum_{i=r,g,b} E_{t}^{i}(\mathbf{x})\right|
        \end{equation}
    }
    \item \textit{Color Consistency Loss}:
    We define the color consistency loss between two enhanced frames as
    \rev{
        \begin{equation}\label{equ:ada:color}
            \mathcal{L}_{col, temporal} = \sum_{x}\sum_{k=r,g,b} \left|\frac{E_{t+1}^{k}(\mathbf{x})+c}{\sum_{i=r,g,b} E_{t+1}^{i}(\mathbf{x}_i)+c} - \frac{E_{t}^{k}(\mathbf{x})+c}{\sum_{i=r,g,b} E_{t}^{i}(\mathbf{x}_i)+c}\right|
        \end{equation}
    }
\end{itemize}
where $E^{i}(\mathbf{x})$ denotes the enhanced image of one channel, where $i=r,g,b$, which represents the RGB channels.
\rev{$c$ is a delta value empirically set to 0.0001 to avoid division by zero in case of dark pixels}.
The exposure consistency loss sums over the RGB channels to avoid the impact of colors when calculating the temporal consistency.
The color consistency loss, in contrast, is divided by the exposure to mitigate the impact of pixel-wise exposure when calculating the temporal consistency.

\rev{Note that to avoid the influence of motion between frames, we employ the Gunnar-Farneback optical flow \cite{bib:farneback2003two}, a dense optical flow alignment method as a pre-processing step.
Specifically, we calculate motions between the two frames (frame $t+1$ and frame $t$) and get the motion data of each pixel of frame $t$, denoted as $(dx,dy)$.
Then we modify the location of each pixel in frame $t$ as $(x+dx,y+dy)$.
The resulting fame as named as the aligned frame $t$.
Finally, we calculate the absolute difference between the aligned frame $t$ and frame $t+1$.}

\subsection{Putting It Together}
\label{subsec:algo:together}
In short, our fast low-light video enhancement module is a deep curve-based enhancement scheme with a learned pixel-wise Gamma correction based curve \emph{i.e.},\xspace \equref{equ:ada:curve}.
The curve parameters are trained by considering both the conventional image enhancement loss as well as the temporal consistency loss \emph{i.e.},\xspace \equref{equ:ada:exposure} and \equref{equ:ada:color}.
Below we explain the detailed model architecture, its training strategy and how it is designed to trade off between video enhancement quality and platform-imposed requirements at runtime.

\subsubsection{Model Architecture}
\label{subsubsec:algo:model}
We apply a U-Net \cite{bib:MICCAI15:Ronneberger} to learn the parameters $\Gamma_n(\mathbf{x})$ in \equref{equ:ada:curve}.
For lower latency, we decrease the number of layers and the channels of the original U-Net.
We also change the contracting paths in the original U-Net to dense connections as DenseNet \cite{bib:CVPR17:Huang} for computation reuse (see \secref{subsubsec:algo:behaviour}).
\figref{fig:nn} shows the model architecture.
\tabref{tab:nn} lists the detailed hyperparameters.
The hyperparameters are empirically tuned to trade off between enhancement quality and latency.
The ablation studies on hyperparameters are in \secref{sec:exp_reuse}.

\begin{figure}[t]
  \centering
  \includegraphics[width=.85\textwidth]{image/loss.pdf}
  \caption{Model architecture and training scheme to learn the pixel-wise curve parameters in \equref{equ:ada:curve}. The model inside the  dashed box is used for inference.}
  \label{fig:nn}
  \vspace{-5mm}
\end{figure}

\begin{table}[t]
\centering
\scriptsize
\caption{Detailed hyperparameters of the neural network.} \label{tab:nn}
\vspace{-3mm}
\begin{tabular}{|c|c|c|c|c|c|}
\hline
\textbf{Layer} & \textbf{Parameters} & \textbf{Layer} & \textbf{Parameters} & \textbf{Layer} & \textbf{Parameters} \\ \hline
Conv\_1 & \begin{tabular}[c]{@{}c@{}}K=1, Cin=3, Cout=1, S=1, G=1, P=0 \\ K=3, Cin=1, Cout=1, S=1, G=1, P=0\end{tabular} & Conv\_2 & \begin{tabular}[c]{@{}c@{}}K=1, Cin=4, Cout=1, S=1, G=1, P=0 \\ K=3, Cin=1, Cout=1, S=1, G=1, P=1\end{tabular} & Conv\_2 & \begin{tabular}[c]{@{}c@{}}K=1, Cin=4, Cout=3\\ S=1, G=1, P=0\end{tabular} \\ \hline
\end{tabular}
\end{table}

We combine the loss function from Zero-DCE \cite{bib:CVPR20:Guo}, \emph{i.e.},\xspace image enhancement loss, with the two temporal consistency losses defined in \equref{equ:ada:exposure} and \equref{equ:ada:color} as the final loss function.
\begin{equation}\label{equ:ada:loss}
    \mathcal{L}_{total} = \mathcal{L}_{Zero-DCE} + \alpha \mathcal{L}_{exp, temporal} + \beta \mathcal{L}_{col, temporal}
\end{equation}
where $\alpha$ and $\beta$ can be empirically tuned.

\subsubsection{Training Strategy}
\label{subsubsec:algo:training}
Due to the lack of video datasets with evenly distributed exposure levels, it is challenging to train a network that minimizes \equref{equ:ada:loss}.
Therefore, we propose to learn the curve by training the image enhancement loss and the temporal consistency losses in a siamese mode as in \cite{bib:ICCV19:Chen, bib:CVPR21:Zhang}.
\figref{fig:nn} shows the training scheme.
In each pass, we input one low-light image from the image dataset and two adjacent low-light video frames from the video dataset into the neural network in a siamese way \emph{i.e.},\xspace three neural networks share the same weights.
The low-light image will be used to calculate the image enhancement loss, whereas the two frames will be used to compute the two temporal consistency losses.

At inference time, the model takes a low-light video frame as input, and generates an enhanced frame in almost real-time ($0.05$ second delay or $20$ frames per second for $270 \times 480$ RGB images on the Raspberry Pi 4 (4GB) platform.
More quantitative evaluations on enhancement quality and latency are in \secref{subsec:exp_compare}.

\subsubsection{Towards Runtime Adaptation}
\label{subsubsec:algo:behaviour}
Although our low-light video enhancement module achieves near real-time performance, there are other requirements \emph{e.g.},\xspace energy, which the users may trade with enhancement quality.
Our video enhancement module supports such runtime adaptation by adjusting the parameters below.
\begin{itemize}
    \item \textit{Computation Reuse}.
    Recall that our video enhancement model consists of three convolutional layers.
    Due to temporal similarities between adjacent frames, we can further reduce the computations by enforcing computation reuse during frame enhancement.
    We allow reuse at both the frame level (see \figref{fig:behaviour:frame}), \emph{i.e.},\xspace reuse the enhanced output of frame $t$ for frame $t+1$, and the layer level (see \figref{fig:behaviour:layer}), \emph{i.e.},\xspace reuse the intermediate results and skip some layers during enhancement.
    \item \textit{Frame Resolution}.
    Down-sampling the resolution of the input frame naturally reduces the computation of the video enhancement process (see \figref{fig:behaviour:resolution}).
\end{itemize}
We utilize these parameters to trade off between enhancement quality and energy at runtime, as explained next.

\begin{figure}[t]
  \centering
   \subfloat[]{\label{fig:behaviour:frame}
   \includegraphics[width=0.3\textwidth]{image/parameter1.pdf}}
   \quad
   \subfloat[]{\label{fig:behaviour:layer}
   \includegraphics[width=0.3\textwidth]{image/parameter2.pdf}}
   \quad
   \subfloat[]{\label{fig:behaviour:resolution}
   \includegraphics[width=0.3\textwidth]{image/parameter3.pdf}}
\caption{Illustrations of adjustable parameters of the video enhancement model: computation reuse as (a) frame level and (b) layer level; (c) down-sample the resolution of input frames.}
\label{fig:behaviour}
\vspace{-3mm}
\end{figure}

\section{Conclusion}
\label{sec:conclude}

This paper presents {\sf UbiVE}\xspace, an energy-aware low-light video enhancement system that achieves near real-time performance with competitive enhanced visual quality.
It consists of a novel model for fast low-light video enhancement as well as an agile energy-aware controller to dynamically adjust the enhancement behaviors for energy conservation.
And experimental results show that {\sf UbiVE}\xspace outperforms existing image and video enhancement methods in latency and temporal stability while retaining satisfactory system energy efficiency.

\newrev{
There are several limitations of our work that may need future exploration.
\textit{(i)}
Although {\sf UbiVE}\xspace achieves near real-time processing for RGB videos, the frames are resized to $270\times480\times3$.
To support videos with higher resolutions, a more lightweight and efficient video enhancement model is necessary.
We plan to exploit the fixed-point operations common on mobile platforms and explore quantization and other model compression techniques to further accelerate the video enhancement model.
An interesting follow-up is to extend {\sf UbiVE}\xspace to videos in raw format to handle extremely low-light conditions.
\textit{(ii)}
The proposed energy profiler can be integrated into other mobile systems.
The energy profiler of {\sf UbiVE}\xspace is built for mobile GPUs or CPUs only for tractability.
Also, more efforts are needed to design the modular and extensible energy profiler for devices with heterogeneous computation resources, \emph{e.g.},\xspace GPU-CPU co-execution.
}

\section{Energy-aware Adaptation Controller}
\label{sec:controller}

This section presents the {\sf UbiVE's}\xspace energy-aware adaptation controller to dynamically adjust the above mentioned video enhancement model's behaviors for energy conservation.
The focus of this controller is to optimize the trade-off between energy and the video quality, because the above design (\secref{subsec:algo:gamma}) has already guaranteed the near real-time video processing.
We first introduce the design of the overall controller (\secref{subsec:design}), then the runtime energy profiler (\secref{subsec:profiler}), and finally the auxiliary modules to automate the control loop (\secref{subsec:loop}).

\subsection{Controller Design}
\label{subsec:design}
At a high level, the energy-aware adaptation controller of {\sf UbiVE}\xspace is a runtime heuristic optimizer for a multi-objective optimization problem, as formulated below.
\subsubsection{Formulation}
\label{subsubsec:formulation}
Mathematically, the controller aims to continuously optimize two metrics, \emph{i.e.},\xspace the visual quality $Q_{video}$ of the output video, and the energy consumption $E_{enhancer}$ of the video enhancement model.
We formulate it as the following time-varying optimization problem.
\begin{eqnarray}\label{equ_opt}
    \mathop{arg max}_{\theta_l, \theta_f, \theta_d}  &  Q_{video} = q(\theta_l, \theta_f, \theta_d, video(t)) ,\,\,
    \mathop{arg min}_{\theta_l, \theta_f, \theta_d}  &  E_{enhancer} = e(\theta_l, \theta_f, \theta_d, resource(t))
\end{eqnarray}
where $\theta_l$, $\theta_f$, and $\theta_d$ are three dynamically adjustable parameters of the video enhancement model, \emph{i.e.},\xspace the amount of layer-level computation reuse, the amount of frame-level computation reuse, and the down-sampled video frame resolution (see \secref{subsubsec:algo:behaviour}).
It is intractable to obtain a closed-form solution to the above dynamic multi-objective optimization problems because dealing with two distinct yet related spaces, \emph{i.e.},\xspace \textit{objective space} and \textit{variable space}, is challenging.
Specifically, the \textit{objective space} consisting of the visual metric $Q_{video}$ and the energy metric $E_{enhancer}$.
Regarding the \textit{variable space} related to the objective performance, some variables (\emph{i.e.},\xspace $\theta_l$, $\theta_f$, and $\theta_d$) are available for objective optimization, we regard them as \textit{decision variables}.
Other variables (\emph{e.g.},\xspace the available execution resource $resource(t)$, the video stream $video(t)$) are imposed by the external context, which are time-varying but independent from the optimization objective variables.

\subsubsection{Optimizer}
\label{subsubsec:optimizer}
To solve the above problem, we propose a heuristic optimizer.
Specifically, we solve the original optimization problem in two stages, which correspond to two typical dynamic optimization problems~\cite{bib:TEC2004:farina}.
In the offline stage, we regard this problem as a static optimization problem and adopt existing evolutionary algorithm to find a widely distributed set of solutions and derive the Pareto-optimal front.
When looking for the Pareto frontier, we do not set any relative importance coefficients for multiple optimization objectives (\emph{i.e.},\xspace $Q$ and $E$), because we need to get the unbiased opinion from this set of solutions.
In the online stage, the optimal decision variables (\emph{i.e.},\xspace $\theta_l$, $\theta_f$, $\theta_d$) changes, whereas the optimal objective space (\emph{i.e.},\xspace $Q, E$) does not change, which is the Type 2 dynamic multiple-objective optimization problem~\cite{bib:TEC2004:farina}.
Therefore, we leverage an analytical hierarchy process to set the dynamic importance coefficients $\lambda$ of the different criteria.
Then we multiply the respective performance of each solution with that coefficients, to get a total score for each solution, \emph{i.e.},\xspace $\lambda E_{enhancer} +(1-\lambda)Q_{video}$, indicating which suits best the current wishes.
For example, if the the enhancement model's demand stray beyond resource levels, the controller re-selects the dynamic importance coefficients to pick a sole optimal solution (\emph{i.e.},\xspace $\theta_l, \theta_f, \theta_d$) from the Pareto frontier at runtime.

\begin{figure}[t]
  \centering
  \includegraphics[width=.80\textwidth]{image/optimizer.pdf}
  \caption{\rev{Workflow of the optimizer in the energy-aware adaption controller.}}
  \label{fig:optimizer}
  \vspace{-4mm}
\end{figure}
\subsection{Runtime Energy Profiler}
\label{subsec:profiler}
For the controller to function properly, it is essential to provide an accurate and timely estimate of $E_{enhancer}$ to the controller.
In {\sf UbiVE}\xspace, we propose an energy model for the video enhancement model that estimates its energy consumption given the adjustable parameters $\theta_l$, $\theta_f$, and $\theta_d$ at runtime.

However, the neighbor solutions on the Pareto frontier in the objective space are not always neighbors in the decision variable space, which makes the search of decision variable time-consuming.
In view of this challenge, we design an extra parametric neural network based regression module for building the map between the state (\emph{i.e.},\xspace the input video frame and the previous reference frame), decision variables (\emph{i.e.},\xspace $DO, FO, DR$), and the objective outcomes (\emph{i.e.},\xspace $Q, E$).
The network consists of two fully-connected layers.
The input of the neural network is \rev{the splicing matrix containing the pixel matrix of the input frame and the previous reference frame.
The output is the vector of $\theta_l$, $\theta_f$, $\theta_d$ as well as the objective outcomes (\emph{i.e.},\xspace $Q, E$).}
We collect $400$ samples to train this regression network.
\rev{At run-time, the network outputs all the optional parameters $\theta_{f}$, $\theta_{d}$, $\theta_{l}$ and the corresponding $Q, E$.
The decision space built by $\theta_{f} \in \{0,1,2...10\}$, $\theta_{d} \in \{1,1/2,1/3\}$, $\theta_{l} \in \{0,1,2,3\}$ is limited.
Therefore, together with the dynamic importance coefficient $\lambda$, it is easy to get best computation reuse parameters by calculating $\lambda E_{enhancer} +(1-\lambda)Q_{video}$.
\figref{fig:optimizer} briefly describes the optimizer workflow.}

\subsubsection{Principles}
As discussed in \secref{subsec:related:energy}, it is challenging to model the energy consumption of a neural network because the energy cost is tightly coupled with the available execution resources of the given platform.
The principles of our runtime energy profiler are two-fold.
\begin{itemize}
    \item
    To estimate the energy cost of a neural network with dynamically adjustable hyperparameters, we decompose the network into layers and strive to model the energy cost on a layer basis.
    The layer-wise decomposition is reasonable because mobile devices with CPU/GPU tend to execute a neural network layer-by-layer due to limited resources \cite{bib:rtas2020:lee}.
    \item
    To estimate the energy cost in presence of time-varying execution resources, we convert the resource dynamics into a single parameter, the cache-hit-rate, which is directly measurable at runtime.
    This is because we empirically observe that the energy cost of memory accesses is dominated by the off-chip memory accesses, which can be estimated by the cache-hit-rate.
    \figref{fig:cache} shows an example of to execute a convolution operation on a mobile platform (\emph{e.g.},\xspace with a $45$nm CMOS ARM chip).
    There are three types of memory accesses for loading the weights and activations of a neural network and writing back results (see \figref{fig:cache:a}), and the DRAM access dominates the energy cost (see \figref{fig:cache:b}).
    Since DRAM (off-chip) accesses are only necessary when the on-chip cache misses, we can use the cache-hit-rate as the multiplier to estimate the energy cost when executing a layer.
\end{itemize}
These principles reduce the overhead to estimate the energy cost to execute a neural network on a platform.

\begin{figure*}[t]
  \centering
   \subfloat[]{\label{fig:cache:a}
   \includegraphics[height=0.18\textwidth]{image/cache_access.pdf}}
   \hspace{15pt}
   \subfloat[]{\label{fig:cache:b}
   \includegraphics[height=0.18\textwidth]{image/DRAM_energy.pdf}}
\caption{(a) Memory access and (b) energy cost of a convolutional operation on a mobile device with 45nm CMOS chip.}
\label{fig:cache}
\vspace{-3mm}
\end{figure*}

\begin{figure}[t]
  \centering
  \includegraphics[width=.9\textwidth]{image/dataflow.pdf}
  \caption{Counting the memory accesses from the underlying operation loops and data flow of different layers.}
  \label{fig:dataflow}
  \vspace{-3mm}
\end{figure}

\subsubsection{Energy Model}
We first propose the energy model to execute a single layer $l$ \rev{on mobile CPU/GPU platforms}.
\rev{
\begin{equation}\label{equ_energy}
    E_l = \delta_C \times C_l + \epsilon \times \delta_{cache}\times M_l + (1-\epsilon) \times \delta_{DRAM} \times M_l + M_l \times \delta_{SM}
\end{equation}
}
where $\delta_C$, $\delta_{cache}$, $\delta_{DRAM}$, \rev{and $\delta_{SM}$} are the unit energy cost of MAC computation, cache access, DRAM access, \rev{and the shared memory access};  $C_l$ and $M_l$ are the total number of MAC and memory access of layer $l$, respectively; and $\epsilon$ is the runtime cache-hit-rate.
For a given hardware platform, these parameters can be derived as follows.
\begin{itemize}
    \item \textit{Unit Energy Cost $\delta_C$, $\delta_{cache}$,  $\delta_{DRAM}$}, and $\delta_{SM}$.
    These parameters can be measured offline.
    \rev{We empirically set $\delta_C : \delta_{cache}: \delta_{DRAM} : \delta_{SM} = 1:6:200:2$ for mobile GPU platforms.}
    \rev{As for mobile CPU platforms, $\delta_{SM} =0$ since they do not have such shared memory space,} and thus $\delta_C : \delta_{cache}: \delta_{DRAM} = 1:6:200$.
    \rev{According to our evaluations, these parameters work with different DNN inference frameworks, \emph{e.g.},\xspace Raspberry Pi 4B (CPU) + NCNN, Hornor 9 (CPU) + Pytorch Mobile, and Nvidia Jetson Nano (GPU). }
    \item \textit{Number of MAC Computation $C_l$}.
    The amount of MAC can be directly derived from the layer architecture of layer $l$, \emph{e.g.},\xspace $C_{conv_l}=K^2* C_o*C_i* H* W$.
    \item \textit{Number of Memory Accesses $M_l$}.
    The memory access during the tensor computation consists of the accesses of layer parameters, input feature maps, and intermediate iterations.
    We compute $M_l$ for a given layer type according to the data flow mapping to the given hardware platform.
    \figref{fig:dataflow} illustrates the data flow operations of different layer types used by our video enhancement model.
    For example, the total amount of memory accesses $M_l$ for a convolution layer $l$ can be calculated by the six-fold operation loops, \emph{i.e.},\xspace $M_{conv_l} = 2*C_o*C_i*H*W*K^2 + C_o*H*W$.
    \item \textit{Runtime Cache-Hit-Rate $\epsilon$}.
    It accounts for the dynamics of the execution resources and thus is measured at runtime.
    We measure $\epsilon$ as the ratio of the actual MAC execution amount per unit time (\emph{i.e.},\xspace $ms$) to the MAC amount with the simulated $100\%$ cache-hit-rate.
    The amount of MAC executed with a $100\%$ cache-hit-rate is profiled offline for the given platform.
    \rev{Observing that framework/compiler-level optimization such as operator fusion may affect the cache hit rate $\epsilon$ even for the same model~\cite{bib:mobisys21:zhang, bib:osdi2018:chen}, we measured the amount of MAC executed with a $100\%$ cache-hit-rate for multiple mainstream frameworks.}
\end{itemize}

Given the energy model for a single layer $l$ as in \equref{equ_energy}, we can estimate the total energy consumption of $n_{frame}$ video frames, with adjustable parameters $\theta_l$, $\theta_f$, and $\theta_d$ as follows.
\begin{equation}\label{equ_total_energy}
    E_{enhancer} = \sum_{f=1}^{n_{frame}} \sum_{l=1}^{L}{\theta_{f_{(index_f)}} \theta_{l_{(index_l)}} E_l \theta_{d_f} }
\end{equation}
where $\theta_{f_{(index_f)}}$ and $\theta_{l_{(index_j)}}$ are Boolean value, and 0 means skipping the computation of frame $f$ or layer $l$.
And $\theta_d$ is the down-sampling rate ($\%$) that affects the resolution of the input frame $f$ by $\theta_{d_f}*H*W$.

\subsubsection{Profiler Workflow}
The energy profiler in {\sf UbiVE}\xspace works in two stages, \textit{offline} and \textit{online}.
\begin{itemize}
    \item \textit{Offline Stage}.
    The unit energy cost $\delta_C, \delta_{cache}$,  $\delta_{DRAM}$, \rev{$\delta_{SM}$}, as well as the MAC throughput of the simulated $100\%$ cache-hit-rate are measured offline for the given platform.
    Specifically, during the offline data collection stage, we use a digital power monitor, \emph{i.e.},\xspace Monsoon AAA10F, to sample the ground truth of power consumption by profiling the device through its external power input.
    The energy cost of accessing the Cache, DRAM, and shared memory normalized to that of a MAC operation is determined by the platform based on the ground truth.
    The MAC execution amount per second in the simulated 100\% cache-hit-rate is the hardware-specified computation frequency times the computational parallelism (\emph{e.g.},\xspace 16 bits).
    The result is an energy profile with a sequence samples $\{catch-hit-rate/power~ drawn \}$.
    \item \textit{Online Stage}.
    During the online profiling stage, the profiler takes the current model hyperparameters and the runtime cache-hit-rate $\epsilon$ as input, and predicts the energy demand using the energy model in \equref{equ_total_energy}.
\end{itemize}
\rev{Note that the primary goal of the profiler is to ensure consistent \textit{ranking} between the \textit{estimated} and the \textit{actual} energy cost tested on the mobile device.
The consistency in ranking suffices to provide accurate feedback to dynamically adjust the video enhancement model’s hyperparameters for energy conservation.}

\subsection{Automated Adaptation Loop}
\label{subsec:loop}

\begin{figure}[t]
  \centering
  \includegraphics[width=.83\textwidth]{image/loop.pdf}
  \caption{Energy-aware adaptation controller workflow.}
  \label{fig:loop}
  \vspace{-3mm}
\end{figure}

\figref{fig:loop} shows the overall workflow of the energy-aware adaptation controller.
The adaptation loop consists of the \textit{resource monitor}, the \textit{runtime energy profiler}, the \textit{analyzer}, and the \textit{optimizer}.
The \textit{resource monitor} tracks the energy supply of the platform, while the \textit{energy profiler} (\secref{subsec:profiler}) predicts the energy demand of the video enhancement process with the current configurations.
If the energy demand exceeds the supply, the \textit{analyzer} notifies the \textit{optimizer}, which then re-selects $\theta_l, \theta_f, \theta_d$ as in \secref{subsubsec:optimizer}.
The control loop routinely checks for changes in the system and performs adaptations at a pre-defined frequency (\emph{e.g.},\xspace 1 second).

\section{Evaluation}
\label{sec:experiment}

\subsection{Experiment Setups}
\label{sec_exp_setup}

\subsubsection{Datasets.}
\label{subsec:datasets}
We experiment with seven datasets, six open low-light image/video enhancement benchmark datasets (SCIE~\cite{bib:tip2018:cai}, Loli-Phone~\cite{bib:TPAMI21:Li}, LOL~\cite{bib:arXiv2018:wei}, VV~\cite{data:vv}, LIME~\cite{bib:TIP16:Guo},  NightOwls~\cite{data:NightOwls}) and one self-collected dataset (MobileScene).
\tabref{tb_dataset} lists the details of these datasets.

We use SICE \cite{bib:tip2018:cai} and Loli-Phone~\cite{bib:TPAMI21:Li} for training.
SCIE contains images of different exposure levels.
We use it to train the single-frame image's enhancement performance of exposure level, smoothness, neighborhood, and color balance.
Loli-Phone contains multiple video clips taken by smartphones, and we adopt it for training the exposure and color consistency of the video enhancement model.
Specifically, we use SICE as the image training dataset as Zero-DCE \cite{bib:CVPR20:Guo}, and Loli-Phone as the video training dataset for the fast low-light video enhancement model of {\sf UbiVE}\xspace.
All the images and video frames are resized as $256\times256\times3$.
We use Adam as the optimizer and train the model for $100$ epochs.
The learning rate is set to $0.0001$ with the batch size of $4$.

The remaining datasets are for testing.
We consider datasets with both paired and unpaired samples.
VV~\cite{data:vv} and LIME~\cite{bib:TIP16:Guo} are unpaired low-light image datasets, NightOwls~\cite{data:NightOwls} is an unpaired low-light video dataset, and LOL~\cite{bib:arXiv2018:wei} is a paired low-light image dataset.
MobiScene is a self-collected dataset including four scenarios (\emph{i.e.},\xspace moving/static cameras with moving/static objects), which is used to measure the enhanced video's temporal stability.
The concrete performance metrics of each dataset are explained in the setups of each experiment below.

\begin{table}[t]
\scriptsize
\caption{Overview of low-light image and video datasets.}
\vspace{-3mm}
\begin{tabular}{|l|l|l|l|}
\hline
\multicolumn{1}{|c|}{\textbf{Datasets}} & \multicolumn{1}{c|}{\textbf{Data type}} & \multicolumn{1}{c|}{\textbf{Sample number}} & \multicolumn{1}{c|}{\textbf{Description}} \\ \hline
\textbf{1. SCIE~\cite{bib:tip2018:cai}} & Images & 2002 pieces & Diverse exposure levels \\ \hline
\textbf{2. Loli-Phone~\cite{bib:TPAMI21:Li}} & Videos  & 300 frames & Taken by real-world smartphones \\ \hline
\textbf{3. VV~\cite{data:vv}} & Unpaired images & 24 pieces & Low-light images \\ \hline
\textbf{4. LIME~\cite{bib:TIP16:Guo}} & Unpaired images & 10 pieces & Low-light images \\ \hline
\textbf{5. NightOwls~\cite{data:NightOwls}} & Unpaired videos & 500 piece &  Low-light videos \\ \hline
\textbf{6. LOL~\cite{bib:arXiv2018:wei}} & Paired images & 500 pairs & Low-light images with the corresponding high-light images \\ \hline
\textbf{7. MobiScene} & Unpaired low-light videos & 40 pieces & Self-collected videos with moving/static cameras and moving/static objects \\ \hline
\vspace{-3mm}
\end{tabular}
\label{tb_dataset}
\end{table}

\subsubsection{Baselines}
\label{sec:baseline}

We compare the performance of {\sf UbiVE}\xspace with the following baselines.
\begin{itemize}
    \item \textbf{Zero-DCE}~\cite{bib:CVPR20:Guo}.
    It is a lightweight low-light image enhancement scheme which adopts zero-shot learning to avoid training with paired data and employs an iterative curve-based architecture to map a low-light image to its enhanced version.
    \item \textbf{Zero-DCE++}~\cite{bib:TPAMI21:Li2}.
    It is the more resource-efficient version of Zero-DCE by compressing the neural network to learn the curve parameters by depth-wise separable convolutions and curve parameter map reuse.

    \item \textbf{MELLEN}~\cite{bib:BMVC18:Lv}.
    It is a recent low-light image enhancement method that achieves high enhancement quality.
    It extracts rich features via multiple sub-networks and generates the output with multi-branch fusion.
    \item \textbf{StableLLVE}~\cite{bib:CVPR21:Zhang}.
    It is a state-of-the-art low-light video enhancement model which enforces the temporal stability among frames.
    \item \textbf{{\sf UbiVE}\xspace Basic}.
    It is the variant of {\sf UbiVE}\xspace without incorporating the temporal consistency loss.
    \item \rev{\textbf{{\sf UbiVE}\xspace W/O\_Align}.
    It is the variant of {\sf UbiVE}\xspace without the optical flow alignment pre-process.}
\end{itemize}

\subsubsection{Implementation}
We implement all the compared methods in Pytorch.
Training of the algorithms is performed on a server with NVIDIA GeForce RTX 2070 GPU and Intel Core i5-10400F CPU at 2.90GHz and CUDA 10.0.
We deploy each method to mobile devices for low-light enhancement.
Specifically, we test three commercial mobile camera-embedded platforms, including a personal smartphone, \emph{i.e.},\xspace \rev{Honor 9 with Octa-core CPU processor (device 1), an embedded development board, \emph{i.e.},\xspace raspberry Pi 4B with Quad-core CPU (device 2), and a mobile platform, \emph{i.e.},\xspace NVIDIA Jetson Nano with 128-core NVIDIA Maxwell GPU (device 3)}.
For mobile deployment of {\sf UbiVE}\xspace, \rev{we use the Pytorch Mobile on the Android 12.0 platform (device 1) the NCNN framework on Raspberry Pi OS (device 2), and Pytorch 1.4 on Ubuntu 18.04 platform (device 3).}
\rev{For platforms with both mobile CPU and GPU, we configure the computing tasks to execute on either the GPU or CPU to avoid excessive GPU-CPU communication.}
We use OpenCV to invoke the embedded camera for video shooting at a rate of 20 frames per second.
The video frames are resized to $270\times{480}$ and fed into each algorithm for enhancement.

\subsection{Low-light Video Enhancement Performance}
\label{subsec:exp_compare}
This section presents the results on low-light enhancement compared with the baselines.

\subsubsection{Performance on Open Benchmarks}
This experiment compares the performance of {\sf UbiVE}\xspace and different baseline methods on open low-light enhancement datasets.

\textbf{Setups}.
We use three low-light image datasets, \emph{i.e.},\xspace VV (task3), LIME (task4), and LOL (task6) to assess the enhancement performance on a single frame, and 10 video streams from the low-light video dataset NightOwls (task5).
We compare the visual quality of the enhanced images or videos and the latency to process each image or video frame.
We use four visual quality metrics, \emph{i.e.},\xspace structural similarity (SSIM)\cite{bib:TIP2004:wang}, peak signal-to-noise ratio (PSNR), mean absolute error (MAE), and natural image quality evaluator (NIQE)\cite{bib:SPL2012:mittal}.
SSIM, PSNR, and MAE measure the visual disparity of low-light images/videos from the labeled high-quality ones, while NIQE is a no-reference metric to measure the deviations from statistical regularities observed in natural images, without training on paired data.
The experiments are conducted on Raspberry Pi 4B \rev{(with CPU, device 2)}.
We run the measurements for six times and report the average values.

\begin{table}[t]
\centering
\scriptsize
\caption{Performance comparison of {\sf UbiVE}\xspace with four baselines in terms of parameter size, latency, and visual quality.}
\vspace{-3mm}
\begin{tabular}{|c|c|c|cccccc|}
\hline
 &  &  & \multicolumn{6}{c|}{\textbf{Output visual quality}} \\ \cline{4-9}
 &  &  & \multicolumn{3}{c|}{\textbf{NIQE $\downarrow$}} & \multicolumn{1}{c|}{\textbf{SSIM $\uparrow$}} & \multicolumn{1}{c|}{\textbf{PSNR $\uparrow$}} & \textbf{MAE $\downarrow$} \\ \cline{4-9}
\multirow{-3}{*}{\textbf{Methods}} & \multirow{-3}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Parameter\\  size\end{tabular}}} & \multirow{-3}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Average latency \\ per frame (s)\end{tabular}}} & \multicolumn{1}{c|}{\textbf{VV (task3)}} & \multicolumn{1}{c|}{\textbf{LIME (task4)}} & \multicolumn{1}{c|}{\textbf{NightOwls (task5)}} & \multicolumn{3}{c|}{\textbf{LOL (task6)}} \\ \hline
\textbf{Zero-DCE~\cite{bib:CVPR20:Guo}} & 308.6KB & 11.743 & \multicolumn{1}{c|}{2.751} & \multicolumn{1}{c|}{{\color[HTML]{000000} 3.812}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 3.541}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 0.846}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 14.132}} & {\color[HTML]{000000} 52.172} \\ \hline
\textbf{Zero-DCE++~\cite{bib:TPAMI21:Li2}} & 39.1KB & 0.193 & \multicolumn{1}{c|}{2.547} & \multicolumn{1}{c|}{{\color[HTML]{000000} 3.82}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 3.159}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 0.864}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 14.596}} & {\color[HTML]{000000} 49.871} \\ \hline
\textbf{StableLLVE~\cite{bib:CVPR21:Zhang}} & 15.8MB & 1.2 & \multicolumn{1}{c|}{2.224} & \multicolumn{1}{c|}{{\color[HTML]{000000} 3.725}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 3.793}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 0.864}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 17.142}} & {\color[HTML]{000000} 34.921} \\ \hline
\textbf{MBLLEN~\cite{bib:BMVC18:Lv}} & 1.7MB & 6.225 & \multicolumn{1}{c|}{2.76} & \multicolumn{1}{c|}{{\color[HTML]{000000} 3.763}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 3.333}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 0.899}} & \multicolumn{1}{c|}{{\color[HTML]{000000} 17.229}} & {\color[HTML]{000000} 34.296} \\ \hline
\textbf{AdaEnlight} & 0.17KB & 0.05 & \multicolumn{1}{c|}{\rev{2.57}} & \multicolumn{1}{c|}{{\color[HTML]{000000} \rev{3.723}}} & \multicolumn{1}{c|}{{\color[HTML]{000000} \rev{2.882}}} & \multicolumn{1}{c|}{{\color[HTML]{000000} \rev{0.854}}} & \multicolumn{1}{c|}{{\color[HTML]{000000} \rev{17.232}}} & {\color[HTML]{000000} \rev{33.784}} \\ \hline
\end{tabular}
\label{tb_compare}
\vspace{-3mm}
\end{table}

\textbf{Results}.
\tabref{tb_compare} summarizes the results.
{\sf UbiVE}\xspace outperforms the baselines in execution latency.
In fact, only {\sf UbiVE}\xspace achieves real-time low-light enhancement.
The average delay of {\sf UbiVE}\xspace is about $0.05s$ per frame, which is smaller than the input streaming speed (\emph{i.e.},\xspace 0.1s per frame for the 10FPS video stream from NightOwls~\cite{data:NightOwls}).
In comparison, the delays of StableLIVE, MBLLEN, Zero-DCE, and Zero-DCE++ are $1.2$, $6.225s$, $11.743s$, and $0.193s$ per frame.
The low latency of {\sf UbiVE}\xspace attributes to the non-iterative enhancement with Gamma correction-based curve design (see \secref{subsec:algo:gamma}).
Despite its low latency, {\sf UbiVE}\xspace achieves competitive visual quality.
Specifically, {\sf UbiVE}\xspace achieves the best NIQE on NightOwls (video) and LIME (image), and is at least as good as Zero-DCE++ in the NIQE metric for the VV task~\cite{data:vv}.
\rev{Also, {\sf UbiVE}\xspace achieves the best PSNR and MAE on LOL \cite{bib:TIP16:Guo}.}
These outcomes validate the effectiveness of the proposed curve-based enhancement.

\subsubsection{Performance on Real-world Mobile Scenarios}
\label{exp_video}
This experiment assesses the temporal stability of the videos enhanced by {\sf UbiVE}\xspace and different baseline methods across four typical real-world mobile scenarios.
Note that both the camera and the objects in videos may be in motion in mobile shooting scenarios, making it challenging for a low-light enhancement algorithm to deliver stable video streams.

\textbf{Setups}.
We test four real-world mobile scenarios:
\textit{Scenario A:} a moving camera capturing videos of moving objects,
\textit{Scenario B:} a moving camera capturing videos of static objects,
\textit{Scenario C:} a static camera capturing videos of moving objects,
and \textit{Scenario D:} a static camera capturing videos of static objects.
We use our self-collected video dataset MobiScene and videos from NightOwls \rev{on Raspberry Pi 4B (with CPU, device 2)}.
We quantify the temporal stability of videos by temporal structural similarity index measure (TSSIM) and mean absolute brightness differences (MADB).
TSSIM is a common metric for the temporal structural similarity of adjacent frames in the video stream, and MADB measures the pixel difference between adjacent frames.

\textbf{Results}.
\figref{fig:scene} compares TSSIM and MABD of different algorithms in all the four scenarios.
\rev{Compared with prior methods, {\sf UbiVE}\xspace achieves the best results in (a), (b), (g) and the second best results in (c), (d), (e), (f).
Furthermore, {\sf UbiVE}\xspace outperforms its \textbf{Basic} variant in all the four scenarios, which validates the necessity of the temporal consistency loss.
Compared with the variant \textbf{W/O\_align}, {\sf UbiVE}\xspace achieves better results in the TSSIM metric, although {\sf UbiVE}\xspace is slightly worse than \textbf{W/O\_align} in the MABD metric.
The results show that {\sf UbiVE}\xspace achieves a better structural-level stability due to the optical flow alignment.}

\subsubsection{Performance on Downstream On-device Tasks: Face Detection}

This experiment shows the performance gains of {\sf UbiVE}\xspace for downstream vision tasks such as face detection.

\textbf{Setups}.
We use the widely used face detection algorithm, DSFD~\cite{bib:cvpr2019:li}, to detect human faces in the raw low-light frames and those enhanced by Zero-DCE, Zero-DCE++, and {\sf UbiVE}\xspace, respectively.
We use the dark face~\cite{bib:top20:yang} dataset to simulate the raw low-light frames.
The experiment was conducted \rev{on Raspberry Pi 4B (with CPU, device 2)} and we measure the precision and recall of face detection.

\begin{figure}[t]
  \centering
   \subfloat[\rev{Moving cam, moving obj}]{\label{fig:t_scene1}
   \includegraphics[width=0.245\textwidth]{image/t_scene1.pdf}}
   \subfloat[\rev{Moving cam, static obj}]{\label{fig:t_scene2}
   \includegraphics[width=0.245\textwidth]{image/t_scene2.pdf}}
   \subfloat[\rev{Static cam, moving obj}]{\label{fig:t_scene3}
   \includegraphics[width=0.245\textwidth]{image/t_scene3.pdf}}
   \subfloat[\rev{Static cam, static obj}]{\label{fig:t_scene4}
   \includegraphics[width=0.245\textwidth]{image/t_scene4.pdf}}\\
   \subfloat[\rev{Moving cam, moving obj}]{\label{fig::m_scene1}
   \includegraphics[width=0.245\textwidth]{image/m_scene1.pdf}}
   \subfloat[\rev{Moving cam, static obj}]{\label{fig::m_scene2}
   \includegraphics[width=0.245\textwidth]{image/m_scene2.pdf}}
   \subfloat[\rev{Static cam, moving obj}]{\label{fig::m_scene3}
   \includegraphics[width=0.245\textwidth]{image/m_scene3.pdf}}
   \subfloat[\rev{Static cam, static obj}]{\label{fig::m_scene4}
   \includegraphics[width=0.245\textwidth]{image/m_scene4.pdf}}
\caption{\rev{Temporal stability quantified by TSSIM and MABD, in four mobile scenarios. Scenario A: moving camera and moving objects \{(a),(e)\}, Scenario B: moving camera and static objects \{(b),(f)\}, Scenario C: static camera and moving objects \{(c),(g)\}, and Scenario D: static camera and static objects \{(d),(h)\}. A high TSSIM and a low MABD mean good temporal stability.}}
\label{fig:scene}
\vspace{-3mm}
\end{figure}

\begin{figure*}[t]
  \centering
\includegraphics[height=.22\textwidth]{image/PR_figure.pdf}
  \caption{Performance comparison for the downstream on-device video application of mobile face detection.}
  \label{fig_ficial}
  \vspace{-3mm}
\end{figure*}

\textbf{Results}.
\figref{fig_ficial} compares the precision-recall trade-off of different algorithms.
{\sf UbiVE}\xspace shows the highest area under the precision-recall curve, representing the best recall and the highest precision.
Specifically, the high precision and recall relate to a low false-positive rate and a low false-negative rate.
We can conclude that the enhanced video frames can improve facial detection performance than the raw low-light frame {\sf UbiVE}\xspace brings better facial detection benefits than Zero-DCE and Zero-DCE++.

\begin{table}[t]
\scriptsize
\caption{Comparison of {\sf UbiVE}\xspace with other commercial solutions.}
\vspace{-3mm}
\begin{tabular}{|c|c|c|c|c|c|}
\hline
\textbf{Solutions} & \textbf{Special hardware} & \textbf{Enhancement} & \textbf{Internet} & \textbf{Latency} & \textbf{Usage shortcoming} \\ \hline
\textbf{Night vision camera} & Infrared camera & On-device & None & Real-time & Lost color \\ \hline
\textbf{Adobe premiere pro} & None & PC & None & 0.04s/frame & Expert only \\ \hline
\textbf{Baidu contrast enhance API} & None & Cloud server & Dependent & 1.8s/frame & Unstable \\ \hline
\textbf{iPhone 13 camera} & None & On-device & None & 1.3s/image & No video support \\ \hline
\textbf{AdaEnlight} & None & On-device & None & 0.03s/frame & None \\ \hline
\end{tabular}
\label{tb_exp_comp_sys}
\end{table}

\begin{figure*}[t]
  \centering
   \subfloat[On-device video enhancement]{
   \includegraphics[height=0.115\textwidth]{image/exp_sys_en2.pdf}}
   \quad
   \subfloat[Remote video enhancement after internet streaming]{
   \includegraphics[height=0.115\textwidth]{image/exp_sys_en1.pdf}}

   \vspace{-3mm}
\caption{Visual quality comparison of the enhanced frame before and after commercial video encoding technique.}
\label{fig:sys_encode}
\vspace{-3mm}
\end{figure*}

\subsubsection{Comparison with Commercial Low-light Enhancement Solutions.}
\label{sec:exp_sys}
\tabref{tb_exp_comp_sys} summarizes {\sf UbiVE's}\xspace main characteristics compared with other commercial low-light enhancement solutions.
{\sf UbiVE}\xspace requires no special devices, and conducts video enhancement without network dependency.

\figref{fig:sys_encode} illustrates that the output visual quality of locally enhanced video, before encoding (\emph{e.g.},\xspace H.264 encoding), is better than the remotely enhanced one, with the intermediate encoding/decoding interference.
The video encoding (\emph{e.g.},\xspace H.264 encoding) technique is widely used to stream video over the internet.
It verifies \rev{the \textit{visual benefit of on-device video enhancement}, demonstrating the need for on-device video enhancement.}
Specifically, the natural image quality metric (NIQE) value of the locally enhanced and remotely enhanced video frame is 3.0 and 3.7, respectively.
The smaller the NIQE score, the higher the quality.

\subsection{System Performance}
\label{sec:exp_system}
This subsection presents the evaluations of {\sf UbiVE}\xspace in terms of various system performance metrics.

\subsubsection{Adaptation to Dynamic Energy Budgets of Mobile Platforms}
\label{sec:exp_budget}

\begin{table}[t]
\centering
\scriptsize
\caption{{\sf UbiVE's}\xspace performance with diverse mobile energy budgets \rev{on Honor 9 (with CPU, device 1)}.}
\vspace{-3mm}
\begin{tabular}{|c|c|c|c|c|c|c|ccc|}
\hline
\multicolumn{1}{|l|}{\multirow{2}{*}{\textbf{No.}}} & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Diverse \\ energy supply\end{tabular}}} & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Power\\ (W)\end{tabular}}} & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Energy per \\ frame(mJ)\end{tabular}}} & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Power cost \\ per video (mAh)\end{tabular}}} & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Visual quality\\ (VMAF)\end{tabular}}} & \multirow{2}{*}{\textbf{\begin{tabular}[c]{@{}c@{}}Latency\\ per frame (ms)\end{tabular}}} & \multicolumn{3}{c|}{\textbf{Optimal Hyperparameters}} \\ \cline{8-10}
\multicolumn{1}{|l|}{} &  &  &  &  &  &  & \multicolumn{1}{c|}{\textbf{$\theta_f$}} & \multicolumn{1}{c|}{\textbf{$\theta_l$}} & \textbf{$\theta_d$} \\ \hline
1 & 89\% & 4.1 & 200 & 1.7 & 99.8 & 49 & \multicolumn{0}{c|}{0} & \multicolumn{1}{c|}{0} & 1 \\ \hline
2 & 78\% & 3.8 & 191 & 1.5 & 93.2 & 47 & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1 \\ \hline
3 & 69\% & 3.7 & 182 & 1.4 & 90.2 & 45 & \multicolumn{1}{c|}{1} & \multicolumn{1}{c|}{1} & 1/2 \\ \hline
4 & 50\% & 3.5 & 175 & 1.3 & 86.2 & 42 & \multicolumn{1}{c|}{2} & \multicolumn{1}{c|}{2} & 1 \\ \hline
5 & 48\% & 3.2 & 138 & 1.1 & 73.7 & 37 & \multicolumn{1}{c|}{3} & \multicolumn{1}{c|}{1} & 1/2 \\ \hline
6 & 32\% & 3.2 & 93 & 0.8 & 72.1 & 30 & \multicolumn{1}{c|}{3} & \multicolumn{1}{c|}{3} & 1/3 \\ \hline
\end{tabular}
\label{tb_energy}
\end{table}

\begin{table}[t]
\centering
\scriptsize
\caption{\rev{{\sf UbiVE's}\xspace performance with diverse mobile energy budgets on Jetson Nano (with GPU, device 3)}.}
\vspace{-3mm}
\begin{tabular}{|c|c|c|c|c|c|c|ccc|}
\hline
\multicolumn{1}{|l|}{\multirow{2}{*}{\rev{\textbf{No.}}}} & \multirow{2}{*}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Diverse \\ energy supply\end{tabular}}}} & \multirow{2}{*}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Power\\ (W)\end{tabular}}}} & \multirow{2}{*}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Energy per \\ frame(mJ)\end{tabular}}}} & \multirow{2}{*}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Power cost \\ per video (mAh)\end{tabular}}}} & \multirow{2}{*}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Visual quality\\ (VMAF)\end{tabular}}}} & \multirow{2}{*}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Latency\\ per frame (ms)\end{tabular}}}} & \multicolumn{3}{c|}{\rev{\textbf{Optimal Hyperparameters}}} \\ \cline{8-10}
\multicolumn{1}{|l|}{} &  &  &  &  &  &  & \multicolumn{1}{c|}{\rev{\textbf{$\theta_f$}}} & \multicolumn{1}{c|}{\rev{\textbf{$\theta_l$}}} & \rev{\textbf{$\theta_d$}} \\ \hline
\rev{1} & \rev{95\%} & \rev{3.3} & \rev{82.3} & \rev{0.68} & \rev{100} & \rev{25} & \multicolumn{0}{c|}{\rev{0}} & \multicolumn{1}{c|}{\rev{0}} & \rev{1} \\ \hline
\rev{2} & \rev{81\%} & \rev{3.2} & \rev{66.2} & \rev{0.55} & \rev{96.9} & \rev{20} & \multicolumn{1}{c|}{\rev{1}} & \multicolumn{1}{c|}{\rev{1}} & \rev{1} \\ \hline
\rev{3} & \rev{76\%} & \rev{3.2} & \rev{57.6} & \rev{0.48} & \rev{94.2} & \rev{18} & \multicolumn{1}{c|}{\rev{4}} & \multicolumn{1}{c|}{\rev{1}} & \rev{1} \\ \hline
\rev{4} & \rev{40\%} & \rev{3.0} & \rev{48.1} & \rev{0.40} & \rev{75.1} & \rev{12} & \multicolumn{1}{c|}{\rev{0}} & \multicolumn{1}{c|}{\rev{0}} & \rev{1/2} \\ \hline
\end{tabular}
\label{tb_energy_nano}
\end{table}

This experiment evaluates how {\sf UbiVE}\xspace optimizes visual quality under different energy budget.
Given a specific energy budget, {\sf UbiVE's}\xspace controller actively adjusts its behaviors (\emph{i.e.},\xspace the frame-level computation reuse, the layer-level computation reuse, and the frame resolution) by re-selecting the suitable hyperparameters $\theta_f$, $\theta_l$, and $\theta_d$.

\textbf{Setups}. We leverage the NightOwls video dataset (task 5) for evaluation.
We normalize the resolution of all video frames into the same size, \emph{i.e.},\xspace 270P, for fair comparison.
And we sample \rev{diverse stations of energy supply (\emph{i.e.},\xspace the remaining battery) on Honor 9 (with CPU, device 1) and Jetson Nano (with GPU, device 3)}, to evaluate {\sf UbiVE's}\xspace hyperparameter selection on computation reuse and down-sampling.
The computation reuse and down-sampling operations for energy conservation will bring about a decline in visual quality.
Thus, we adopt a full-reference video quality metric, \emph{i.e.},\xspace video multimethod assessment fusion (VMAF) to quantify the quality decrease of computation reuse in different levels.

\textbf{Results}.
\tabref{tb_energy} and \tabref{tb_energy_nano} summarize the performance and the corresponding hyperparameters with different energy supply.
We make the following observations.
First, {\sf UbiVE}\xspace can select suitable hyperparameters with various energy budgets.
For example, \rev{at $78\%$ energy supply on Honor 9 (with CPU, device 1)}, the optimal hyperparameter is to reuse 1 frame and 1 layer without down-sampling.
At $48\%$ energy supply, it becomes reusing 3 frames and 3 layers and down-sampling by $1/3$.
Second, different hyperparameters setups of $\theta_f$, $\theta_l$, and $\theta_d$ can dynamically tune the trade-off between the output video quality and the energy consumption.
Third, the computation reuse and down-sampling behaviors can further reduce the execution latency of the video enhancement model, ranging from $49ms$ to $30ms$ per frame.
Therefore, all these hyperparameters ensure real-time constraints.
\rev{Finally, for completeness, we run {\sf UbiVE}\xspace on Honor 9 (device 1, with $3200$ mAH battery) with different computation reuse parameters to simulate diverse user preference on visual quality  and energy consumption.
Experiments show that {\sf UbiVE}\xspace will drain the battery in $2.5h \sim 4h$ under these settings.}

\begin{figure}[t]
  \centering
   \subfloat[Latency per frame]{\label{fig:fps_latency_frame}
   \includegraphics[width=0.2\textwidth]{image/fps_latency_f.pdf}}
   \hfill
   \subfloat[Latency per video]{\label{fig:fps_latency_video}
   \includegraphics[width=0.2\textwidth]{image/fps_latency_v.pdf}}
   \hfill
   \subfloat[Power]{\label{fig:fps_power}
   \includegraphics[width=0.2\textwidth]{image/fps_power.pdf}}
    \hfill
   \subfloat[Energy cost per frame]{\label{fig:fps_energy}
   \includegraphics[width=0.2\textwidth]{image/fps_energy.pdf}}
   \vspace{-3mm}
\caption{{\sf UbiVE's}\xspace performance across diverse inputs with varying video frame rates (FPS), \emph{i.e.},\xspace 10 FPS, 20FPS, and 30FPS.}
\label{fig:fps}
\vspace{-3mm}
\end{figure}

\begin{figure*}[t]
  \centering
  \includegraphics[height=0.29\textwidth]{image/various_device.pdf}
  \caption{\rev{{\sf UbiVE's}\xspace performance on three typical camera-embedded platforms.}}
  \label{fig_device}
  \vspace{-3mm}
\end{figure*}

\subsubsection{Adaptation to Input Video Frame Rates}
We now test {\sf UbiVE's}\xspace performance over diverse  video streams.

\textbf{Setups}.
We use $20$ video clips of $10s$ duration with three different video frame rates, \emph{i.e.},\xspace 10 frame per second (FPS), 20FPS, and 30 FPS.
We test the {\sf UbiVE's}\xspace execution latency on \rev{Honor 9 (with CPU, device 1)}.
We use the digital power monitor of Monsoon AAA10F to power the smartphone and measure its energy cost.

\textbf{Results}.
\figref{fig:fps} shows the performance of {\sf UbiVE}\xspace across diverse input video streams.
First, {\sf UbiVE}\xspace achieves real-time processing for all frame rates.
Specifically, for the 10FPS, 20FPS, and 30FPS input video streams, the average processing latency per frame is $50ms$, $47ms$, and $31ms$, respectively.
Furthermore, the average delay of {\sf UbiVE}\xspace to process the entire video clip with these three frame rates is $\leq 10s$, which is smaller than the streaming speed of the input video, thereby ensuring real-time video stream enhancement.
Second, the energy consumption of {\sf UbiVE}\xspace on the Honor 9 platform is adjustable for diverse input videos with the same energy budget.
Specifically, the energy cost of {\sf UbiVE}\xspace for the 10FPS, 20FPS, and 30FPS input videos is $227 mJ$, $158 mJ$, and $97mJ$, respectively.
For video streams over 30FPS, we can further reduce the execution latency and energy consumption by adjusting the computation reuse and frame resolution hyperparameters.

\subsubsection{Performance on Diverse Mobile Platforms}
\label{subsec:exp_device}
This experiment evaluates {\sf UbiVE}\xspace on different  devices.

\textbf{Setups}.
We test three platforms: Honor 9 \rev{(using CPU, device 1)}, Raspberry Pi 4B \rev{(with CPU, device 2)}, and \rev{NVIDIA Jetson Nano (with GPU, device 3)}.
Different devices have diverse resource availability (\emph{e.g.},\xspace computation throughput and memory hierarchy) and \rev{frameworks (Pytorch mobile, NCNN, Pytorch)}, which lead to different execution latency.
We use the NightOwls dataset to extract video samples as the input of {\sf UbiVE}\xspace.
We use nine input videos of 10s at 270P, and a frame rate of 10FPS for testing to obtain the overall processing delay.

\textbf{Results}.
\figref{fig_device} compares the {\sf UbiVE}\xspace performance on these three different platforms.
On all three resource-constrained mobile devices, {\sf UbiVE}\xspace achieves real-time processing with $20ms \sim 51.5ms$ execution latency.

\subsection{Ablation Studies}
\label{subsec:exp:bench}

This section evaluates the impact of various setups on the {\sf UbiVE's}\xspace performance.

\begin{figure*}[t]
  \centering
   \subfloat[$Ite=1$, Zero-DCE++]{
   \includegraphics[width=0.2\textwidth]{image/dce_plus_ite1.pdf}}
   \hfill
   \subfloat[$Ite=8$, Zero-DCE++]{
   \includegraphics[width=0.2\textwidth]{image/dce_plus_ite8.pdf}}
  \hfill
  \subfloat[\rev{$Ite=1$, {\sf UbiVE}\xspace}]{
  \includegraphics[width=0.2\textwidth]{image/our_ite1.pdf}}
  \hfill
  \subfloat[\rev{$Ite=8$, {\sf UbiVE}\xspace}]{
  \includegraphics[width=0.2\textwidth]{image/our_ite8.pdf}}
  \vspace{-3mm}
\caption{\rev{Impact of iteration numbers $Ite$ on  visual quality by Zero-DCE++ and our Gamma correction-based curve.}}
\label{fig:iterations}
\vspace{-3mm}
\end{figure*}

\begin{figure}[t]
  \centering
   \subfloat[Iteration number]{\label{fig_ite_num}
   \includegraphics[width=0.24\textwidth]{image/ite_num.pdf}}
   \hfill
   \subfloat[Architecture]{\label{fig_layer_num}
   \includegraphics[width=0.24\textwidth]{image/layer_num.pdf}}
   \hfill
   \subfloat[Reuse computation]{\label{fig_reuse}
   \includegraphics[width=0.24\textwidth]{image/reuse.pdf}}
   \hfill
   \subfloat[frame resolution]{\label{fig_resolution}
   \includegraphics[width=0.24\textwidth]{image/downsample.pdf}}
\caption{Benchmark of various parameters, \emph{i.e.},\xspace (a) iteration number, (b) layer and channel number, (c) frame-level and layer-level computation reuse, and (d) frame resolution, on {\sf UbiVE's}\xspace performance.}
\label{fig_bench}
\end{figure}

\subsubsection{Iteration Number in Gamma Correction-based Curve}
\label{subsec:exp_ite}
As discussed in \secref{subsec:algo:gamma}, we propose the non-iterative enhancement method to break the latency bottleneck of traditional curve-based image enhancement.
As shown in \figref{fig:iterations}, the visual quality enhanced by {\sf UbiVE}\xspace does not notably increase with more iterations.
Thus, we set the iteration number as 1, which is equivalent to a non-iterative model.
\figref{fig_ite_num} compares the execution latency of {\sf UbiVE's}\xspace enhancement model and Zero-DCE with different iteration numbers in their curve functions.
{\sf UbiVE}\xspace avoids the long latency due to the iterative enhancement process in Zero-DCE.

\subsubsection{Layer and Channel Number}
To tune the architecture hyperparameters (\emph{i.e.},\xspace the number of layer and channel) of {\sf UbiVE's}\xspace enhancement model, we compare the performance of different architecture settings.
We randomly selected six video segmentations from NightOwls dataset with the frame resolution of $270\times{480}$ for testing, each video has a duration of 10S, a frame rate of 10FPS, and a frame resolution of $270\times{480}$.
\figref{fig_layer_num} illustrates the model's execution latency with four settings.
All the settings result in the competitive visual quality, while 3 layers with 4 channels has the lowest latency \rev{on Raspberry Pi 4B (with CPU, device 2)}.
Thus, we set the layer number as 3 and the channel number as 6 in {\sf UbiVE's}\xspace enhancement model by default.

\subsubsection{Impact of Computation Reuse}
\label{sec:exp_reuse}
As discussed in \secref{subsubsec:algo:behaviour}, due to temporal similarities between adjacent video frames, we can dynamically adjust the model complexity by enforcing computation reuse.
\figref{fig_reuse} illustrates the energy consumption of {\sf UbiVE's}\xspace video enhancement model \rev{on Honor 9 (with CPU, device 1)}, under different frame and layer reuse settings.
We compare five options, \emph{i.e.},\xspace reuse\#1 (frame = 1 layer = 1),  reuse\#2 (frame = 1, layer = 2), reuse\#3 (frame = 1, layer = 3), and reuse\#4 (frame = 2, layer = 2), and reuse\#5 (frame = 2, layer = 3).
The outcomes verify that these computation reuse hyperparameters can effectively tune the model's energy cost.
\rev{It is worth mentioning that such computation reuse also reduces the overall latency by $10\% \sim 50\%$.
For example, given a fixed frame-level computation reuse parameter 5, if we set layer-level computation reuse parameter as 1,2 and 3, the latency decreases by 12.5\%, 35.1\% and 57.6\%, respectively.
However, since {\sf UbiVE}\xspace without computation reuse already achieves near real-time processing \emph{i.e.},\xspace 50ms per frame (shown in \tabref{tb_compare}), a reduction of $50\%$ in delay does not introduce perceivable improvement in user experience.
Therefore, we mainly focus on the improvement in energy consumption due to computation reuse.}

\subsubsection{Impact of Frame Resolution}
This experiment verifies the effect of input frame resolution on the energy consumption of {\sf UbiVE's}\xspace video enhancement, \rev{on Honor 9 (with CPU, device 1)}.
\figref{fig_resolution} compares the energy cost of three down-sampling levels (\emph{i.e.},\xspace $1\sim 3$), representing the full resolution, down to $1/2$ resolution, and down to $1/3$ resolution, respectively.
The energy consumption increases linearly with the frame resolution.
Thus, down-sampling the resolution of the input video frame can naturally tune the model's energy demand.

\begin{table}[t]
\centering
\scriptsize
\caption{Cross-time and cross-model energy profiler measurement on Honor 9 (with CPU, device 1).}
\vspace{-3mm}
\begin{tabular}{|cccccc|}
\hline
\multicolumn{6}{|c|}{\textbf{Cross-time evaluation}} \\ \hline
\multicolumn{1}{|c|}{\textbf{\begin{tabular}[c]{@{}c@{}}Prediction at time 1\\ (resource state 1)\end{tabular}}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}Measurement at time 1\\ (resource state 1)\end{tabular}}} & \multicolumn{1}{c|}{\textbf{Error}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}Prediction at time 2\\ (resource state 2)\end{tabular}}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}Measurement at time 2\\ (resource state 2)\end{tabular}}} & \textbf{Error} \\ \hline
\multicolumn{1}{|c|}{0.192mJ} &
\multicolumn{1}{c|}{0.2mJ} &
\multicolumn{1}{c|}{4\%} &
\multicolumn{1}{c|}{0.141mJ} &
\multicolumn{1}{c|}{0.138mJ} &
\multicolumn{1}{c|}{1.4\%} \\ \hline
\multicolumn{6}{|c|}{\textbf{Cross-model evaluation}} \\ \hline
\multicolumn{1}{|c|}{\textbf{\begin{tabular}[c]{@{}c@{}}Prediction for model 1\\ (4 layers )\end{tabular}}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}Measurement for model 1\\ (4 layers)\end{tabular}}} & \multicolumn{1}{l|}{\textbf{Error}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}Prediction for model 2\\ (5 layers)\end{tabular}}} & \multicolumn{1}{c|}{\textbf{\begin{tabular}[c]{@{}c@{}}Measurement for model 2\\ (5 layers)\end{tabular}}} & \multicolumn{1}{l|}{\textbf{Error}} \\ \hline
\multicolumn{1}{|c|}{0.285mJ} &
\multicolumn{1}{c|}{0.27mJ} &
\multicolumn{1}{c|}{5.6\%} &
\multicolumn{1}{c|}{0.368mJ} &
\multicolumn{1}{c|}{0.38mJ} &
\multicolumn{1}{c|}{3.2\%} \\ \hline
\end{tabular}
\label{tb_exp_general}
\end{table}

\begin{table}[t]
\centering
\scriptsize
\caption{\rev{Cross-time and cross-model energy profiler measurement on Jetson Nano (with GPU, device 3).}}
\vspace{-3mm}
\begin{tabular}{|cccccc|}
\hline
\multicolumn{6}{|c|}{\rev{\textbf{Cross-time evaluation}}} \\ \hline
\multicolumn{1}{|c|}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Prediction at time 3\\ (resource state 1)\end{tabular}}}} & \multicolumn{1}{c|}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Measurement at time 3\\ (resource state 1)\end{tabular}}}} & \multicolumn{1}{c|}{\rev{\textbf{Error}}} & \multicolumn{1}{c|}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Prediction at time 4\\ (resource state 2)\end{tabular}}}} & \multicolumn{1}{c|}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Measurement at time 4\\ (resource state 2)\end{tabular}}}} & \rev{\textbf{Error}} \\ \hline
\multicolumn{1}{|c|}{\rev{0.36mJ}} & \multicolumn{1}{c|}{\rev{0.38mJ}} & \multicolumn{1}{c|}{\rev{5.6\%}} & \multicolumn{1}{c|}{\rev{0.541mJ}} & \multicolumn{1}{c|}{\rev{0.574mJ}} &
\rev{6.1\%} \\ \hline
\multicolumn{6}{|c|}{\rev{\textbf{Cross-model evaluation}}} \\ \hline
\multicolumn{1}{|c|}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Prediction for model 1\\ (4 layers )\end{tabular}}}} & \multicolumn{1}{c|}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Measurement for model 1\\ (4 layers)\end{tabular}}}} & \multicolumn{1}{l|}{\rev{\textbf{Error}}} & \multicolumn{1}{c|}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Prediction for model 3\\ (3 layers)\end{tabular}}}} & \multicolumn{1}{c|}{\rev{\textbf{\begin{tabular}[c]{@{}c@{}}Measurement for model 3\\ (3 layers)\end{tabular}}}} & \multicolumn{1}{l|}{\rev{\textbf{Error}}} \\ \hline

\multicolumn{1}{|c|}{\rev{0.129mJ}} & \multicolumn{1}{c|}{\rev{0.128mJ}} & \multicolumn{1}{c|}{\rev{0.8\%}} & \multicolumn{1}{c|}{\rev{0.090mJ}} & \multicolumn{1}{c|}{\rev{0.093mJ}} &
\rev{3.3\%} \\ \hline
\end{tabular}
\label{tb_exp_general_gpu}
\end{table}

\subsubsection{Generalization of Energy Profiler}
This experiment demonstrates the cross-time and cross-model generalization performance.
We refer to the evaluation methodology in nn-Meter~\cite{bib:mobisys21:zhang}.
We use the existing energy profiler trained for Honor 9 \rev{(with CPU, device 1) and Jetson Nano (with GPU, device 3) to predict model energy consumption for four different times and three models.
We evaluate our energy profiling for CPU at time 1 and time 2 with model 1 and model 2; while for GPU at time 3 and time 4 with model 1 and model 3.}
At different times, the available execution resources of the platform differ, which will affect the cache hit rate and thus the energy demand.
The three models have different architectures: model 1 has 3 Conv and activation layers, model 2 has 4 Conv and activation layers, \rev{and model 3 has 2 Conv and activation layers.}
As shown in \tabref{tb_exp_general} and \tabref{tb_exp_general_gpu}, the energy profiler achieves \rev{$\geq 93.9\%$} accuracy for cross-time prediction, and \rev{$\geq 94.4\%$} for cross-model prediction.
\rev{Importantly, the results show that our energy profiler ensures consistent ranking between the estimated and the actual energy cost, which is crucial for the controller to adapt the hyperparameters to the energy demand.}

\rev{\subsection{Summary of Main Experimental Results}}
\rev{We summarize the main results of our evaluations as follows.
\begin{itemize}
    \item \textit{Near real-time processing on diverse mobile devices}.
    \newrev{{\sf UbiVE's}\xspace enhancement model} achieves near real-time video enhancement on different mobile devices, \emph{e.g.},\xspace $51$ ms per frame on Raspberry Pi 4B, $47$ ms per frame on Honor 9, and $20$ ms per frame on NVIDIA Jetson Nano.
    \newrev{This is because the proposed non-iterative model breaks the latency bottleneck of the state-of-the-art enhancement schemes}.

    \item \textit{Better trade-off between processing delay and visual quality}.
    \newrev{{\sf UbiVE's}\xspace enhancement model} achieves the lower video enhancement latency while retaining competitive visual quality.
    {\sf UbiVE's}\xspace latency is only 1/4 of Zero-DCE++ \cite{bib:TPAMI21:Li2}, the fastest low-light enhancement scheme.
    Meanwhile, {\sf UbiVE}\xspace still yields high visual quality.
    For example, {\sf UbiVE}\xspace achieves a PSNR of $17.232$, which is almost the same as MBLLEN \cite{bib:BMVC18:Lv}, the state-of-the-art video enhancement solution (with a PSNR of $17.229$).

    {\sf UbiVE}\xspace consistently outputs stable and high-quality videos on various datasets (LOL, NightOwls, LIME and VV), and different scenarios (static/mobile camera, static/mobile objects).
    \newrev{This is mainly due to the enforced temporal consistency between video.}

    \item \textit{Adaptive to diverse energy supply}.

    \newrev{{\sf UbiVE's}\xspace energy-aware controller} is able to adapt the enhancement quality according to the energy budget at runtime.
    As a concrete example, when the energy supply is reduced from 95\% to 40\%, the power consumption of the {\sf UbiVE}\xspace is reduced from 0.68mAh to 0.4mAh (see \tabref{tb_exp_general_gpu}).
    \newrev{It proves the necessity of the energy-aware controller in mobile systems.}

\end{itemize}
}
\section{Introduction}
\label{sec:intro}

The pervasive deployment of camera-embedded mobile devices \emph{e.g.},\xspace smartphones, wearables, tablets, and robots has stimulated a wide spectrum of novel mobile video based applications.
Examples include face detection on smartphones for user authentication \cite{bib:percom2020:fan}, outdoor surveillance with mobile robots \cite{bib:IDST2021:al, bib:INDIN12:Juang}, and object detection with drones~\cite{bib:ICCV2019:zhu}, \emph{etc.}\xspace
These applications are features with two characteristics.
\textit{(i)}
There is a growing interest in \textit{on-device processing} of these video streams rather than cloud offloading for privacy concerns or real-time interactions \cite{bib:Computer17:Ananthanarayanan}.
\textit{(ii)}
The video streams captured by mobile devices often suffer from \textit{low-light effect} since they are typically taken by non-professional users with narrow-aperture cameras \cite{bib:ConTEL19:Teli, bib:TIP19:Ren}.
Low-light videos can notably impair user experience \emph{e.g.},\xspace for video display, and downstream application accuracy, \emph{e.g.},\xspace face detection.

Despite extensive research on low-light image and video enhancement \cite{bib:ICCV19:Chen, bib:CVPR20:Guo, bib:TIP16:Guo, bib:ICCV19:Jiang, bib:BMVC18:Lv, bib:TPAMI21:Li, bib:TPAMI21:Li2, bib:CVPR21:Zhang}, they are unfit for deployment to mobile devices because they fail to meet the following requirements.
\begin{itemize}
    \item \textit{Real-time Processing on Resource-constrained Mobile Devices.}
    The \textit{real-time} processing of video stream is necessary for application's responsiveness, and \textit{on-device} processing benefits the user privacy.
    However, mainstream low-light video enhancement schemes are computation-intensive \cite{bib:BMVC18:Lv, bib:ICCV19:Jiang, bib:ICCV19:Chen, bib:CVPR21:Zhang}.
    Even the state-of-the-art lightweight image enhancement model Zero-DCE++ \cite{bib:TPAMI21:Li2} fails to achieve real-time processing on mobile devices such as a Raspberry Pi ($5$ $frames/s$ for $270 \times 480$ RGB images, see \secref{sec:experiment}).
    \item \textit{Adapting to Diverse Energy Supply of Mobile Devices.}
    \textit{Energy awareness} is crucial for the long-term operation of the above video processing since mobile devices are often battery-powered~\cite{bib:SOSP99:Flinn, bib:todaes2000:benini}.
    It is desired that the low-light video processing module can adapt its enhancement quality according to the energy budget at runtime.
    Despite prior studies on adaptive mobile computer vision \cite{bib:sigcomm2020:kim, bib:mobicom2019:baig, bib:hotcloud2019:wang, bib:SIGCOMM2016:sun, bib:mobicom2019:lee, bib:tmc2020:yi} and general energy profiling of neural networks \cite{bib:CVPR17:Yang, bib:TCS20:Lee, bib:ssc2017:yang, bib:ICCAD2019:wu,bib:islped2018:ke, bib:ISPASS2019:parashar}, there lacks a runtime energy profiler and adaptation loop dedicated to deep learning enabled low-light video enhancement.
\end{itemize}

In this paper, we propose {\sf UbiVE}\xspace, an energy-aware on-device low-light video enhancement solution for mobile platforms.
It is challenging to achieve high enhanced visual quality, low execution latency, and adaptation to energy budget on mobile platforms.
{\sf UbiVE}\xspace tackles these challenges via a modular system design and a set of novel algorithms.
Observing that latency bottleneck of the state-of-the-art image enhancement schemes \cite{bib:CVPR20:Guo, bib:TPAMI21:Li2} lies in the iterative enhancement curve function, {\sf UbiVE}\xspace devises a novel non-iterative curve function for acceleration without compromising enhancement quality.
{\sf UbiVE}\xspace also incorporates temporal consistency among frames to mitigate the flicking problem \cite{bib:TPAMI21:Li} when enhancing video streams.
To adapt to the dynamic energy budget imposed by mobile platforms, {\sf UbiVE}\xspace exploits a runtime energy profiler to estimate the energy cot of the video enhancement process at runtime and configures its hyper-parameters (\emph{e.g.},\xspace computation reuse and frame resolution) accordingly.
We implement {\sf UbiVE}\xspace as a compact video pre-processing middleware to provide enhanced video streams for various downstream mobile video applications such as facial recognition in automatic package delivery and object detection in video surveillance (see \figref{fig:middleware_overview}).

\begin{figure}[t]
  \centering
  \includegraphics[width=.9\textwidth]{image/pipeline.pdf}
  \caption{Illustration of {\sf UbiVE}\xspace as a video enhancement middleware for downstream mobile video applications.}
  \label{fig:middleware_overview}
  \vspace{-3mm}
\end{figure}

Our main contributions are summarized as follows.
\begin{itemize}
    \item
    To the best of our knowledge, {\sf UbiVE}\xspace is the first \textit{on-device} low-light \textit{video} enhancement system for \textit{mobile} scenarios. It achieves near real-time and high-quality processing on commodity mobile platforms and stays adaptive to the dynamic energy budget.
    \item
    The key technical novelties include a non-iterative low-light enhancement model that breaks
    the latency bottleneck of the state-of-the-art enhancement schemes and enforces temporal consistency between video frames.
    Also, it adopts a runtime energy-aware controller that integrate a runtime energy profiler to adjust the video enhancement behaviors adaptively.
    \item
    We evaluate {\sf UbiVE}\xspace on four public benchmarks and real-world mobile scenarios on three mobile platforms (Honor 9 smartphone, raspberry Pi 4B, and Jetson AGX Xavier).
    Experiments show that {\sf UbiVE}\xspace achieves near real-time ($30\sim50ms$ per frame) video enhancement with competitive visual quality to existing image and video enhancement schemes.
    {\sf UbiVE}\xspace also enables agile self-adaptation to satisfy the dynamic energy budgets at runtime.
\end{itemize}

In the rest of this paper, we present an overview of {\sf UbiVE}\xspace as well as its functional modules in \secref{sec:overview}, \secref{sec:algorithm}, and \secref{sec:controller}.
We show the evaluations in \secref{sec:experiment}, review related work in \secref{sec:related}, and conclude in \secref{sec:conclude}.

\section{Motivations}
\label{sec:motivation}

\subsection{Intractable QoE Demands on Ubiquitous Video Stream Enhancement}
Both identifying and boosting the user demands on the quality of experience (QoE) of ubiquitous video streams are intractable.
Specifically, the first challenge is building the relationship between QoE measurement and the mobile users' satisfaction.
The QoE metrics we intend to adopt aim to capture the quality of the video enhancement process from both the view of \textit{user experience} and \textit{mobile system performance}.
Depart from the existing efforts on adaptive video streaming, which leverage the coarse-grained resolution or bit rate of each frame as the visual quality metrics.
In this paper, we present to optimize the sequential relevance-aware video stream in the pixel level and introduce the following three QoE demands for enhancing the ubiquitous video stream(s) at the mobile user's end:
\begin{itemize}
    \item \textit{Visual Quality}: maximize the fine-grained visual quality $Q$ of the processed video stream, \emph{e.g.},\xspace the signal-to-noise ratio $Q_{n}$ of video display, the recognition accuracy specified by the subsequent video applications;
    \item \textit{Resource efficiency}: minimize the \textit{response latency} $T_{en}$ of video enhancement process on diverse deployment devices with tightly limited resource budgets (\emph{e.g.},\xspace computation $C_{bgt}$, memory $M_{bgt}$);
    minimize the energy consumption $E_{en}$ imposed by running video enhancement model on the deployment devices.
    \item  \textit{Agility}: minimize the \textit{response latency} $T_{en}$ of video enhancement process on diverse deployment devices with tightly limited resource budgets (\emph{e.g.},\xspace computation $C_{bgt}$, memory $M_{bgt}$);
\end{itemize}
And the second challenge is, none of existing video QoE enhancement techniques can achieve a great tradeoff between real-time processing efficiency and resource efficiency. \TODO{Prof. Zhou, could you please revise this sentence about existing efforts and check the following preliminary experiment settings.}
Furthermore, to achieve the above three video QoE enhancement objectives, we evaluate the feasibility of three classes of existing video QoE improvement techniques, \emph{i.e.},\xspace video frame-based enhancement methods xx, video stream-based enhancement methods xx, and video frame-based compressed enhancement methods.
As we will show in $\S$~\ref{sec:experiment}, none of them can achieve all of these three QoE objectives.

Given these challenges, we intend to provide a novel solution, \emph{i.e.},\xspace ~ {\sf UbiVE}\xspace middleware, to be compatible with resource heterogeneity and optimize all of the above three QoE metrics.

In response to this problem,

However, it is non-trivial to profile the energy usage of the DNN-based video process for providing exact runtime feedback on \textit{unknown resource availability} with \textit{agnostic video stream and DNN topology}.
The majority of existing energy profiling methods heavily depend on the processing model's topology (\emph{e.g.},\xspace model structure, computation path)~\cite{bib:Mobisys18:Liu} and the deployment platform's hardware resource configurations (\emph{e.g.},\xspace on-chip memory capability, computation bandwidth)~\cite{bib:CVPR17:Yang, bib:TCS20:Lee}.
That is, once the video process models or deployment devices change at runtime, the above pre-defined energy profiling methods will fail to provide the precise energy prediction.
However, this work intends to adaptively adjust the video enhancement model's computation according to the run-time prediction results of energy cost at unknown deployment devices.
Therefore, a \textit{runtime DNN-aware} and \textit{one-fit-all} energy prediction method is necessary.

Depart from the above methods, {\sf UbiVE's}\xspace profile the DNN energy based on the following two experimental observations:
\textit{(i)} despite the dynamic topology of video processing models, the underlying tensor operation (\emph{i.e.},\xspace the energy-dominating operation loops) and the data flow are stable.
\textit{(ii)} In spite of the dynamic hardware resource supply (\emph{e.g.},\xspace on-chip memory capability, computation bandwidth), the hit ratio of on-chip memory access shows up the stable rules and within a small sampling set.
\section{{\sf UbiVE}\xspace Overview}
\label{sec:overview}
This section presents our problem scope and the design overview of {\sf UbiVE}\xspace.

\subsection{Problem Scope}
\label{subsec:scope}
In short, we target at on-device low-light enhancement of ubiquitous video streams.
We elaborate on the concrete motivations and requirements below.

\begin{figure}[t]
  \centering
  \includegraphics[width=.97\textwidth]{image/study.pdf}
  \caption{An example study of the ubiquity of low-light video streams in the mobile video surveillance scenario.}
  \label{fig:study}
  \vspace{-3mm}
\end{figure}

\subsubsection{Motivations}
\label{subsubsec:motivations}
Our work is motivated by the ubiquity of camera-embedded devices (\emph{e.g.},\xspace smartphones, wearables, tablets, and robots) in everyday life for various video applications ranging from video surveillance \cite{bib:INDIN12:Juang} to video conferencing \cite{app:zoom}.
These video streams, which we call as \textit{ubiquitous video streams}, are often captured by \textit{non-professional users} and are likely to experience \textit{low-light} conditions.
The reasons are two-fold.
\textit{(i)}
The limited camera aperture size on mobile devices restrict the overall amount of light that reaches the camera sensor and thereby decrease the signal-to-noise ratio of the captured videos~\cite{bib:ConTEL19:Teli}.
\textit{(ii)}
Non-professional users may take videos under uncontrolled lighting conditions, \emph{e.g.},\xspace back-lit, weak light, and extremely dark environments \cite{bib:TIP19:Ren}.
Also, we conduct an example study using a mobile robot (\emph{i.e.},\xspace Yahboom) to show the ubiquity of low-light video streams in the daily video surveillance scenario.
The robot continuously collects video streams by random walking indoor and outdoor for 6 hours, at daytime and nighttime.
\figref{fig:study} shows the details of four groups of study.
The proportion of low-light video streams is up to $87.5\%$ (\emph{i.e.},\xspace group No.4) and $10.5\%$ (\emph{i.e.},\xspace group No.1) at nighttime and daytime, respectively.

\rev{We consider video enhancement \textit{locally on-device} rather than \textit{remotely on cloud} for the following reasons.}
\begin{itemize}
    \item
    The ubiquitous video streams may be immediately consumed on the mobile devices \cite{bib:Computer17:Ananthanarayanan}.
    For example, a surveillance drone may run object recognition algorithms on video streams taken in dim light and only uploads suspicious frames to cloud servers to save bandwidth.
    An automatic delivery robot may authenticate the target recipient before delivery by running face recognition algorithms locally for privacy concerns.
    \item
    \rev{On-device processing avoids the extra loss in visual quality due to video encoding during data transmission.
    Specifically, offloading video enhancement to cloud follows an ``encode-decode-enhance'' pipeline.
    Frames are encoded at the mobile device, transmitted via wireless networks, decoded at the cloud, and finally fed into the video enhancement module.
    Video encoding is necessary due to the limited wireless bandwidth.
    Yet it can decrease the enhanced visual quality because many mainstream video encoding schemes \emph{e.g.},\xspace H.264 \cite{bib:H2642003overview} are lossy.
    \figref{fig:encode_unencode} shows an example, where the upper row shows the frames enhanced following cloud processing \emph{i.e.},\xspace encode-decode-enhance, while the lower row shows the results of local processing \emph{i.e.},\xspace directly enhance.
    There is notable loss in detection precision (53.4\% accuracy loss in the upper left figures) and visual quality (30.6 VMAF loss in the upper right figures).
    \figref{fig:bitrate_vmaf} provides a more quantitative study.
    We enhance one example video clip from our self-collected dataset (MobileScene, see \secref{sec_exp_setup} for details) following the ``encode-decode-enhance'' pipeline using the standard H.264 video codec for encoding/decoding and {\sf UbiVE}\xspace for enhancement.
    We vary the input bitrate into the codec and measure the visual quality of the enhanced frames by the VMAF index \cite{bib:li2016toward}.
    As is shown, the bitrate (and thus the network bandwidth) limits the visual quality of video enhancement.
    The VMAF index is only $55$ at the bitrate of $1$ Mbps.
    Even at a bitrate of $64$ Mbps (higher than the uplink bandwidth of commercial 5G networks \cite{t-mobile-5G-report}), the VMAF index is only around $60$, which is lower than that of local processing (see \tabref{tb_energy} and \tabref{tb_energy_nano} in our evaluations).}
\end{itemize}

\begin{figure*}[t]
  \centering
    \subfloat[]{\label{fig:encode_unencode}
    \includegraphics[height=0.22\textwidth]{image/compressed_compare.pdf}}
    \hspace{15pt}
    \subfloat[]{\label{fig:bitrate_vmaf}
    \includegraphics[height=0.22\textwidth]{image/Bitrate_VMAF.pdf}}
    \vspace{-3mm}
\caption{\rev{Comparison of video enhancement locally on-device and remotely on-cloud. (a) Example frames enhanced by local processing and remote processing. The upper row follows the encode-decode-enhance pipeline of remote processing, while the lower row are directly enhanced as in local processing. (b) Enhanced video quality of the encode-decode-enhance pipeline under different frame bitrates into the video codec.}}
\label{fig:local_compare}
\vspace{-3mm}
\end{figure*}

\subsubsection{Requirements}
\label{subsubsec:requirements}
To support the low-light enhancement of ubiquitous video streams on mobile devices, a solution should meet the following criteria.
\begin{itemize}
    \item \textit{Near Real-time Video Enhancement}.
    As with other mobile computer vision tasks ~\cite{bib:icce2014:song, bib:nca2020:wijnands, bib:cvpr2018:liu},

    the low-light enhancement model should be lightweight to fit in the limited execution resources on mobile platforms and to achieve near real-time processing of the video streams.
    \item \textit{Energy-aware Adaptation}.
    Since many mobile devices are battery-powered, it is crucial that the video enhancement is energy-aware.
    Particularly, it is desirable that the video enhancement process can trade off between the output visual quality and the platform-imposed energy budget at runtime.
\end{itemize}
Despite prior research on the low-light image/video enhancement \cite{bib:ICCV19:Chen, bib:CVPR20:Guo, bib:TIP16:Guo, bib:ICCV19:Jiang, bib:BMVC18:Lv, bib:TPAMI21:Li, bib:TPAMI21:Li2, bib:CVPR21:Zhang} and adaptive mobile computer vision \cite{bib:sigcomm2020:kim, bib:mobicom2019:baig, bib:hotcloud2019:wang, bib:SIGCOMM2016:sun, bib:mobicom2019:lee, bib:tmc2020:yi}, no solution fulfills the above requirements simultaneously, which motives the design of {\sf UbiVE}\xspace.
\rev{Note that we focus on enhancement of RGB videos taken in low light.
We leave extreme low light enhancement to future work because such cases often require videos stored in raw formats \cite{bib:ICCV19:Chen, bib:ICCV19:Jiang, bib:TPAMI21:Li}, which are not pervasively supported in mobile and embedded platforms.}

\subsection{Solution Overview}
\label{subsec:solution}
Our solution, {\sf UbiVE}\xspace, is an energy-aware low-light video enhancement scheme that achieves near real-time video processing with competitive enhanced visual quality against the state-of-the-arts \cite{bib:CVPR20:Guo, bib:TPAMI21:Li2, bib:BMVC18:Lv}, while allowing runtime enhancement behavior adaptation to platform-imposed energy budgets.
This is achieved by decoupling the requirements in \secref{subsubsec:requirements} with two functional modules: \textit{fast low-light video enhancement} and \textit{energy-aware adaptation controller}.
\begin{itemize}
    \item \textit{Fast Low-Light Video Enhancement (\secref{sec:algorithm})}.
    This module aims at near real-time low-light video enhancement on mobile platforms without compromising enhancement quality.
    Prior low-light enhancement schemes are unfit for our purposes because they either induce unacceptable latency \cite{bib:BMVC18:Lv, bib:ICCV19:Jiang} or are not optimized for videos \cite{bib:CVPR20:Guo, bib:TPAMI21:Li2} \emph{e.g.},\xspace suffering from the flicking problem \cite{bib:TPAMI21:Li}.
    In {\sf UbiVE}\xspace, we design a new low-light video enhancement model by removing the latency bottleneck of a state-of-the-art low-light image enhancement model \cite{bib:CVPR20:Guo} and enforcing temporal consistency between frames.
    \item \textit{Energy-aware Adaption Controller (\secref{sec:controller})}.
    This module dynamically adjusts the frame resolution and computation reuse of the video enhancement model to maximize the enhanced video quality while minimizing the energy consumption.
    Such adaptation is non-trivial due to the multi-objective nature of the optimization problem and the challenge to estimate energy consumption at runtime.
    In {\sf UbiVE}\xspace, we develop an optimizer to heuristically solve the multi-objective optimization problem and propose a runtime energy profiler to estimate the energy cost of the video enhancement model at runtime.
    This module also contains a resource monitor and analyzer to automate the adaptation process.
\end{itemize}

\begin{figure}[t]
  \centering
  \includegraphics[width=.85\textwidth]{image/overview.pdf}
  \caption{Workflow of {\sf UbiVE}\xspace.}
  \label{fig:overview}
  \vspace{-8mm}
\end{figure}

\figref{fig:overview} shows the workflow of {\sf UbiVE}\xspace.
The ubiquitous video streams captured by a camera-embedded mobile device are fed into {\sf UbiVE's}\xspace fast low-light video enhancement module for visual quality enhancement.
Meanwhile, {\sf UbiVE's}\xspace energy-aware adaptation module adds an extra control flow to configure the video enhancement module's parameters according to the dynamic energy budgets of the mobile device.

We implement {\sf UbiVE}\xspace as a compact video pre-processing middleware.
It transparently augments the low-light video streams into high-quality ones for downstream on-device video applications such as facial recognition and object detection.
As a middleware, {\sf UbiVE}\xspace can resolve the application diversity and the system complexity and effectively deal with multiple co-existing demands and constraints from different aspects (\emph{e.g.},\xspace visual quality, latency, energy).
Application developers do not have to initiate the video enhancement or be involved in the details of system monitoring and analysis.

\section{Related Work}
\label{sec:related}

\subsection{Low-light Image and Video Enhancement}
\label{subsec:related:low-light}

Low-light enhancement improves the perception or interpretability of images and videos captured in dim light \cite{bib:TPAMI21:Li, bib:imwut20:janveja}.
Most prior efforts focus on image enhancement \cite{bib:CVPR20:Guo, bib:TIP16:Guo, bib:TCE07:Ibrahim, bib:TIP97:Jobson, bib:TIP13:Lee, bib:TPAMI21:Li2} while video enhancement \cite{bib:ICCV19:Chen, bib:BMVC18:Lv, bib:ICCV19:Jiang, bib:CVPR21:Zhang} is regarded as an extension of image enhancement by avoiding the flicking problem between frames.
We refer readers to \cite{bib:TPAMI21:Li} for a comprehensive review.
Of our particular interest is Zero-DCE \cite{bib:CVPR20:Guo} and its follow-up, Zero-DCE++ \cite{bib:TPAMI21:Li2}, a state-of-the-art zero-shot learning based low-light image enhancement scheme.
The reasons are two-fold.
\textit{(i)}
There lacks paired training data \emph{i.e.},\xspace low-light images or videos and the corresponding well-illuminated versions, in real-world mobile video applications.
Therefore, zero-shot learning based solutions \cite{bib:CVPR20:Guo, bib:TPAMI21:Li2} are preferable over supervised learning ones \cite{bib:PR17:Lore, bib:ICCV19:Chen} since the former requires neither paired nor unpaired training data.
\textit{(ii)}
Zero-DCE \cite{bib:CVPR20:Guo} and Zero-DCE++ \cite{bib:TPAMI21:Li2} apply a model architecture that consists of a deep neural network and an image-to-curve mapping, making it one of the most lightweight solution while achieving competitive image enhancement performance.

The primary challenge to extend low-light enhancement from images to videos is the flicking problem \cite{bib:TPAMI21:Li}.
Some studies implicitly solve the problem by substituting the 2D network architectures in image enhancement models to the corresponding 3D versions \cite{bib:BMVC18:Lv, bib:ICCV19:Jiang}.
However, they require specialized equipment to collect video datasets for training the new 3D architectures.
A more practical strategy is to incorporate temporal consistency into the loss function of image enhancement models \cite{bib:ICCV19:Chen, bib:CVPR21:Zhang}.
For example, Chen \emph{et al.}\xspace~\cite{bib:ICCV19:Chen} proposed a self-consistency loss to tolerate minor differences between inputs while keeping the output stable.
Zhang \emph{et al.}\xspace \cite{bib:CVPR21:Zhang} enforces temporal consistency by ensuring stable optical flows estimated from the videos.

To this end, the low-light video enhancement module of {\sf UbiVE}\xspace extends existing low-light enhancement solutions from images to videos.
\rev{And {\sf UbiVE}\xspace advances the prior arts by introducing novel designs}.
\textit{(i)}
\rev{It notably reduces the processing time of low-light enhancement schemes, by replacing the iterative image-curve mapping using a non-iterative mapping function.
For example,
{\sf UbiVE}\xspace dramatically outperforms Zero-DCE \cite{bib:CVPR20:Guo} in terms of processing latency, one of the most lightweight low-light image enhancement method.}
\textit{(ii)}
It designs the novel temporal consistency loss and its training scheme compatible to zero-shot learning based image enhancement schemes without the need for large-scale paired video datasets.

\subsection{Energy-awareness in DNNs}
\label{subsec:related:energy}

Existing DNN energy profiling methods couple with either the DNN topology (\emph{e.g.},\xspace model structure, computation path)~\cite{bib:Mobisys18:Liu} or the hardware platform (\emph{e.g.},\xspace on-chip memory capability, memory bandwidth)~\cite{bib:CVPR17:Yang, bib:TCS20:Lee}.
Tien-Ju \emph{et al.}\xspace~\cite{bib:ssc2017:yang} propose a layer-wise estimation method, which takes approximately 10 seconds to output the normalized energy consumption of a DNN based on its architecture, sparsity, and bitwidth.
Yannan \emph{et al.}\xspace ~\cite{bib:ICCAD2019:wu}present the hardware architecture-dependent energy profiling method.
Some other works generate the design-specific tables with predefined attributes to estimate DNN energy cost~\cite{bib:islped2018:ke, bib:ISPASS2019:parashar}.
However, the above methods cannot profile the DNN energy usage for providing exact runtime feedback on \textit{dynamic resource availability} with \textit{agnostic DNN topology}, because the sampling for the specific DNN prototype or platform is costly.

Unlike existing methods, {\sf UbiVE}\xspace builds the one-fits-all profiler for DNN energy usage based on the following two observations:
\textit{(i)} despite the dynamic topology of video processing models, the underlying tensor operation (\emph{i.e.},\xspace the energy-dominating operation loops) and the data flow are stable.
\textit{(ii)} Despite the dynamic hardware resource supply (\emph{e.g.},\xspace on-chip memory capability, computation bandwidth), the hit ratio of on-chip memory access shows up the stable rules and within a small sampling set.

\subsection{Adaptive Vision Tasks on Mobile Devices}
\label{subsec:related:vision_tasks}

Adaptive vision tasks involve image/video delivery between distributed mobiles or between the mobile and cloud.
Examples include live video streaming~\cite{bib:sigcomm2020:kim}, video-on-demand~\cite{bib:mobicom2019:baig}, and virtual reality~\cite{bib:hotcloud2019:wang}.
The key challenge for such tasks is the reliance on networking conditions.
To tackle this challenge, researchers propose two categories of solutions, \emph{i.e.},\xspace adaptive bitrate~\cite{bib:ICDCS2019:chen,bib:SIGCOMM2016:sun, bib:hotcloud2019:wang}, and super-resolution~\cite{ bib:mobicom2019:lee,bib:tmc2020:yi, bib:IMWUT2020:liu}.
They co-design the task demands (\emph{e.g.},\xspace visual quality, latency, or inference accuracy) with the streaming process.
In bitrate adaptation, each video is split into segments.
These segments are encoded using multiple bitrates and streamed with the suitable bitrate based on the network conditions \cite{bib:hotcloud2019:wang}.
In super-resolution scheme, compact content with low resolution or quality is transmitted, which is then enhanced at the receiver~\cite{bib:mobicom2019:lee,bib:tmc2020:yi}.
For example, Yeo \emph{et al.}\xspace \cite{bib:hotnet2017:yeo} leverages the super-resolution DNNs to enable the content-aware video delivery.
Lee \emph{et al.}\xspace ~\cite{bib:mobicom2019:lee} present MobiSR to boost the performance of on-device super-resolution.

Nevertheless, the concerns of these efforts are coupled with other edge or cloud devices. {\sf UbiVE}\xspace serves as a locally deployed middleware to directly provide video enhancement services for local applications.

\section{Introduction}
ACM's consolidated article template, introduced in 2017, provides a
consistent \LaTeX\ style for use across ACM publications, and
incorporates accessibility and metadata-extraction functionality
necessary for future Digital Library endeavors. Numerous ACM and
SIG-specific \LaTeX\ templates have been examined, and their unique
features incorporated into this single new template.

If you are new to publishing with ACM, this document is a valuable
guide to the process of preparing your work for publication. If you
have published with ACM before, this document provides insight and
instruction into more recent changes to the article template.

The ``\verb|acmart|'' document class can be used to prepare articles
for any ACM publication --- conference or journal, and for any stage
of publication, from review to final ``camera-ready'' copy, to the
author's own version, with {\itshape very} few changes to the source.

\section{Template Overview}
As noted in the introduction, the ``\verb|acmart|'' document class can
be used to prepare many different kinds of documentation --- a
double-blind initial submission of a full-length technical paper, a
two-page SIGGRAPH Emerging Technologies abstract, a ``camera-ready''
journal article, a SIGCHI Extended Abstract, and more --- all by
selecting the appropriate {\itshape template style} and {\itshape
  template parameters}.

This document will explain the major features of the document
class. For further information, the {\itshape \LaTeX\ User's Guide} is
available from
\url{https://www.acm.org/publications/proceedings-template}.

\subsection{Template Styles}

The primary parameter given to the ``\verb|acmart|'' document class is
the {\itshape template style} which corresponds to the kind of publication
or SIG publishing the work. This parameter is enclosed in square
brackets and is a part of the {\verb|documentclass|} command:
\begin{verbatim}
  \documentclass[STYLE]{acmart}
\end{verbatim}

Journals use one of three template styles. All but three ACM journals
use the {\verb|acmsmall|} template style:
\begin{itemize}
\item {\verb|acmsmall|}: The default journal template style.
\item {\verb|acmlarge|}: Used by JOCCH and TAP.
\item {\verb|acmtog|}: Used by TOG.
\end{itemize}

The majority of conference proceedings documentation will use the {\verb|acmconf|} template style.
\begin{itemize}
\item {\verb|acmconf|}: The default proceedings template style.
\item{\verb|sigchi|}: Used for SIGCHI conference articles.
\item{\verb|sigchi-a|}: Used for SIGCHI ``Extended Abstract'' articles.
\item{\verb|sigplan|}: Used for SIGPLAN conference articles.
\end{itemize}

\subsection{Template Parameters}

In addition to specifying the {\itshape template style} to be used in
formatting your work, there are a number of {\itshape template parameters}
which modify some part of the applied template style. A complete list
of these parameters can be found in the {\itshape \LaTeX\ User's Guide.}

Frequently-used parameters, or combinations of parameters, include:
\begin{itemize}
\item {\verb|anonymous,review|}: Suitable for a ``double-blind''
  conference submission. Anonymizes the work and includes line
  numbers. Use with the \verb|\acmSubmissionID| command to print the
  submission's unique ID on each page of the work.
\item{\verb|authorversion|}: Produces a version of the work suitable
  for posting by the author.
\item{\verb|screen|}: Produces colored hyperlinks.
\end{itemize}

This document uses the following string as the first command in the
source file:
\verb|\documentclass[acmlarge]{acmart}|.

\section{Modifications}

Modifying the template --- including but not limited to: adjusting
margins, typeface sizes, line spacing, paragraph and list definitions,
and the use of the \verb|\vspace| command to manually adjust the
vertical spacing between elements of your work --- is not allowed.

{\bfseries Your document will be returned to you for revision if
  modifications are discovered.}

\section{Typefaces}

The ``\verb|acmart|'' document class requires the use of the
``Libertine'' typeface family. Your \TeX\ installation should include
this set of packages. Please do not substitute other typefaces. The
``\verb|lmodern|'' and ``\verb|ltimes|'' packages should not be used,
as they will override the built-in typeface families.

\section{Title Information}

The title of your work should use capital letters appropriately -
\url{https://capitalizemytitle.com/} has useful rules for
capitalization. Use the {\verb|title|} command to define the title of
your work. If your work has a subtitle, define it with the
{\verb|subtitle|} command.  Do not insert line breaks in your title.

If your title is lengthy, you must define a short version to be used
in the page headers, to prevent overlapping text. The \verb|title|
command has a ``short title'' parameter:
\begin{verbatim}
  \title[short title]{full title}
\end{verbatim}

\section{Authors and Affiliations}

Each author must be defined separately for accurate metadata
identification. Multiple authors may share one affiliation. Authors'
names should not be abbreviated; use full first names wherever
possible. Include authors' e-mail addresses whenever possible.

Grouping authors' names or e-mail addresses, or providing an ``e-mail
alias,'' as shown below, is not acceptable:
\begin{verbatim}
  \author{Brooke Aster, David Mehldau}
  \email{dave,judy,steve@university.edu}
  \email{firstname.lastname@phillips.org}
\end{verbatim}

The \verb|authornote| and \verb|authornotemark| commands allow a note
to apply to multiple authors --- for example, if the first two authors
of an article contributed equally to the work.

If your author list is lengthy, you must define a shortened version of
the list of authors to be used in the page headers, to prevent
overlapping text. The following command should be placed just after
the last \verb|\author{}| definition:
\begin{verbatim}
  \renewcommand{\shortauthors}{McCartney, et al.}
\end{verbatim}
Omitting this command will force the use of a concatenated list of all
of the authors' names, which may result in overlapping text in the
page headers.

The article template's documentation, available at
\url{https://www.acm.org/publications/proceedings-template}, has a
complete explanation of these commands and tips for their effective
use.

\section{Rights Information}

Authors of any work published by ACM will need to complete a rights
form. Depending on the kind of work, and the rights management choice
made by the author, this may be copyright transfer, permission,
license, or an OA (open access) agreement.

Regardless of the rights management choice, the author will receive a
copy of the completed rights form once it has been submitted. This
form contains \LaTeX\ commands that must be copied into the source
document. When the document source is compiled, these commands and
their parameters add formatted text to several areas of the final
document:
\begin{itemize}
\item the ``ACM Reference Format'' text on the first page.
\item the ``rights management'' text on the first page.
\item the conference information in the page header(s).
\end{itemize}

Rights information is unique to the work; if you are preparing several
works for an event, make sure to use the correct set of commands with
each of the works.

\section{CCS Concepts and User-Defined Keywords}

Two elements of the ``acmart'' document class provide powerful
taxonomic tools for you to help readers find your work in an online
search.

The ACM Computing Classification System ---
\url{https://www.acm.org/publications/class-2012} --- is a set of
classifiers and concepts that describe the computing
discipline. Authors can select entries from this classification
system, via \url{https://dl.acm.org/ccs/ccs.cfm}, and generate the
commands to be included in the \LaTeX\ source.

User-defined keywords are a comma-separated list of words and phrases
of the authors' choosing, providing a more flexible way of describing
the research being presented.

CCS concepts and user-defined keywords are required for all short- and
full-length articles, and optional for two-page abstracts.

\section{Sectioning Commands}

Your work should use standard \LaTeX\ sectioning commands:
\verb|section|, \verb|subsection|, \verb|subsubsection|, and
\verb|paragraph|. They should be numbered; do not remove the numbering
from the commands.

Simulating a sectioning command by setting the first word or words of
a paragraph in boldface or italicized text is {\bfseries not allowed.}

\section{Tables}

The ``\verb|acmart|'' document class includes the ``\verb|booktabs|''
package --- \url{https://ctan.org/pkg/booktabs} --- for preparing
high-quality tables.

Table captions are placed {\itshape above} the table.

Because tables cannot be split across pages, the best placement for
them is typically the top of the page nearest their initial cite.  To
ensure this proper ``floating'' placement of tables, use the
environment \textbf{table} to enclose the table's contents and the
table caption.  The contents of the table itself must go in the
\textbf{tabular} environment, to be aligned properly in rows and
columns, with the desired horizontal and vertical rules.  Again,
detailed instructions on \textbf{tabular} material are found in the
\textit{\LaTeX\ User's Guide}.

Immediately following this sentence is the point at which
Table~\ref{tab:freq} is included in the input file; compare the
placement of the table here with the table in the printed output of
this document.

\begin{table}
  \caption{Frequency of Special Characters}
  \label{tab:freq}
  \begin{tabular}{ccl}
    \toprule
    Non-English or Math&Frequency&Comments\\
    \midrule
    \O & 1 in 1,000& For Swedish names\\
    $\pi$ & 1 in 5& Common in math\\
    \$ & 4 in 5 & Used in business\\
    $\Psi^2_1$ & 1 in 40,000& Unexplained usage\\
  \bottomrule
\end{tabular}
\end{table}

To set a wider table, which takes up the whole width of the page's
live area, use the environment \textbf{table*} to enclose the table's
contents and the table caption.  As with a single-column table, this
wide table will ``float'' to a location deemed more
desirable. Immediately following this sentence is the point at which
Table~\ref{tab:commands} is included in the input file; again, it is
instructive to compare the placement of the table here with the table
in the printed output of this document.

\begin{table*}
  \caption{Some Typical Commands}
  \label{tab:commands}
  \begin{tabular}{ccl}
    \toprule
    Command &A Number & Comments\\
    \midrule
    \texttt{{\char'134}author} & 100& Author \\
    \texttt{{\char'134}table}& 300 & For tables\\
    \texttt{{\char'134}table*}& 400& For wider tables\\
    \bottomrule
  \end{tabular}
\end{table*}

\section{Math Equations}
You may want to display math equations in three distinct styles:
inline, numbered or non-numbered display.  Each of the three are
discussed in the next sections.

\subsection{Inline (In-text) Equations}
A formula that appears in the running text is called an inline or
in-text formula.  It is produced by the \textbf{math} environment,
which can be invoked with the usual
\texttt{{\char'134}begin\,\ldots{\char'134}end} construction or with
the short form \texttt{\$\,\ldots\$}. You can use any of the symbols
and structures, from $\alpha$ to $\omega$, available in
\LaTeX~\cite{Lamport:LaTeX}; this section will simply show a few
examples of in-text equations in context. Notice how this equation:
\begin{math}
  \lim_{n\rightarrow \infty}x=0
\end{math},
set here in in-line math style, looks slightly different when
set in display style.  (See next section).

\subsection{Display Equations}
A numbered display equation---one set off by vertical space from the
text and centered horizontally---is produced by the \textbf{equation}
environment. An unnumbered display equation is produced by the
\textbf{displaymath} environment.

Again, in either environment, you can use any of the symbols and
structures available in \LaTeX\@; this section will just give a couple
of examples of display equations in context.  First, consider the
equation, shown as an inline equation above:
\begin{equation}
  \lim_{n\rightarrow \infty}x=0
\end{equation}
Notice how it is formatted somewhat differently in
the \textbf{displaymath}
environment.  Now, we'll enter an unnumbered equation:
\begin{displaymath}
  \sum_{i=0}^{\infty} x + 1
\end{displaymath}
and follow it with another numbered equation:
\begin{equation}
  \sum_{i=0}^{\infty}x_i=\int_{0}^{\pi+2} f
\end{equation}
just to demonstrate \LaTeX's able handling of numbering.

\section{Figures}

The ``\verb|figure|'' environment should be used for figures. One or
more images can be placed within a figure. If your figure contains
third-party material, you must clearly identify it as such, as shown
in the example below.
\begin{figure}[h]
  \centering
  \includegraphics[width=\linewidth]{sample-franklin}
  \caption{1907 Franklin Model D roadster. Photograph by Harris \&
    Ewing, Inc. [Public domain], via Wikimedia
    Commons. (\url{https://goo.gl/VLCRBB}).}
  \Description{The 1907 Franklin Model D roadster.}
\end{figure}

Your figures should contain a caption which describes the figure to
the reader. Figure captions go below the figure. Your figures should
{\bfseries also} include a description suitable for screen readers, to
assist the visually-challenged to better understand your work.

Figure captions are placed {\itshape below} the figure.

\subsection{The ``Teaser Figure''}

A ``teaser figure'' is an image, or set of images in one figure, that
are placed after all author and affiliation information, and before
the body of the article, spanning the page. If you wish to have such a
figure in your article, place the command immediately before the
\verb|\maketitle| command:
\begin{verbatim}
  \begin{teaserfigure}
    \includegraphics[width=\textwidth]{sampleteaser}
    \caption{figure caption}
    \Description{figure description}
  \end{teaserfigure}
\end{verbatim}

\section{Citations and Bibliographies}

The use of \BibTeX\ for the preparation and formatting of one's
references is strongly recommended. Authors' names should be complete
--- use full first names (``Donald E. Knuth'') not initials
(``D. E. Knuth'') --- and the salient identifying features of a
reference should be included: title, year, volume, number, pages,
article DOI, etc.

The bibliography is included in your source document with these two
commands, placed just before the \verb|\end{document}| command:
\begin{verbatim}
  \bibliographystyle{ACM-Reference-Format}

Computer Vision and Pattern Recognition (cs.CV)