
Early Exploration (2015-2017)
In the early exploration stage, the efforts focused on the understanding
of the roles played by the filter weights and the nonlinear activation
unit in a computational neuron of CNNs. The main objective was to shed
light on the superior performance from the signal processing viewpoint
since it was difficult to get engineering insights from the end-to-end
non-convex optimization problem formulation. The necessity of nonlinear
activation was explained in [1]} as a mechanism
to resolve the sign confusion problem in neural network training.
The filter weights were explained as the memory of an CNN learning
system [2]} and used to capture local, mid-level and
global intrinsic representations of objects [3]}.
Although the explanation of the role of filter weights was preliminary,
it did inspire the later work on unsupervised representation learning.

FF-CNN, Saab Transforms and channel-wise Saab Transforms (2018-2019)
Based on the foundation laid by [2]}, [3]},
novel ways to determine filter weights of the convolutional layers in
CNNs in a feedforward manner were proposed in
[6]}. The design is called the feedforward CNN
(FF-CNN). The filter weights in convolutional layers of FF-CNN are
determined by the Saab (or Saak) transforms. The transform operations
are unsupervised since no class labels are used. The filter weights in
FC layers of FF-CNN are determined by the label-guided least-squared
regression (LAG). The main objective was to lower the training
complexity with an interpretable design.

PixelHop, PixelHop++, PointHop, PointHop++, and GL Applications
(2020-2021)
PixelHop [7]} and PixelHop++
[8]} were two first-generation GL systems. They
designs depart from the neural networks completely. Instead of
demanding end-to-end connectivity of neurons, GL systems adopt
feedforward and ensemble learning principle. Both PixelHop and
PixelHop++ have three modules. They use multi-stage Saab transforms
and pooling operation to yield joint spatial-spectral representations with no
supervision in the first module. The LAG operation is exploited to
reduce the representation dimension before being fed into classifiers in the
second stage. A traditional ML classifier is used in the third stage.
The main difference between PixelHop and PixelHop++ lies in the
multi-stage “standard" and “channel-wise" Saab transforms. While
PixelHop and PixelHop++ target at the image classification application,
PointHop [9]} and PointHop++
[10]} were developed for the point cloud
classification. Due to the irregular spatial structure of a 3D point
cloud scan, a novel octant representation was introduced before the Saab
transform in PointHop.
We have witnessed many successful applications of GL since 2020. Their
performance is comparable with that of state-of-the-art DL solutions
with significantly less resource. Examples include image classification
[11]}, image enhancement [12]},
image quality assessment [13]}, [14]},
deepfake image/video detection [15]}, [16]}, [17]}, [18]}, point cloud
classification, segmentation, registration [19]}, [20]}, [21]}, [22]}, [23]}, [24]}, face biometrics [25]}, [26]}, texture analysis and synthesis
[27]}, [28]}, [29]}, graph node
classification [30]}, [31]}, data
compression [32]}, [33]}, joint image
compression and classification [34]}, 3D
medical image analysis [35]}, [36]}, etc. We
will present several successful examples to provide insights for future
development of GL in Sec. .

Advanced GL Tools and Broader Applications (2022- )
Two advanced GL tools were developed in 2022: 1) DFT/RFT for supervised
discriminant/relevant feature selection [37]} (see
Sec. REF ) and 2) SLM/SLR as a new classifier/regressor
that exploits both feature combinations and decision combinations (see
Sec. REF ).
Furthermore, a more advanced GL system called IPHop was proposed in
[38]}. It replaces the traditional max-pooling
operation with the absolute-max-pooling operation and yields more
unsupervised representations by introducing the spatial Saab transform at each
channel in the first module of PixelHop. Furthermore, it replaces the
LAG operation with DFT in the second module of PixelHop.

