Piecewise Linear Neural Networks and Deep Learning
As a powerful modelling method, PieceWise Linear Neural Networks (PWLNNs)
have proven successful in various fields, most recently in deep learning. To
apply PWLNN methods, both the representation and the learning have long been
studied. In 1977, the canonical representation pioneered the works of shallow
PWLNNs learned by incremental designs, but the applications to large-scale data
were prohibited. In 2010, the Rectified Linear Unit (ReLU) advocated the
prevalence of PWLNNs in deep learning. Ever since, PWLNNs have been
successfully applied to extensive tasks and achieved advantageous performances.
In this Primer, we systematically introduce the methodology of PWLNNs by
grouping the works into shallow and deep networks. Firstly, different PWLNN
representation models are constructed with elaborated examples. With PWLNNs,
the evolution of learning algorithms for data is presented and fundamental
theoretical analysis follows up for in-depth understandings. Then,
representative applications are introduced together with discussions and
outlooks.