Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization

Abstract: While deep learning is successful in a number of applications, it is not yet well understood theoretically. A satisfactory theoretical characterization of deep learning however, is beginning to emerge. It covers the following questions: 1) representation power of deep networks 2) optimization of the empirical risk 3) generalization properties of gradient descent techniques --- why the expected error does not suffer, despite the absence of explicit regularization, when the networks are overparametrized? In this review we discuss recent advances in the three areas. In approximation theory both shallow and deep networks have been shown to approximate any continuous functions on a bounded domain at the expense of an exponential number of parameters (exponential in the dimensionality of the function). However, for a subset of compositional functions, deep networks of the convolutional type can have a linear dependence on dimensionality, unlike shallow networks. In optimization we discuss the loss landscape for the exponential loss function and show that stochastic gradient descent will find with high probability the global minima. To address the question of generalization for classification tasks, we use classical uniform convergence results to justify minimizing a surrogate exponential-type loss function under a unit norm constraint on the weight matrix at each layer -- since the interesting variables for classification are the weight directions rather than the weights. Our approach, which is supported by several independent new results, offers a solution to the puzzle about generalization performance of deep overparametrized ReLU networks, uncovering the origin of the underlying hidden complexity control.

\section{Introduction}
\dropcap{I}n the last few years, deep learning has been tremendously
successful in many important applications of machine
learning. However, our theoretical understanding of deep
learning, and thus the ability of developing principled
improvements, has lagged behind.  A satisfactory theoretical
characterization of deep learning is emerging. It covers the
following areas: 1) {\it approximation} properties of deep
networks 2) {\it optimization} of the empirical risk 3) {\it
	generalization} properties of gradient descent techniques
-- why the expected error does not suffer, despite the
absence of explicit regularization, when the networks are
overparametrized?

\subsection{When Can Deep  Networks Avoid the Curse of Dimensionality?}

We start with the first set of questions,
summarizing results in \cite{HierarchicalKernels2015,Hierarchical2015,poggio2015December},
and \cite{Mhaskaretal2016,MhaskarPoggio2016}. The main result is that deep  networks have
the theoretical guarantee, which shallow networks do not have, that
they can avoid the {\it curse of dimensionality}  for an important class of problems,
corresponding to {\it compositional functions}, that is functions of
functions. An especially interesting subset of such compositional
functions are  {\it hierarchically local compositional
  functions} where all the constituent functions are local in the
sense of bounded small dimensionality. The deep networks that can
approximate them without the curse of dimensionality are of the deep
convolutional type -- though, importantly, weight sharing is not necessary.

Implications of the theorems likely to be relevant in practice are:

 a) {\it Deep convolutional architectures} have the theoretical
  guarantee that they can be {\it  much better} than one layer architectures such
  as kernel machines for certain classes of problems;
 b) the problems for which certain deep networks are guaranteed to avoid
the {\it curse of dimensionality} (see for a nice review
  \cite{Donoho00high-dimensionaldata})  correspond to input-output
  mappings that are {\it compositional with local constituent
    functions};
c) the key aspect of convolutional networks that can give them an
  exponential advantage is {\it not weight sharing} but {\it locality} at each
  level of the hierarchy.

\subsection{Related Work}

Several papers in the '80s focused on the approximation power and
learning properties of one-hidden layer networks (called shallow
networks here). Very little appeared on multilayer networks, (but see
\cite{mhaskar1993approx, mhaskar1993neural, chui1994neural, chui1996, Pinkus1999}).
By now, several papers \cite{poggio03mathematics,MontufarBengio2014,
  DBLP:journals/corr/abs-1304-7045} have appeared. \cite{Anselmi2014,anselmi2015theoretical,poggioetal2015,
  LiaoPoggio2016, Mhaskaretal2016} derive new upper bounds for
the approximation by deep networks of certain important classes of
functions which avoid the curse of dimensionality. The upper bound for
the approximation by shallow networks of general functions was well
known to be exponential. It seems natural to assume that, since there
is no general way for shallow networks to exploit a compositional
prior, lower bounds for the approximation by shallow networks of
compositional functions should also be exponential. In fact, examples
of specific functions that cannot be represented efficiently by
shallow networks have been given,  for instance in
\cite{Telgarsky2015, SafranShamir2016, Theory_I}. An interesting
review of approximation of univariate functions by deep networks has
recently appeared \cite{2019arXiv190502199D}.

\begin{figure
\centering
\includegraphics[trim=10 33 30 73, width=0.9\linewidth,clip]{Figures/Example_2_functions.pdf}
\caption{The top graphs are associated to {\it functions}; each of the
  bottom diagrams depicts the ideal {\it network} approximating the
  function above. In a) a shallow universal network in 8 variables and
  $N$ units approximates a generic function of $8$ variables
  $f(x_1, \cdots, x_8)$. Inset b) shows a  hierarchical
  network at the bottom in $n=8$ variables, which approximates well
  functions of the form
  $f(x_1, \cdots, x_8) = h_3(h_{21}(h_{11} (x_1, x_2), h_{12}(x_3,
  x_4)), \allowbreak h_{22}(h_{13}(x_5, x_6), h_{14}(x_7, x_8))) $ as
  represented by the binary graph above.  In the approximating network
  each of the $n-1$ nodes in the graph of the function corresponds to
  a set of $Q =\frac{N}{n-1}$ ReLU units computing the ridge function
  $\sum_{i=1}^Q a_i(\scal{\mathbf{v}_i}{\mathbf{x}}+t_i)_+$, with
  $\mathbf{v}_i, \mathbf{x} \in \R^2$, $a_i, t_i\in\R$. Each term in
  the ridge function corresponds to a unit in the node (this is
  somewhat different from todays deep networks, but equivalent to
  them \cite{Theory_I}). Similar to the shallow network, a hierarchical network is
  universal, that is, it can approximate any continuous function; the
  text proves that it can approximate a compositional functions
  exponentially better than a shallow network. Redrawn from
  \cite{MhaskarPoggio2016}.
 }
\label{example_functions}
\end{figure}

\subsection{Degree of approximation}\label{approxsect}

The general paradigm is as follows. We are interested in determining
how complex a network ought to be to {\it theoretically guarantee}
approximation of an unknown target function $f$ up to a given accuracy
$\epsilon>0$.  To measure the accuracy, we need a norm $\|\cdot\|$ on
some normed linear space $\mathbb{X}$. As we will see the norm used in
the results of this paper is the $sup$ norm in keeping with the
standard choice in approximation theory. As it turns out, the results
of this section
require the sup norm in order to be independent from the unknown
distribution of the input data.

Let $V_N$ be the be set of all networks of a given kind with $N$ units
(which we take to be or measure of the complexity of the approximant
network). The \textit{degree of approximation} is defined by
$\mathsf{dist}(f, V_N)=\inf_{P\in V_N}\|f-P\|.$ For example, if
$\mathsf{dist}(f, V_N)=\O(N^{-\gamma})$ for some $\gamma>0$, then a
network with complexity $N=\O(\epsilon^{-\frac{1}{\gamma}})$ will be
sufficient to guarantee an approximation with accuracy at least
$\epsilon$. The only a priori information on the class of target
functions $f$, is codified by the statement that $f\in W$ for some
subspace $W\subseteq \mathbb{X}$. This subspace is a smoothness and compositional class,
characterized by the parameters $m$ and $d$ ($d=2$ in the example of
Figure \ref{example_functions} ; it is the size of the kernel
in a convolutional network).

\subsection{Shallow and deep networks}
\label{subprevious}

This section characterizes conditions under which deep networks are
``better'' than shallow network in approximating functions.  Thus we
compare shallow (one-hidden layer) networks with deep networks as
shown in Figure \ref{example_functions}.  Both types of networks use
the same small set of operations -- dot products, linear combinations,
a fixed nonlinear function of one variable, possibly convolution and
pooling. Each node in the networks   corresponds to
a node in the graph of the function to be approximated, as shown in
the Figure. A unit is a neuron which computes
$(\scal{x}{w}+b)_+$,  where $w$ is the vector of weights on the vector input
$x$. Both $w$ and the real number $b$ are parameters tuned by
learning. We assume here that each node in the networks computes the
linear combination of $r$ such units $\sum_{i=1}^r c_i
(\scal{x}{w_i}+b_i)_+$. Notice that in our main example of a network corresponding to a
function with a binary tree graph, the resulting architecture is an
idealized version of deep convolutional neural networks described in
the literature. In particular, it has only one output at the top
unlike most of the deep architectures with many channels and many
top-level outputs. Correspondingly, each node computes a single value
instead of multiple channels, using the combination of several
units. However our results hold also for these more
complex networks (see \cite{Theory_I}).
\noindent

The sequence of results is as follows.

\begin{itemize}

\item {\it Both shallow (a) and deep (b) networks are universal}, that
  is they can approximate arbitrarily well any continuous function of
  $n$ variables on a compact domain. The result for shallow networks
  is classical.

\item We consider a special class of functions of $n$ variables on a
  compact domain that are {\it hierarchical compositions of local
    functions}, such as
  $f(x_1, \cdots, x_8) = h_3(h_{21}(h_{11} (x_1, x_2), h_{12}(x_3,
  x_4)), \allowbreak h_{22}(h_{13}(x_5, x_6), h_{14}(x_7, x_8))) $

\noindent The structure of the function in Figure  \ref{example_functions} b)
is represented by a graph of the binary tree type, reflecting dimensionality $d=2$ for the constituent functions $h$. In
general, $d$ is arbitrary but fixed and independent of the
dimensionality $n$ of the compositional function $f$.
\cite{Theory_I} formalizes the more general compositional case using
directed acyclic graphs.

\item The approximation of functions with a {\it compositional
    structure} -- can be achieved with the same degree of accuracy by
  deep and shallow networks but  the number of parameters are much
  smaller for the deep networks than for the shallow network with
  equivalent approximation accuracy.
\end{itemize}

We approximate functions with networks in which the activation
nonlinearity is a smoothed version of the so called ReLU, originally
called {\it ramp} by Breiman and given by
$\sigma (x) = x_+ = max(0, x)$ .  The architecture of the deep
networks reflects the function graph with each node $h_i$ being a
ridge function, comprising one or more neurons.

Let $I^n=[-1,1]^n$, $\mathbb{X}=C(I^n)$ be the space of all continuous
functions on $I^n$, with $\|f\|=\max_{x\in I^n}|f(x)|$.
Let
$\mathcal{S}_{N,n}$ denote the class of all shallow networks with $N$
units of the form
$$
x\mapsto\sum_{k=1}^N a_k\sigma(\scal{{w}_k}{x}+b_k),
$$
where ${w}_k\in\R^n$, $b_k, a_k\in\R$. The number of trainable
parameters here is $(n+2)N\sim n$. Let $m\ge 1$ be an integer, and
$W_m^n$ be the set of all functions of $n$ variables with continuous
partial derivatives of orders up to $m < \infty$ such that $\|f\|+\sum_{1\le
  |\k|_1\le m} \|D^\k f\| \le 1$, where $D^\k$ denotes the partial
derivative indicated by the multi-integer $\k\ge 1$, and $|\k|_1$ is
the sum of the components of $\k$.

For the hierarchical binary tree network, the analogous spaces are
defined by considering the compact set $W_m^{n,2}$ to be the class of
all compositional functions $f$ of $n$ variables with a binary tree
architecture and constituent functions $h$ in $W_m ^2$.  We define the
corresponding class of deep networks $\mathcal{D}_{N,2}$ to be the set
of all deep networks with a binary tree architecture, where each of
the constituent nodes is in $\mathcal{S}_{M,2}$, where $N=|V|M$, $V$
being the set of non--leaf vertices of the tree. We note that in the
case when $n$ is an integer power of $2$, the total number of
parameters involved in a deep network in $\mathcal{D}_{N,2}$ is $4N$.

The first theorem is about shallow networks.

\begin{theorem}
\label{optneurtheo}
Let $\sigma :\R\to \R$ be infinitely differentiable, and not a polynomial.  For $f\in W_m^n$ the complexity of shallow networks that
provide accuracy at least $\epsilon$ is
\begin{equation}
N= \O(\epsilon^{-n/m})\,\, and\,\,
is\,\, the\,\, best\,\, possible.
\end{equation}
\end{theorem}

The estimate of Theorem \ref{optneurtheo} is the best possible if the only a priori
information we are allowed to assume is that the target function
belongs to $f\in W_m^n$. The exponential dependence on the
dimension $n$ of the number $e^{-n/m}$ of parameters needed to
obtain an accuracy $\O(\epsilon)$ is known as the {\it curse of
  dimensionality}. Note that the constants involved in $\O$ in the theorems will depend upon
the norms of the derivatives of $f$ as well as $\sigma$.

Our second and main theorem is about deep networks with smooth
activations (preliminary versions appeared in
\cite{poggio2015December,Hierarchical2015,Mhaskaretal2016}). We
formulate it in the binary tree case for simplicity but it extends
immediately to functions that are compositions of constituent
functions of a fixed number of variables $d$ (in convolutional
networks $d$ corresponds to the size of the kernel).

\begin{theorem}
\label{deeptheo}
For $f\in W_m^{n,2}$ consider a deep network with the same
compositional architecture and with an activation function
$\sigma :\R\to \R$ which is infinitely differentiable, and
not a polynomial. The complexity of the
network to provide approximation with
accuracy at least $\epsilon$ is
\begin{equation}
N =\mathcal{O}((n-1)\epsilon^{-2/m}).
\label{deepnetapprox}
\end{equation}
\end{theorem}

The proof is in \cite{Theory_I}.  The assumptions on $\sigma$ in the
theorems are not satisfied by the ReLU function $x\mapsto x_+$, but
they are satisfied by smoothing the function in an arbitrarily small
interval around the origin. The result of the
theorem can be extended to non-smooth ReLU\cite{Theory_I}.

In summary, when the only a priori assumption on the target function
is about the number of derivatives, then to {\it guarantee} an
accuracy of $\epsilon$, we need a shallow network with
$\O(\epsilon^{-n/m})$ trainable parameters. If we assume a
hierarchical structure on the target function as in
Theorem~\ref{deeptheo}, then the corresponding deep network yields a
guaranteed accuracy of $\epsilon$ with $\O(\epsilon^{-2/m})$ trainable
parameters. Note that Theorem~\ref{deeptheo} applies to all $f$ with a
compositional architecture given by a graph which correspond to, or is
a subgraph of, the graph associated with the deep network -- in this
case the graph corresponding to $W_m^{n,d}$.

\section{ The Optimization Landscape of Deep
  Nets with Smooth Activation Function}
\label{BezoutBoltzman}

The main question in optimization of deep networks is to the landscape
of the empirical loss in terms of its global minima and local critical
points of the gradient.

\subsection{Related work}

There are many recent papers studying optimization
in deep learning. For optimization we mention work based on the idea
that noisy gradient descent \cite{DBLP:journals/corr/Jin0NKJ17,
  DBLP:journals/corr/GeHJY15, pmlr-v49-lee16, s.2018when} can find a
global minimum.  More recently, several authors studied the dynamics of
gradient descent for deep networks with assumptions about the input
distribution or on how the labels are generated. They obtain global
convergence for some shallow neural networks
\cite{Tian:2017:AFP:3305890.3306033, s8409482,
  Li:2017:CAT:3294771.3294828, DBLP:conf/icml/BrutzkusG17,
  pmlr-v80-du18b, DBLP:journals/corr/abs-1811-03804}. Some local
convergence results have also been proved
\cite{Zhong:2017:RGO:3305890.3306109,
  DBLP:journals/corr/abs-1711-03440, 2018arXiv180607808Z}. The most
interesting such approach is \cite{DBLP:journals/corr/abs-1811-03804},
which focuses on minimizing the training loss and proving that
randomly initialized gradient descent can achieve zero training loss
(see also \cite{NIPS2018_8038, du2018gradient,
  DBLP:journals/corr/abs-1811-08888}). In summary, there is by now an extensive
literature on optimization that formalizes and refines to different
special cases and to the discrete domain our results of  \cite{theory_II, theory_IIb}.

\subsection{Degeneracy of global and local minima under the
  exponential loss}

The {\it first part} of the argument of this section relies on the
obvious fact (see \cite{theory_III}), that for RELU networks under the
hypothesis of an exponential-type loss function, there are {\it no
  local minima that separate the data} -- the only critical points of
the gradient that separate the data are the global minima.

Notice that the global minima are at $\rho = \infty$, when the
exponential is zero. As a consequence, the Hessian is identically zero
with all eigenvalues being zero. On the other hand any point of the
loss at a finite $\rho$ has nonzero Hessian: for instance in the
linear case the Hessian is proportional to $\sum_n^N x_n x^T_n$.
The local minima which are not global minima  must
misclassify. How degenerate are they?

Simple arguments \cite{theory_III} suggest that the critical points
which are not global minima  cannot be completely degenerate. We thus have the following

\begin{property}
  Under the exponential loss, global minima are completely degenerate
  with all eigenvalues of the Hessian ($W$ of them with $W$ being the
  number of parameters in the network) being zero. The other critical
  points of the gradient are less degenerate, with at least one -- and typically $N$
  -- nonzero eigenvalues.
\end{property}

For the general case of non-exponential loss and smooth
nonlinearities instead of the RELU the following conjecture has been
proposed \cite{theory_III}:

\begin{conjecture}: For appropriate overparametrization, there are a
  large number of global zero-error minimizers which are degenerate;
  the other critical points -- saddles and local minima -- are
  generically (that is with probability one) degenerate on a set of
  much lower dimensionality.
\end{conjecture}

\subsection{SGD and Boltzmann Equation}
The second part of our argument (in \cite{theory_IIb}) is that SGD
concentrates in probability on the most degenerate minima. The argument is based on the
similarity between a Langevin equation  and SGD and on the fact
that the Boltzmann distribution is exactly the asymptotic ``solution'' of the
stochastic differential Langevin equation and also of SGDL, defined as
SGD with added white noise (see for instance
\cite{raginskyetal17}). The Boltzmann distribution is

\begin{equation}
p(f) = \frac{1}{Z}e^{-\frac{L}{T}},
\label{Bolzman}
\end{equation}

\noindent where $Z$ is a normalization constant, $L(f)$ is the loss
and $T$ reflects the noise power. The equation implies that SGDL
prefers degenerate minima relative to non-degenerate ones of the same
depth. In addition, among two minimum basins of equal depth, the one
with a larger volume is much more likely in high dimensions as shown
by the simulations in \cite{theory_IIb}. Taken together, these two
facts suggest that SGD selects degenerate minimizers corresponding to
larger isotropic flat regions of the loss. Then SDGL shows concentration --
{\it because of the high dimensionality} -- of its asymptotic
distribution Equation \ref{Bolzman}.

Together \cite{theory_II} and \cite{theory_III} suggest the following

\begin{conjecture}: For appropriate
  overparametrization of the deep network, SGD selects with high
  probability the global minimizers of the empirical loss, which are
  highly degenerate.
\end{conjecture}

\begin{figure}
\centering
\includegraphics[width=1.0\linewidth ]{Figures/SGD_vs_SGDL.pdf}
\caption{ Stochastic Gradient Descent and Langevin Stochastic Gradient
  Descent (SGDL) on the $2$D potential function shown above leads to
  an asymptotic distribution with the histograms shown on the left. As
  expected from the form of the Boltzmann distribution, both dynamics
  prefer degenerate minima to non-degenerate minima of the same
  depth. From \cite{theory_III}.
}
\label{wedge_rbf_sgdl}
\end{figure}

\section{Generalization}
\label{generalization}
Recent results by \cite{2017arXiv171010345S} illuminate the apparent
absence of ''overfitting” (see Figure \ref{no-overfitting}) in the
special case of linear networks for binary classification.  They prove
that minimization of loss functions such as the logistic, the
cross-entropy and the exponential loss yields asymptotic convergence
to the maximum margin solution for linearly separable datasets,
independently of the initial conditions and without explicit
regularization.  Here we discuss the case of nonlinear multilayer DNNs
under exponential-type losses, for several variations of the basic
gradient descent algorithm. The main results are:
\begin{itemize}
\item classical uniform convergence bounds for generalization suggest
  a form of complexity control on the dynamics of the weight {\it
    directions $V_k$}: minimize a surrogate loss subject to a
  unit $L_p$ norm constraint;

\item gradient descent on the exponential loss with an explicit $L_2$
  unit norm constraint is equivalent to a well-known gradient descent
  algorithms {\it weight normalization} which is closely related to
  batch normalization;

\item unconstrained gradient descent on the exponential loss yields a
  dynamics with the same critical points as weight normalization: the
  dynamics implicitly respect a $L_2$ unit constraint on the
  directions of the weights $V_k$.
\end{itemize}

We observe that several of these results {\it  directly apply to kernel
machines} for the exponential loss under the
separability/interpolation assumption, because  kernel machines are
one-homogeneous.

\subsection{Related work}

A number of papers have studied gradient descent for deep networks
\cite{NIPS2017_6836, DBLP:journals/corr/abs-1811-04918,
  Arora2019FineGrainedAO}. Close to the approach summarized here
(details are in \cite{theory_III}) is the paper
\cite{Wei2018OnTM}. Its authors study generalization assuming a
regularizer because they are -- like us -- interested in normalized
margin. Unlike their assumption of an explicit regularization, we show
here that commonly used techniques, such as weight and batch
normalization, in fact minimize the surrogate loss margin while
controlling the complexity of the classifier without the need to add a
regularizer or to use weight decay. Surprisingly, we will show that
even standard gradient descent on the weights implicitly controls the
complexity through an ``implicit'' unit $L_2$ norm constraint. Two
very recent papers (\cite{2019arXiv190507325S} and
\cite{DBLP:journals/corr/abs-1906-05890}) develop an elegant but
complicated margin maximization based approach which lead to some of
the same results of this section (and many more).  The important question of
which conditions are necessary for gradient descent to converge to the
maximum of the
margin of $\tilde{f}$ are studied by \cite{2019arXiv190507325S} and
\cite{DBLP:journals/corr/abs-1906-05890}. Our approach does not need the
notion of maximum margin but our theorem \ref{margin-maxTheorem}
establishes a connection with it and thus with the results of
\cite{2019arXiv190507325S} and
\cite{DBLP:journals/corr/abs-1906-05890}. Our main goal here (and in
\cite{theory_III}) is to achieve a simple understanding of where the
complexity control underlying generalization is hiding in the training
of deep networks.

\subsection{Deep networks: definitions and properties}

We define a deep network with $K$ layers with the usual
coordinate-wise scalar activation functions
$\sigma(z):\quad \mathbf{R} \to \mathbf{R}$ as the set of functions
$f(W;x) = \sigma (W^K \sigma (W^{K-1} \cdots \sigma (W^1 x)))$, where
the input is $x \in \mathbf{R}^d$, the weights are given by the
matrices $W^k$, one per layer, with matching dimensions. We sometime
use the symbol $W$ as a shorthand for the set of $W^k$ matrices
$k=1,\cdots,K$. For simplicity we consider here the case of binary
classification in which $f$ takes scalar values, implying that the
last layer matrix $W^K$ is $W^K \in \mathbf{R}^{1,K_l}$. The labels
are $y_n\in\{-1,1\}$. The weights of hidden layer $l$ are collected in
a matrix of size $h_l\times h_{l-1}$.  There are no biases apart form
the input layer where the bias is instantiated by one of the input
dimensions being a constant. The activation function in this section
is the ReLU activation.

For ReLU activations the following
important positive one-homogeneity property holds
$\sigma(z)=\frac{\partial \sigma(z)}{\partial z} z$.
A consequence of one-homogeneity is a structural lemma (Lemma 2.1 of
\cite{DBLP:journals/corr/abs-1711-01530}) $\sum_{i,j} W^{i,j}_k \left(\frac{\partial f(x)}{\partial
    W^{i,j}_k}\right)= f(x)$ where $W_k$ is here the vectorized representation of the weight
matrices $W_k$ for each of the different layers (each matrix is a
vector).

For the network, homogeneity implies
$f(W;x)=\prod_{k=1}^K \rho_k f(V_1,\cdots,V_K; x_n)$, where
$W_k=\rho_k V_k$ with the matrix norm $||V_k||_p=1$. Another property
of the Rademacher complexity of ReLU networks that follows from
homogeneity is
$\mathbb{R}_N(\mathbb{F}) = \rho \mathbb{R}_N(\tilde{\mathbb{F}})$
where $\rho=\rho_1 \prod_{k=1}^K \rho_k$, $\mathbb{F}$ is the class of
neural networks described above.

We define $f= \rho \tilde{f}$; $\tilde{\mathbb{F}}$ is the
associated class of normalized neural networks (we call
$f(V;x)=\tilde{f}(x)$ with the understanding that $f(x)=f(W;x)$).
Note that
$\frac{\partial f}{\partial \rho_k} = \frac{\rho}{\rho_k}\tilde{f}
\label{rho}$ and that the definitions of $\rho_k$, $V_k$ and
$\tilde{f}$ all depend on the choice of the norm used in
normalization.

In the case of training data that can be separated by the networks
$f(x_n) y_n>0 \quad \forall n=1,\cdots,N$. We will sometime write
$f(x_n)$ as a shorthand for $y_n f(x_n)$.

\subsection{Uniform convergence bounds: minimizing a surrogate loss
  under norm constraint}
\label{Early stopping}

Classical {\it generalization bounds for regression}
\cite{Bousquet2003} suggest that minimizing the empirical loss of a
loss function such as the cross-entropy
subject to constrained {\it complexity of the minimizer} is a way to
to attain  generalization, that is an expected loss which is close to the
empirical loss:

\begin{proposition}
The following generalization bounds
apply to $\forall f \in \mathbb{F}$ with probability at least
$(1-\delta)$:
\begin{equation}
L(f) \leq \hat{L}(f) + c_1\mathbb{R}_N(\mathbb{F}) + c_2 \sqrt
\frac{\ln(\frac{1}{\delta})}{2N}
\label{bound}
\end{equation}
\end{proposition}
\vskip0.1in
\noindent where $L(f) = \mathbf E [\ell(f(x), y)]$ is the expected
loss, $\hat{L}(f)$ is the empirical loss, $\mathbb{R}_N(\mathbb{F})$
is the empirical Rademacher average of the class of functions
$\mathbb{F}$, measuring its complexity; $c_1, c_2$ are constants that
depend on properties of the Lipschitz constant of the loss function,
and on the architecture of the network.

Thus minimizing under a constraint on the
Rademacher complexity  a surrogate function such as the
cross-entropy (which becomes the logistic loss in the binary
classification case) will minimize  an upper bound on the expected classification
error because such surrogate functions are upper bounds on the $0-1$
function. We can choose a class of functions $\mathbf{\tilde{F}}$ with
normalized weights and write $f(x)=\rho \tilde{f}(x)$ and
$\mathbb{R}_N(\mathbb{F})=\rho \mathbb{R}_N(\mathbb{\tilde{F}})$. One
can choose any fixed $\rho$ as a (Ivanov) regularization-type
tradeoff.

In summary, the problem of generalization may be approached by minimizing
the exponential loss -- more in general an exponential-type loss, such
the logistic and the cross-entropy -- under a unit norm constraint on
the weight matrices, since we are interested in the directions of the weights:
\begin{equation}
\lim_{\rho \to \infty} \arg\min_{||V_k||=1, \  \forall k} L(\rho \tilde{f})
\label{UnitNormMin}
\end{equation}
\noindent where we write $f(W) = \rho \tilde{f}(V)$ using the
homogeneity of the network. As it will become clear later, gradient
descent techniques on the exponential loss automatically increase
$\rho$ to infinity. We will typically consider the sequence of
minimizations over $V_k$ for a sequence of increasing $\rho$. The key
quantity for us is $\tilde{f}$ and the associated weights $V_k$;
$\rho$ is in a certain sense an auxiliary variable, a constraint that
is progressively relaxed.

In the following we explore the implications for deep networks of this classical
approach to generalization.

\subsubsection{Remark: minimization of an exponential-type loss implies margin maximization
}

Though not critical for our approach to the question of generalization
in deep networks it is interesting that constrained minimization of
the exponential loss implies margin maximization. This property
relates our approach to the results of several recent papers
\cite{2017arXiv171010345S,
  2019arXiv190507325S,DBLP:journals/corr/abs-1906-05890}. Notice that
our theorem \ref{margin-maxTheorem} as in
\cite{DBLP:conf/nips/RossetZH03} is a {\it sufficient condition for
  margin maximization}. Necessity is not true for general loss
functions.

To state the margin property more formally, we adapt to our setting a
different result due to \cite{DBLP:conf/nips/RossetZH03} (they
consider for a linear network a vanishing $\lambda$ regularization
term whereas we have for nonlinear networks a set of unit norm
constraints). First we recall the definition of the empirical loss
$L(f)=\sum_{n=1}^N \ell(y_n f(x_n))$ with an exponential loss function
$\ell(yf)= e^{-yf}$.  We define $\eta(f)$ a the {\it margin} of $f$,
that is $\eta(f)=\min_n f(x_n)$.

Then our margin maximization theorem (proved in \cite{theory_III})  takes the form

\begin{theorem}
    Consider the set of $V_k, k=1,\cdots, K$ corresponding to

  \begin{equation}
 \min_{{||V_k||}=1} L(f(\rho_k, V_k))
\label{V(rho)}
  \end{equation}
  \noindent where the norm $||V_k||$ is a chosen $L_p$ norm and
  $L(f)(\rho_k, V_K) = L(\tilde{f}(\rho)) = \sum_ n \ell(y_n \rho f(V; x_n))$ is the
  empirical exponential loss. For each layer consider a sequence of increasing
  $\rho_k$. Then the associated sequence of $V_k$ defined by Equation
  \ref{V(rho)}, converges for $\rho \to \infty$ to the maximum
  margin of $\tilde{f}$, that is to
  $\max_{||V_k|| \leq 1} \eta(\tilde{f})$ .
\label{margin-maxTheorem}
\end{theorem}

\subsection{Minimization under unit norm constraint: weight normalization}
\label{OurWeightNormalization}

The approach is then to minimize the loss function
$ L(f(w))=\sum_{n=1}^N e^{- f(W;x_n) y_n }= \sum_{n=1}^N e^{- \rho
  f(V_k;x_n) y_n }$, with $\rho= \prod \rho_k$, subject to
$||V_k||^p_p =1 \ \forall k$, that is under a unit norm constraint for
the weight matrix at each layer (if $p=2$ then
$\sum_{i,j} (V_k)_{i,j}^2= 1$ is the Frobenius norm), since $V_k$ are
the directions of the weights which are the relevant quantity for
classification. The minimization is understood as a sequence of
minimizations for a sequence of increasing $\rho_k$. Clearly these
constraints imply the constraint on the norm of the product of weight
matrices for any $p$ norm (because any induced operator norm is a
sub-multiplicative matrix norm). The standard choice for a loss
function is an exponential-type loss such the cross-entropy, which for
binary classification becomes the logistic function. We study here the
exponential because it is simpler and retains all the basic
properties.

There are several gradient descent techniques that given the
unconstrained optimization problem transform it into a {\it
  constrained} gradient descent problem. To provide the background let us
formulate the standard unconstrained gradient descent problem for the
exponential loss as it is used in practical training of deep networks:

\begin{equation}
	\dot{W}^{i,j}_k  = -\frac{\partial L}{\partial W^{i,j}_k}= \sum_{n=1}^N y_n \frac{\partial{f(x_n; w)}} {\partial W^{i,j}_k} e^{- y_n
		f(x_n;W)}
\label{standardynamicsW}
\end{equation}
\noindent where $W_k$ is the weight matrix of layer $k$. Notice that,
since the structural property implies that at a critical point we have
$\sum_{n=1}^N y_n f(x_n; w) e^{- y_n f(x_n;W)} = 0$, the only critical
points of this dynamics that separate the data (i.e.
$y_n f(x_n; w)>0 \ \forall n$) are global minima at infinity. Of
course for separable data, while the loss decreases asymptotically to
zero, the norm of the weights $\rho_k$ increases to infinity, as we
will  see later. Equations \ref{standardynamicsW} define a dynamical
system in terms of the gradient of the exponential loss $L$.

The set of gradient-based algorithms enforcing a unit-norm constraints
\cite{845952} comprises several techniques that are equivalent for small
values of the step size. They are all good approximations of the true
gradient method. One of them is the {\it Lagrange multiplier method};
another is the {\it tangent gradient method} based on the following theorem:

\begin{theorem} \cite{845952}  Let $||u||_p$ denote a vector norm that is
  differentiable with respect to the elements of $u$ and let $g(t)$ be
  any vector function with finite $L_2$ norm.  Then, calling
  $\nu(t)=\frac{\partial ||u||_p}{\partial u}_{u=u(t)}$, the equation
\begin{equation}
 \dot{u}=h_g(t)=Sg(t)= (I-\frac{\nu \nu^T}{||\nu||_2^2}) g(t)
\label{dot_u}
\end{equation}
\noindent with $||u(0)|| =1$, describes the flow of a vector $u$ that
satisfies $||u(t)||_p=1$ for all $t \ge 0$.
\label{Theorem1}
\end{theorem}

In particular, a form for $g$ is $g(t)= \mu(t) \nabla_u L$, the
gradient update in a gradient descent algorithm. We call $Sg(t)$ the
tangent gradient transformation of $g$.  In the case of $p=2$ we replace $\nu$ in Equation
\ref{dot_u} with $u$ because
$\nu(t)=\frac{\partial ||u||_2}{\partial u}=u$. This gives
$S= I-\frac{u u^T}{||u||_2^2}$ and $\dot{u}=Sg(t).$

Consider now the empirical loss $L$ written in terms of $V_k$ and
$\rho_k$ instead of $W_k$, using the change of variables defined by
$W_k=\rho_k V_k$ but without imposing a unit norm constraint on $V_k$.
The flows in $\rho_k,V_k$ can be computed as
$\dot{\rho_k}=\frac{\partial W_k}{\partial \rho_k} \frac{\partial
  L}{\partial W_k}= V_k^T \frac{\partial L}{\partial W_k}$ and
$\dot{V_k}=\frac{\partial W_k}{\partial V_k} \frac{\partial L}{\partial W_k} =
\rho_k \frac{\partial L}{\partial W_k}$, with $\frac{\partial
  L}{\partial W_k}$ given by Equations \ref{standardynamicsW}.

We now enforce the unit norm constraint on $V_k$ by using the tangent gradient
transform on the $V_k$ flow. This yields

\begin{equation}
\dot{\rho_k}= V_k^T \frac{\partial  L}{\partial W_k} \quad
\dot{V_k}= S_k \rho_k \frac{\partial L}{\partial W_k}.
\label{v-flow-withunitnorm}
\end{equation}

Notice that the dynamics above follows from the classical approach of
controlling the Rademacher complexity of $\tilde{f}$ during optimization (suggested
by bounds such as Equation \ref{bound}.  The approach and the
resulting dynamics for the directions of the weights may seem  different from the standard
unconstrained approach in training deep networks. It turns out,
however, that the dynamics described by Equations
\ref{v-flow-withunitnorm} is the same dynamics of {\it Weight
  Normalization}.

The technique of {\it Weight normalization} \cite{SalDied16} was
originally proposed as a small improvement on standard gradient descent
``to reduce covariate shifts''. It was defined for each layer in
terms of $w=g \frac{v}{||v||}$, as
\begin{equation}
\dot{g}=\frac{v}{||v||} \frac{\partial L}{\partial w}
\dot{v}=\frac{g}{||v||} S \frac{\partial L}{\partial w}
\label{W-normalization}
\end{equation}
\noindent with $S=I- \frac{v v^T}{||v||^2}$.

It is easy to see that Equations \ref{v-flow-withunitnorm} are the
same as the weight normalization Equations \ref{W-normalization}, if
$||v||_2=1$. We now observe, multiplying Equation
\ref{v-flow-withunitnorm} by $v^T$, that $v^T \dot{v}=0$ because
$v^T S=0$, implying that $||v||^2$ is constant in time with a constant
that can be taken to be $1$. Thus the two dynamics are the
same.

\subsection{Generalization with hidden
  complexity control}
\label{nounitnorm}
Empirically it appears that GD and SGD converge to solutions that can
generalize even without batch or weight normalization.  Convergence
may be difficult for quite deep networks and generalization may not be
as good as with batch normalization but it still occurs. How is this
possible?

We study the dynamical system
$\dot{W_k}^{i,j}$ under the reparametrization
$W^{i,j}_k = \rho_k V^{i,j}_k$ with $||V_k||_2=1$.  We consider  for each
weight matrix $W_k$ the corresponding ``vectorized'' representation in
terms of vectors $W_k^{i,j} = W_k$.  We use the following definitions
and properties (for a vector $w$):

\begin{itemize}
\item Define $\frac{w}{||w||_2}=\tilde{w}$; thus $w=||w||_2\tilde{w}$ with
  $||\tilde{w}||_2=1$. Also define $S={I-\tilde{w}\tilde{w}^T}=I- \frac {w
      w^T}{||w||_2^2}$.
\item The following relations are easy to check:
\begin{enumerate}
\item $\frac{\partial ||w||_2}{\partial w}=\tilde{w}$
\item $\frac{\partial \tilde{w}}{\partial
    w}=\frac{S}{||w||_2}$.
\item $Sw=S \tilde{w}=0$
\item $S^2=S$
\label{Relations}
\end{enumerate}
\end{itemize}

The gradient descent dynamic system used in training deep networks for
the exponential loss is given by Equation \ref{standardynamicsW}.
Following the chain rule {\it for the time derivatives}, the dynamics
for $W_k$ is exactly (see \cite{theory_III}) equivalent to the
following dynamics for $||W_k||=\rho_k$ and $V_k$:
\begin{equation}
\dot{\rho_k}= \frac{\partial ||W_k||}{\partial W_k} \frac{\partial
  W_k}{\partial t}= V^T_k \dot{W_k}
\label{rhodot}
\end{equation}
\noindent and
\begin{equation}
\dot{V_k}= \frac{\partial V_k}{\partial W_k} \frac{\partial
  W_k}{\partial t}= \frac {S_k}{\rho_k} \dot{W_k}
\label{vdot}
\end{equation}
\noindent where $S_k= I- V_kV_k^T$.  We used property 1 in
\ref{Relations} for Equation \ref{rhodot} and property 2 for Equation
\ref{vdot}.

The key point here is that the dynamics of $\dot{V_k}$ includes a unit
$L_2$ norm constraint: using the tangent gradient transform will not
change the equation because $S^2=S$.

As separate remarks , notice that if for $t>t_0$, $f$ separates all
the data, $\frac{d}{dt}{\rho_k} >0$, that is $\rho$ diverges to
$\infty$ with $\lim_{t \to \infty}\dot{\rho}=0$. In the 1-layer
network case the dynamics yields $\rho \approx \log t$
asymptotically. For deeper networks, this is
different. \cite{theory_III} shows (for one support vector) that the
product of weights at each layer diverges faster than logarithmically,
but each individual layer diverges slower than in the 1-layer case.
The norm of the each layer grows at the same rate $\dot{\rho_k^2}$,
independent of $k$. The $V_k$ dynamics has stationary or critical points given by

\begin{equation}
\sum \alpha_n(\rho(t)
\left(\frac{\partial{\tilde{f}(x_n)}} {\partial
                  V_k^{i,j}}-V_k^{i,j}  \tilde{f}(x_n) \right),
\label{wdot4}
\end{equation}
\noindent where $\alpha_n= e^{-y_n \rho(t) \tilde{f}(x_n)}$.
We examine later the linear one-layer case $\tilde{f}(x)=v^T x$ in
which case the stationary points of the gradient are given by $\sum \alpha_n(\rho(t)
(x_n - v v^T x_n)$ and of course coincide with the solutions obtained
with Lagrange multipliers. In the general case the
critical points correspond for $\rho \to \infty$ to degenerate zero
``asymptotic minima'' of the loss.

To understand whether there exists an implicit complexity control in
standard gradient descent of the weight directions, we  check whether there exists an
$L_p$ norm for which unconstrained normalization is equivalent to
constrained normalization.

From Theorem \ref{Theorem1} we expect
the constrained case to be given by the action of the following
projector onto the tangent space:
\begin{equation}
  S_{p} = I-\frac{\nu \nu^T}{||\nu||_2^2} \quad\textnormal{with}\quad \nu_i=\frac{\partial ||w||_p}{\partial w_i} = \textnormal{sign}(w_i)\circ\left(\frac{|w_i|}{||w||_p}\right)^{p-1}.
\end{equation}
The constrained Gradient Descent is then
\begin{equation}
\dot{\rho_k}= V^T_k \dot{W_k}  \quad
\dot{V_k} = \rho_k S_p \dot{W_k}.
\label{ConstrainedGradP}
\end{equation}

On the other hand,  reparametrization of
the unconstrained dynamics in the $p$-norm gives (following Equations \ref{rhodot} and \ref{vdot})
\begin{equation}
\begin{split}
\dot{\rho_k}&= \frac{\partial ||W_k||_p}{\partial W_k} \frac{\partial
	W_k}{\partial t}= \textnormal{sign}(W_k)\circ\left(\frac{|W_k|}{||W_k||_p}\right)^{p-1} \cdot \dot{W_k} \\
\dot{V_k}&= \frac{\partial V_k}{\partial W_k} \frac{\partial
	W_k}{\partial t}= \frac {I - \textnormal{sign}(W_k) \circ\left(\frac{|W_k|}{||W_k||_p}\right)^{p-1}W_k^T}{||W_k||_p^{p-1}} \dot{W_k}.
\end{split}
\end{equation}
These two dynamical systems are clearly different for generic
$p$ reflecting the presence or absence of a regularization-like
constraint on the dynamics of $V_k$.

As we have seen however, for $p=2$ the 1-layer dynamical system obtained by
minimizing $L$ in $\rho_k$ and $V_k$ with $W_k=\rho_k V_k$ under the constraint
$||V_k||_2=1$, is the weight normalization dynamics

\begin{equation}
\dot{\rho_k}=V_k^T \dot{W_k} \quad
\dot{V_k}= S \rho_k \dot{W_k} ,
\label{WN}
\end{equation}

\noindent which is quite similar to the standard gradient
equations
\begin{equation}
\dot{\rho_k}= V_k^T \dot{W_k}  \quad
\dot{v} =\frac{S}{\rho_k} \dot{W_k}.
\label{StandardGrad}
\end{equation}

\begin{figure*}[t!]\centering
		\includegraphics[width=1.0\textwidth]{Figures/fig2.pdf}
		\caption{\it The top left graph shows testing vs
                  training cross-entropy loss for networks each
                  trained on the same data sets (CIFAR10) but with a
                  different initializations, yielding zero
                  classification error on training set but different
                  testing errors. The top right graph shows the same
                  data, that is testing vs training loss for the same
                  networks, now normalized by dividing each weight by
                  the Frobenius norm of its layer. Notice that all
                  points have zero classification error at
                  training. The red point on the top right refers to a
                  network trained on the same CIFAR-10 data set but
                  with randomized labels. It shows zero classification
                  error at training and test error at chance
                  level. The top line is a square-loss regression of
                  slope $1$ with positive intercept. The bottom line
                  is the diagonal at which training and test loss are
                  equal.  The networks are 3-layer convolutional
                  networks. The left can be considered as a
                  visualization of Equation \ref{bound} when the
                  Rademacher complexity is not controlled. The right
                  hand side is a visualization of the same relation
                  for normalized networks that is
                  $L(\tilde{f}) \leq \hat{L}(\tilde{f}) +
                  c_1\mathbb{R}_N(\mathbb{\tilde{F}}) + c_2 \sqrt
                  \frac{\ln(\frac{1}{\delta})}{2N}$. Under our
                  conditions for $N$ and for the architecture of the
                  network the terms
                  $c_1\mathbb{R}_N(\mathbb{\tilde{F}}) + c_2 \sqrt
                  \frac{\ln(\frac{1}{\delta})}{2N}$ represent a small
                  offset. From
                  \cite{DBLP:journals/corr/abs-1807-09659}.
}
\label{main}
	\end{figure*}

        The two dynamical systems differ only by a $\rho_k^2$ factor
        in the $\dot{V_k}$ equations. However, the critical points of
        the gradient for the $V_k$ flow, that is the point for which
        $\dot{V_k}=0$, are the same in both cases since for any $t>0$
        $\rho_k(t)>0$ and thus $\dot{V_k}=0$ is equivalent to
        $S\dot{W_k}=0$.  Hence, gradient descent with unit $L_p$-norm
        constraint is equivalent to the standard, unconstrained
        gradient descent but only when $p = 2$. Thus

        \begin{fact} The standard dynamical system used in deep
          learning, defined by
          $\dot{W_k}=-\frac{\partial L}{\partial W_k}$, implicitly
          respectss a unit $L_2$ norm constraint on $V_k$ with
          $\rho_k V_k =W_k$. Thus, under an exponential loss, if the
          dynamics converges, the $V_k$ represent the minimizer under
          the $L_2$ unit norm constraint.
	\label{w(T)}
\end{fact}

Thus standard GD implicitly enforces the $L_2$ norm constraint on
$V_k=\frac{W_k}{||W_k||_2}$, consistently with Srebro's results on
implicit bias of GD. Other minimization techniques such as coordinate
descent may be biased towards different norm constraints.

\subsection{Linear networks and rates of convergence}

The linear ($f(x)=\rho v^T x$) networks case
\cite{2017arXiv171010345S} is an interesting example of our analysis
in terms of $\rho$ and $v$ dynamics. We start with unconstrained
gradient descent, that is with the dynamical system

 \begin{equation}
\dot{\rho}=  \frac{1}{\rho} \sum_{n=1}^N  e^{-
                \rho v^Tx_n} v^Tx_n \quad \dot{v}=\frac{1}{\rho}\sum_{n=1}^N  e^{- \rho v^T x_n}
(x_n- v  v^T x_n).
\label{Ttt}
\end{equation}

If gradient descent in $v$ converges to $\dot{v}=0$ at finite time,
$v$ satisfies $ v v^T x = x$, where $x= \sum_{j=1}^C \alpha_j x_j$
with positive coefficients $\alpha_j$ and $x_j$ are the $C$ support
vectors (see \cite{theory_III}).  A solution $v^T = ||x|| x^\dagger$
then {\it exists} ($x^\dagger$, the pseudoinverse of $x$, since $x$ is
a vector, is given by $x^\dagger= \frac{x^T}{||x||^2}$). On the other
hand, the operator $T$ in $v(t+1)=T v(t)$ associated with equation
\ref{Ttt} is non-expanding, because $||v||=1,\ \forall t$. Thus there
is a fixed point $v \propto x$ which is {\it independent of initial
  conditions} \cite{Ferreira1996} and unique (in the linear case)

The rates of convergence of the solutions $\rho(t)$ and $v(t)$,
derived in different way in \cite{2017arXiv171010345S}, may be read
out from the equations for $\rho$ and $v$. It is easy to check that a
general solution for $\rho$ is of the form $\rho \propto C \log t$. A
similar estimate for the exponential term gives
$e^{- \rho v^T x_n} \propto \frac{1}{t}$. Assume for simplicity a
single support vector $x$. We claim that a solution for
the error $\epsilon= v-x$, since $v$ converges to $x$, behaves as
$\frac{1}{\log t}$.  In fact we write $v =x+\epsilon$ and plug it in
the equation for $v$ in \ref{T}. We obtain (assuming normalized input $||x||=1$)
\begin{equation}
\dot{\epsilon}=\frac{1}{\rho}  e^{- \rho v^T x} (x- (x+\epsilon)
(x+\epsilon)^T x) \approx \frac{1}{\rho}  e^{- \rho v^T x} ( x- x - x
\epsilon^T - \epsilon x^T),
\label{T}
\end{equation}
\noindent which has the form
$\dot{\epsilon}=-\frac{1}{t \log t} (2 x \epsilon^T)$. Assuming
$\epsilon$ of the form
$\epsilon \propto \frac{1}{\log t}$ we obtain
$-\frac{1}{t \log^2 t }= -B \frac{1}{t \log^2 t}$. Thus the error
indeed converges as
$\epsilon \propto \frac{1}{\log t}$.

A similar analysis for the weight normalization equations \ref{WN}
considers the same dynamical system with a change in the equation for
$v$, which becomes

\begin{equation}
\dot{v} \propto e^{-\rho}  \rho  (I- v v^T) x.
\label{T-WN}
\end{equation}
This equation differs by a factor $\rho^2$ from
equation \ref{T}. As a consequence equation \ref{T-WN} is of the form
$\dot{\epsilon}=-\frac{\log t}{t} \epsilon$, with a general solution
of the form $\epsilon \propto t^{-\frac{1}{2}\log t}$. In
summary, {\it GD with weight normalization converges faster to the
  same equilibrium than standard gradient descent: the rate for
  $\epsilon= v- x$ is  $t^{-\frac{1}{2} log(t)}$ vs  $\frac{1}{\log t}$.}

Our goal was to find
$\lim_{\rho \to \infty} \arg \min_{||V_k||=1, \ \forall k} L(\rho
\tilde{f}) $. We have seen that various forms of gradient descent
enforce different paths in increasing $\rho$ that empirically have
different effects on convergence rate. It will be an interesting
theoretical and practical challenge to find the optimal way, in terms
of generalization and convergence rate, to grow $\rho\rightarrow \infty$.

Our analysis of simplified batch normalization \cite{theory_III}
suggests that several of the same considerations that we used for
weight normalization should apply (in the linear one layer case BN is
identical to WN). However, BN differs from WN in the multilayer case
in several ways, in addition to weight normalization: it has for
instance separate normalization for each unit, that is for each row of
the weight matrix at each layer.

\begin{figure
	\centering
	\includegraphics[width=0.9\linewidth]{Figures/no-overfitting.pdf}
	\caption{Empirical and expected error in CIFAR 10 as a
          function of number of neurons in a 5-layer convolutional
          network. The expected classification error does not increase
          when increasing the number of parameters beyond the size of
          the training set in the range we tested.}
	\label{no-overfitting}
\end{figure}

\section{Discussion}

A main difference between shallow and deep networks is in terms of
{\it approximation} power or, in equivalent words, of the ability to
learn good representations from data based on the compositional
structure of certain tasks. Unlike shallow networks, deep local
networks -- in particular convolutional networks -- can avoid the
curse of dimensionality in approximating the class of hierarchically
local compositional functions. This means that for such class of
functions deep local networks represent an appropriate hypothesis
class that allows good approximation with a minimum number of
parameters. It is not clear, of course, why many problems encountered
in practice should match the class of compositional functions. Though
we and others have argued that the explanation may be in either the
physics or the neuroscience of the brain, these arguments are not
rigorous. Our conjecture at present is that compositionality is
imposed by the wiring of our cortex and, critically, is reflected in
language. Thus compositionality of some of the most common visual
tasks may simply reflect the way our brain works.

{\it Optimization} turns out to be surprisingly easy to perform for
overparametrized deep networks because SGD will converge with high
probability to global minima that are typically much more degenerate for
the exponential loss than other local critical points.

More surprisingly, gradient descent yields {\it generalization} in
classification performance, despite overparametrization and even in
the absence of explicit norm control or regularization, because
standard gradient descent in the weights enforces an implicit unit
($L_2$) norm constraint on the {\it directions of the weights} in the
case of exponential-type losses.

In summary, it is tempting to conclude that the practical success of
deep learning has its roots in the almost magic synergy of unexpected
and elegant theoretical properties of several aspects of the
technique: the deep convolutional network architecture itself, its
overparametrization, the use of stochastic gradient descent, the
exponential loss, the homogeneity of the RELU units and of the
resulting networks.

Of course many problems remain open on the way to develop a full
theory and, especially, in translating it to new architectures. More
detailed results are needed in approximation theory, especially for
densely connected networks. Our framework for optimization is missing
at present a full classification of local minima and their dependence
on overparametrization for general loss functions. The analysis of
generalization should include an analysis of convergence of the
weights for multilayer networks (see \cite{2019arXiv190507325S} and
\cite{DBLP:journals/corr/abs-1906-05890}). A full theory would also
require an analysis of the trade-off  between
approximation and estimation error, relaxing the separability
assumption.

\showmatmethods{}

\acknow{We are grateful to Sasha Rakhlin and Nate Srebro for useful
  suggestions about the structural lemma and about separating critical
  points. Part of the funding is from the Center for Brains, Minds and
  Machines (CBMM), funded by NSF STC award CCF-1231216, and part by
  C-BRIC, one of six centers in JUMP, a Semiconductor Research
  Corporation (SRC) program sponsored by DARPA.}

\showacknow{}

Machine Learning (cs.LG); Machine Learning (stat.ML)


Algebraic stories from one and from the other pockets

Abstract: We present a number of questions in commutative algebra posed on the problem solving seminar in algebra at Stockholm University during the period Fall 2014 - Spring 2017.

\section{The Waring problem for complex-valued forms}
The following famous  result on binary forms was   proven by J.~J.~Sylvester in 1851. (Below we use the terms ``forms" and ``homogeneous polynomials" as synonyms.)

\begin{Theorem}[Sylvester's Theorem \cite{Syl2}]\label{th:Sylv}
\rm{(i)} A general binary form $f$ of odd degree $k=2s-1$ with complex coefficients can be written as
$$ f(x, y) =\sum_{j=1}^s(\al_jx+\be_jy)^k.$$

\noindent
\rm{(ii)} A general binary form $f$ of even degree $k=2s$ with complex coefficients can be written as
$$ f(x,y)=\la x^k +\sum_{j=1}^s(\al_jx+\be_jy)^k.$$
\end{Theorem}

Sylvester's result was the starting point of the study of the so-called \emph{Waring problem for polynomials} which we discuss below.

\medskip
 Let $S = \bC[x_1,\ldots,x_n]$ be the polynomial ring  in $n$ variables with complex coefficients. With respect to the standard grading, we have $S = \bigoplus_{d\geq 0} S_d$, where $S_d$ denotes the vector space of  all forms of degree $d$.

\begin{defi+}\label{def1}
Let $f$ be a form of degree $k$ in $S$. A presentation of $f$ as a sum of $k$-th powers of linear forms, i.e., $f=l_1^k+\ldots+l_s^k$, where $l_1,\ldots,l_s \in S_1$, is called a \emph{Waring decomposition} of $f$.
The minimal length of such a decomposition is called the \emph {Waring rank} of $f$, and we denote it as $\rk(f)$.
 By $\rk^\circ(k,n)$ we denote  the Waring rank of a {\em general} complex-valued form of degree $k$ in $n$ variables.
\end{defi+}

\begin{Rmk}
Besides being a natural question from the point of view of algebraic geometry, the Waring problem  for polynomials is partly motivated by its celebrated prototype, i.e., the Waring problem for natural numbers.  The latter was posed in 1770 by a British number theorist E. Waring
who claimed that, for any positive integer $k$, there exists a minimal number $g(k)$ such that every natural number can be written as sum of at most $g(k)$ $k$-th powers of positive integers. The famous Lagrange's four-squares Theorem (1770) claims that $g(2) = 4$  while the existence of $g(k)$, for any integer $k\ge 2$, is due to D. Hilbert (1900). Exact values of $g(k)$ are currently known only in a few cases.
\end{Rmk}

\subsection{Generic $k$-rank}
In terms of Definition~\ref{def1}, Sylvester's Theorem claims that the Waring rank of a general binary complex-valued  form of degree $k$ equals $\left\lfloor \frac{k}{2} \right\rfloor$.
More generally, the important result of J. Alexander and A. Hirschowitz \cite{al-hi} completely describes the Waring rank $\rk^\circ(k,n)$ of  general forms of any degree and in any number of variables.

\begin{Theorem}[Alexander-Hirschowitz Theorem, 1995]\label{th:AH} For all pairs of positive integers $(k,n)$,  the generic Waring rank  $\rk^\circ(k,n)$ is given by
\begin{equation}\label{rgen}
\rk^\circ(k,n)=\left\lceil \frac{\binom{n+k-1}{n-1}}{n} \right\rceil,
\end{equation}
except for the following cases:
\begin{enumerate}
	\item $k = 2$, where $\rk^{\circ}(2,n) = n$;
	\item $k=4,\; n=3,4,5$; and $k=3, n=5$, where, $r_{gen}(k,n)$ equals the r.h.s of \eqref{rgen} plus $1$.
\end{enumerate}
\end{Theorem}

Going further,  R.F. and B.S. jointly with G.~Ottaviani considered the following natural version of the Waring problem for complex-valued forms, see \cite{FOS}.

\begin{defi+}
Let $k,d$ be positive integers. Given a form $f$ of degree $kd$, a {\em $k$-Waring decomposition} is a presentation of $f$ as a sum of $k$-th powers of forms of degree $d$, i.e., $f = g_1^k + \cdots + g_s^k$, with $g_i \in S_d$. The minimal length of such an expression is called the {\em $k$-rank} of $f$ and is denoted by $\rk_k(f)$.
We denote by $\rk_k^\circ(kd,n)$ the $k$-rank of a general complex-valued form of degree $kd$ in $n$ variables.
 \end{defi+}

 In this notation, the case $d = 1$ corresponds to the classical Waring rank, i.e., if $k = \deg(f)$, then $\rk(f) = \rk_{k}(f)$ and $\rk^\circ(k,n)=\rk_k^\circ(k,n)$. Since the case $k = 1$ is trivial,  we assume below that $k \geq 2$.

\begin{problem}
Given a triple of positive integers $(k,d,n)$, calculate  $\rk_k^\circ(kd,n).$
\end{problem}

The main  result of \cite{FOS}  states  that, for any triple $(k,d,n)$ as above,
\begin{equation}\label{ineq:main}
\rk^\circ_k(kd,n)\le k^{n-1}.
\end{equation}
At the same time, by a simple parameter count, one has an obvious  lower bound for  $\rk^\circ(k,n)$ given by
\begin{equation}\label{ineq:lower bound}
\rk^\circ_k(kd,n)\geq \left\lceil \frac{\binom{n+kd-1}{n-1}} {\binom{n+d-1}{n-1}} \right\rceil.
\end{equation}

\medskip
A remarkable fact about the upper bound given by \eqref{ineq:main} is that it is independent of $d$. Therefore, since the right-hand side of \eqref{ineq:lower bound} equals  $k^n$ when $d \gg 0$,  we get that for large values of $d$, the bound in \eqref{ineq:main} is actually sharp. As a consequence of this remark, for any fixed $n\ge 1$ and $k\ge 2$, there exists a positive integer $d_{k,n}$ such that $\rk^\circ_k(kd,n)= k^n,$ for all $d\ge d_{k,n}$.

In the case of binary forms, it has been proven that \eqref{ineq:lower bound} is actually an equality \cite{Re1, LuOnReSh}. Exact values of $d_{k,n}$, and the behaviour of $\rk_k(kd,n)$ for $d \leq d_{k,n}$, have also been computed  in a few other cases, see \cite[Section 3.3]{On}. These results agree with the following illuminating conjecture suggested by G.~Ottaviani in 2014.

\begin{conjecture}\label{conj:main}
The $k$-rank of a general form of degree $kd$ in $n$ variables is given by
\begin{equation}\label{eq:main}
\rk_k^\circ(kd,n)=\begin{cases} \min \left\{s\ge 1 ~|~ s\binom{n+d-1}{n-1}-\binom {s}{2}\ge \binom{n+2d-1}{n-1}\right\}, & \text{ for } k=2;\\
\min \left\{s\ge 1 ~|~ s\binom{n+d-1}{n-1}\ge \binom{n+kd-1}{n-1}\right\}, & \text{ for } k\ge 3.
\end{cases}
\end{equation}
\end{conjecture}

Observe that, for $k\ge 3$, Conjecture~\ref{conj:main} claims that the na\"ive bound \eqref{ineq:lower bound} obtained by a parameter count is actually sharp, while, for $k = 2$, due to an additional group action there are many {\it defective} cases where the inequality is strict.

\begin{Rmk}
	Problems about additive decompositions including the above Waring problems can be usually rephrased geometrically in terms of {\em secant varieties}. In the case of $k$-Waring decompositions, we need to consider the {\em variety of powers} $V_{n,kd}^{(k)}$, i.e., the variety of $k$-th powers of forms of degree $d$ inside the (projective) space of forms of degree $kd$. The {\em $s$-th secant variety} $\sigma_s(V_{n,kd}^{(k)})$ is the Zariski  closure of the union of all linear spaces spanned by $s$-tuples of points lying on $V_{n,kd}^{(k)}$. In other words, it is the closure of the set of forms whose $k$-rank is at most $s$. Since the variety of powers is non-degenerate, i.e., it is not contained in any proper linear subspace of the space of forms of degree $kd$, the sequence of secant varieties  stratifies the latter space  and coincides with it for all sufficiently large $s$. Hence, the $k$-rank of a general form is the smallest value of $s$ for which the $s$-th secant variety of $V_{n,kd}^{(k)}$ coincides with the space of all forms of degree $kd$. Hence, Conjecture \ref{conj:main} can be rephrased as a conjecture about the dimensions of the secant varieties of $V_{n,kd}^{(k)}$. (We refer to \cite[Section 1.3.3]{On} for more details.)
\end{Rmk}

\subsection{Maximal $k$-rank}

A harder problem which is largely open even in the classical case of Waring decompositions, deals with the  computation of  the $k$-rank of an {\em arbitrary} complex-valued form of degree divisible by $k$.

\begin{defi+}
Given a triple $(k,d,n)$,  denote by $\rk^{\max}_k(kd,n)$  the minimal number of terms such that {\em every}  form of degree $kd$  in $n+1$ variables can be represented as the sum of at most  $\rk^{\max}_k(kd,n)$ $k$-th powers of  forms of degree $d$. The number $\rk^{\max}_k(kd,n)$ is called the {\it maximal $k$-rank}.

(Similarly to the  above, we omit the subscript when considering the classical Waring rank, i.e., for $d = 1$.)
\end{defi+}

In \cite[Theorem 5.4]{Re1}, B. Reznick proved that the maximal Waring rank of binary forms of degree $k$ equals $k$. Moreover, the maximal value $k$ is attained exactly on the binary forms representable as $\ell_1 \ell_2^{k-1},$ where $\ell_1$ and $\ell_2$ are any two
non-proportional
linear binary forms. (Apparently these claims have  been known much earlier, but have never been carefully written down with a complete proof.)

\begin{problem}
Given a triple of positive integers $(k,d,n)$, calculate  $\rk_k^{\max}(kd,n).$
\end{problem}

At the moment, we have an explicit conjecture about the maximal $k$-rank only  in the case of binary forms.

\begin{conjecture}\label{conj:maxrank}
For any positive integers $k,d$, the maximal $k$-rank $\rk^{\max}_k(kd,2)$ of binary forms equals $k$. Additionally, in the above notation, binary forms representable by $\ell_1 \ell_2^{kd-1}$, where $\ell_1$ and $\ell_2$ are non-proportional linear forms, have the latter maximal $k$-rank.
\end{conjecture}

Conjecture~\ref{conj:maxrank} is obvious for $k=2$ since, for any binary form $f$ of degree $2d$, we can write
\begin{equation} \label{eq:k2}
	f = g_1 g_2 = \left(\frac{1}{2} (g_1+g_2)\right)^2 + \left(\frac{i}{2} (g_1-g_2)\right)^2 \text{ with } g_1,g_2 \in S_d.
\end{equation}
The first non-trivial case is the one of binary sextics, i.e., $k=3, d=2$, which has been settled in \cite{LuOnReSh} where it has also  been  shown that the $4$-rank of $x_1x_2^7$ is equal to $4$.

\begin{Rmk} The best known general result about maximal ranks is due to G. Bleckherman and Z. Teitler, see \cite{BT} where they prove that the maximal rank is always at most twice as big as the generic rank. (This fact  is true  both for the classical ($d=1$) and for the higher ($d\geq 2$) Waring ranks.)

In the classical case of Waring ranks, this bound is (almost) sharp for binary forms, but in many other cases it is rather crude. At present, better bounds are known only in few special cases of low degrees \cite{BD13, Je14}. To the best of our knowledge, the exact values of the maximal Waring rank are only known for binary forms (classical, see \cite{Re1}), quadrics (classical), ternary cubics (see \cite{Seg42, LT10}), ternary quartics \cite{Kl99}, ternary quintics \cite{DeP15} and quaternary cubics \cite{Seg42}.
\end{Rmk}

\subsection{The $k$-rank of monomials}
Let $m = x_1^{a_1} \cdots  x_n^{a_n}$ be a monomial
 with $0 < a_1 \leq a_2 \leq \cdots \leq a_n.$ It has been shown in \cite{ca-ch-ge} that the classical Waring rank of $m$ is equal to $\frac{1}{(a_1+1)}\prod_{i=1,\ldots,n} (a_i+1)$.

\medskip
Later E. Carlini and A.O. settled the  case of the $2$-rank, see \cite{ca-on}. Namely,  if $m$ is a monomial of degree $2d$, then we can write $m=m_1 m_2$, where $m_1$ and $m_2$ are monomials of degree $d$. From  identity (\ref{eq:k2}), it follows that the $2$-rank of $m$ is at most two. On the other hand, $m$ has rank  one exactly when we can choose $m_1 = m_2$, i.e., when  the power of each variable in  $m$ is even.

While the cases $k=1$ and $k=2$ are solved, for $k\ge 3$, the question about the $k$-rank of monomials of degree $kd$,  is still open. At present, we are only aware of two general results in this direction. Namely, \cite{ca-on} contains the bound
 $\rk_k(m) \leq 2^{k-1}$,  and recently, S.L., A.O., B.S., together with B. Reznick, have shown that $\rk_k(m) \leq k$ when $d \geq n(k-2)$, see \cite{LuOnReSh}.
Thus, for fixed $k$ and $n$, all but a finite number of monomials of degree divisible by $k$ have $k$-rank less than $k$.

\begin{problem} \label{prob:monrk}
Given $k \geq 3$ and a monomial $m$ of degree $kd$, determine the monomial $k$-rank $\rk_k(m)$.

 \end{problem}
In the case of binary forms, a  bit more is currently known which motivates the following question.

\begin{problem}\label{prob:C}
Given $k \geq 3$ and a monomial $x^a y^b$ of degree $a+b=kd$, it is known that $\rk_k(x^{a}y^{b})  \leq \max(s,t)+1$, where $s$ and $t$ are the remainders of the division of  $a$ and $b$  by $k$, see  \cite{ca-on}. Is it true that the latter inequality is, in fact, an equality?
\end{problem}

\subsection{Degree of the Waring map}

Here again, we  concentrate on the case of binary forms (i.e., $n=2$). As we mentioned above, in this case, it is proven that
$$\rk_k^\circ(kd,2)= \left \lceil \frac{\dim S_{kd}}{\dim S_{d}}\right \rceil= \left \lceil\frac{kd+1}{d+1}\right \rceil.$$

\begin{defi+}
We say that a pair $(k,d)$ is {\it perfect} if $\frac{kd+1}{d+1}$ is an integer.
\end{defi+}

All perfect pairs are easy to describe.

\begin{Lemma}\label{lm:perfect} The set of all pairs $(k,d)$ for which
$\frac{kd+1}{d+1}\in\bN$ splits
into the disjoint sequences
$E_j := \{(jd+j+1,d) ~|~ d = 1,2,\ldots \}$. In each $E_j$, the corresponding quotient equals $jd+1$.
\end{Lemma}

Given a perfect pair $(k,d)$, set $s:=\frac{kd+1}{d+1}$. Consider the map
$$W_{k,d}: S_d \times \ldots \times S_d \to S_{kd}, ~~(g_1,\ldots,g_s) \mapsto g_1^k + \ldots + g_s^k.$$
Let $\widetilde{W}_{k,d}$ be the same map, but defined up to a permutation of the $g_i$'s. We call it the {\it Waring map}. By \cite[Theorem 2.3]{LuOnReSh}, $\widetilde{W}_{k,d}$ is a generically finite map of complex linear spaces of the same dimension. By definition, its {\it degree} is the cardinality of the inverse image of a generic form in $S_{kd}$.

\begin{problem}\label{pr:deg}
Calculate the degree of $\widetilde{W}_{k,d}$ for  perfect pairs $(k,d)$.
\end{problem}

For the classical Waring decomposition ($d = 1$), we have a perfect pair if and only if $k$ is odd. From Sylvester's Theorem, we know that in this case the degree of the Waring map is $1$, i.e., the general binary form of odd degree has a {\it unique} Waring decomposition, up to a permutation of its summands.

\begin{Rmk}
	For the case of the classical Waring decomposition, the latter problem has  also been considered  in the case of more  variables. In modern terminology, the cases where the general form of a given degree has a unique decomposition up to a permutation of the summands are called {\it identifiable}. Besides the case of binary forms of odd degree, some other identifiable cases are classically known. These  are the quaternary cubics (Sylvester's Pentahedral Theorem \cite{Syl2}) and the ternary quintics \cite{Hilb, Pal03, Ri04, MM13}. Recently, F.~Galuppi and M.~Mella proved that these are the only possible identifiable cases, \cite{GM16}.
\end{Rmk}

\begin{Rmk}
	Problems dealing with additive decompositions of homogeneous polynomials similar to those we  consider in this section, have a very long story  going back to J.~J.~Sylvester and the Italian school of algebraic geometry of the late 19-th century. In the last decades, these problems received renewed attention due to their potential applications. Namely, homogeneous polynomials can be naturally identified with {\it symmetric tensors} and in several applied branches of science where such tensors are used, for example, to encode multidimensional data sets, {\it additive decompositions of tensors} play a crucial role as an efficient way to code those.
		We refer to \cite{Lan} for an extensive exposition of these connections.
\end{Rmk}

\section{Ideals of generic forms}
Let $I$ be a homogeneous ideal in $S$, i.e., an ideal generated by homogeneous polynomials. The ideal $I$ and the quotient algebra $R = S/I$ inherit the {\it grading} of the polynomial ring.
\begin{defi+}
	Given a homogeneous ideal $I \subset S$, we call the function 	$$

		\HF_R(i) := \dim_{\bC} R_i = \dim_{\bC} S_i - \dim_\bC I_i
	$$
	the {\it Hilbert function} of $R.$
	The power series
	$$
		\HS_R(i) := \sum_{i\in\bN} \HF_R(i) t^i \in \bC[[t]]
	$$
	is called the {\em Hilbert series} of $R$.
\end{defi+}

Let $I$ be a homogeneous ideal generated by  forms $f_1, \dots, f_r$ of degrees $d_1, \dots, d_r$, respectively.
 It was shown in \cite{fr-lo} that, for fixed parameters $(n,d_1,\dots, d_r)$, there exists only a finite number of possible Hilbert series for $S/I$, and that
there is a Zariski open subset in the space of coefficients of the $f_i$'s
on which the Hilbert series of $S/I$ is one and the same and, in the appropriate sense, it is minimal among all possible Hilbert series, see below. We call algebras with this Hilbert series {\it generic}.
There is a longstanding conjecture about this minimal Hilbert series formulated by the first author, see \cite{fr}.

\begin{conjecture}[Fr\"oberg's Conjecture, 1985]\label{conj:fr}
Let $f_1, \dots, f_r$ be generic forms of degrees
$d_1, \dots, d_r$, respectively. Then the Hilbert series of the quotient algebra $R = S/(f_1,\ldots,f_r)$ is given by
\begin{equation}\label{eq:RALF}
\HF_R(t)=\left[\frac{\prod_{i=1}^r(1-t^{d_i})}{(1-t)^n}\right]_+.
\end{equation}
Here
$[\sum_{i\ge0}a_iz^i]_+:=\sum_{i\ge0}b_iz^i$,
with $b_i=a_i$ if $a_j\ge0$ for all $j\le i$ and $b_i=0$. In other words,   $[\sum_{i\ge0}a_iz^i]_+$ is the truncation of a power series at its first non-positive coefficient.
\end{conjecture}

Conjecture~\ref{conj:fr} has been
proven in the following cases: for $r\le n$ (easy exercise, since in this case $I$ is  a complete intersection); for $n\le2$, \cite{fr}; for $n=3$,
\cite{an3}, for
$r=n+1$,  which follows from \cite{st}. Additionally, in  \cite{ho-la} it has been proven that \eqref{eq:RALF}
is correct in the first nontrivial degree $\min_{i=1}^r(d_i+1)$.  There are also other special results in the case  $d_1=\dots =d_r$, see
 \cite{ni, au, fr-ho, mi-mi, ne}.
We should also mention that  \cite{fr-lu} contains a survey of the existing results on the generic series for various algebras and also it studies  the (opposite) problem of finding the maximal Hilbert series for fixed parameters $(n,d_1,\ldots,d_r)$.

\smallskip

It is known that the actual Hilbert series of the quotient ring of any ideal with the same numerical parameters is lexicographically larger than or equal to the conjectured one. This fact implies that if for a  given discrete data $(n,  d_1, \ldots, d_r)$,  one finds just a single example of an algebra with the Hilbert series as in \eqref{eq:RALF}, then Conjecture~\ref{conj:fr} is settled in this case.

\medskip
Although  algebras with the minimal Hilbert series constitute a Zariski open set, they are  hard to find constructively. We are only aware of two explicit constructions giving the minimal series  in the special case $r=n+1$, namely  R.~Stanley's choice $x_1^{d_1}, \ldots,x_n^{d_n}, (x_1+\cdots + x_n)^{d_{n+1}}$, and C. Gottlieb's choice  $x_1^{d_1}, \ldots,x_n^{d_n}, h_{d_{n+1}}$, where $h_{d}$ denotes the complete homogeneous symmetric polynomial of degree $d$, (private communication). To the best of our knowledge,  already in the next  case $r=n+2$ there is no  concrete guess about how to construct a similar example. There is however a substantial computer-based evidence pointing towards the possibility of replacing generic forms of degree $d$ by a product of generic forms of much smaller degrees. We present  some problems and conjectures related to such pseudo-concrete constructions below.

\subsection{Hilbert series of generic power ideals.} Differently from the situation occurring in  R.~Stanley's result, if we consider ideals generated by more than $n+1$ powers of generic linear forms,  there are known examples of $(n,d_1,...,d_r)$ for which algebras generated by powers of generic linear forms  fail to have the Hilbert series as in \eqref{eq:RALF}.

\medskip
Recall that ideals generated by powers of linear forms are usually called {\it power ideals}. Due to their appearance in several areas of algebraic geometry, commutative algebra and combinatorics, they have been  studied more thoroughly. In the next section, we will discuss their relation with the so-called {\it fat points}. (For a more extensive survey of power ideals,  we refer to a nice paper by F. Ardila and A. Postnikov \cite{AP10}.)

\medskip
Studying Hilbert functions of generic power ideals, A.~Iarrobino formulated the following conjecture, usually referred to as the Fr\"oberg-Iarrobino Conjecture, see \cite{ia, Ch}.

\begin{conjecture}[Fr\"oberg-Iarrobino Conjecture]\label{conj:fr-ia}
Given generic linear forms $\ell_1, \ldots, \ell_r$   and  a positive integer $d$, let $I$ be the power ideal generated by $\ell_1^d,\ldots,\ell_r^d$. Then the Hilbert function of $R = S/I$ is as in \eqref{eq:RALF}, except for the cases $(n,r) = (3,7), (3,8), (4,9), (5,14)$ and possibly for $r = n+2$ and $r = n+3$.
\end{conjecture}
This conjecture is still largely open. In \cite{fr-ho} R.F. and J. Hollman checked it for low degrees and low number of variables using the first version of the  software package {\it Macaulay2}. In the last decades, some progress has been made in reformulation of Conjecture~\ref{conj:fr-ia} in terms of the ideals of {\it fat points} and {\it linear systems}. We will return to this topic in the next section.

\subsection{Hilbert series of other classes of ideals}
Computer experiments suggest that in order to always generically get the Hilbert function as in \eqref{eq:RALF} we need to replace power ideals by slightly less special ideals.

\medskip
For example, given a partition $\mu = (\mu_1,\ldots,\mu_k) \vdash d$, we call  by a {\it $\mu$-power ideal}  an ideal generated by forms of the type $({\bf l}_1^\mu,\ldots,{\bf l}_r^\mu)$, where ${\bf l}_i^\mu=l_{i,1}^{\mu_1}\cdots l_{i,k}^{\mu_k}$ and $l_{i,j}$'s are distinct linear forms.
\begin{problem}\label{problem: F}
For $\mu \neq (d)$, does a generic $\mu$-power ideal have the same Hilbert function as in \eqref{eq:RALF}?
\end{problem}

Performed computer experiments  suggest a positive answer to the latter problem. L. Nicklasson has also conjectured that ideals generated  by powers of generic forms of degree $\geq 2$ have the Hilbert series as in \eqref{eq:RALF}.

\begin{conjecture}[\cite{ni}] \label{conj:nic}
For generic forms $g_1,\ldots,g_r$   of degree $d>1$,  the ideal $(g_1^k,\ldots,g_r^k)$
has the same Hilbert series as the one generated by $r$ generic forms of degree $dk$. \end{conjecture}

 It was observed in \cite[Theorem A.3]{LuOnReSh} that Conjecture \ref{conj:nic} implies Conjecture \ref{conj:main}, connecting the two first sections of the present paper. It was also shown that Conjecture \ref{conj:nic} holds in the case of binary form by specializing the $g_i$'s to be $d$-th powers of linear forms and applying the fact that generic power ideals in two variables have the generic Hilbert series \cite{GeSh}. The same idea gives a positive answer to Problem \ref{problem: F}  in the case of binary forms, by specializing $l_{i,1} = \ldots = l_{i,k}$, for  $i = 1,\ldots,r$.

\subsection{Lefschetz properties of graded algebras}
We say that a graded algebra $A$ has the {\it weak Lefschetz property} (WLP) (respectively, the {\it strong Lefschetz property} (SLP)) if the multiplication map $\times l:A_i\rightarrow A_{i+1}$ (respectively,  $\times l^k:A_i\rightarrow A_{i+k}$) has the maximal rank, i.e., it is either injective or surjective, for a generic linear form $l$ and all $i$ (resp., for all $i$ and $k$).
(For more references and open problems about the Lefschetz properties, see \cite{MiNa}.)

\begin{problem} It has been conjectured that each complete intersection $R=S/(f_1,\ldots,f_n)$ satisfies the WLP and also the SLP, see \cite{HMNW}. Does the same hold for $R=S/(f_1,\ldots,f_r)$, with $f_1,\ldots,f_r$ being generic forms, and $r > n$?
\end{problem}

It follows from \cite{St} that monomial complete intersections satisfy the SLP. In \cite{BFL}
the following situation has been studied. For  the ring $T_{n,d,k}=S/(x_1^d,\ldots,x_n^d)^k$, it is shown that for $k\ge d^{n-2}$, $n\ge3$, $(n,d)\ne(3,2)$,  $T_{n,d,k}$
fails the  WLP. For $n=3$, there is an explicit  conjecture when the WLP holds.  Additionally,  there is some information about  $n>3$.

\begin{problem} When are the WLP and the SLP true for $T_{n,d,k}$?
\end{problem}

We now introduce the concept of the \emph{$\mu$-Lefschetz properties}.
Let $\mu=(\mu_1,\ldots,\mu_k)$ be a partition of $d$, i.e.,  $\sum_{i=1}^k\mu_i=d$. We say that an algebra has the {\it $\mu$-Lefschetz property} if $\times{\bf l}^\mu:A_i\rightarrow A_{i+d}$ has maximal rank for all $i$, where ${\bf l}^\mu=l_1^{\mu_1}\cdots l_k^{\mu_k}$, and $l_i$'s are
generic linear forms.

\begin{problem}
For $R= S/(f_1,\ldots,f_r)$, where $f_1,\ldots,f_r$ are generic forms, does $R$ satisfy the $\mu$-Lefschetz property for all partitions $\mu$?
 \end{problem}

\section{Symbolic powers}

For a prime ideal $\wp$ in a Noetherian ring $R$, define  its {\it $m$-th symbolic power} $\wp^{(m)}$ as  $$\wp^{(m)}=\wp^mR_\wp\cap R.$$
It is the $\wp$-primary component of $\wp^m$. For a general ideal $I$ in $R$, its {\it $m$-th symbolic power} is defined as
$I^{(m)}=\cap_{\wp\in{\rm Ass}(I)}(I^mR_\wp\cap R)$.

\subsection{Hilbert functions of fat points.}
Let $I_X$ be the ideal in $\bC[x_1,\ldots,x_n]$ defining a scheme of reduced points $X = P_1 + \ldots + P_s$ in $\mathbb P^{n-1}$, say $I_X = \wp_1 \cap \ldots \cap \wp_s$ where $\wp_i$ is the prime ideal defining the point $P_i$. Then, the $m$-th symbolic power $I^{(m)}$ is the ideal $I_X^{(m)} = \wp_1^m \cap \ldots \cap \wp_s^m$ which defines the scheme of {\it fat points} $X = mP_1 + \ldots + mP_s$.

Ideals of $0$-dimensional schemes are  classical objects of study since the beginning of the last century.  Their Hilbert functions are of particular interest. Study of these ideals and calculation of their Hilbert functions can be often  related to the so-called {\it polynomial interpolation problem}. Indeed, the homogeneous part of degree $d$ of the ideal $I_X$ is the space of hypersurfaces of degree $d$ in $\mathbb P^{n-1}$ passing through the $P_i$'s up to order $m-1$, i.e., the space of polynomials of degree $d$ whose partial differentials up to order $m-1$ vanish at every $P_i$.

It is well-known that the Hilbert function of $0$-dimensional schemes is strictly increasing until it reaches the {\it multiplicity} of the scheme, see \cite[Theorem 1.69]{IK06}. Hence, since the degree of a $m$-fat point in $\mathbb P^{n-1}$ is ${n-1+m-1 \choose n-1}$, the expected Hilbert function is $$\HF_{S/I_X}(d) = \min\left\{{n-1+d \choose n-1}, s{n-1+m-1 \choose n-1}\right\}.$$ In the case of simple generic points, i.e., for $m=1$, it is known that  the actual Hilbert function is as expected.

In the case of double points ($m=2$), counterexamples were known since the end of the 19-th century. In 1995, after a series of important papers, J. Alexander and A. Hirschowitz proved that the classically known examples were the only counterexamples. For higher multiplicity, very little is known at present. In the case of projective plane, a series of equivalent conjectures have been given by  B. Segre \cite{Seg61}, B. Harbourne \cite{Har86}, A. Gimigliano \cite{Gim87} and A. Hirschowitz \cite{Hir89}. These are known as the {\it SHGH-Conjecture}, see \cite{Har00} for a survey of this topic.

\smallskip
{\it Apolarity Theory} is a very useful tool in studying of the ideals of fat points and it is  connecting  all the algebraic stories we have told above. In particular, the following lemma is crucial.
(We refer to \cite{IK06} and \cite{Ger96} for an  extensive description of this issue.)

\begin{Lemma}[Apolarity Lemma]
	Let $X = P_1+\ldots+P_s$ be a scheme of reduced points in $\mathbb P^{n-1}$ and let $L_{1},\ldots, L_{s}$ be linear forms in $\bC[x_0,\ldots,x_n]$ such that, for any $i$, the coordinates of $P_i$ are the coefficients of $L_i$. Then, for every $m\geq d$,
	$$
		\HF_{S/I^{(m)}_X}(d) = \dim_{\bC}[(L_1^{m-d+1},\ldots,L_s^{m-d+1})]_d.
	$$
\end{Lemma}

Using this statement we obtain that calculation of  the Hilbert function of a scheme of fat points is equivalent to the calculation of  the Hilbert function of the corresponding power ideal. In particular, Fr\"oberg-Iarrobino conjecture (Conjecture \ref{conj:fr-ia}) can be rephrased as a conjecture about the Hilbert function of ideals of generic fat points.

\medskip
Recently R.F. raised the question about what happens in case of the ideals of generic fat points in a multi-graded space. A point in multi-projective space $P \in \mathbb P^{n_1-1}\times\ldots\times \mathbb P^{n_t-1}$ is defined by a prime ideal $\wp$ in the multi-graded polynomial ring $S = \bC[x_{1,1},\ldots,x_{1,n_1};\ldots;x_{t,1},\ldots,x_{t,n_t}] = \bigoplus_{I \subset \bN^t} S_I$, where $S_I$ is the vector space of multi-graded polynomials of multi-degree $I = (i_1,\ldots,i_t) \in \bN^t$. A scheme of fat points $X = mP_1+\ldots+mP_s$ is the scheme associated with the multi-graded ideal $\wp_1^m\cap\ldots\cap\wp_s^m$.

\begin{problem}
	Given a scheme of generic fat points $X \subset \mathbb P^{n_1-1}\times\ldots\times \mathbb P^{n_t-1}$, what is the multi-graded Hilbert function $\HF_{S/I_X}(I)$, for $I \in \bN^t$?
\end{problem}

This question was first considered by M. V. Catalisano, A. V. Geramita and A. Gimigliano who solved it in the case of double points, i.e., for $m=2$ in $\mathbb P^1 \times \ldots \times \mathbb P^1$. Recently, A.O. jointly with E. Carlini and M. V. Catalisano    resolved the case of triple points ($m=3$) in $\mathbb P^1 \times \mathbb P^1$ and computed the Hilbert function for an arbitrary multiplicity except for a finite region in the space of multi-indices, see \cite{CCO17}.

\subsection{Symbolic powers vs. ordinary powers.}
As we mentioned above, if $I$ is the ideal defining  a set $X$ of points, the $m$-th symbolic power of $I$ is the ideal of polynomials vanishing up to order $m-1$ at all points in $X$ or, in other words, the space of hypersurfaces which are singular at all points in $X$ up to order $m-1$. For this reason,  symbolic powers are interesting from a geometrical point of view, but they are more difficult to study  compared to the usual powers which carry  less geometrical information.  Hence, it is important to find relations between them.  Observe that the inclusion $I^m \subset I^{(m)}$ is trivial.

\smallskip
{\it Containment problems} between the ordinary and the symbolic powers of ideals of points have been  studied in substantial details. One particularly interesting question is  to understand for which pairs of positive integers $(m,r)$,  $I^{(m)} \subset I^r$. A very important result in this direction  is the fact that, for any ideal $I$ of reduced points in $\mathbb P^n$ and any $r > 1$, we have $I^{(nr)}\subset I^r$. This statement was proven in \cite{ELS01} by L. Ein, R. Lazersfeld and K. Smith for characteristic $0$  and by M. Hochster and C. Huneke in positive characteristic, see \cite{HH02}. At present, the important question is whether the bound in the latter statement is sharp. In \cite{DSTG13}, M. Dumicki, T. Szemberg and H. Tutaj-Gasi\'nska provided the first example of a configuration of points such that $I^{(3)} \not\subset I^2$. (We refer to \cite{SS17} for a complete account on this topic.)

\medskip
In the recent paper \cite{GGSVT16}, F. Galetto, A. V. Geramita, Y. S. Shin and A. Van Tuyl defined the {\it $m$-th symbolic defect of an ideal} as the number of minimal generators of the quotient ideal $I^{(m)} / I^m$. If $I$ defines a set of general points in projective space, it was already known that $I^{(m)} = I^m$ if and only if  $I$ is a complete intersection. Additionally, in \cite[Theorem 6.3]{GGSVT16} the authors characterize all cases of $s$ points in $\mathbb P^2$ having the $2$-nd symbolic defect equal to $1$. These cases are exactly $s = 3,5,7,8$.

\begin{problem} For the ideal $I$ of $s$ general points in $\mathbb P^{n-1}$, what is the difference between the Hilbert series of the $m$-th symbolic power and the $m$-th ordinary power?
\end{problem}

\section{Miscellanea}
\subsection{Hilbert series of numerical semigroup rings}
Let $\mathcal{S}=\langle s_1,\ldots,s_k\rangle$ be a numerical semigroup, i.e. $\mathcal{S}$ consists of all linear combinations with non-negative integer coefficients of the positive integers
$s_i$, and let $k[x^{s_1},\ldots,x^{s_k}]=k[\mathcal{S}]$ be the semigroup ring. The Hilbert series of $k[\mathcal{S}]$ is of the form $p(t)/q(t)$, where $p,q$ are polynomials with integer coefficients.
A semigroup is called \emph {cyclotomic} if the polynomial $p(t)$ has all its roots in the unit circle (which in fact, implies that they lie on the unit circle). (Detailed information about numerical semigroups can be found in  e.g., \cite{Ci}.)

\begin{conjecture} $\mathcal{S}$ is cyclotomic if and only if $k[\mathcal{S}]$ is a complete intersection.
\end{conjecture}

\subsection{Non-negative forms}  The next circle of problems is related to the celebrated article  \cite{Hi} of D.~Hilbert and to a number of results formulated in  \cite{CLR}.

Denote by  $P_{n,m}$ the set of all non-negative real forms, i.e., real homogeneous polynomials  of (an even) degree $m$ in $n$ variables which never attain negative values; denote by $\Sigma_{n.m}\subseteq P_{n,m}$ the subset of non-negative forms which can be represented as sums of squares of real forms of degree $\frac{n}{2}$.
(In \cite{Hi} D.~Hilbert proved that $\Delta_{n,m}=P_{n,m}\setminus\Sigma_{n,m}$ is non-empty unless the pair $(n,m)$ is of the form $(n,2)$, $(2,m)$  or $(4,3)$.)  Finally, if $\Z(p)$ stands for the real zero locus  of a real form $p$,  denote by $B_{n,m}$ (resp. $B'_{n,m}$) the supremum of $|\Z(p)|$ over $p\in P_{n,m}$ such that $|\Z(p)|<\infty$
(resp. over $p\in\Sigma_{n,m}$ such that $|\Z(p)|<\infty$). In other words, $B(n,m)$ is the supremum of the number of zeros of non-degenerate forms under the assumption that all these roots are isolated (and similarly for
$B'_{n,m}$).  Obviously, $B'_{n,m}<B_{n,m}$.

\medskip
The following basic question  was posed in \cite{CLR}.

\begin{problem} Are  $B_{n,m}$ and $B'_{n,m}$ finite for any pair $(n,m)$ with even $n$? \end{problem}

In \cite{CLR} it was shown that the answer to this problem is  positive  for $m=2,3$ and for the pair $(4,4)$. Relatively recently,  in \cite{Stu} the following upper bound for $B_{n,m}$ was established
$$B_{n,m}\le 2\frac{(m-1)^{n+1}-1}{m-2}.$$
However this bound can not be sharp, as shown in \cite{Ko}.  In case of $B'_{n,m}$, the following guess seems  quite plausible and is proven for $m=3$.

\begin{conjecture}  For any given pair $(n,m)$ with even $n$, $B'_{n,m}=\left(\frac{n}{2}\right)^{m-1}$.

\end{conjecture}

For $B_{n,m}$, no similar guess is known, but some intriguing information  is available  in the case $m=3$, see \cite{CLR}. The following problem is related to the classical Petrovski-Oleinik upper bound on the number of real ovals of real plane algebraic curves.

\begin{problem} Determine $\lim_{n\to\infty}\frac{B_{n,3}}{n^2}$. \end{problem}

The latter limit exists and lies in the interval  $\left[\frac{5}{18},\frac{1}{2}\right]$, see  \cite{CLR}.

\subsection{Polynomial generation}
Let $p$ be a prime number and let $\mathbb{F}_p$ denote the field with $p$ elements. Consider the two maps
$$\phi: \mathbb{F}_p[x_1,\ldots,x_n] \to \mathbb{F}_p[x_1,\ldots,x_n], f \mapsto \sum_{a \in Z(f)} x^a,$$
$$\psi: \mathbb{F}_p[x_1,\ldots,x_n] \to \mathbb{F}_p[x_1,\ldots,x_n], f \mapsto \sum_{a \in \mathbb{F}_p^n} f(a) x^a.$$
Here  $x^a := x_1^{a_1} \cdots x_n^{a_n}$, where each $a_i$ is regarded as an integer, and $Z(f)$ is the zero locus of $f$ in $F_p^n$, i.e.,  $Z(f) := \{a \in \mathbb{F}_p^n \,|\, f(a) = 0\}$.
When $p = 2$, then $\phi$ is a bijective map on the vector space of polynomials of degree at most one in each variable, and $\phi^4(f) = f$, see \cite{lu}.
The map $\psi$, suggested by M. Boij, is a linear bijective map on the vector space of polynomials with degree at most $p-1$ in each variable, and when $p = 2$, these two maps are closely related in the sense that
$\phi(f) = \psi(f) + \sum_{a \in \mathbb{F}_2^n} x^a.$

Consider now the case $n = 1$ and $p > 2$. The map $\phi$ is no longer a bijection, but the sequence $\phi(f),\phi^2(f),\ldots$ will eventually become periodic. It is an easy exercise to show that $0 \mapsto 1 + x + \cdots + x^{p-1} \mapsto x \mapsto 1 \mapsto 0$. When $p \leq 17$, this is the only period, i.e., $\phi(f)^{d(f)} = 0$ for some $d(f)$. For $p=71$,  we have found a period of length two;
$1 +x^{63}
\mapsto x^{23} + x^{26} + x^{34} + x^{39} + x^{41} + x^{51} + x^{70} \mapsto 1 + x^{63}.$ One can show that the length of the period is always an even number, but it is not clear which even numbers that can occur as lengths of periods.

\begin{problem}
For $n=1$ and given $p$, what are the (lengths of the) possible periods of $\phi$?
\end{problem}

Let us now turn to the map $\psi$ and the case $n=1$. For $p =3$,  $\psi^8(f) = f$ for all polynomials $f$ in $\mathbb{F}_3[x]$ of degree at most two.  For $p = 5$, the least $i$ such that $\psi^{i} = {\rm Id}$ on the space of polynomials of degree at most four, is equal to $124$. For $p = 7$, the corresponding number is $1368$.

\begin{problem}
For $n = 1$ and given $p$, find the minimal positive integer  $i$ such that $\psi^i$ is the identity map on the space of polynomials of degree at most $p-1$.
\end{problem}

\subsection{Exterior algebras}
Let $f$ be a generic form of even degree in the exterior algebra $E$ over $\mathbb{C}$ with $n$ generators. Moreno and Snellman showed that the Hilbert series of $E/(f)$ is equal to the expected series $[(1+t)^d(1-t^d)]_+$, see \cite{M-S}. When the degree of $f$ is odd, we have $(f)\subseteq{\rm Ann}{f}$. This annihilator ideal shows an unexpected behaviour. The most striking case is when
$(n,d) = (9,3)$. It turns out that $\dim_{\mathbb{C}}({\rm Ann}{f})_3 = 4$, see \cite{lu-ni}. At the same time a naive guess is that $({\rm Ann}{f})_3$ is spanned by $f$ only.
Additionally, computer experiments suggest that $(f)$ and ${\rm Ann}{f}$ agree in low degrees.

\begin{problem} Let $f$ be a form of odd degree $d$ in $E$.  Is it true that $({\rm Ann}(f))_i = (f)_i$, for $i < (n-d)/2$? \end{problem}

\medskip
We finish our list of problems  with the following conjecture stated in  \cite{cr-lu-ne}, which connects the question about the Hilbert series of generic forms in the exterior algebra with the Hilbert series of power ideals in the commutative setting.
\begin{conjecture}
Let $f$ and $g$ be generic quadratic forms in $E$ and let $\ell_1$ and $\ell_2$ be two generic linear forms in $S$. Then the Hilbert series of $E/(f,g)$ is equal to the Hilbert series of $S/(x_1^2,\ldots,x_n^2,\ell_1^2,\ell_2^2)$ and is given by
$1 + a(n,1) t + a(n,2) t^2 + \cdots + a(n,s)t^s + \cdots$, where $a(n,s)$ is the number of lattice paths inside the rectangle $(n+2-2s)\times (n+2)$ starting from the bottom-left corner and ending at the top-right corner by using only moves of two types: either $(x,y)\rightarrow (x+1,y+1) \textrm{\ or\ } (x-1,y+1)$.

\end{conjecture}

\medskip\noindent
{\bf Acknowledgements.}   The authors want to  thank all the participants of the problem-solving seminar at Stockholm University for their contributions and patience. The
fourth author is sincerely grateful to Dr.~Kh.~Khozhasov for pointing out references \cite{Stu, Ko}.