
model architecture: the design of a model as a set of trainable parameters without specification on the exact weight of the parameters [1]}.

loss landscape: the geometry of the global loss associated with a model architecture[2]}.

model instance: given a model architecture with certain amount of trainable parameters, a set of weights associated to the trainable parameters defines an instance of model; during the training process, each optimization step leads to a model instance.

optimization[1]}: neural network is trained by optimizing a object function, usually in the form of minimizing a loss function.

global/local optimum: under the optimization formalization, the optimum point is at the end of the optimization process. As explained in [4]}, in an ideal world where the complete distribution of data is available to train a model, the optimum is global; while any stop point for any certain subdistribution is a local optimum.

model initialization: optimization always starts with an initialization of an model instance.

pretraining[5]}: train a model on a pretext task before training on the target task; the trained model instances will become the initialization of the target task.

finetuning[5]}: train a model on a target task with initialization pretrained task.

Independent and Identifically distributed (IID)[7]}: if given a set of data \({x_i}\) , each of these \(x_i\)  observations is an independent draw from a fixed (“stationary”) probability distribution.

Out-of-distribution (OOD) generalization[8]}: Generalization consists in reducing the gap in performance between training data and testing data. When the data generating process from the training data is indistinguishable from that of the test data, it's “in-distribution”, if not, it's called out-of-distribution generalization problem [8]}. As in [10]}, considering datasets \(Data_{e}:=\lbrace (x_{i}^{e},y_{i}^{e} )\rbrace _{i=1}^{n_{e}}\)  collected under multiple domains \(e\) , each containing samples IID with a probability distribution \(D(X^{e},Y^{e})\) . The goal of OOD generalization is to use these datasets to learn a predictior \(Y\approx f(X)\) , which performs well across a large set of unseen but related domains \(e\in \mathcal {E}_{all}\) . Namely, the goal is to minimize
\(R^{OOD}(f) = \max _{e\in \mathcal {E}_{all}} R^{e}(f)\) 
, where
\(R^{e}(f) := E_{X^{e}, Y^{e}}[l(f(X^{e}),Y^{e})]\) 
is the risk under domain \(e\) . Here the set \(\mathcal {E}_{all}\)  contains all possible domains.

generalization[1]}: the most general goal of generalization is to enable the model make reliable prediction on unseen data; out-of-distribution prediction (OOD) is a more challenging type of generalization problem which requires the model to be generalizable to unseen data distribution.

mini-batch[1]}: as a common practice for robustness and computation memory concerns, no matter how large a data set is available, only a sub set of data is sampled to train a model at each optimization step.

representation[1]}: as coined by the line of work named “representation learning”, the word representation is interachangable with “embedding”, referring to a vector/matrix of learned features.

transfer model parameter: a technique that is related to pretraining-finuetuning; for the implementation, a simple initialization of part of the target model using the pretrained model could serve the purpose.

