A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization Problems in Unsupervised Learning <s> Scientific measurements frequently involve large numbers of variables whose complex interactions are not easily found. A practical computing method termed ISODATA, which finds the cluster structure of such data, is described. The resulting description of the data provides a fit to the data of a set of cluster centers that tends to minimize the sum of the squared distances of each data point from its closest cluster center. An application to the grouping or clustering of the answers of 209 people to an 80-question sociological survey illustrates the utility of the method. <s> BIB001 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization Problems in Unsupervised Learning <s> It has often been asserted that since hierarchical clustering algorithms require pairwise interobject proximities, the complexity of these clustering procedures is at least O(N 2 ). Recent work has disproved this by incorporating efficient nearest neighbour searching algorithms into the clustering algorithms. A general framework for hierarchical, agglomerative clustering algorithms is discussed here, which opens up the prospect of much improvement on current, widely-used algorithms. This 'progress report' details new algorithmic approaches in this area, and reviews recent results. <s> BIB002 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization Problems in Unsupervised Learning <s> Abstract Principal component analysis of a data matrix extracts the dominant patterns in the matrix in terms of a complementary set of score and loading plots. It is the responsibility of the data analyst to formulate the scientific issue at hand in terms of PC projections, PLS regressions, etc. Ask yourself, or the investigator, why the data matrix was collected, and for what purpose the experiments and measurements were made. Specify before the analysis what kinds of patterns you would expect and what you would find exciting. The results of the analysis depend on the scaling of the matrix, which therefore must be specified. Variance scaling, where each variable is scaled to unit variance, can be recommended for general use, provided that almost constant variables are left unscaled. Combining different types of variables warrants blockscaling. In the initial analysis, look for outliers and strong groupings in the plots, indicating that the data matrix perhaps should be “polished” or whether disjoint modeling is the proper course. For plotting purposes, two or three principal components are usually sufficient, but for modeling purposes the number of significant components should be properly determined, e.g. by cross-validation. Use the resulting principal components to guide your continued investigation or chemical experimentation, not as an end in itself. <s> BIB003 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization Problems in Unsupervised Learning <s> Principal component analysis (PCA) is a ubiquitous technique for data analysis and processing, but one which is not based upon a probability model. In this paper we demonstrate how the principal axes of a set of observed data vectors may be determined through maximum-likelihood estimation of parameters in a latent variable model closely related to factor analysis. We consider the properties of the associated likelihood function, giving an EM algorithm for estimating the principal subspace iteratively, and discuss the advantages conveyed by the definition of a probability density function for PCA. <s> BIB004 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization Problems in Unsupervised Learning <s> When large multivariate datasets are analyzed, it is often desirable to reduce their dimensionality. Principal component analysis is one technique for doing this. It replaces the p original variables by a smaller number, q, of derived variables, the principal components, which are linear combinations of the original variables. Often, it is possible to retain most of the variability in the original variables with q very much smaller than p. Despite its apparent simplicity, principal component analysis has a number of subtleties, and it has many uses and extensions. A number of choices associated with the technique are briefly discussed, namely, covariance or correlation, how many components, and different normalization constraints, as well as confusion with factor analysis. Various uses and extensions are outlined. ::: ::: ::: Keywords: ::: ::: dimension reduction; ::: factor analysis; ::: multivariate analysis; ::: variance maximization <s> BIB005
Clustering algorithms  , BIB002 ,  , BIB001 divide a group of samples into multiple clusters ensuring that the differences between the samples in the same cluster are as small as possible, and samples in different clusters are as different as possible. The optimization problem for the k-means clustering algorithm is formulated as minimizing the following loss function: where m is the number of clusters, x is the feature vector of samples, µ i is the center of cluster i, and S i is the sample set of cluster i. The implication of this objective function is to make the sum of variances of all clusters as small as possible. The dimensionality reduction algorithm ensures that the original information from data is retained as much as possible after projecting them into the low-dimensional space. Principal component analysis (PCA) BIB003 , BIB005 , BIB004 is a typical algorithm of dimensionality reduction methods. The objective of PCA is formulated to minimize the reconstruction error as where M represents the number of samples, e j is the base vector, x i is a D-dimensional vector, x i is the reconstruction of Another optimization goal is to find an optimal probability density function of p(x), which maximizes the logarithmic likelihood function (MLE) of the training samples, In the framework of Bayesian methods, some prior distributions are often assumed on parameter θ, which also has the effect of alleviating overfitting.