An optional step in the TinyML workflow is to directly reduce the dimensionality of the data. Linear methods for dimensionality reduction commonly used in large-scale data-mining include matrix factorization and principal component analysis (PCA) techniques such as singular value decomposition [1]}, flattened convolutions [1]}, non-negative matrix factorization [3]}, independent component analysis [4]}, and linear discriminant analysis [5]}. PCA is used to maximize the preservation of variance of the data in the low-dimensional manifold [6]}. Models operating on intrinsic dimensions of the data are computationally tractable and mitigate the curse of dimensionality. Non-linear methods to handle non-linear sampling of low-dimensional manifolds by high-dimensional vectors include uniform manifold approximation and projection [7]}, t-distributed stochastic neighbor embedding [8]}, locally linear embedding [9]}, kernel PCA [10]}, and autoencoders [11]}. Non-linear methods minimize a distance metric (e.g., fuzzy embedding topology [7]}, Kullback-Leibler divergence [13]}, local neighbourhoods [9]}, and Euclidean norm [10]}) between the high-dimensional data and a low-dimensional latent representation. Feature engineering uses domain expertise to extract tractable features from the raw data [16]}. Domain-specific feature extraction is generally more suited for microcontrollers over linear and non-linear dimensionality reduction techniques. Feature selection can reduce the number of redundant features further during model development [17]}. Feature selection methods include statistical tests, correlation modeling, information-theoretic techniques, tree ensembles, and metaheuristic methods (e.g., wrappers, filters, and embedded techniques) [18]}.
