In general, the tremendous success and achievements of the many flavors of machine learning (ML) are based on variations of gradient descent algorithms that minimize some version of a cost or loss function ([1]}, [2]}. At their core, existing algorithms take advantage of the stochastic convergence of weights in neural networks, with individual nodes typically expressed as some version of the classical perceptron model, and the network's ability to capture latent non-trivial statistical associations that encode inputs. A fundamental limitation, however, is the need to train these systems in either supervised or unsupervised ways by exposing them to large numbers of training examples. In some situations these limitations pose significant issues, such as when there simply is not sufficient data (or enough high quality data) for training, or when tasks such as learning and classification need to be done `on the fly' in near real time to support just-in-time inference or decision making. In addition to requiring large amounts of labeled data for training, state-of-the-art ML models such as GPT3, at 175 billion parameters ([3]}), require a huge compute infrastructure. The compute cost of the GPT3 training cycle alone is estimated to be $4.6M (https://lambdalabs.com/blog/demystifying-gpt-3/). Current large data, large compute, and large model trends in ML will not scale. And while some existing ML can perform near real time learning, it still requires expensive pre-trained models. Clearly a more data, compute and energy efficient paradigm for ML is needed.
