Decremental learning. Contrary to conventional machine learning tasks, decremental learning attempts to remove the impact of training data from the trained model efficiently. This line of research dates back to some early work on incremental and decremental versions of Support Vector Machines [1]}, [2]}, while it has recently gained traction due to the rising concerns for privacy in ML models. Ginart et al. [3]} study data deletion mechanisms for variations of k-means clustering. Schelter et al. [4]} propose decremental learning methods for a selection of non-iterative ML models such as item-based collaborative filtering and KNN. Recent work from Guo et al. [5]} studies this problem for neural networks and shows promising theoretical results albeit apparent scalability bottlenecks due to the requirement of reversing Hessian matrix. Bourtoule et al. [6]} propose a framework in which training samples are divided into shards and trained in silo to obtain a combination of weaker learners, and later on, combined to form a stronger model. When there is a deletion request concerning specific user data, it's sufficient to update the corresponding impacted weaker learner.
