1. A method comprising: maintaining a model at an online system, the model receiving a plurality of features and generating a response based on the received features; determining a set of new features available for the model based on data maintained by the online system; determining a superset of features for the model, the superset of features comprising the plurality of features used by the model and the new features available for the model; determining measures of feature impact for the features in the superset of features, a measure of feature impact for a feature corresponding to a contribution of the feature to an error term of the model proportional to a change in the error term when the feature is removed from the model; generating a first group of features from the determined superset of features for the model, each feature in the first group of features having a measure of feature impact at least exceeding a first measure of feature impact threshold; generating an additional group of features from the determined superset of features, each feature in the additional group of features having a measure of feature impact at least exceeding a second measure of feature impact threshold that is different relative to the first measure of feature impact threshold, the additional group of features including the first group of features and an additional set of features in addition to the first group of features; training a first version of the model using the first group of features and a second version of the model using the additional group of features, each group of features resulting in a performance metric for the model; and presenting, for display in a user interface, information describing the performance metric for each version of the model.