When dealing with time series with complex non-stationarities, low retrospective regret on individual realizations is a more appropriate goal than low prospective risk in expectation. Online learning algorithms provide powerful guarantees of this form, and have often been proposed for use with non-stationary processes because of their ability to switch between different forecasters or experts''. However, existing methods assume that the set of experts whose forecasts are to be combined are all given at the start, which is not plausible when dealing with a genuinely historical or evolutionary system. We show how to modify the fixed shares'' algorithm for tracking the best expert to cope with a steadily growing set of experts, obtained by fitting new models to new data as it becomes available, and obtain regret bounds for the growing ensemble.
Turning to more conventional approaches, econometrics has a large literature on detecting non-stationarity (of the basically-harmless integrated'' type characteristic of random walks), and finding structural breaks'' (change points), after which models must be re-estimated or re-specified @cite_3 . Economists do not seem to have considered an ensemble method like ours, perhaps due to their laudable (if unfulfilled) ambition to capture the exact data-generating process in a single parsimonious model. Similarly, most work on data-set shift and concept drift in machine learning @cite_13 deals with how a single model should be learned (or modified) so as to be robust to various changes in the joint distribution of inputs and outputs. Unlike all these approaches, we do not have to assume that any of our models are well-specified, nor assume anything about the nature of the data-generating process or how it changes over time.
@cite_13 @cite_3
2162651021 1559361175
Dataset shift is a common problem in predictive modeling that occurs when the joint distribution of inputs and outputs differs between training and test stages. Covariate shift, a particular case of dataset shift, occurs when only the input distribution changes. Dataset shift is present in most practical applications, for reasons ranging from the bias introduced by experimental design to the irreproducibility of the testing conditions at training time. (An example is -email spam filtering, which may fail to recognize spam that differs in form from the spam the automatic filter has been built on.) Despite this, and despite the attention given to the apparently similar problems of semi-supervised learning and active learning, dataset shift has received relatively little attention in the machine learning community until recently. This volume offers an overview of current efforts to deal with dataset and covariate shift. The chapters offer a mathematical and philosophical introduction to the problem, place dataset shift in relationship to transfer learning, transduction, local learning, active learning, and semi-supervised learning, provide theoretical views of dataset and covariate shift (including decision theoretic and Bayesian perspectives), and present algorithms for covariate shift. Contributors: Shai Ben-David, Steffen Bickel, Karsten Borgwardt, Michael Brckner, David Corfield, Amir Globerson, Arthur Gretton, Lars Kai Hansen, Matthias Hein, Jiayuan Huang, Takafumi Kanamori, Klaus-Robert Mller, Sam Roweis, Neil Rubens, Tobias Scheffer, Marcel Schmittfull, Bernhard Schlkopf, Hidetoshi Shimodaira, Alex Smola, Amos Storkey, Masashi Sugiyama, Choon Hui Teo Neural Information Processing series Economies evolve and are subject to sudden shifts precipitated by legislative changes, economic policy, major discoveries, and political turmoil. Macroeconometric models are a very imperfect tool for forecasting this highly complicated and changing process. Ignoring these factors leads to a wide discrepancy between theory and practice. In their second book on economic forecasting, Michael Clements and David Hendry ask why some practices seem to work empirically despite a lack of formal support from theory. After reviewing the conventional approach to economic forecasting, they look at the implications for causal modeling, present a taxonomy of forecast errors, and delineate the sources of forecast failure. They show that forecast-period shifts in deterministic factors--interacting with model misspecification, collinearity, and inconsistent estimation--are the dominant source of systematic failure. They then consider various approaches for avoiding systematic forecasting errors, including intercept corrections, differencing, co-breaking, and modeling regime shifts; they emphasize the distinction between equilibrium correction (based on cointegration) and error correction (automatically offsetting past errors). Their results on forecasting have wider implications for the conduct of empirical econometric research, model formulation, the testing of economic hypotheses, and model-based policy analyses.