Empty Crystal Ball on White Background Photo: Marco Verch - License: CC BY 2.5
Anyone can make a prediction but how good are they?
January 20, 2020By Tom Austin1 CommentReading time: 4 minutes
It's hard to make meaningful, substantive, accurate long-term predictions. Change happens — unexpected change.
Predictions are harbingers of change. It's easy to predict from the past: Extrapolate an unchanging future. You may be right more than you're wrong, but your misses will kill you. Unexpected change ruins predictions and predictors.
Predictions can be of specific events, such as when the Dow Industrial Index hits 36,000, or broader trends, like the ascendance of cloud computing or the shift away from globalization.
Predicting new trends before they're obvious is far more valuable to long term planning than predicting a particular future event.
Specific future values can be important too, for instance, to stock day-traders and sports-bettors. But for people making strategic, long term decisions, trends are more useful. They have shape, size, texture, and other dimensions. They imply a whole range of changing values and related clusters of predictions.
Meta-trends consist of several underlying trends that, together, drive other trends. Human generational change has been happening forever. It's a frothy wave on the horizon, with youth challenging experience, as in "OK, Boomer!" discounting the wisdom of older generations. That meta-trend is an unfair caricature of various generations because of individual differences. But there's some wisdom in caricatures as well.
The most actionable predictions deal with future states that are significantly different than the current state
How to evaluate the prediction waves that break on the news shore every January?
1. Evaluate completeness.
Examine carefully whether or not the analysis provides:
A description of the trend or event itself, with appropriate antecedents and dependencies
Evidence, both supporting and contradictory with open access to data and algorithms where relevant
Impact assessment: Opportunities and threats posed by the change
Likelihood factors and the product of likelihood times impact
Alternative potential outcomes and influencers
Magnitude and rate of change over time – perhaps predicting when the trend will hit both the toe and knee of the curve
Other significant waypoints and indicators
Recommendations for various constituencies
Don't expect to see many predictions anywhere that fully meet those requirements.
2. Dig out often unstated assumptions.
Example: AI solves the prediction problem
Given the "AI Revolution," isn't it possible someone could develop an AI model to predict the long-term future?
Unfortunately, AI models are trained with data from the past, not data from the future! For a taste of the problem this presents, consider the Wall Street Journal's recent article "Use AI for Picking Stocks? Not So Fast"[1], which shows that no matter how much training and back-testing you do of a model, in the real world, it can't reliably predict the long-term future. (It's a different story for very short-term events, as in algorithmic and high-frequency trading, but HFT predates recent hype around using AI for picking stocks and, in any case, its predictive validity is generally measured in microseconds.)
The AI boosters will point to experiments like DeepMind's AlphaGo, a game playing "AI" that came up with chess and Go strategies that challenged the conventional wisdom of world champions and textbooks in ways to beat the games' world champions. Certainly, that creativity proves we could build an AI to predict the future in more worldly arenas. Right? Wrong again, I'm afraid. AlphaGo dealt with a limited search space and could develop far more sophisticated solutions than humans. But predicting the future in the real-world often requires evaluating an infinite search space, something doable only with an infinite amount of time.
3. Assume there is no such thing as magic
Can we flag a new trend and analyze its implications if we don't understand the technology well enough to be able to explain how it works in practical, defensible terms? It's currently vogue to talk about AI and Machine Learning as technology classes that learn on their own and AI as a form of intelligence. These metaphors, of artificial intelligence and machines that learn, lead people to erroneous conclusions.
It would be less deceptive if they just said it's magic, as in Arthur C. Clarke's statement
Any sufficiently advanced technology is indistinguishable from magic. [2]
Don't accept magic. Look for the hidden human operator behind the curtain (as in the movie the Wizard of OZ) or find very real, practical implementations that are in use and seek out opposing points of view on what's really going on.
4. Consider various conventions
These conventions have little predictive value, but they can be used to test the prediction and stimulate the formation of alternative hypotheses.
A. Amara's law [3] says, "We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run." This is a very clever observation. It plays to our cognitive biases but offers no ways to calibrate the size or timing of the over and underestimates. A variant on this law is the observation we tend to overestimate the speed with which a technology will diffuse through the economy in the short run.
B. The "OK, boomer" meta-trend (introduced above)
Every generation creates its own world view – one that's different from that of previous generations. This process is an essential source of variation, testing, challenging, rejecting, adjusting, and just disrupting the heck out of everything prior generations had already figured out. At least half of whatever the kids are doing, particularly early technology adoption patterns, is going to stick and become the "new normal" – only to be displaced by the next generation's rebellion. The trick, of course, is figuring out which half of whatever they're doing is going to disrupt the status quo and come to dominate conventional wisdom 20 years hence.
C. It takes a lot of failures before a big new idea finally catches on
Many presumed trends die in their first incarnation, only to be successfully resurrected decades later.
Witness "network computers" circa 1996. Miserable failure. Twenty years later? That's what tablets in general and Chromebooks as a variant, really are.
Application Service Providers (ASPs) were the next big thing in the late '90s. SaaS was their reincarnation a decade later.
The Internet of Things (IoT) died a quiet death late last century, only to be reborn 20 years later. Perhaps overhyped and at emerging more slowly than expected, it seems to have finally taken root (and will grow as edge computing technology and 5G emerge.)
5. Go beyond the predictions.
Don't just examine the predictions and data — history and motives matter.
Anyone who's 100% accurate isn't trying hard enough.
Likewise, a source always swimming with the current — or always contrary.
Look for evidence of both false positives and false negatives.
[1] https://www.wsj.com/articles/use-ai-for-picking-stocks-not-so-fast-11578279960?mod=hp_jr_pos1
[2] https://en.wikipedia.org/wiki/Clarke%27s_three_laws
[3] https://en.wikipedia.org/wiki/Roy_Amara#Amara's_law
Prev postPrev post:
Analyst Syndicate CEO and Founder, former world-wide Head of Research and Fellow at Gartner for almost 25 years. Brain sciences academic background. At Gartner, Tom drove research and advisory services on Office Systems, Client/Server, Network Computing, Workgroup Systems, Portals, Content and Collaboration, Social Software, Cloud Computing, and Artificial Intelligence. He is probably most renowned for his hugely successful disruptive Maverick innovation research program. Six weeks after leaving Gartner in June 2018, he began planning the Analyst Syndicate which went commercial October 2019 and now has over 30 senior analysts. See his work on thansyn.com and follow his posts on LinkedIn.
The views and opinions in this analysis are my own and do not represent positions or opinions of The Analyst Syndicate. Read more on the Disclosure Policy.
Tom Austin's Tweet-Wrangling Newsletter #2 - The Analyst Syndicate
[…] [23] https://thansyn.com/anyone-can-make-a-prediction-but-how-good-are-they/ […]
Keep me informed about:
Protect, preserve and defend
Disruption Pandemic Planning Business Strategy coronaviris playbook acquisition COVID-19 Coronavirus Machine Learning digital transformation Artificial Intelligence Remote work CIO Security IBM AI supply chain Cybersecurity WFH risk management data privacy