SPIES: A Framework For Improving Forecasting & Decisions Under Uncertainty

This article has been republished from New World Navigator, our Strategy and AI newsletter, a publication dedicated to helping organisations and individuals navigate disruptive change. Sign up here to receive and participate in thought-provoking discussions, expert insights, and valuable resources!

SPIES is not exactly the 007 type of spy, but actually a simple yet powerful tool for helping to improve the quality and accuracy of forecasting under uncertainty.

Let’s face it. Most of us are pretty rubbish at assessing uncertain outcomes. Sadly, my perennially dismal attempts at stock-picking and “timing the market” are classic examples. Fortunately, there are tools and frameworks which can guide us to improve the accuracy of our forecasts (which I may do well to start applying to my stock-picking attempts!).

The Pitfalls of Decision-Making

Firstly, the bad news. Human beings are remarkably poor at assessing uncertain outcomes, especially when we do not have historical data or a reliable frame of reference. Even when we do, we are often skewed by various biases. These include recency bias, where we give greater weight to recent trends, and sunk cost fallacy, where we make non-optimal choices simply because we are unconsciously motivated by a need to justify our past choices (even when the past choices have been invalidated by present circumstances), among many others. We also have a predilection for underestimating, and therefore outright dismissing, the likelihood of “black swan” events, which are extremely high-impact events that have a seemingly low (but non-negligible) probability of occurrence.

The consequences can be dire when we underestimate the likelihood of “black swan” events. The financial crisis of 2008 (precipitated by the “unexpected” collapse of the subprime mortgage market), as well as the meltdown of Japan’s Fukushima nuclear reactor (resulting from the failure of the reactor’s design to accommodate “unlikely” earthquakes above 7.9 on the Richter scale) were catastrophic and a direct result of decision-makers not sufficiently accounting and preparing for seemingly low probability events.

Despite our fallibilities in this regard, the need to make subjective decisions under uncertainty nonetheless remains a critical and unavoidable part of business and more broadly, life. Nearly all decisions we make involve some degree of uncertainty, and the level of uncertainty tends to increase with seniority in responsibility and age. Experience (or “gut feel”) can certainly carry you some way toward better judgments, but not in situations where a reliable frame of reference is missing.

I recently worked with senior management at a client to evaluate the probabilities of occurrence associated with various scenarios and projects. Although subjective in nature, this exercise helped to surface implicit assumptions, thereby facilitating constructive debate on steps that each team could take to improve each project’s likelihood of success. While many elements of the projects were familiar to the management team, several projects were some way outside of their usual focus areas, which made their experience and frame of reference significantly less relevant to evaluating probabilities.

I SPIES A Solution

But here is the good news. Subjective Probability Interval Estimates or SPIES (in my opinion, one of the best acronyms of all time) is a tool that can be helpful in such instances and, more broadly, when assessing probabilities. For those of us who are more statistically-inclined, SPIES is a tool for subjectively estimating confidence intervals (CI). A CI indicates the probability that a certain parameter will fall between a set of values. Taking one of my real-life examples, I am confident that for 9 out of 10 trips I make from my home to the local swimming pool, my journey will take 25 minutes or less, implying a 90% CI of 25 minutes.

Simple to use and adopt in any organisation or project, SPIES has also been demonstrated to be hugely effective. A Carnegie Mellon University study for instance, has shown that forecasts made using SPIES were more than twice as accurate as compared to other methods (or when no specific method is used). Specifically, when asked to estimate temperatures, study participants using SPIES estimated CIs that included the correct answer 74% of the time, as opposed to 30% of the time when SPIES was not used.

The Benefits of SPIES

In most decision-making processes, we tend to draw ranges too narrowly when assessing potential outcomes, with two binary endpoints (e.g. on-time or delayed, best case or worst case scenario) being the most common approach taken. Yet as the world is seldom black and white, so are potential outcomes seldom simply “1” or “0”.

SPIES is so effective because it prompts us to consider the full range of potential outcomes. Instead of settling on binary scenarios and outright dismissing certain outcomes as improbable and not planning for them to occur, the SPIES framework invites decision-makers to accommodate and plan for a range of scenarios within a probability-based framework. Consequently, decision-makers are more likely to end up allocating appropriate time and resources to address both common occurrence outcomes as well as low-probability but high impact scenarios (i.e., “black swan” events).

Estimating Confidence Intervals with SPIES

Let’s adopt a relatable example to illustrate how SPIES could be used. Assume that you are the scheduler for the train company that operates the rail between London to Glasgow, and you are looking to estimate potential journey times. Unfortunately massive and unprecedented train strikes have been announced (a phenomenon that those of us in the UK have become very well-acquainted with over the past few months!). Historical train journey times, even from previous smaller scale strikes, are of course no longer representative.

The first step in SPIES involves defining a set of intervals that will cover the full range of possible outcomes. The full range of possible outcomes for the duration of the journey could potentially extend from 6.5 hours (everything goes swimmingly and the train is slightly ahead of time) to several days (Murphy’s Law: never underestimate the potential for things to go wrong!).

Using SPIES, we would demarcate intervals (e.g., 0–7 hours, 7–9 hours, 9–12 hours, 12–16 hours, 16–24 hours, 1–2 days, 2+ days etc.) covering the full range of potential outcomes. The next step would be to estimate probabilities for each of the intervals (e.g., 25% probability for 0–7 hours, 20% probability for 7–9 hours etc.). We would then aim to arrive at a confidence interval (usually 90–95%) for when the train is expected to arrive at Glasgow. This example is illustrated in the table below where the 90% confidence interval for the duration of the train journey is 24 hours or less.

In the absence of SPIES, we might have been tempted to settle into a default binary assessment comprising a simple “best case” (e.g., the train is not late or only slightly late) and “worst case” (e.g., the train is 3–4 hours late). We would probably have dismissed (or not even considered) what we thought were edge (but high impact) cases, such as the entire network becoming crippled due to cascading effects arising from multiple simultaneous strikes, which would have resulted in the train journey taking upwards of 16 or even 24 hours.

Hypothetical example of subjective estimates of potential duration for a train journey from London to Glasgow under strike conditions, and the resulting 90% confidence interval (i.e., 24 hours or less).

How to Use SPIES to Your Advantage

SPIES is particularly salient for situations that are novel, where historical precedent and human experience are unavailable or irrelevant. Trying to ascertain how human society might react to the arrival of an alien on Earth is one (extreme) example. Within the context of business, an example could include a company looking to predict customers’ reactions to the adoption of a novel, highly unusual product. I have also personally used SPIES to estimate the potential duration of a project and for project budget forecasting purposes, to good effect.

As with any forecasting method, SPIES is subject to the adage of “garbage in, garbage out”. Since empirical data is not (or is less likely to be) an input into SPIES, debate and discourse is particularly important. As was the case with the senior management team at my client organisation, having an open discussion about subjective probability estimates should prompt implicit assumptions and biases to surface, ultimately leading to a recalibration and improvement of the forecast.

Last but not least — and this one applies to all forecasts, rather than just those that are SPIES-derived — make sure to review your forecasts after the dust has settled. In such post-mortem exercises, whether your prediction actually came to pass is less important than a reflective examination of whether the forecast was “contaminated” by hidden assumptions and unspoken biases.

Happy SPYING!