How might novel strategies for uncovering hidden data help us navigate uncertainty? When planning for the future, hidden and unknown variables that influence its course are always present if we know where to look.
By sharpening organisational tools to catch emerging signals of change, companies can prepare for the future and anticipate the unexpected. In fact, as this article shows, the unknown often turns out to be merely the unacknowledged.
Evolving perspectives of uncertainty
It’s no mystery that the coronavirus pandemic has reshaped the way we think about uncertainty. In management literature, uncertainty is understood as a state of imperfect or complete lack of information about a present or future event, and is often divided into three types: foreseen uncertainties, unforeseen uncertainties and chaos, the pure unknown. Risk management practices tend to identify risk with uncertainty, but scholars point out that the constructs are different.
Risk can be quantified numerically by calculating the probability of an event based on previous data, and models and scenarios can be built that fit previous experience. On the other hand, uncertainty is a state in which we do not necessarily have the advantage of historical data or wisdom derived from past experience. Uncertain events are often too novel, complex and ambiguous to predict.
Covid-19 context and pandemic planning
In the context of the pandemic, debates have arisen about whether the event could be defined as a “black swan”, a highly unpredictable event outside the realm of regular expectations or a or a “grey rhino”, a highly probable yet neglected event.
Alas, there is no doubt that chaos is the condition society and businesses experienced during these months, as the pandemic forced us into a novel and uncharted territory for modern humanity. Despite that, some businesses and societies have done better than others. Why is that?
How can we plan the night before in a really careful sort of way that takes into account the unfortunate things that might happen, so that we can react very quickly when the day comes and these uncertain things happen?
Imperial’s Professor Ruth Misener’s work with Royal Mail and other businesses plans uncertainty into algorithms to ensure the best decisions can be made as actions beyond control come to impact outcomes. She joined Dr Michel-Alexandre Cardin from Imperial’s Dyson School of Design Engineering and Maria Angeles Diaz at an Imperial Business Partners event about the topic of decision making under uncertainty.
Debates have arisen about whether the pandemic is a “black swan”, a highly unpredictable event, or a or a “grey rhino”, a highly probable but neglected event.
As Dr Jack Jacquier, Reader in Mathematical Finance at Imperial College London, noted in a recent interview for Imperial Tech Foresight: ‘statistical predictions depend on the data used to train the model, as the model is limited to the training set, extreme and outlier events are not predicted.
Limited data based on geographical historical evidence gave some indication of how the pandemic might spread – but not the full picture that was needed.
Historical data on SARS was geographically limited and insufficient in quantity to allow us to build precise models to make predictions and forward plans for the novel coronavirus.’ Looking at the future, he says novel AI techniques such as GAN — generative adversarial networks — that operate on quantum computers will be able to tackle data input scarcity by generating similar, mimicking examples of limited time-series (sets of data points) to feed the statistical models. These can provide novel information that can be used to build more accurate predictions and scenarios, overcoming data set limitations arising from insufficient input quantity.
Missing data could be crucial
Professor David Hand, in his book Dark Data, shows how missing or overlooked data can be more important than the data we draw on.
Increasing a business’s ability to identify relevant information and to generate and transform data into models, is fundamental to reducing and tackling uncertainty. Often though, crucial data is already available to companies and data scientists but is overlooked. Donald Rumsfeld, in his famous 2002 briefing at the Pentagon, described known knowns, known unknowns and unknown unknowns. What’s unexplored in this categorisation, despite complementing the concepts of foreseen uncertainty, unforeseen uncertainty or chaos, is the idea of “unknown knowns”.
Data about events that we don’t know that we (could) know, or that we have but don’t recognise. This enters in the realm of dark data, data that is missing, that is hidden, not analysed or that businesses collect but don’t pay attention to, the blind spots that can lead to erroneous assumptions and conclusions.
Knowing where to look for insights
Nokia failed to recognise 200 patents that Apple filed around the iPhone as a signal of potential disruption to their business. Working on the assumption that Apple was a computer company, the signals were beyond their competitive radar.
In Dark Data, Professor Hand shows how missing or overlooked data can be more important than the data we draw on. He provides a taxonomy of types of dark data, showing how they can be recognised and used to our advantage. Stressing the importance of information generation, he describes statistical techniques to generate data that could have arisen in the observations — such as creating extra randomly perturbed copies of data sets or copies of misclassified data points — that can fill information gaps and improve statistical models. Recognising and controlling for dark data can help us take appropriate decisions even in the face of ignorance.
As a concrete example of unknown knowns around competitive and technological uncertainties in industry, consider the 200 patents that Apple filed around the iPhone, which Nokia failed to recognise as a signal of potential disruption to their business. Working on the assumption that Apple was a computer company, the signals were beyond their competitive radar. Another example is Blockbuster, which failed to foresee and adapt to the impact of e-commerce and on-demand streaming on their business, allowing Netflix to pioneer this space. They failed to ask a simple question:
– What could be the new emerging ways of delivering the same service? According to Imperial’s Professor Jan Ross, one basic strategy for dealing with uncertainty and environmental change is imitation. This heuristic works both with close competitors and distant ones (e.g. Nokia and Apple).
Strategic imitative co-opetition (cooperation and competition) might result in companies reinforcing their market position, as happened during recent years between Samsung and Apple. This is especially relevant in the context of the coronavirus, where states failed to quickly imitate the strategies adopted by early hit nations.
Indicators of disruption: Certainty in times of crisis
Business environments transform because of technology, climate change, geopolitical shifts, consumer preference under quickly shifting trends. Disruptions also lead to behavioural uncertainty. Often businesses assume consumers to behave rationally, but as the pandemic taught us, people — especially under disruptions — tend to make decisions based on sentiments and gut-feeling.
A concrete example was the panic-buying of toilet paper or the over stocking of food. More research and attention should be paid to these kinds of unknown knowns, the psychology of decision-making processes influenced by biases, ideology, instinct, groupthink or sentiments such as fear and greed, but also fairness and ingenuity.
Solutions to the challenge
How can we best manage uncertainties and the unknown? One solution is to map the source and impact of contingencies that could arise in the interconnected and interdependent ecosystem in which companies operate.
Most importantly, we can create frameworks to move from risk management to uncertainty management.
In fact, the ability to benchmark uncertainties from low to high degree and gauge their potential impact is essential to assess the eventual cost and urgency of appropriate countermeasures. Organisational capacity to influence the occurrence and mitigate the impact of uncertain events highly depend on their source. Different approaches are required for internal and external uncertainties.
While internal uncertainties should be approached with a cause-driven response, aiming to reduce gaps of knowledge within organisational processes, external types need a consequence-based one, anticipatory and adaptive with respect to ecosystem changes.
An AI epidemiologist developed by Canadian company BlueDot, a health monitoring platform, spotted early warning signals of the epidemic days before the word ‘outbreak’ started to spread on official media.
To deal with uncertainties, companies should excel in knowledge-generation and social learning, paying attention to data and dark data, in addition to emerging trends and technologies. Managers should be ready to challenge assumptions and question their long-term hypotheses, creating a culture of innovation and information sharing.
As Professor Mark Girolami stressed during his Imperial Tech Foresight talk on quantifying uncertainty in 2018, subjective assumptions change expected outcomes, ‘we need to know what we don’t know.’ Internally, data-driven analysis of processes and technologies such as digital twins that can perform 3D simulations of very complex scenarios, promise to greatly reduce various blind-spots and minimize uncertainties across all levels of a project.
Redesigning our approach to uncertainty
Design thinking techniques are also important instruments to explore uncertainties arising from socio-technical interactions between product and customer, revealing potentially unexpected downsides in design that could lead to market failures. Externally, AI algorithms can be used for trendspotting and early-warning signal detection, as they can automatically scan the web for novel and unacknowledged patterns.
An example is the AI epidemiologist developed by Canadian company BlueDot, a health monitoring platform, that spotted early warning signals of the epidemic days before the word “outbreak” started to spread on official media and warned its customers ahead of time.
The unknown often hides an opportunity. By actively investigating the future with speculative scenarios and narratives, foresight methodologies can help businesses to transform unknown potential into competitive advantage.
Contingency theory states that the effectiveness of an organization is related to its “fit” with the environment. Different conditions require different ways of organising processes and the project’s success depends on how appropriate the project organisation is, relative to the emergent prevailing external conditions. Rapidly changing environments require rapid adaptation of the organisational ecosystem, quick trial and error cycles and flexible management, maximising variability of output with minimum disruptions.
Companies need to build a flexible, redundant and adaptive infrastructure for their businesses, to be able to act quickly and collectively across the whole organisation as the challenge arises. Looking where they “least wanna look”, reframing (e.g. Blockbuster vs Netflix), overcoming blindspots and biases such as overconfidence. It’s fundamental to stress knowledge boundaries to find new sources of uncertainty, contingencies, but also possibilities for innovation and upheaval.
How research leads to strategy, and how hindsight leads to foresight
Businesses should perform exploratory modelling, scenario discovery and scanning for early warning signals, understanding also the potential of systems thinking and counterfactuals to reveal unexpected angles to problems.
Fostering a type of learning that is Bayesian in nature, organisations should update their knowledge and beliefs about the world and the markets by constantly incorporating new and relevant information as it becomes available. Imperial Tech Foresight adopts methods such as horizon scanning, STEEPV, integral futures, causal layered analysis, to provide frameworks by which environmental change can be anticipated and organisational fitness updated. Pockets of the future are already embedded in the present, this is true every time we observe a disruptive event in hindsight, tracing its emergence. Often the unknown is potential that we can’t intuitively grasp, chaos that hides opportunity.
By identifying key questions and actively investigating the future with speculative scenarios and narratives, foresight methodologies can help businesses to transform unknown potential into competitive advantage. Dealing with complexity requires an understanding of multiple futures and disruptions and the exploration of a variety of options that might create more resilient systems.
When events that can’t be predicted happen, anti-fragility — the ability to thrive and grow under uncertainty — comes from the capacity to quickly react and adapt to novel environments and the ability to absorb failures and learn from them.
Having a knowledge edge is fundamental. Data-driven planning should reduce timescales and predictive models need to update very quickly — incorporating less historical long-run data and more fresh inputs — while imitation, as a crucial heuristic, should always be part of the strategy of businesses.