Operating successfully in a world with lots of theories and little data

You may have seen the recent brouhaha in the financial press about a 2010 research paper by Carmen M. Reinhart and Kenneth Rogoff of Harvard which found a correlation between high levels of government debt and low or negative economic growth. This paper was important because it underpinned the austerity policies adopted by many governments around the world since the financial crash – not least here in the UK. The brouhaha has come about because economists from the University of Massachusetts have repeated their analysis using the same data but different weightings and found that the median growth at highly indebted economies was much higher.

In the meantime other studies of different data have apparently found similar results to the original 2010 paper from Reinhart and Rogoff.

This is a big deal because whether to pursue austerity policies is perhaps the most important decision facing most developed economy politicians right now.

The problem is that there isn’t much data to go on and the data quality itself is questionable. There haven’t been many periods in history when governments were as indebted as they are now, and it is impossible to know if economics have changed to a sufficient extent that historical data isn’t relevant. If we understood economics well enough to have a causal theory linking debt and growth then minimal data proving that theory would be helpful, but in the absence of that causal theory we are simply looking at correlations of limited data sets. In this situation people with convictions based on their political persuasion use data to press their case rather than to determine the right answer. A very dangerous game.

This problem also exists in business where most of our ‘understanding’ is based on observed correlations with little understanding of causal relationships. With the best intentions in the world this can lead to slavish following of trends to the detriment of business performance. The problem is, if anything, more pernicious in business because the advice of most business gurus includes is backed up by some level of causal logic as well as observed correlations with success. The 1970-2000 mantra that a simple focus on shareholder value was the best way for companies to generate success is a good example. It started with an observation that there was a correlation – successful companies like Coca Cola were shareholder value focused – and then added the causal theory to back it up (in simplified form a company’s shareholders want value, which can only come from generating good cash and profits, which are the essence of a sustainable business), but the correlation has recently fallen apart and the causal link between a focus shareholder value and long term success is now hotly disputed. The problem arose because the causal theory was based on an erroneous assumption that shareholders would distinguish between short term profit creation that destroyed long term value and truly building for the long term.

So what to do?

I think there is huge value in drawing insight from the limited data available but it’s critical to sense check the conclusions that others come to before following their advice. The most important sense check is whether it feels right – intuitively it makes sense to me that indebted countries will grow more slowly because they are having to use more of their resources to repay debt and hence have less to promote growth. In a business context this intuition often has its roots in the vision and values of a company. Advice that is inconsistent with that world view won’t feel right.

The next sense check is the credibility of the theorist – most importantly, are they impartial, looking at the data and searching for conclusions, or did they start with the conclusion and then go looking for data to back it up.

Thirdly – do research. If an idea is important then it is worth investing time to form a considered view.

Finally – stay open minded and be prepared to try things and change. Don’t make the oh-so-common mistake of ignoring everything that doesn’t fit with existing beliefs (behavioural psychologists call this the ‘confirmation bias’) and be prepared to disucss half formed opinions and lightly held convictions. In a fast moving world with little data it’s the only way to progress.

These ideas apply to many decisions that startups have to take on an ongoing basis – should I have a freemium business model? should I sell director or via channel? should I pursue growth over profitability? etc. etc. etc.