Good startup hypotheses must be falsifiable

Karl Popper was perhaps the pre-eminent philosopher of science of the 20th century and central to his work is the notion that for a theory to be useful it must be falsifiable. Benedict Evans posted this quote on Twitter yesterday in a debate about the applicability of Clayton Christensen’s innovators dilemma theory to the iPhone. I’ve reproduced it here because it’s useful for startups generally.

Astrology did not pass the test. Astrologers were greatly impressed, and misled,by what they believed to be confirming evidence_so much so that they were quite unimpressed by any unfavourable evidence. Moreover, by making their interpretations and prophecies sufficiently vague they were able to explain away anything that might have been a refutation of the theory had the theory and the prophecies been more precise. In order to escape falsification they destroyed the testability of their theory. It is a typical soothsayer’s trick to predict things so vaguely that the predictions can hardly fail: that they become irrefutable.

The Marxist theory of history, in spite of the serious efforts of some of its founders and followers, ultimately adopted this soothsaying practice. In some of its earlier formulations (for example in Marx’s analysis of the character of the “coming social evolution’) their predictions were testable, and in fact falsified. Yet instead of accepting the refutations the followers of Marx reinterpreted both the theory and the evidence in order to make them agree. In this way they rescued the theory from refutation; but they did so at the price of adopting a device which made it irrefutable. They thus gave a “conventionalist twist” to the theory; and by this stratagem they destroyed its much advertised claim to scientific status.

Steve Blank defines a startup as an organisation built to search for a repeatable and scalable business model that starts with a series of hypotheses about pieces of their business model. Best practice is to quickly test the hypotheses and move the company around the build measure learn loop as efficiently as possible. Doing this well requires a lot of discipline, and Steve Blank has talked about many aspects of that at length, especially having the discipline to get out of the room and test hypotheses by speaking with customers.

One element of discipline that doesn’t get talked about much is to making the hypotheses falsifiable. Popper had to write about this in the context of science because it somewhat goes against the grain of human nature. To make a hypotheses testable is to invite failure and that is difficult, even for those with the best of intentions, as can  be seen from the example with the Marxists above. Moreover, the older and more established a hypothesis (or company) is the easier it is for proponents to slip into vagueness.

With startups the challenge of maintaining falsifiability is more difficult than it is in science. As well as the having the discipline of regularly looking failure in the eye founders have to operate in a fast moving world where the evidence base and hence hypotheses are anyway continually evolving. The laws of physics are the same today as they always have been whilst founders change their hypotheses in real time as they learn about their market.

In practice maintaining falsifiability means defining targets for things like conversion rates, CPAs, and even positive responses from customer interviews and committing to re-examining key hypotheses if the numbers come in below target. At Forward Partners Right from the earliest stages we encourage companies to have a broad range of targets, and we define success as making good progress towards all of them, or maybe nailing half of them and giving up on the other half, all the while staying in constant dialogue and being prepared to switch out targets or change the levels as we learn. Failure is, of course, the inverse of success.