Good startup hypotheses must be falsifiable

Karl Popper was perhaps the pre-eminent philosopher of science of the 20th century and central to his work is the notion that for a theory to be useful it must be falsifiable. Benedict Evans posted this quote on Twitter yesterday in a debate about the applicability of Clayton Christensen’s innovators dilemma theory to the iPhone. I’ve reproduced it here because it’s useful for startups generally.

Astrology did not pass the test. Astrologers were greatly impressed, and misled,by what they believed to be confirming evidence_so much so that they were quite unimpressed by any unfavourable evidence. Moreover, by making their interpretations and prophecies sufficiently vague they were able to explain away anything that might have been a refutation of the theory had the theory and the prophecies been more precise. In order to escape falsification they destroyed the testability of their theory. It is a typical soothsayer’s trick to predict things so vaguely that the predictions can hardly fail: that they become irrefutable.

The Marxist theory of history, in spite of the serious efforts of some of its founders and followers, ultimately adopted this soothsaying practice. In some of its earlier formulations (for example in Marx’s analysis of the character of the “coming social evolution’) their predictions were testable, and in fact falsified. Yet instead of accepting the refutations the followers of Marx reinterpreted both the theory and the evidence in order to make them agree. In this way they rescued the theory from refutation; but they did so at the price of adopting a device which made it irrefutable. They thus gave a “conventionalist twist” to the theory; and by this stratagem they destroyed its much advertised claim to scientific status.

Steve Blank defines a startup as an organisation built to search for a repeatable and scalable business model that starts with a series of hypotheses about pieces of their business model. Best practice is to quickly test the hypotheses and move the company around the build measure learn loop as efficiently as possible. Doing this well requires a lot of discipline, and Steve Blank has talked about many aspects of that at length, especially having the discipline to get out of the room and test hypotheses by speaking with customers.

One element of discipline that doesn’t get talked about much is to making the hypotheses falsifiable. Popper had to write about this in the context of science because it somewhat goes against the grain of human nature. To make a hypotheses testable is to invite failure and that is difficult, even for those with the best of intentions, as can  be seen from the example with the Marxists above. Moreover, the older and more established a hypothesis (or company) is the easier it is for proponents to slip into vagueness.

With startups the challenge of maintaining falsifiability is more difficult than it is in science. As well as the having the discipline of regularly looking failure in the eye founders have to operate in a fast moving world where the evidence base and hence hypotheses are anyway continually evolving. The laws of physics are the same today as they always have been whilst founders change their hypotheses in real time as they learn about their market.

In practice maintaining falsifiability means defining targets for things like conversion rates, CPAs, and even positive responses from customer interviews and committing to re-examining key hypotheses if the numbers come in below target. At Forward Partners Right from the earliest stages we encourage companies to have a broad range of targets, and we define success as making good progress towards all of them, or maybe nailing half of them and giving up on the other half, all the while staying in constant dialogue and being prepared to switch out targets or change the levels as we learn. Failure is, of course, the inverse of success.

  • Denny Wong

    Interesting read and concept. Similarly like the saying “difference between the journey and destination”, the key in the”falsifiable” concept is in the journey.

    My takeaways are:

    Firstly, the action of testing (hence the effort to falsify or validate the different assumptions) and the ability to adapt, correct or pivot is the key to success.

    Secondly, which is tougher. Is for the founder to have the courage and humbleness to know that one does not know all. Especially during the selection and interpretation of metrics. While getting brutally honest feedback (weighing, internalising and acting). Followed by measuring it.

  • OK, but every technology takes time to be adopted. Founders these days are giving up and pivoting very quickly, sometimes you need to grind instead of pivot. Read Fred’s Wilson post here:

    If you are obsessed with the problem you are solving you should keep trying until you nail it. Most of our numbers used to be below our targets during the 1.5 year, if we heard some of the investors and advisors recommendations we would gave up long time. Just lately things started to work and we are getting high conversion rates for some channels such as SEO and PPC. I think founders should stay focused and never get pushed to give up or declare failure if they have the will to wakeup everyday in the morning and work for 12hrs until they run out of money.

    Still enjoying your posts Nic since I first found you 5 years ago 🙂

  • Hi Abdullah – glad you like the blog! Thanks for reading.

    I agree founders shouldn’t give up too early, but at the same time some companies aren’t meant to be. Hypotheses should be large and small, all falsifiable. If the small hypotheses fail their tests then it is no reason to pivot or give up e.g. if hte hypothesis is that display retargeting will dive CPAs down and it doesn’t, then look for another marketing hypothesis, or go to work on conversion or something, don’t give up. However, if after thorough testing a large hypothesis fails – e.g. we will drive customers via digital marketing – then a rethink is required. The judgement call is knowing when a hypothesis has been thoroughly tested.