I just read Wyatt Jenkins’ 10 A/B testing lessons I learned the hard way. Here’s number one:
1. It’s a hypothesis. Don’t fall in love. Most tests don’t turn out the way you plan them. There’s a roughly 70% chance that you are wrong. Try lots of ideas quickly and cheaply. Consider excluding difficult browsers like IE and excluding customer segments that introduce lots of edge cases. Do what it takes to get a test out fast.
In common with many people in the startup ecosystem we are big believers in keeping things lean. At it’s heart this means being clear about your assumptions and viewing the startup process as a learning journey. We even structure our due diligence process to tease out all the assumptions a startup is making and then help build a workplan to validate them.
What we’ve seen is that despite best intentions it’s incredibly easy to become fixed on a given strategic choice – e.g. which customer segment to go for, product features, company positioning, tone of voice, or elements of the visual identity. My colleague Dharmesh is fond of saying that it’s important to test every new idea within a couple of weeks because after that the brain somehow starts to see it as permanent and stops wanting to test it. I like to explain aversion to changing plans as an emotional version of the sunk cost fallacy – where the cost is the emotion we’ve invested in arriving at and supporting the idea.
It is easy to write a blog post saying we shouldn’t fall in love with our hypotheses, but in practice it’s difficult not to, constantly questioning everything is tiring, and when we pitch our ideas we want to stay consistent with them. As with many aspects of good execution the key is discipline, in focusing in on the most important assumptions and leaving the rest till later, and in staying with it even when you’d rather be doing something else. Having other people in your team to keep you honest also helps.
But even if it is hard, Jenkins is right, we shouldn’t fall in love with our hypotheses. Keeping our decision making rational massively increases our chances of success, and remember that our hypotheses are hypotheses is one way to do that.