If you are building an app with limited time and money, experiments are a necessity. It is invaluable to learn what works, and what doesn't, before you invest time and money into an idea.
It's true that some people get it correct the first time. But, that is rare and risky to assume that you got it correct too. Running experiments helps you gain confidence that you are creating a correct solution.
In this post, I will outline the components of a Lean experiment. I use Lean experiments because they emphasize spending the least amount of time to build a testable solution.
Setting Up a Lean Experiment
A Lean experiment follows a straightforward path:
- Establish your riskiest assumption
- Identify the customer behavior associated with that assumption
- Commit to a single metric you believe in
- Design a solution
- Capture your hypothesis
- Build the minimum viable feature/product
- Document the results
- Record your observations
- Add your insights
Let's go more into detail about each part of a Lean experiment.
Identify the Riskiest Assumption
The riskiest assumption is an assumption you are making about a key-stone element of your product. You accept it as true without evidence. It can take form as a belief about your audience, a recognition of a business necessity, a projection about market size, and many other things. It is best to start with the riskiest assumptions first. If we don't address this assumption, this business idea might fail.
The riskiest assumption is not easy to find [1]. When trying to find your riskiest assumption, start with universal risks. Be sure to use your intuition and challenge your own assumptions. Follow up by talking with domain experts.
The Customer's Action
How does your riskiest assumption relate to your customer’s actions?
Let’s pretend that your riskiest assumption is "customers will trust your service with their personal information". The related action is the customer giving you their personal information.
When creating a Lean experiment, you want to identify an action that you can test and measure. Some examples:
- Choses option a over b
- Pays you for the product
- Signs up for a newsletter
- Comes back to the application the next day
- Refers a friend
The best types of actions are real “currency” exchanges. This is when a person give you something of value to them. It demonstrates a high level of honesty in their action. The more currency that the person has to spend to complete this action, the more value we can associate with that action.
Target Metric
Commit to a minimum success threshold. Choose a number high enough to get you and your team excited if true and relative to the assumption you are testing.
Experiment
Build the absolute minimum required to test your assumption, as fast as possible. Don’t just confirm, learn. Include ways to capture surprises as well as real behavior important to validating your business activity is successful in influencing customer behavior.
Hypothesis
A specific description of the customer behavior you’re measuring, your target metric and the experiment you’re running to test your assumption. If we do X, then Y% of customers will behave in way Z.
Results
What actually happened? Record the actual metrics generated during your experiment, paying close attention to new behaviors and surprises. For example, 50% (5/10) people behaved in way Z.
Observations
Why did the customer behave this way? Record the observable behavior, live feedback and reactions you witnessed firsthand, which influenced the actual results.
Insights
Insights change the way you think about your customers or your idea. Record the new information you generated during this experiment, as well as unique observations made in the field.
Decision
Was your riskiest assumption true? Look for a trend in your evidence over time. No single experiment holds all the answers. Iterate? Persevere? Pivot?
If you are running many experiments it can be a good idea to create a visual map of those experiments so your team can be aware of the progress you are making [2].
Summary
Remember, there is no perfect solution, but there are certainly ones that are better than others. Don't assume that you got it right. Assume that you don't know, but the people using your application will tell and show you what "right" feels like.