Learn to Earn: An Experimentation Guide
How can we be sure that the features we are developing will deliver value?
This is a problem we’ll quickly run into when we start thinking about delivering value instead of features. The answer is simple: we often can’t know in advance what is valuable. Yet, it’s risky to proceed when we don’t. We should make sure we’re not wasting resources chasing after the wrong things; to allow much more resources to build the right thing.
This is why we need to experiment. Experiments are at the core of the scientific method, which is undoubtedly a valid method for generating evidence.
When there’s a lack of evidence, we have to treat ideas differently than if we knew they’re true. We have to treat them as assumptions. As the Lean Start-Up movement suggests, we can express our assumptions as part of a hypothesis. Then we can run an experiment to test that hypothesis and see whether our assumptions were right or wrong. The insights we’ll get from the generated evidence will reduce the risk and uncertainty of our idea.
Experimentation is all about building to learn, to hit the nail more effectively when we’re building to earn.
Experimentation Guidelines
There are plenty of ways to run experiments, but they all must be used appropriately and in the proper context.
Here are some rules of thumb:
- Pick the right experiment. There are many experiences to choose from, as we’ll see right after. We can find the one right for our context by answering these questions: What type of hypothesis are we testing? How much evidence do we already have? How much time do we have until the next decision point or until we run out of money?
- Go cheap and quick at the beginning. At the start, we generally know little and just want to discover more. Later we can run more experiments to produce stronger evidence for validation. Nevertheless, we should always pick the experiment that produces the strongest evidence given the context.
- Evidence doesn’t speak on its own. We need to analyze it to get insights that will support or refute the hypothesis we’ve been testing. Then we’ll decide if we’re going to persevere with the idea, pivot it to a new trajectory, or kill it.
- Experiments are not long-lasting solutions. We need to limit its exposure to customers and be transparent on why we’re doing it and what we plan to do next. If the idea succeeds, we must figure out how to make it sustainable and scalable.
Types of Experiments
As I said, there are many types of experiments to choose from. We can find an exhaustive list in Testing Business Ideas. And we can also use our own creativity to develop different ones if they fit our learning purpose best.
Here, I’ll brief some I’m familiar with, organized by the purpose they might fit best:
- Discovery and Ideation
- Demand Validation
- Solution Testing
- Success Evaluation
During the Discovery and Ideation phase, we can use more generative experiments. They’ll help us gain awareness around customer problems and solution desires.
📋 Survey
Evidence Strength■■□□□□□□□□Resources Needed ■■■■□□□□□□
Collecting information through a closed-ended questionnaire from a sample of customers. It’s ideal for when we already have qualitative insights from other methods that don’t scale.
👂Interview
Evidence Strength■■■■□□□□□□Resources Needed ■■□□□□□□□□
Talking with our customers. It sounds straightforward on paper, but there are a few hacks we need to know to make these conversations worth it. It’s ideal for gaining qualitative insights into the fit between our value proposition and the customer segment.
👾 Forums Analysis
Evidence Strength■■■■□□□□□□Resources Needed ■■■■□□□□□□
Searching in discussion forums to find what people are saying about our product or competitors’ products. It’s ideal for finding out about patterns of workarounds customers do to get the product to do what they need.
📞 Sales / Support Feedback
Evidence Strength■■■■□□□□□□Resources Needed ■■□□□□□□□□
Using the sales force and customer support feedback when they exist. It’s ideal for uncovering unmet needs, pains and gains.
👥 Shadowing
Evidence Strength■■■■■■□□□□Resources Needed ■■■■□□□□□□
Working alongside or observing a customer for the entire day. Also known as ethnography. Needless to say that it requires consent. It’s ideal for learning about customer behavior in the real world.
💁♂ Concierge
Evidence Strength■■■■■■■■■■Resources Needed ■■■■□□□□□□
Offering one-on-one manual help to accomplish the task our proposed feature or product would do. It’s labor-intensive and doesn’t scale, but we get plenty of direct feedback to later iterate on. It’s ideal for learning firsthand whether what we are doing is helpful and necessary and what steps we’ll need to go through.
During the Demand Validation phase, we can use experiments about selling the proposed feature or product and figuring out who buys it. They’ll help us to gain awareness around customer segments and to triage ideas.
🖼 Landing Page
Evidence Strength■■■■□□□□□□Resources Needed ■■■■■□□□□□
Pitching all of the benefits of a future product on a simple landing page, with a call to action to register interest (just like the Fake Door). We’ll have to drive traffic into it, e.g., with a Campaign. It’s ideal for determining if our value proposition resonates with our customer segment.
📢 Campaign
Evidence Strength■■■■■■□□□□Resources Needed ■■■■■■□□□□
Running online/social media/e-mails campaigns that clearly articulates a value proposition for a targeted customer segment. We’ll need a destination for the target audience to visit once they click the ad, e.g., a Landing Page. It’s ideal for quickly testing our value proposition with a customer segment.
🚪 Fake Door
Evidence Strength■■■■■■■□□□Resources Needed ■■□□□□□□□□
Placing a button that supposedly links to a new feature. When the customers click that button, it registers that they did it and explains that this is yet to come. As people believe the button is real, we can consider their behavior valid. It’s ideal for triaging nice-to-have features.
🔗 Referral Program
Evidence Strength■■■■■■■■□□Resources Needed ■■■■■■■□□□
Promoting the product to new customers through word of mouth or digital codes. It requires passionate customers and discounts. It’s ideal for testing with customers how to organically scale our business.
💰Crowdfunding
Evidence Strength■■■■■■■■■■Resources Needed ■■■■■■■■■□
Funding a product by raising many small amounts of money from a large number of people. Typically through a pre-order page. It’s ideal not only for funding but also to know if our potential customers are committed.
During the Solution Testing phase, we should use more evaluative experiments. They’ll help us to learn if a specific solution and its experience resonate with our customers.
✏️ Prototype
Evidence Strength■■■■□□□□□□Resources Needed ■■■■□□□□□□
Sketching the interface on paper. Or representing it with clickable zones to simulate reactions to interaction. Prototypes can be built with various degrees of fidelity. Then we interview customers around them to see what got them excited and what did confuse them. It’s ideal to rapidly test concepts, and user flows.
🧙♀️ Wizard of Oz
Evidence Strength■■■■■■■■■■Resources Needed ■■■■■■■□□□
Creating a customer experience but delivering value manually. Unlike the Concierge, it looks to be complete from the front, but all the tasks that automated systems would typically be doing are carried out manually behind the scenes. Someone is pulling strings — just like the Wizard of Oz. It’s ideal for validating the experience while still discovering if it delivers value.
🍱 Mash-up
Evidence Strength■■■■■■■■■■Resources Needed ■■■■■■■□□□
Combining multiple existing services to deliver value. There’re tons of services out there for almost anything we can think of doing. With some creativity, we can stick them into a Frankenstein that gets the job done until we can justify using real resources. It’s ideal for learning if the solution resonates with customers.
🛹 MVP
Evidence Strength■■■■■■■■■■Resources Needed ■■■■■■■■□□
Building a functioning product with the minimum viable to provide value and test our assumption. Even if cutting corners on a great experience in the process. It’s ideal for learning if the core promise of our solution resonates with customers.
Note: The MVP term is also used to mean experiment — any kind. Here I’m just using it for this specific one.
During the Success Evaluation phase, we should use more quantitative experiments. They’ll help us know if we reached the results we were expecting and understand rationally if our proposed solution was successful.
📊 Data Analysis
Evidence Strength■■■■□□□□□□Resources Needed ■■■□□□□□□□
Using our software data collection to look for customer behavior patterns. It’s ideal for validating the success of an implemented solution and finding drop-off points on a user flow — circling back to the Discovery phase.
⚖️ Split Test
Evidence Strength■■■■■■□□□□Resources Needed ■■■■■□□□□□
Comparing two versions, control A against variable B, and determining which one performs better. It can also be done with multiple versions and not only two. It’s ideal for testing different versions of value propositions, prices, and features to see what resonates best with customers.
