Measuring and Learning: The Lean Startup Approach

Poor management is one of the main reasons why most startups and new products fail, despite the talent and dedication of their founders. Traditional management is not suitable for dealing with the high uncertainty that startups face, but adopting a non-management or absolute chaos approach is also ineffective. In this regard, the Lean Startup method offers a solution based on the Lean Manufacturing method used by Toyota in the manufacturing of physical goods, but adapted to startups and the creation of technological products in highly uncertain environments. This approach promotes the formation of multidisciplinary teams, working in short iterations with constant feedback, and measuring progress through validated learning. Instead of simply doing things and waiting to see what happens, the importance of conducting experiments to validate predefined hypotheses is emphasized. The recommended approach is iterative, with a build-measure-learn cycle, and it encourages the creation of Minimum Viable Products (MVPs) to understand customer needs before investing resources in large-scale projects that could result in resource waste (having a working product does not guarantee its value to customers).

Defining hypotheses

The lean startup method works like the scientific method: in each iteration, the assumptions to be validated are defined, and the results are measured to see if they are correct or not. The two main hypotheses to define are the value hypothesis and the growth hypothesis.

Value hypothesis

The value hypothesis focuses on determining whether a product or service truly provides value to the customer and if they are willing to pay for it. To do this, several key questions are raised:

• Target consumer segment: What is the group of people who could be customers of the product or service?
• Need or problem it solves: What specific need or problem does the product or service address? How can it improve the customer’s life?
• Perceived value and willingness to pay: How much value does the customer perceive in the product or service? Would they be willing to pay for it, either with money or their attention?

Example: “We believe that users between the ages of 20 and 40 who are interested in their physical well-being and do not have access to a gym in their area will be willing to pay for an application that provides them with personalized exercise routines and tracks their progress.” In this case, the hypothesis focuses on a specific consumer segment (users between the ages of 20 and 40 interested in their physical well-being) and the problem it solves (lack of access to a gym). It also states that these customers would be willing to pay for a solution in the form of an application that provides them with personalized exercise routines and tracks their progress.

Growth hypothesis

The growth hypothesis refers to the assumption of how sustainable growth can be achieved in a business or project, that is, how to ensure that new consumers come from actions taken by past consumers. There are three growth engines that can be used according to the author to achieve this goal:

• Sticky engine: This growth engine aims to attract and retain customers in the long term; the customer encounters exit barriers and it is difficult for them to leave or stop using our product. Companies that use this growth engine rely on having a high customer retention rate and therefore closely monitor the churn rate, which is defined as the fraction of customers who, during a given period of time, do not remain loyal to the product. If the rate of acquiring new customers is higher than the churn rate, the company will grow. The growth rate is determined by the capitalization rate, which is simply the result of subtracting the churn rate from the natural growth rate. Example: “We believe that customers will encounter exit barriers in our product as they feel committed and emotionally connected to their progress and the community surrounding it.”
• Viral Engine: It is based on growth through the usage of the product or service, where knowledge about the product spreads rapidly from person to person, not because customers ‘spread the word,’ but as a side effect of its usage. The viral engine is driven by a feedback loop that can be quantified, known as the viral loop, and its speed is determined by the viral coefficient. The viral coefficient measures how many new consumers will use the product as a result of a new consumer being registered, in other words, it measures how many customers each consumer will bring. The higher the coefficient, the faster the product expands. Example: ‘We believe our users will share their achievements and progress on social media, which will spark interest among their friends and followers. Upon seeing these posts, some people are likely to try our application and register as new users.’
• Paid Engine: Companies using this growth engine pay to acquire customers. To increase the growth rate, one can either increase the revenue provided by each consumer or the Customer Lifetime Value (LTV), or minimize the cost of acquiring a new one (CPA), maximizing marginal profit. If the LTV is lower than the CPA, growth will slow down, and it can be compensated with ‘one-time’ tactics such as investing in capital or advertising campaigns.

While it is possible to combine these growth engines, it is recommended to focus on one and specialize in it. At the end of the day, it is crucial to use the relevant metrics mentioned above to validate learnings and measure progress. It is also important to note that growth eventually stalls, so it is necessary to adapt and seek new strategies to continue growing.

Build

The proposed method moves away from extensive planning on a whiteboard and instead promotes the construction of Minimum Viable Products (MVPs), which do not necessarily have to be technological, to test and empirically validate hypotheses with early adopters, real users. Working in small iterations/batches allows for obtaining valuable information early on and facilitates the early detection of flaws and issues, reducing the accumulation of unfinished tasks and maintaining a clear and organized focus on product development. Regarding quality, it is important to move away from assumed standards and focus on what truly contributes to the intended learning. Prioritizing speed and obtaining real data is key, and quality is addressed as feedback is obtained and iterative improvements are made to the product. When it comes to design, it is essential to consider that the primary objective is to learn and validate hypotheses, so it is recommended to avoid efforts that do not directly contribute to that learning. This does not mean completely neglecting design but making conscious decisions and focusing on what is necessary to obtain the required user information.

Example: An online financial advisory project by the United States government started as a simple informative hotline (0800) without agents to measure and validate public interest and determine the most relevant topics for them (value hypotheses). Through this approach, valuable information could be gathered, and the feasibility of proposed solutions could be validated before investing significant resources.

Measure and learn

It is essential to measure the results of each experiment to achieve our objectives. Once we understand the situation, adjustments must be made to move closer to the desired ideal and, ultimately, make the decision to pivot or persevere based on the validated learning obtained. In this regard, it is important to note that user stories should not be considered complete solely because they have been delivered by the development team but rather when the hypothesis and the intended value have been validated. To avoid falling into the trap of ‘vanity metrics’ which are metrics that can improve or worsen without necessarily being the result of the development team’s effort, it is recommended to conduct multivariate experiments. These experiments allow us to effectively measure the impact on behavior and determine if an improvement or decrease in a metric is directly related to a specific action. It is crucial for the measurements to be accessible, understandable, and reliable. This implies that the information must be consistent, verifiable, and accurate in order to make informed decisions.

Pivoting or Persevering

One of the most challenging decisions for entrepreneurs is choosing between persevering or pivoting. Available funds are like a runway for a startup, so it is crucial to gain learning at a lower cost and in less time. To pivot effectively, it is essential to have clear hypotheses supported by actionable metrics that allow us to make decisions and replicate actions to achieve the desired behavior changes. It is recommended to schedule regular meetings to assess the startup’s progress and direction, usually every few weeks or months. When making a decision, it is important to overcome the fear of failure or potential negative reactions to significant changes in the company’s strategy. We must also consider that investing too much effort can create false hopes and consume energy on a path that does not bring us closer quickly to the desired scenario. There are several types of pivots that can be considered, such as the zoom-in pivot (focusing on an individual feature that becomes the complete product), the zoom-out pivot (expanding the product’s focus), the customer segment pivot (targeting a different customer than anticipated), the customer need pivot (addressing a new problem raised by the customer), the platform pivot, the business architecture pivot (switching between B2B and B2C business models or between high margins and low volume), the value capture pivot, the growth engine pivot (adopting a spiral, sticky, or paid approach), the distribution channel pivot, and the technology pivot, among others. Each pivot represents a new strategic hypothesis that needs to be validated with a new MVP and becomes a resilience tool for the startup.

In summary, the Lean Startup approach aims to eliminate inefficiency and reduce waste by focusing on minimizing time in each “build-measure-learn” cycle. While speed is important, quality should not be compromised as defects can negatively impact idea generation and validation, leading to incorrect decisions.

If you want to go deeper into this topic, you can read the book on which this post was based: “The Lean Startup” by Eric Ries