The hidden cost of successful experiments

But not all successful experiments are created. Some experiments lead to changes that seem positive in the short term (eg, lead to more profits, customer maintenance, etc.), but in fact they are smuggled by a new complexity in the operation of the company itself. This makes it more difficult to further improve the system and perform future experiments, slowing the engine itself that makes innovation possible.

“The fundamental question is, what does it mean to say that something ‘worked’? I ask Sébastien martinAssistant Business Professor in Kellogg. “Right now it can be amazing for me and my clients. But in the long run. Does it affect my ability to innovate?”

To explore the hidden costs for experimentation -based companies, Martin and Yudi Huang’s associates (also from Kellogg School) and Zhiwei Qin (former Lyft scientist) created a mathematical model that records the complexity of change in a company. They have discovered that the friction created by this complexity unites over time, such as the interest in a debt. In other words, every successful change becomes increasingly difficult to discover, demanding more complex and time-consuming experiments-and these slows often remain detected.

However, the researchers also found that after a company crossing a certain limit of complexity, it is no longer worth resisting it. Thus, despite the fact that we have to deal with the slowest and most favorable experiments, it actually pays to get more “complexity” debt, even when the performance in experimentation is reduced. “The only solution in this case is often rebuilding from scratch instead of trying to prevent additional complexity,” Huang explains.

“The model says something very opposite,” Martin admits. “He underlines a result almost no one speaks.”

Understanding this often hidden effect of experimentation can be particularly valuable to technology companies that invest many resources in experiments. “Technology companies have huge groups focused on experimentation, successful experiments are the ones that promote people,” says Martin. “But saying if something” works “in the long run is a surprisingly difficult task.”

The hidden cost of success

Martin experienced this difficulty firsthand. While working as a researcher in Lyft in 2020, he helped test a new reinforcement-learning algorithm to match drivers with riders.

“It was very expensive to run these experiments, but it is definitely worth it,” he recalls. “Increased drivers’ revenue, making customers happier. Lyft worldwide developed the new algorithm in 2021. Everything was processed – or did it?

“I realized that when you make a complex change like this it becomes more difficult for other groups [at the company] To innovate, “Martin says.” It also makes the process of experimenting harder and more expensive. ”

For one thing, having a sophisticated mechanical learning algorithm that changes over time means that the next experimental idea becomes more difficult to find and more difficult to apply. “I have to predict what this crazy algorithm will do in reaction to whatever I want to try,” Martin explains, “so it is more difficult to know if my idea is good.”

In addition, composite experiments throughout the system are not as simple as the A/B that tests the color of a button. Like a large rock fell into a rush of rush, the new Lyft algorithm can change the flow of the whole system in unpredictable ways – so the only way to try was to activate it for everyone to see what is happening and then turn it off and compare the results. These so -called “Change Experiments” It had to be repeated many times, and for a longer period of time, in order to create reliable results.

Martin felt that the debt of complexity from these experiments could prevent future future innovation efforts, but could not be sure. “There is almost no way to measure them during the experimentation process,” he says. “I began to guess myself. And that led to an interesting mathematical exercise.”

Modeling

Like debtThe “complexity” within a company is difficult to measure because it is difficult to determine.

“He records so many things -writers, software, depth,” says Martin. “For our purposes, complexity has only one meaning: the possibility of performing a successful experiment in your company on any given day is lower when your complexity is higher.”

Using this definition of complexity, Martin and his colleagues modeling how a company changes over time in response to constant experimentation. They appreciated this idealized company in terms of the two basic quantities. The first, called the percentage of utility, represents the main measurements that the company is trying to maximize through experimentation, such as profit or commitment of users. The second term represents complexity: The higher the complexity, the more it removes the rate of experimentation.

From there, the compromises are relatively simple. Over time, successful experiments increase the company’s common use rate – as well as the new Lyft algorithm, has led to an increase in revenue, commitment and efficiency. At the same time, any change resulting from a successful experiment either increases its complexity or leaves it unchanged.

Trap adjustment

After analyzing the model’s behavior, the researchers found that complexity is indeed a real problem, manifested in three distinctive standards or “traps”.

The first trap is that the negative impact of complexity debt has no ceiling. It will simply grow and grow without a plateau. “The idea is that if you continue to apply changes – blindly following successful experimentation results, all the time – there is no limit to how bad it can get,” Martin says.

The second trap occurs when the debt of a company’s complexity becomes self-contained. In the model, a company can make choices that keep this complexity from getting worse. However, after achieving a certain limit of complexity, even this approach becomes pointless: the optimum choice for an extremely complex company is to continue experimenting and more debt.

Why? Because management of complexity is expensive short -term and only beneficial in the long run.

“Public companies have investors who are interested in a maximum of five to ten years from now,” Martin explains. “And when you are a big company and your complexity is high, the improvements are rare, so you have a much stronger incentive to only get any improvement you can,” regardless of the long -term cost of complexity.

However, companies with low complexity (such as newly formed businesses) should be particularly careful for its maintenance in the vagina. These companies tend to be even more interested in short -term revenue and growth than companies most mature and are particularly sensitive to the third trap, where they feel the need to experiment as quickly as possible. This is the dilemma often faced by technology businesses: because they have to show relatively rapid growth in their investors to survive in the next round of funding, they tend to undertake “greedy” experiments that increase the complexity to (and above) the threshold. And when they begin to grow, the debt of complexity is already at this point.

“The newly established businesses are very different from the companies after the iPO,” says Martin. “Any change you could make that seem to help you grow should be applied – otherwise you will die.”

No free lunch

So is the complexity – and its sneaky impact on experimentation and innovation – certainly inevitably, like death and taxes?

Perhaps, but Martin warns the excessive interpretation of the behavior of a single mathematical model. “Our document was intended to point out a result in the context of experimentation, which then allows people to start thinking more clearly,” he says. “When you try to solve a problem, half of the game is to know it.”

Unlike what many companies seem to believe, Martin adds, ongoing experimentation is not a free ticket for continuous improvement. “Based on my own experience in the technological sphere, I think this idea would be very controversial,” he says.

But it doesn’t have to be. So -called “Degradation Experiments” Act like tests of change in opposition, measuring what happens when a seemingly positive change is temporarily returning later. If there are no harmful results, maybe change – and its resulting complexity – can be permanently reversed, Martin says, because the “system has evolved” and no longer needs. And in other areas based on experiments such as pharmaceuticals, long-term study study-not just short-term effects-is the rule.

“We need a little more than this thought in the technology industry,” says Martin. “It puts the idea in people’s minds that the changes can be temporary because there may be a value for them – and that the return or at least the review, it’s okay.”