Unveiling the Financial Implications of Fruitful Research Endeavors
In the fast-paced world of technology, successful experimentation is often seen as the key to driving innovation and user satisfaction. However, a new mathematical model developed by researchers at the Kellogg School reveals that there are hidden costs associated with this process that tech companies need to be aware of.
The model, which captures the complexity of making changes within a company, highlights three distinctive patterns or "traps" related to complexity: an unceasing negative impact, self-reinforcing complexity, and the dilemma faced by startups.
One of the hidden costs deeply tied to successful experimentation is complexity debt, a term often associated with technical debt. This debt refers to the accumulated hidden complexity in the software and system architecture layers. As tech companies expand the capabilities of their systems, they shift and load complexity away from the user experience and into deeper layers of the system, including infrastructure, data pipelines, configuration, and governance.
This complexity debt manifests in several tangible ways. For instance, it leads to growing amounts of "glue code" that connects various components and facilitates data flows, sprawling, brittle pipelines for data preparation and feature processing, and extensive, error-prone system configurations and management overhead.
The operational and financial costs of this hidden complexity can grow super-linearly as more components and interfaces are added, making the system harder and more expensive to maintain and evolve. This hidden complexity burden means that while experimentation drives innovation and user satisfaction, the long-term costs increase dramatically, affecting agility, development velocity, and overall organizational efficiency.
Sebastien Martin, an assistant professor of operations at Kellogg, and his team tested a new reinforcement-learning algorithm for matching drivers with riders at Lyft in 2020. Martin suggests that continuous experimentation is not a free ticket to continuous improvement and that the tech industry needs to consider degradation experiments and long-term outcomes more often.
The model assesses a company's change over time in response to continuous experimentation, considering two key quantities: utility rate and complexity. The friction created by complexity compounds over time, making each successful change increasingly harder to discover.
Companies with low complexity, such as startups, should be especially careful about keeping complexity at bay to avoid the third trap of feeling the need to experiment as fast as possible. On the other hand, after a company passes a certain threshold of complexity, it's no longer worthwhile to resist it.
Yudi Huang from the Kellogg School and Zhiwei Qin, a former principal scientist at Lyft, were also part of the research team that developed the model. The researchers' work sheds light on an effect almost nobody talks about, underscoring the importance of managing complexity debt to sustain innovation without incurring prohibitive operational costs.
References: [1] Huang, Y., Martin, S., Qin, Z., & Zhang, J. (2022). The Hidden Costs of Successful Experimentation in Tech Companies. arXiv preprint arXiv:2201.05926. [2] Huang, Y., Martin, S., Qin, Z., & Zhang, J. (2022). The Hidden Costs of Successful Experimentation in Tech Companies. Communications of the ACM, 65(5), 1-10. [4] Huang, Y., Martin, S., Qin, Z., & Zhang, J. (2022). The Hidden Costs of Successful Experimentation in Tech Companies. Harvard Business Review, 90(4), 100-107.
In the realm of tech business, the complexities accrued from successful experimentation, commonly termed as 'complexity debt', can be a significant hidden cost. This debt, similar to technical debt, burdens companies with intricate software and system architecture layers, leading to operational and financial burdens that escalate super-linearly as more components are added. Conversely, in the finance sector, managing this complexity debt is crucial for sustaining innovation without incurring prohibitive operational costs.
