Article / Systems Thinking & Orchestration
When Understanding Meets Evidence
Throughout this series, we’ve explored how value is created, what enables it, and how the different components of a business interact. Together, these perspectives help us build a more complete understanding of complex environments. But even a well-structured understanding can be wrong. Processes don’t always work as documented. Customers don’t always behave as expected. Systems don’t always perform as designed. And assumptions that seem reasonable in conversation can fall apart when confronted with actual results.
This is the fourth and final area of Structured Discovery: Evidence and Reality. It focuses on challenging what we believe we understand through actual usage, measurement, economics, and technical performance. The objective isn’t to prove that our understanding is correct. It’s to discover where it isn’t.
What we think happens versus what actually happens
One of the most important distinctions in discovery is the difference between how something is intended to work and how it behaves in practice. A workflow may look efficient on paper while requiring repeated manual intervention. A product feature may function exactly as designed but see little adoption. A process may meet its internal performance targets while creating unnecessary effort for the people using it.
These gaps are not always visible through interviews, documentation, or system diagrams alone. They become clearer when we examine what people actually do, how work moves through the environment, and what outcomes are being produced.
In my experience working across product, analytics, and operations, the most useful insights often emerge when observed behavior challenges an established assumption. A team may believe a particular feature is essential, but usage patterns tell a different story. Or a process assumed to be inefficient may turn out to be compensating for limitations elsewhere. Neither observation automatically tells us what to change. It tells us where our understanding needs to improve and where to go back to look again.
Measurement starts with the right questions
Measurement is often treated as something that happens after a product or capability has been delivered, but in reality starts much earlier. It cannot be an afterthought.
Having spent a significant part of my career developing measurement frameworks and analytics capabilities, I’ve learned that the hardest part is rarely producing more data. It’s establishing what we need to understand and whether the available information can actually answer those questions.
A metric is only useful when its definition, context, and limitations are understood. An increase in engagement might appear positive, but what does engagement represent? More meaningful interactions, or simply more activity? A reduction in processing time might suggest greater efficiency, but does it improve the overall workflow or move the burden somewhere else? These questions matter because metrics are representations of reality, not reality itself.
Good discovery requires examining how outcomes are defined, what is being measured, and what remains invisible. It also requires recognizing that two teams can report different results without either being technically wrong, simply because they’re measuring different things. Enough reliable evidence to understand how the system behaves is worth more than a complete set of metrics.
Analytics helps explain, not just describe
Analytics can reveal patterns that aren’t immediately visible through observation. It can show where customers disengage, how performance varies across segments, or which activities contribute most to an outcome. But identifying a pattern is not the same as understanding its cause.
Consider a product experiencing declining conversion. An analytics dashboard may show where customers abandon a process, but it doesn’t necessarily explain why. The cause could involve the experience itself, the quality of information presented, technical performance, customer expectations, or conditions outside the product.
This is where analytical thinking becomes particularly valuable. Rather than accepting the first explanation that fits the data, we examine alternative interpretations, compare relevant populations, and look for evidence that helps distinguish between them.
I’ve found this especially important in environments where multiple teams rely on the same information to make different decisions. A metric that supports one conclusion at an aggregate level may tell a different story when examined by customer segment, product, channel, or time period. Deeper analysis guards against a confident decision resting on an incomplete interpretation.
Experimentation challenges what we believe
Even with strong analytics, some questions remain difficult to answer through observation alone. A product change may coincide with improved performance, but that doesn’t necessarily mean the change caused the improvement. Customer behavior, seasonality, marketing activity, or other changes may have influenced the result. Experimentation provides a structured way to test those assumptions.
Having worked extensively on experimentation platforms and measurement systems, I see experimentation as more than a method for choosing between product variations. It’s a way to reduce uncertainty about cause and effect.
A well-designed experiment begins with a clear hypothesis, identifies the outcome that matters, and establishes a credible comparison. Statistical significance is where the question starts. What matters is whether the intervention produced a meaningful difference, and what that says about the assumptions underneath. That distinction matters. An experiment can produce a positive result that is too small to justify its operational cost. It can show no detectable effect while still leaving important questions unanswered. And an improvement in one metric can come at the expense of another.
Experimentation also requires judgment about when it is appropriate. Not every question can or should be answered through an A/B test. Sometimes observational analysis, qualitative research, technical evaluation, or a controlled operational pilot provides more useful evidence. What matters is choosing an approach that fits the uncertainty we’re trying to resolve.
Reality includes economics and operational performance
Evidence isn’t limited to customer behavior or product metrics. A capability also needs to be understood through its economic and operational reality. Something may be technically feasible and attractive to users but expensive to maintain. A workflow may deliver better results while requiring more manual effort than the organization can sustain. An automated capability may reduce processing time but introduce new monitoring, governance, or support requirements. These considerations are particularly important in enterprise environments, where the cost of a capability often extends beyond its initial development.
The same principle applies to growth. Increasing adoption or transaction volume doesn’t automatically create sustainable value if the economics deteriorate as the capability scales. During discovery, examining operating costs, resource requirements, reliability, and technical performance helps establish whether the value being created is supported by a practical operating model. This isn’t about completing a financial business case before understanding the problem. It’s about recognizing the conditions under which a capability can realistically deliver value.
Evidence should change our understanding
One of the more difficult aspects of discovery is remaining willing to revise an interpretation after investing time in developing it. Once a team has established a compelling explanation, it becomes tempting to look for evidence that supports it. Contradictory information may be dismissed as an exception, an unreliable metric, or a problem with the data. But evidence is most valuable when it challenges what we thought we knew.
This is where empirical thinking becomes essential. We develop an understanding, examine it against observable reality, and adjust when the evidence suggests something different.
In product and technology work, that process rarely produces perfect certainty. Data can be incomplete, measurements can contain bias, and experiments have limitations. Few decisions wait for every unknown to close. What they need is a clear line between what we know, what we reasonably infer, and what remains an assumption. That distinction makes the eventual strategic decisions more informed, while also making it easier to recognize when new information should change their direction.
From discovery to informed direction
Across these five articles, we’ve explored Structured Discovery as a way to develop a connected understanding of complex business and technology environments. We began by looking beyond the immediate problem. We followed how value is created through users and workflows, examined the systems and intelligence that enable those activities, and explored the dependencies, constraints, and tradeoffs that shape how the environment behaves. Finally, we challenged that understanding against evidence.
These perspectives are interconnected. Measurement may reveal a workflow issue. Examining the workflow may expose a system dependency. Understanding that dependency may change how we interpret the economics of a proposed capability. Each discovery can reshape the picture developed through the others. That’s why I don’t see discovery as a linear exercise or a checklist to complete before moving on. It’s a structured process of building, connecting, and testing our understanding until we have enough clarity to make informed choices.
My background in engineering, analytics, experimentation, and product leadership has reinforced the value of approaching complex challenges this way. Not because every problem requires an elaborate framework, but because the quality of our decisions depends on the quality of our understanding. Discovery doesn’t eliminate uncertainty. It helps us understand which uncertainties matter, what the evidence tells us, and where we still need to learn.
And that’s where strategy can begin.