Language, mindset, and discipline in AI product development

My approach comes from building and operating systems, investigating their behavior, and turning that experience into engineering practices and frameworks I have applied across teams. This work connects product discovery, architecture, evaluation, delivery, and learning from production.

I bring that accumulated judgment into new projects and expect the work to develop it further. Developing the product and improving the methods used to understand, build, and operate it are connected responsibilities. Both belong in the engineering work.

Language and engineering responsibility#

The language we use should make the work and its responsibilities clear. Familiar terms can conceal different expectations. Owning evaluation, for example, could mean improving existing scores, or investigating whether the evaluations capture the required behavior and establishing methods that guide development. These responsibilities require different scope and authority.

The same care applies to discovery and delivery. A candidate solution, a discovery prototype, and a production solution represent different commitments. Evidence that justifies investing in a production implementation leaves work to establish whether it is ready for release. Whether the live product achieves the intended outcome is another question. These distinctions need to remain visible when we describe something as validated, ready, or done.

For delivery, I make the production commitment explicit: which users and situations we support, what behavior they can depend on, the operating constraints, and how failures will be detected and handled. An initial release can support a narrow scope while meeting production requirements within it.

I expect confidence to be explainable: what it rests on, where it applies, and which questions remain open. This applies to claims about the product and to the methods used to assess it.

Scientific mindset and experienced judgment#

For me, a scientific mindset means distinguishing observations from assumptions, making reasoning available for examination, and investigating what we do not yet understand. It includes reporting failures and revising conclusions when the evidence warrants it.

Experience helps me identify consequential questions, anticipate failure modes, and choose useful investigations. I expect that judgment to influence the work. Its reasoning should be available for examination, and further experience should help refine it.

Before an experiment, I identify the uncertainty, the decision it affects, the evidence that could change that decision, and the limits of the test. The experiment can then be small and fast. When it affects real users or systems, it also needs safeguards, observation, and a way to stop it.

Our methods require the same scrutiny. We may need to reconsider what an evaluation measures, which situations it represents, or whether its findings justify the proposed decision. My frameworks remain open to examination and revision as part of this work.

Building shared understanding#

Shared understanding develops through examining concrete situations together.

Consider a hypothetical support assistant whose answer follows a general company policy but omits an applicable exception. One reviewer accepts the answer because its statements are correct; another rejects it because the omission could change the customer’s decision. Examining the case helps make the expectation explicit: the answer must include an exception when it changes what the customer should understand or do.

Domain experts explain the consequences and relevant distinctions. Engineers investigate how the behavior arises and how it can be changed. Users help establish whether the complete workflow is useful, including the effort required to check and correct the result. These contributions address different parts of the product decision.

Discovery and delivery are continuous activities of the same team. The people investigating a solution remain involved in building and operating it. Shared understanding develops through that work, including findings from implementation and production.

People can begin with different terminology and methods. They need to be able to examine the reasoning, contribute relevant knowledge, and support the work that follows from the conclusions. I expect their expertise to help improve my understanding as well.

Discipline in discovery and delivery#

Discipline means applying this reasoning consistently to the next decision.

In discovery, the team investigates whether and how a candidate solution could address the problem. Findings can justify further investigation, production investment, a change of approach, or stopping work on the current solution. The required evidence depends on the decision and its consequences.

In delivery, the team builds and maintains the behavior it has committed to support. Learning continues, with obligations to the people depending on the product. Before release, we need evidence about supported behavior and its boundaries, satisfied operating constraints, a controllable rollout, and the means to observe and respond to failures.

For each investigation, we should retain enough information to explain the question, what was changed, how the result was judged, what happened, and what decision followed. This lets others examine the reasoning and lets later work build on what was learned.

A finding may change the implementation, the supported scope, or our understanding of the solution. Evidence about technical behavior, completion of the user’s job, improvement over alternatives, and the intended outcome needs to remain distinguishable. Progress on one does not establish the others.

The recurring practice is to frame the decision, act within justified limits, inspect the result, and revise the next commitment. Discovery and delivery differ in purpose and obligations; both require thinking, action, and learning.

What the company must make possible#

This way of working requires honesty, trust, and cooperation. People need to be able to report failures, question assumptions, ask for help, and contribute knowledge across roles. Their findings need a practical route into decisions.

I can lead the development of missing engineering practices and help make unfamiliar problems understandable. That work needs agreed responsibilities, decision authority, allocated time, access to relevant people and data, and leadership support. Findings must be able to affect scope, architecture, priorities, and release decisions.

The team’s methods can still be developing. The commitment to support this work needs to be established before I accept the role. I am not looking to spend the role repeatedly negotiating permission to investigate the system or waiting for the company to recognize that the work is necessary.

These are the conditions under which I want to apply my experience, build useful production systems, and continue developing the engineering practice through substantive problems.

Related notes: My approach to production AI engineering and Evaluation in AI product discovery and delivery.