An AI initiative is easier to evaluate when it starts with a decision or a workflow, rather than a model. “Add an agent” describes a mechanism. “Help an operator resolve this exception with traceable evidence” describes a useful job.
Map the job before the architecture
Write down the trigger, the information available, the decision to make, and the person responsible for the outcome. Include the current process and its weak points. This gives the team a baseline to improve.
Ask three questions:
- What would a good result look like to the person using the system?
- What evidence does the system need, and how fresh must it be?
- What should happen when the evidence is incomplete or conflicting?
Make authority explicit
Retrieving information, drafting a recommendation, and taking an external action have different consequences. Treat them as separate capabilities. A system that summarizes a record does not automatically need permission to change that record.
For each action, define who can authorize it, what should be logged, and how a mistake can be recovered. Keep human review where judgment or consequential commitments require it.
Evaluate the whole workflow
Start with representative tasks, including ambiguous and unsuccessful cases. Measure usefulness, incorrect actions, latency, and cost together. A fluent answer is not enough if it does not help the user complete the job.
The first deliverable can be a simple decision map and an evaluation set. They often reveal whether you need an agent, a retrieval feature, conventional automation, or a better interface.