A good AI project does not begin with a model. It begins with work that is expensive, repeated and sufficiently well understood. If cost and quality are not measured before automation, it becomes impossible to prove either benefit or harm after launch. This framework helps select a first process and run a constrained, controlled pilot.
A strong first candidate has
- Enough volume and visible manual cost
- Digital and accessible input data
- A clearly defined good result
- Tolerable error risk or a human approval point
1. Score the process—not the popularity of the technology
List 10–20 repetitive processes and score each on frequency, handling time, error cost, rule clarity and input quality. Start with work that has meaningful volume but manageable downside.
A first pilot should not autonomously move money, terminate employment or create an irreversible legal commitment. AI may prepare a recommendation, but an accountable person should approve the final action.
- Monthly frequency
- Average manual handling time
- Financial or reputational cost of error
- Share of exceptional cases
- Data availability and quality
- Can a good answer be objectively evaluated?
2. Capture the baseline before automating
Monthly manual cost equals case volume × minutes per case × fully loaded cost per minute. Add the cost of correction, delay and missed opportunity only where it can be defended.
Automation value is not 100% of that amount. Subtract human review, model and infrastructure usage, maintenance and exception handling. Model conservative, base and optimistic cases.
3. Separate the system into three layers
Reliable automation is rarely one large AI agent. A deterministic layer controls integrations, states, permissions and business rules. The AI layer classifies, extracts or generates where rules are insufficient. A control layer validates outputs, stops uncertain cases and leaves an audit trail.
This separation lets a team replace the model without rebuilding the system and identify exactly where a failure occurred.
- Workflow: triggers, state, integrations and retry logic
- AI: a narrow task with explicit input and output
- Guardrails: schema, allowed values and business rules
- Human-in-the-loop: approval based on risk threshold
- Audit: model version, input, output and resulting action
4. Measure quality separately from speed
An automation that saves 80% of handling time but creates 5% costly errors may destroy value. Before the pilot, build a representative set of real examples with human-approved correct outcomes. Reuse it to compare every change.
In production, monitor error categories—not only average accuracy. A critical error should weigh more than a cosmetic one. Uncertain cases should route to a person rather than disappear inside a high aggregate score.
- Task success rate
- Critical error rate
- Human escalation rate
- Average cost and duration per case
- Corrections after AI output
- Adoption and actual use
5. A four-week pilot must end with a decision
Week one defines the baseline, risk and success threshold. Week two builds one narrow end-to-end flow using representative data in a safe environment. Week three runs in shadow mode, where the system recommends but does not act. Week four reviews the evidence and decides whether to scale, revise or stop.
A pilot is not successful because the demo works. It succeeds when benefit exceeds full cost, errors stay within the agreed boundary and the team can operate the system.
- Week 1: baseline, data, risk and success criteria
- Week 2: one end-to-end flow
- Week 3: shadow mode and error analysis
- Week 4: ROI, quality, operability and go/no-go
Free calculator
AI automation ROI calculator
Enter conservative assumptions for one process. The calculator does not collect or transmit your data.
Use with your team
AI pilot go/no-go checklist
Mark an item complete only when you have evidence—a document, metric, demonstration or an accountable owner's confirmed process.
- 01The process has a named owner
- 02Current cost and time are measured
- 03Representative examples exist
- 04Correct output is defined
- 05Critical error categories are known
- 06Sensitive data is inventoried
- 07Data transfer and retention are understood
- 08The AI task is intentionally narrow
- 09Business rules do not live only in the model
- 10Uncertain cases can stop
- 11Risky actions need human approval
- 12Every decision is auditable
- 13Model and prompt versions are traceable
- 14Full cost per case is estimated
- 15A quality evaluation set exists
- 16Acceptable error thresholds are explicit
- 17Provider or model failure has a fallback
- 18A named owner makes the go/no-go decision
FAQ
Common questions
Which process should be automated first?+
Choose frequent, digital and objectively verifiable work with moderate rather than catastrophic error cost. Good examples include document classification, reviewed extraction, response drafts and report preparation.
How much data does a pilot need?+
It depends on process variation. Representativeness matters more than raw volume: frequent cases, rare exceptions and critical error categories must all appear in the evaluation set.
Do we need our own AI model?+
Usually not. First validate the business hypothesis with a reliable provider, clear data policy and strong control layer. A custom model is justified only by specific quality, cost, privacy or latency requirements.
When should a process not be automated?+
When it is rare, constantly changing, lacks a defined good outcome, has inaccessible data or creates disproportionate consequences from a single error that human approval cannot contain.
Primary references
For deeper technical work:
NIST AI Risk Management Framework