Find your gaps, then plan one team's 90 days
Work in pairs or as a table. Each step has a time box. Your answers save in this browser as you type, and you can copy them out at the end to share with your team or the Vanguard team.
- 1Evidence, not opinion. If you would have to ask someone, the answer is no.
- 2Write down the awkward answers. They decide what you build first.
- 3"Don't know" is a gap. Mark it and move on.
Run the five checks
Each of these has an answer you could get this week. Most business units are surprised by at least two of them. Mark each one, then note what you know or what you would need to find out.
Can your pipeline stop a bad change, including one from an admin?
Points at 01 PipelineNot whether CI runs. Whether a failing check blocks the merge for everyone.
Gap if: required checks are empty, or admins can bypass them.
How to find out today
# what is actually enforced on main gh api repos/:owner/:repo/branches/main/protection \ --jq '{checks: .required_status_checks.contexts, admins_enforced: .enforce_admins.enabled}' # checks: [] or admins_enforced: false means start with 01
How much of your ready queue says what it touches and what data is involved?
Points at 02 ClassificationIf a ticket does not say what it touches and what data it carries, no rule can act on it. This is usually the quickest gap to close.
Gap if: more than a handful of ready items are missing module, data class or customer risk.
How to find out today
-- items ready to start that nothing could safely start
project = CUS AND status = "Ready for Delivery"
AND ("Data Class" is EMPTY
OR "Customer Risk" is EMPTY
OR "Module" is EMPTY)Do the same changes get the same approvals?
Points at 03 Approval rulesTake one sensitive area (billing, payments, anything customer-facing) and look at who signed off the last twenty changes to it.
Gap if: different names each time, and some with no architect at all.
How to find out today
# who approved changes to one module last month
gh pr list --state merged --limit 50 \
--search "path:src/billing merged:>=2026-10-01" \
--json number,reviews \
--jq '.[] | {pr: .number, approvers: [.reviews[].author.login] | unique}'Could you answer the four questions for three random tickets from last month?
Points at 04 EvidenceWithout asking a person: what risk class was it, who approved it and on what basis, what was tested, and what risk was accepted? Step 2 lets you try it.
Gap if: you had to ask someone, or open four systems.
How to find out today
-- three at random, no cherry-picking
project = CUS AND status = Done
AND resolved >= -30d
ORDER BY rand()How flaky is the test suite?
Points at 01 PipelineAgents treat a red test as something to fix. A flaky suite teaches them to "fix" code that was never broken.
Gap if: more than 5% of runs need a rerun. The Level 3 target is under 2%.
How to find out today
# workflow runs that needed a rerun, last 100
gh api "repos/:owner/:repo/actions/runs?per_page=100" \
--jq '[.workflow_runs[] | select(.run_attempt > 1)] | length'Commands are illustrative. Substitute Azure DevOps, GitLab or your own equivalents.
Try the test on three tickets
If you have a laptop and access, pick three tickets closed in the last 30 days at random. Answer each question from the ticket and its pull request alone. No asking, no memory, no reconstruction. If not, take this away and do it this week.
| Question |
|---|
The full test uses ten tickets. Ten out of ten is Level 3. Eight out of ten means the process works when people remember, which is Level 2.
Check the tests under the code agents will change
For most business units the barrier to Level 3 is test coverage, not AI. Level 3 depends on the pipeline catching what agents get wrong, and a pipeline that tests very little cannot do that. Agents treat the tests as the definition of correct. Where the tests are thin, a green pipeline proves very little.
List the three modules agents will change most
An average across the codebase hides the problem. What matters is coverage where the work lands, and how bad it would be if that code broke.
Start where low coverage, frequent change and high impact meet. That module caps how much you can hand to agents.
Which of these are true today?
0 of 6 in placeWhere does test coverage need investment before anything else?
That programme is the investment, even though none of it is AI. Agents can write the tests. A person has to judge whether they describe the right behaviour.
Plan 90 days for one team
One team passing the test is a pilot, not a business unit claim. Pick the team and the two foundations your gaps point at. Coverage on the weakest module starts in the Foundations stage and runs alongside everything after it.
Which team and product would you start with, and why?
A good first team has a pipeline in reasonable shape, a product owner who will classify, and a product that is not your oldest or least tested.
Which main revenue products have to follow, and in what order?
The first team proves the approach. The Level 3 goal is met only when your main revenue products are delivered this way. List them, with the team behind each and the biggest blocker, which is often test coverage.
Which two of the five would you put in place first?
Pick two. Each one sits on the one below it.
Put dates and an owner against each stage
Each stage has an exit test. If the team cannot pass it, it does not move on. Timings assume a pipeline already in reasonable shape; an old product with few tests will take longer.
Look
- Run the five checks. Build nothing yet.
- Write the answers down.
- Pick the two workstreams they point at.
Foundations
- Make the pipeline able to say no.
- Classify every ready ticket.
- Start coverage on the weakest module.
Rules and evidence
- Write the approval rules in a file.
- Record evidence on the ticket.
- Give agent questions a turnaround time.
Test, then turn on
- Score ten random changes.
- Walk it through with Vanguard.
- Switch on pickup only after a pass.
Choose what you will measure
Choose the measures and thresholds up front, then report what they say. Avoid lines of code and percentage AI-written. Both reward volume.
What can you measure this week with the data you already have?
Which change in habits will be hardest in your business unit?
Who looks at the numbers each week, and what will they do with them?
Run the five checks on one team and write down the answers
Make it yours. Who will you sit down with, and when?
Five foundations, built in this order
Level 2 to Level 3 adds no speed. It adds the ability to show how any change was classified, approved and tested, months later, without asking anyone to remember. Only the last foundation is an AI project. The first four make handing work over defensible.
Answered from the record alone. Choosing the best ten does not count.
Before any work starts, without anyone chasing.
A test suite agents can trust.
No required check skipped, by anyone.
Test coverage decides how far you can go
For most business units the thing stopping Level 3 is not the agents. It is the tests. Every foundation below rests on the pipeline catching what an agent gets wrong, and a pipeline that tests very little cannot do that. At Calderwick, the fictional Level 3 unit in the guidance, the limit on delegation was coverage in a twenty-year-old billing module.
Why it matters
- Agents treat a green pipeline as done. Thin tests mean green proves little.
- Agents treat a red test as something to fix. Flaky tests teach them to change code that was never broken.
- The four questions ask what was tested. Without tests there is no good answer.
- Volume rises at Level 3. Tests are what keep the defect rate flat as it does.
What good looks like
- Coverage known per module, not just as one average.
- A floor that fails the build, raised as coverage improves.
- Each pull request shows coverage before and after, and it goes up.
- Under 2% of runs need a rerun.
How to start
- Find the module that is changed often, matters most and is tested least.
- Have agents write characterisation tests there. A person judges whether they describe the right behaviour.
- Explore AI computer control, where an agent drives the application through its screens as a user would. Use it to turn manual regression scripts into repeatable end-to-end tests for older products with little automation. Run it against test environments only, with no customer data.
- Fund it before more licences or a better agent.
- Until it is covered, classify its changes as higher risk.
Four questions for each change
Take ten changes at random from last month and answer these from the ticket and its pull request alone. A team assesses itself first, then walks the result through with Vanguard with the evidence on the table.
- What risk class and data class was it, recorded before work started?
- Which approvals fired, and can you show the rule that made them fire?
- Who approved it, when, and what were they looking at?
- What was tested, and what residual risk was knowingly accepted?
From the three business units already at Level 3
What worked
- Putting the rules in place first, then switching the agent on.
- Experiments on real tickets, keeping only what worked.
- Widening autonomy one phase at a time, each with an exit test.
- Marking rescued tickets, so the team sees how often the agent needs help.
What did not work
- Starting with the orchestrator because it is the visible part.
- Gating everything. People wave changes through to keep the queue moving.
- Fourteen classification fields. Four that are always filled in beat fourteen half filled.
- One good team, then declaring the business unit Level 3.
- Measuring lines of code or percentage AI-written.
- Treating any rise in defect counts as failure. Watch the rate.
What we would do differently
- Fund test coverage in old modules before more licences.
- Give agent questions a turnaround time and keep to it.
- Track the defect rate as volume rises, and keep it visible.
- Build automatic pickup last.
Six habits that have to change
A small team can build the foundations in a quarter. These habits are why it takes longer than the plan says.
Definition of ready
Enforced by the workflow transition. If a field is blank, the item does not move. Expect about two weeks of complaints.
Product owner"It looked small"
Stops being a reason to skip classification. Small changes to fragile modules are the ones that need an approval, and the ones people wave through.
EveryoneThe architect's job
Moves from reviewing everything to owning the approval rules. Fewer reviews, and the ones that reach the architect need them.
ArchitectWhat quality means
Moves from finding defects to making the pipeline trustworthy. That is a promotion, and it needs saying out loud.
Test and qualityAgent questions get a turnaround time
An agent waiting on a question is capacity doing nothing. If questions sit for a day, throughput collapses and people decide the model does not work.
Product ownerWhat gets funded
The next investment is usually test coverage in an old module, not more licences or a better agent. It is an AI investment even though none of it is AI.
BU leaderEveryone moves up a layer
No role disappears and none gets easier. Each moves from doing the work to deciding what the work should be and whether it was done properly. Responsibility for what reaches a customer stays with the people whose names are on the approvals.
| Role | At Level 2 | At Level 3 |
|---|---|---|
| Product owner | Writes tickets, then answers questions about them all day. | Classifies work once, up front. Sees only the questions nobody else can answer. |
| Developer | Drives each story from start to finish. | Handles new problems and agent questions. Reviews the evidence on the pull request, risk section first. |
| Architect | Finds out at code review. | Owns the approval rules. Sees the decisions that matter before any code exists. |
| Test and quality | Finds defects. | Makes the pipeline solid enough for the team to trust. |
| Security and compliance | Writes standards. | Reviews changes against the policy file and samples the evidence. |
| Business unit leader | Asks how it is going. | Watches a few numbers and funds test coverage. |
Three business units, assessed in Harris International
All three describe Level 3 the same way: governed before the work starts, evidence as the work happens, built unattended. In all three the agent cannot approve or merge its own work.
Level 1 to Level 3. An AI developer runs around the clock on approved tickets, six steps from idea to pull request, with a named person approving at five gates. Time and cost recorded per stage.
Average time to close an AI-directed issue, June to September. Four digital workers, four risk classes, 117 merged pull requests, no hotfixes. Class D work is never built by the agent.
One AI developer across six products with three risk levels. Anything touching financial data is at least medium. A question from the agent gets a reply within a working day.