Path to Level 3 Workshop (Copy)
Harris International Group Vanguard AI Group · Leadership 2026
AI-DLC workshop

Your Path to Level 3

Level 3 is decided by evidence, and the evidence is only as good as your tests. Use this sheet to find your gaps, plan 90 days for one team and choose the measures you will track. Answer for how your teams normally work, not the best demonstration anyone has given.

L1 · Assisted L2 · Directed L3 · Delegated
Activity sheet

Find your gaps, then plan one team's 90 days

Work in pairs or as a table. Each step has a time box. Your answers save in this browser as you type, and you can copy them out at the end to share with your team or the Vanguard team.

  • 1Evidence, not opinion. If you would have to ask someone, the answer is no.
  • 2Write down the awkward answers. They decide what you build first.
  • 3"Don't know" is a gap. Mark it and move on.
1
Your gaps · 8 min

Run the five checks

Each of these has an answer you could get this week. Most business units are surprised by at least two of them. Mark each one, then note what you know or what you would need to find out.

Can your pipeline stop a bad change, including one from an admin?

Points at 01 Pipeline

Not whether CI runs. Whether a failing check blocks the merge for everyone.

Gap if: required checks are empty, or admins can bypass them.

Answer
How to find out today
# what is actually enforced on main
gh api repos/:owner/:repo/branches/main/protection \
  --jq '{checks: .required_status_checks.contexts,
         admins_enforced: .enforce_admins.enabled}'
# checks: [] or admins_enforced: false means start with 01

How much of your ready queue says what it touches and what data is involved?

Points at 02 Classification

If a ticket does not say what it touches and what data it carries, no rule can act on it. This is usually the quickest gap to close.

Gap if: more than a handful of ready items are missing module, data class or customer risk.

Answer
How to find out today
-- items ready to start that nothing could safely start
project = CUS AND status = "Ready for Delivery"
  AND ("Data Class" is EMPTY
    OR "Customer Risk" is EMPTY
    OR "Module" is EMPTY)

Do the same changes get the same approvals?

Points at 03 Approval rules

Take one sensitive area (billing, payments, anything customer-facing) and look at who signed off the last twenty changes to it.

Gap if: different names each time, and some with no architect at all.

Answer
How to find out today
# who approved changes to one module last month
gh pr list --state merged --limit 50 \
  --search "path:src/billing merged:>=2026-10-01" \
  --json number,reviews \
  --jq '.[] | {pr: .number, approvers: [.reviews[].author.login] | unique}'

Could you answer the four questions for three random tickets from last month?

Points at 04 Evidence

Without asking a person: what risk class was it, who approved it and on what basis, what was tested, and what risk was accepted? Step 2 lets you try it.

Gap if: you had to ask someone, or open four systems.

Answer
How to find out today
-- three at random, no cherry-picking
project = CUS AND status = Done
  AND resolved >= -30d
ORDER BY rand()

How flaky is the test suite?

Points at 01 Pipeline

Agents treat a red test as something to fix. A flaky suite teaches them to "fix" code that was never broken.

Gap if: more than 5% of runs need a rerun. The Level 3 target is under 2%.

Answer
How to find out today
# workflow runs that needed a rerun, last 100
gh api "repos/:owner/:repo/actions/runs?per_page=100" \
  --jq '[.workflow_runs[] | select(.run_attempt > 1)] | length'
Your answers point atMark the five checks to see which foundations need work first.

Commands are illustrative. Substitute Azure DevOps, GitLab or your own equivalents.

2
The Level 3 test · optional

Try the test on three tickets

If you have a laptop and access, pick three tickets closed in the last 30 days at random. Answer each question from the ticket and its pull request alone. No asking, no memory, no reconstruction. If not, take this away and do it this week.

Question
0 / 12Answer all twelve to see the result.

The full test uses ten tickets. Ten out of ten is Level 3. Eight out of ten means the process works when people remember, which is Level 2.

3
Test coverage · 5 min
The usual blocker

Check the tests under the code agents will change

For most business units the barrier to Level 3 is test coverage, not AI. Level 3 depends on the pipeline catching what agents get wrong, and a pipeline that tests very little cannot do that. Agents treat the tests as the definition of correct. Where the tests are thin, a green pipeline proves very little.

List the three modules agents will change most

An average across the codebase hides the problem. What matters is coverage where the work lands, and how bad it would be if that code broke.

ModuleCoverage todayChanged often?Impact if it breaks

Start where low coverage, frequent change and high impact meet. That module caps how much you can hand to agents.

Which of these are true today?

0 of 6 in place

Where does test coverage need investment before anything else?

That programme is the investment, even though none of it is AI. Agents can write the tests. A person has to judge whether they describe the right behaviour.

Until the tests are thereTreat thinly tested modules as higher risk in your classification, so the rules route their changes to a person. Widen what agents can do in that module as coverage rises.
4
Your 90 days · 7 min

Plan 90 days for one team

One team passing the test is a pilot, not a business unit claim. Pick the team and the two foundations your gaps point at. Coverage on the weakest module starts in the Foundations stage and runs alongside everything after it.

Which team and product would you start with, and why?

A good first team has a pipeline in reasonable shape, a product owner who will classify, and a product that is not your oldest or least tested.

Which main revenue products have to follow, and in what order?

The first team proves the approach. The Level 3 goal is met only when your main revenue products are delivered this way. List them, with the team behind each and the biggest blocker, which is often test coverage.

Which two of the five would you put in place first?

Pick two. Each one sits on the one below it.

Put dates and an owner against each stage

Each stage has an exit test. If the team cannot pass it, it does not move on. Timings assume a pipeline already in reasonable shape; an old product with few tests will take longer.

Days 1 to 10

Look

  • Run the five checks. Build nothing yet.
  • Write the answers down.
  • Pick the two workstreams they point at.
Five answers, in writing, shared with the team.
Days 11 to 45

Foundations

  • Make the pipeline able to say no.
  • Classify every ready ticket.
  • Start coverage on the weakest module.
No bypassed checks in 30 days. 95% of ready items classified.
Days 46 to 75

Rules and evidence

  • Write the approval rules in a file.
  • Record evidence on the ticket.
  • Give agent questions a turnaround time.
An approval fires the same way whoever is reviewing.
Days 76 to 90

Test, then turn on

  • Score ten random changes.
  • Walk it through with Vanguard.
  • Switch on pickup only after a pass.
10 out of 10, then automatic pickup goes on.
5
Your measures · 4 min

Choose what you will measure

Choose the measures and thresholds up front, then report what they say. Avoid lines of code and percentage AI-written. Both reward volume.

What can you measure this week with the data you already have?

Which change in habits will be hardest in your business unit?

Habit

Who looks at the numbers each week, and what will they do with them?

6
Before you leave
One action for this week

Run the five checks on one team and write down the answers

Make it yours. Who will you sit down with, and when?

Answers save in this browser as you type.