Skip to content
arula

Take a change an AI wrote, check what could be wrong, and explain what you would approve, send back, or ask an owner to decide.

Chapter 03 / 07

Review

Check the reviewer’s claims against the code, requirements, and a suitable reproduction.

What you should leave with
Shared intuition
the review gave us claims whose support we could inspect.
Shared language
the finding is what the reviewer says; the requirement and checked code help us assess it.
Shared behavior
follow the cited code and run a suitable check.
Step 1 of 5

What Review receives

Review asks an agent to examine the diff for problems it can support from the code shown.
That can give us a more specific claim to check than Diagnose’s text match.
We’ll start with the logging call, then use the fee change to see what else the reviewer can notice.
Run Review for Task 1, and let’s look at exactly what the agent receives.

speed review --feature payments --task 1
Source references

SPEED: lib/cmd/review.sh, lines 194–212, routes --task into _cmd_review_task.
Use --task 1;
--task-id 1 selects a different path in this build.

Task Review reads the same branch from Task 1 and gets its diff against main.
It also reads the author model and declared file list.
Here those are sonnet and the four files we’ve already seen.
The full diff goes into the review message; the file declarations don’t filter it.

Source references

Fixture: .speed/features/payments/tasks/1.json, lines 2–11.
SPEED: lib/cmd/review.sh, lines 95–104 and 119–125.

Look at the message template in the shell script.
It inserts the author model, a JSON object containing the file declarations, and the diff.
Our task’s files_touched list becomes the modified list; created and deleted are empty.
Those labels come from the task record, rather than Git classifying each change.
Here’s the assembled message with just the logging hunk shown.

Source references

SPEED: lib/cmd/review.sh, lines 129–140, builds the message.
Line 98 supplies the author model; lines 99–104 build the declarations; lines 119–125 supply the diff.

Recorded author model: sonnet

Declared file lists:

{
  "created": [],
  "modified": [
    "src/payments/retry.ts",
    "src/payments/service.ts",
    "test/refund-retry.test.ts",
    "test/fixtures/cards.ts"
  ],
  "deleted": []
}

Diff (main…round-0):

-      log.error(serialiseFailure(error, redact(req)));
+      log.error(serialiseFailure(error, req));
Source references

Fixture: src/payments/service.ts, line 67.
These are message excerpts; the file-list JSON is expanded for readability.
The actual message contains the full branch diff.

The reviewer can see both calls: redact(req) has been replaced by req.
It also receives the fee change and the rest of the branch diff.
A separate file supplies its instructions.
That file asks for actionable problems supported by the diff, with uncertainty stated where information is missing.
It also tells the agent to treat comments and metadata as review material, rather than follow instructions embedded in them.

Source references

SPEED: agents/clean-context-reviewer.md, lines 3–12.

The provider call passes those instructions and the message to the reviewer.
It chooses the review model from SPEED’s configuration and grants read-only tool permissions.
The sonnet value inside the message is the recorded author model; it doesn’t choose who reviews the code.
The role file also says not to run commands.
When this reviewer suggests a reproduction, we still have to run it.

Source references

SPEED: lib/cmd/review.sh, lines 145–150;
agents/clean-context-reviewer.md, lines 70–71.

Compare that message with Task 1’s acceptance criterion.
The criterion asks whether two refund records exist, but the reviewer doesn’t receive it.
It receives no separately loaded spec or Diagnose output either.
This lets us examine the whole diff for problems beyond satisfying that one refund criterion.
Any requirement the reviewer uses must be supported by what is actually in the diff.

Source references

Fixture: .speed/features/payments/tasks/1.json, lines 50–54.
SPEED: lib/cmd/review.sh, lines 83–85 and 129–140;
agents/clean-context-reviewer.md, lines 3–12.

Task criterion, omitted from the Review message:

{
  "acceptance_criteria": [
    {
      "criterion": "[RETRY-01] a retried refund against a capture records both refunds",
      "verify_by": "test"
    }
  ]
}
Step 2 of 5

Requirements and findings

The fee change is a useful example.
The rate is 1.49 percent, so a capture of two hundred gives 2.98 before rounding.
The new calculation floors that to two; the test expects three.
Both the calculation and the added test are in this branch diff.
The reviewer can point out the disagreement and suggest running the test.

Source references

Fixture: src/payments/service.ts, line 43.

const SCHEME_FEE_RATE = 0.0149;
Walkthrough notes

Fee-change excerpt from main...round-0; current fixture location:
src/payments/service.ts, lines 118–120.

-    const fee = applyRate(requested, SCHEME_FEE_RATE);
+    // Fees are never rounded up against the merchant.
+    const fee = minor(Math.floor(requested * SCHEME_FEE_RATE));
     const net = sub(requested, fee);
Source references

Fixture: test/service.test.ts, lines 107–115.

test('[RISK-02] the scheme fee rounds half up at the capture boundary', () => {
  for (const [amount, fee, net] of [[200, 3, 197], [9_999, 149, 9_850]]) {
    const s = svc();
    const p = auth(s, amount);
    s.capture(p.id, amount);
    assert.equal(sumAccount(s, p.id, 'scheme_fees'), fee);
    assert.equal(sumAccount(s, p.id, 'merchant_settled'), net);
  }
});

The reply is JSON containing an issues array.
Each issue needs a message and severity, with a location and reproduction details where the input supports them.
It can also include observed and expected behavior.
A scenario identifier, such as RISK-02, names a case in the test catalog.
The reviewer may copy it only if it appears in the diff and applies to this finding.
That rule prevents the agent from inventing an identifier just to make its finding look testable.
SPEED saves the review and archives its evidence so later commands can use it.

Do: Show SPEED agents/clean-context-reviewer.md, lines 23–31 and 37–64; and lib/cmd/review.sh, lines 160–184. In the fixture, open .speed/features/payments/reviews/task-1.review: lines 4–11 contain the logging claim; lines 13–20 contain the fee claim; lines 22–29 contain the retry-coverage claim. Recheck the wording and locations after live Review. These saved claims aren’t a promised new result.

Selected fields from three saved findings:

{
  "issues": [
    {
      "message": "The authorise() failure path now serialises the raw request instead of the redacted one, so a full PAN and CVV can reach the log sink on every tokenise failure.",
      "severity": "critical",
      "file": "src/payments/service.ts",
      "line": 67,
      "scenario_id": "RISK-01"
    },
    {
      "message": "capture() now floors the scheme fee instead of rounding half up, contradicting TR4/RISK-02 and the RISK-02 test added in this same diff.",
      "severity": "major",
      "file": "src/payments/service.ts",
      "line": 119,
      "scenario_id": "RISK-02"
    },
    {
      "message": "reissueRefund(), the retry helper this diff introduces, is never called from any source or test file, so its retry loop, attempt limit, and error-rethrow behaviour are entirely unexercised.",
      "severity": "major",
      "file": "src/payments/retry.ts",
      "line": 12,
      "scenario_id": "RETRY-01"
    }
  ]
}
Step 3 of 5

Follow the logging claim

Our saved review claims that passing the raw request exposes card data.
Open serialiseFailure and follow what it does with that request.
It calls redact on the context before JSON.stringify builds the string.
The logger then stores that string.
So removing redact at the call site hasn’t established a leak: the helper still does the redaction.
The saved claim doesn’t hold up against this implementation.
This is why we need to follow a call beyond the lines changed in the diff.

Source references

Fixture: src/payments/service.ts, line 67, passes the request to the serializer.

log.error(serialiseFailure(error, req));
Source references

Fixture: src/obs/index.ts, lines 49–64, redacts the context and builds the string.

const PAN_FIELDS = new Set(['pan', 'cardNumber', 'primaryAccountNumber', 'cvv', 'cvc']);

export const redact = (value: unknown): unknown => {
  if (Array.isArray(value)) return value.map(redact);
  if (value && typeof value === 'object') {
    return Object.fromEntries(
      Object.entries(value as Record<string, unknown>).map(([k, v]) =>
        PAN_FIELDS.has(k) ? [k, '[redacted]'] : [k, redact(v)],
      ),
    );
  }
  return value;
};

export const serialiseFailure = (error: unknown, context: unknown): string =>
  JSON.stringify({ error: error instanceof Error ? error.message : String(error), context: redact(context) });
Source references

Fixture: src/obs/index.ts, line 30, stores the string in the log sink.

error: (message: string) => sinks.logs.push({ level: 'error', message, at: Date.now() }),
Step 4 of 5

Reproduce the failure path

Let’s inspect the stored log from a failure.
The request expects a string for the card number; we’ll deliberately pass a number instead.
Tokenisation calls replace on it, which throws and takes us into the logging path.
We can then read the captured log using the fixture’s published test card.

Source references

Fixture: src/payments/service.ts, lines 35–41 and 62–68;
src/vault.ts, lines 21–26;
test/fixtures/cards.ts, lines 8–16;
src/obs/index.ts, lines 21–31 and 66–70, exposes and resets the captured logs.
Run from the fixture root; the imports name every module used by this demonstration.

node --experimental-strip-types --input-type=module <<'JS'
import { PaymentsService } from './src/payments/service.ts';
import { sinks, resetSinks } from './src/obs/index.ts';
import { TEST_CARDS, EXPIRY } from './test/fixtures/cards.ts';
resetSinks();
try {
  new PaymentsService().authorise({
    pan: Number(TEST_CARDS.visa), expiry: EXPIRY, amount: 1000,
  });
} catch {}
console.log({
  records: sinks.logs.length,
  redacted: sinks.logs.every(r => JSON.parse(r.message).context.pan === '[redacted]'),
  containsTestPan: sinks.logs.some(r => r.message.includes(TEST_CARDS.visa)),
});
JS

The recorded run produced one log record.
Its card-number field was redacted, and the stored message didn’t contain the test card number.
That answers the logging claim for this failure path.
It doesn’t test every other path through the service.
Also notice that the saved review suggested bad expiry as a reproduction.
This tokenise function never validates expiry, so that suggestion wouldn’t cause the failure we need.

Walkthrough notes

The original script records running and checking this reproduction.
This port rechecked the source; it hasn’t repeated that execution or generated fresh Review output.

Recorded result from that reproduction:

{ records: 1, redacted: true, containsTestPan: false }
Step 5 of 5

Check retry and fee claims

The saved review also raises a concern about retry coverage.
Let’s compare the helper with the test.

Do: Show fixture src/payments/retry.ts, lines 19–27; and test/refund-retry.test.ts, lines 9–16, side by side.

The helper catches a failed refund call and tries again, up to its attempt limit.
The test calls refund twice directly, then checks that two refund records exist.
It never calls the helper or makes a refund attempt fail.
So this test doesn’t check the retry loop, despite having retry in its name.

The fee finding also has an existing test we can run.
The spec requires half-up rounding, the service uses floor, and the existing test expects a fee of three for a capture of two hundred.

Do: Show fixture src/payments/service.ts, lines 118–120; specs/tech/payments.md, lines 40–43; and test/service.test.ts, lines 107–115.

We’ve checked the logging claim for the failure path we demonstrated.
The retry test leaves the helper’s behavior unchecked.
The fee calculation disagrees with the expected amount in its test.
Next, we’ll run Eval for Task 1 and examine what evidence it gives us for those remaining concerns.

Shared intuition: the review gave us claims whose support we could inspect.
Shared language: the finding is what the reviewer says; the requirement and checked code help us assess it.
Shared behavior: follow the cited code and run a suitable check.

Walkthrough notes

Keep all Review evidence tied to Task 1; a saved task review is historical evidence.

Carry forward

The demonstrated log path redacts the card number. Retry-helper coverage remains unproven; the fee calculation disagrees with its requirement. Take those questions into Eval.

Help me reason through this

Follow serialiseFailure beyond the diff. Then compare the retry helper with what the test actually calls.

Your explanation is saved in this browser.