Fixed For You

Your bug.
Fixed for you
first.

Diagram: the ring, drawn in characters, with eight stations in order: report, evidence, fix for one, stop drill, verified for them, told, everyone, flag removed. One lit cell, marked with an at sign, is you, the person who reported the bug. It moves from station to station. At “everyone” the rest of the ring lights up as the flag reaches 25% and then 100% of users.

8 stations From the report to the flag removed

Each station is described below, with what it leaves in GitLab.

When a customer reports a bug, the fix goes live for that customer first, behind a GitLab feature flag switched on for them. It is checked on their own requests in production, they are told, and only then does everyone get it and the flag is removed.

The ticket does not close on a merge. It closes when production shows the reporter’s requests working.

See every ring Review it in five minutes
  • Hands-off: the customer’s report is the only human act.
  • Built on Kutt, the open-source URL shortener, running on Google Cloud.
  • The customers are synthetic. A person writes their reports from real mailboxes.

How one ring runs

Eight stations, one customer report. Every station leaves something in GitLab that you can open and check.

  1. Report

    A customer writes to support on the app’s Contact support page. The conductor files it as an issue and adds the customer to it by email, so every reply reaches their inbox. This is the only human act in the ring.

    Leaves in GitLab

    • Issue, with the customer as an email participant

  2. Evidence

    The conductor finds the customer’s account and posts their failing requests from production traces. The intake agent compares them with the report. A matching failure means reproduced. No failure in production means the ring asks the customer one question and writes no code.

    Leaves in GitLab

    • Evidence note
    • ring:: state on the issue board
    • Issue weight

  3. Fix for one

    Write
    The fix agent writes a regression test that replays the customer’s failing request, and a fix behind the feature flag ring-<issue> that keeps the old code path.
    Check
    Unit tests, the customer’s regression test, the negative control and the charter check, with SAST, secret detection, dependency, IaC and container scanning. Duo Code Review reads every merge request.
    Ship
    The merge request merges itself when every gate passes. One signed image goes to staging, a smoke test, then production: the same digest.
    Switch on
    The flag is switched on for one user list: the reporter, plus the ring’s own probe account.

    Leaves in GitLab

    • Branch and merge request
    • Pipeline and test report
    • Vulnerability report
    • Container image, CycloneDX SBOM, Sigstore signature
    • Staging and production environments
    • Release with notes and evidence
    • Feature flag and its user list

  4. Stop drill

    Before anyone relies on the flag, the ring proves it works as an off switch. It replays the customer’s request in production as its probe account with the flag off, then on, and needs the old behaviour, then the fixed one.

    Leaves in GitLab

    • Drill note on the issue
    • Flag history

  5. Verified for them

    The conductor reads the reporter’s own requests on the flagged path from GitLab Observability. Enough successes and no failures: verified. A failure: the flag goes off, an incident opens, and the ring tries again with the new evidence.

    Leaves in GitLab

    • Observability traces of the reporter’s requests
    • Incident, when they still fail

  6. Told

    The reply agent drafts the message to the customer. A gate checks the draft for anything that must not leave the building, then the conductor posts it on the issue, and GitLab emails it to the customer.

    Leaves in GitLab

    • Reply on the issue, emailed to the customer

  7. Everyone

    The flag ramps to 25% of users, then 100%. It holds at each step while the fleet’s error rate stays under 1%.

    Leaves in GitLab

    • Flag strategy changes in the flag history

  8. Flag removed

    A merge request removes the flag and the old code path. Once it is deployed, the issue closes with the ring’s record: attempts, CI minutes, credits and grams of CO2e.

    Leaves in GitLab

    • Cleanup merge request
    • Closed issue with the ring record

Guardrails and refusals

When production disagrees with a fix, the ring does not argue. The flag goes off, and the ring either tries again with the new evidence or hands the ticket to a person, with the reason written on the issue. Those rings sit in the ledger next to the ones that worked.

Fences around the agents

  • Their notes stay internal

    The three agents are GitLab Duo flows that can only write internal notes. Every public word on a ring issue comes from the conductor.

    Check: .gitlab/duo/flows/*.yml

  • No production credentials

    The agents read the issue and write code or drafts. Reading traces, switching flags, merging, replying and closing are done by the conductor, which is plain deterministic code.

    Check: ring/lib/tick.ts

  • Charter check

    An agent’s change may touch only server/ and test/ring/ and add only its own flag. Anything else, the charter included, fails the pipeline.

    Check: charter:check, charter.yml

  • Negative control

    The regression test must fail on the merge base, pass with the flag on, and fail with the flag off. A test that proves nothing does not merge.

    Check: ring:negative-control

  • Reply gate

    The reply is checked before it is sent. A refused draft never reaches the customer.

    Check: ring/lib/reply.ts

  • Signed spans

    Verification only counts spans that carry a signature over the fields it reads. The observability ingest is not authenticated, so unsigned spans are ignored.

    Check: server/telemetry.js, ring/lib/signing.ts

When production disagrees

  1. If production shows no failing request for the customer

    then the ring asks them one question and writes no code.

    ring::needs-info

  2. If switching the flag off does not bring back the old behaviour

    then the stop drill fails and a person takes the ticket. A fix that cannot be undone does not stay on.

    ring::human

  3. If the reporter’s requests still fail with the fix on

    then the flag goes off at once, with no build and no deploy, an incident opens, and the fix agent tries again with the new evidence.

    rollback, incident

  4. If the second attempt fails too, or the ring spends more credits than its budget

    then a person takes the ticket, with the reason on the issue.

    ring::human

  5. If the fleet’s error rate passes 1% during the ramp

    then the flag goes back off and the ring tries again.

    rollback

  6. If the reporter sends no requests on the fixed path for an hour

    then the fix cannot be verified for them, so a person takes over.

    ring::human

Numbers from the ledger

Read from ledger.json, which the pipeline rebuilds from the GitLab issues every hour and on every push to main.

Credits and grams of CO2e are modeled, not measured. What is counted and what is estimated is in CLAIMS.md.

Open the ring ledger
Rings run
…
Time to fixed for the reporter
…
Time to everyone
…
Fix attempts and rollbacks
…
Handed to a person
…
Credits per finished ring
…
CO2e per resolved report
…

Check it yourself

Nothing here needs taking on trust. Each claim points at a GitLab object, a test or a file.