Automated Web Benchmarking in CI Without Theater

Automated web benchmarking is how methodology survives staffing changes. A wiki page of wrk flags does not. This article is the minimum pipeline that can fail a pull request without becoming political theater.

The minimum pipeline

• A dedicated runner, not a shared GitHub-hosted CPU that also compiles random PRs.
• A lock file for oha, k6, vegeta, and h2load versions.
• A warm-up step that is not scored.
• Thresholds expressed as code.
• Artifacts in object storage plus a short comment on the pull request.

k6 thresholds are the cleanest developer experience for scenario work. For microbenchmarks, compare against a baseline JSON from main. Fail if p99 regresses more than X milliseconds or successful RPS drops more than Y percent, unless a label acknowledges the regression and links an issue.

export const options = {
  scenarios: {
    json: { executor: 'constant-arrival-rate', rate: 8000, duration: '45s', preAllocatedVUs: 200 }
  },
  thresholds: {
    http_req_failed: ['rate<0.001'],
    http_req_duration: ['p(99)<12']
  }
};

Overrides and politics

Automated web benchmarking that cannot be overridden becomes political. Automated web benchmarking that is always overridden becomes theater. The label-plus-issue escape hatch is the compromise that ships. Audit the labels monthly. If every PR carries the escape hatch, you no longer have a gate.

Guard the runner

Disable frequency scaling if you can. Pin IRQs. Reject runs when the box is already loaded. Cloud CI is hostile to tight microbenchmarks. If budget allows one reserved instance family, use it. If it does not, widen thresholds and stop pretending you have laboratory precision. Automated web benchmarking on noisy neighbors still catches catastrophes. It will not catch a 3% serializer regression.

Canary the harness

Schedule a weekly run of the harness against a known binary. When a k6 version or a runner kernel moves the number, you want that isolated from application changes. The yardstick needs its own test.

What the PR comment should contain

Baseline versus candidate p99, RPS, error rate, instance type, and a link to raw JSON. No screenshots of TUI tools. Reviewers should be able to rerun the same command. That is automated web benchmarking as an engineering artifact rather than as a badge.

Pair this with web benchmarking methodology so the pipeline has a taxonomy. Pair it with best practices so humans do not optimize the dashboard.

FAQ

Should every repo have this? Every user-facing HTTP service should. Internal cron jobs can wait.

Nightly or per PR? Microbenchmarks per PR on a dedicated runner. Heavier capability tests nightly.

Build the gate. Then protect the runner. That is the whole job of automated web benchmarking.

Mistakes people make around automated web benchmarking

The first mistake is treating automated web benchmarking as a slogan instead of a scoped experiment. The second is copying a command from a chat log without pinning the client version. The third is publishing a peak while dropping the error column. Any one of those three will make a later review of automated web benchmarking feel like starting over.

A fourth mistake is mixing clusters. The seed web benchmarking is allowed to inform this page. A composite from an unrelated seed is not allowed to finish the claim. If someone pastes a plaintext crown into a discussion of automated web benchmarking, ask which protocol, which payload, and which error gate they used. If they cannot answer, the paste is decoration.

A fifth mistake is skipping warm-up and then blaming the framework. JIT runtimes, page cache, and TLS session tickets all distort the first seconds. How you treat those seconds is part of automated web benchmarking whether or not the original brief mentioned them.

Checklist before you claim automated web benchmarking

• One-sentence hypothesis that contains the phrase automated web benchmarking or an equivalent measurement goal.
• Frozen artifact: digest, flags, workers, runtime version.
• Client and server on disjoint cores or separate machines.
• Warm-up that is not scored.
• At least three accepted samples with an error gate.
• p50, p95, p99, successful RPS, RSS, and the exact command.
• A note on whether the client or the server hit the wall first.
• A link to sibling long-tails in the web benchmarking cluster.

If a box on that list is empty, you are not done with automated web benchmarking. You are drafting. Drafts can live in a branch. They should not live in a decision memo.

Who owns automated web benchmarking inside a company

Give the work a code owner. Platform teams usually own the harness. Product teams usually own the hypothesis, because only they know which route is expensive. Finance does not own automated web benchmarking, but they will quote it. Write the sentence they are allowed to quote and put it at the top of the report.

On-call owns the dashboard that automated web benchmarking is supposed to move. If on-call cannot find the panel in two minutes, the study will not change operations. That is not a documentation nit. That is how laboratory work dies after the launch channel goes quiet.

Security and privacy review the payload and the logs. A test that ships real user bodies into a load generator is not a study of automated web benchmarking. It is an incident with extra steps. Use synthetic documents that match production shape.

Worked example in the web benchmarking cluster

Suppose two engineers disagree about automated web benchmarking after a runtime bump. They rerun the same pinned command on the reserved instance family. One run is client bound. The other used HTTP/2 without saying so. After both mistakes are labeled, the delta shrinks to a number that no longer funds a rewrite. That outcome is a success. Studies that only count rewrites as success will stay loud and stay wrong.

Suppose instead the rerun confirms a 35% p99 improvement on the route that is 40% of CPU. Then automated web benchmarking has earned a migration design, not a migration. The design still needs an APM row, a rollback, and a field check. Lab wins that skip those three become next quarter’s incident review.

Keep raw JSON next to the Git SHA. If you cannot point at a file, you do not have a result for automated web benchmarking. You have a memory. Memories do not survive staffing changes. Put the file path in the ticket before the meeting so the meeting cannot become a debate about vibes.

If this page is the first time someone on the team has seen the phrase automated web benchmarking, send them the seed overview for web benchmarking after they finish the checklist, not before. Reading the cluster in the wrong order is how long-tails collapse back into a single generic argument. Keep the long-tail sharp. That is the only reason this article exists as its own URL.