Measurement protocol

The rules every comparative claim in this catalog is held to. Written before the first number, so that no page can quietly invent its own methodology, and cited by comparison.md (Phase 7) rather than restated there.

Last reviewed: August 28, 2026.

IMPORTANT

The field's published search benchmarks disagree with each other mostly because their corpora, flags and match counts differ — not because the tools do. A comparison that does not hold those constant is measuring its own configuration. This page exists so that this catalog's tables are not another entry in that literature.

---

The two rules that override everything else

1. No cross-tool timing appears in a table unless one harness produced every row, on one host, in one session. A number lifted from an upstream README and set beside a number we measured is not a comparison; it is two measurements of two machines.
2. No accelerator figure is reported without its baseline. A speedup ratio whose denominator is unnamed is not evidence. GPU and FPGA results are reconstructed against a named CPU baseline with an equally good prefilter, end to end including transfer and pattern compilation, or they are labelled [paper-claim] and left uncompared.

Corpora

Measurements run against a fixed corpus, never the working tree — otherwise the numbers in the prose drift with every commit and ci --verify goes red for the wrong reason. Six shapes, because they stress different layers:

Corpus	Shape	What it isolates
single-rare	One large file, one rare literal, few matches	Prefilter quality; the engine barely runs
single-dense	One large file, high match count	Match handling — line counting, formatting, output
many-small	A deep tree of small files	Traversal, syscall and open/close cost
ignored	A tree with deep .gitignore nesting	Filter evaluation, and whether it descends what it should not
non-ascii	UTF-8-heavy prose and CJK source	Unicode case folding, class handling, validation cost
repetitive	Versioned history of one tree	Where run-length and BWT-based indexes are supposed to win

Each is generated by a committed program with a known match distribution, so a claim can name the expected number of hits and a regression shows up as a changed count rather than a changed time.

Query classes

A single query tells you almost nothing. Every comparative table reports the same nine classes, because tools diverge sharply between them:

rare literal · common literal · case-insensitive literal · alternation of literals · regex with an extractable literal prefix · regex with no extractable literal · anchored · bounded repetition (a{2,5}) · Unicode class (\p{L}).

The sixth is the one that separates engines from prefilters: it is the case where literal extraction yields nothing and the automaton runs over every byte.

What is held constant

• Match counts are reported per row. A tool that finds a different number of lines is not a comparable data point — it is answering a different question, and the difference must be explained before the timing is shown.
• Output flags are fixed and stated. Line numbers, colour and counting change the answer materially in some tools and not in others; --no-heading --color=never -equivalents are pinned per tool and recorded with the table.
• Cold and warm cache are both reported, with the drop procedure recorded. A warm-only number is a page-cache measurement wearing a search tool's name.
• Thread count is pinned, and single-threaded is always among the rows — parallel scaling is a separate question from engine cost, and conflating them is how "tool A is 4× faster" becomes "tool A defaulted to more threads".

Avoiding the process-overhead floor

Millisecond-scale results over a small corpus measure fork/exec and dynamic linking, not search. Corpora must be large enough to push the measured phase well above that floor, and the floor itself is measured once per host (a run against an empty corpus) and reported alongside.

This is why the interactive-contracts (Phase 6) question is measured differently: first-result latency and cancellation responsiveness are latency distributions, not throughput, and the harness for them is in-process.

Statistics

Reuse the repository's benchmarking guidelines and the sparkles:test-runner harness rather than inventing either:

• @benchmark with auto-scaling iteration counts; --bench-json for snapshots.
• Repeated trials with bootstrap confidence intervals and a nonparametric test, never a single-run ratio.
• On Linux, --perf hardware counters — retired instructions and cache misses are far more stable than wall time on a shared machine and should be the primary signal where available.
• A regression is actionable only when it exceeds 5% and a 95% confidence interval excludes zero, the threshold sparkles:fuzzy already adopted.

Where measurements are allowed to run

Not in CI. This repository's runners are ubuntu-latest and macos-latest; they are shared, and the macOS leg has been observed stalling a five-minute job for seventy-eight. CI verifies that examples compile and run; it never produces a number this catalog quotes.

Numbers come from a designated local runner, recorded with compiler and version, CPU model, ISA, build flags, corpus identity and sample distribution — the snapshot discipline fuzzy/benchmarks.md already established. A -mcpu=native snapshot never gates a different ISA.

Evidence labels

Every claim in this tree carries one, and the label is part of the sentence rather than a footnote:

Label	Meaning
[source-verified]	Read in pinned implementation, tests, or checked-in design docs
[spec-verified]	Read in an authoritative specification (POSIX, a Unicode UTS, a format spec)
[host-verified: <os/arch>]	Behaviour executed on that host, with command and corpus recorded
[measured: <protocol>]	A number from this repository's harness under this protocol
[paper-claim]	A figure reported by a publication; baseline and hardware named beside it
[artifact-verified]	A paper's artifact built and run locally, with the delta from its figure recorded
[literature]	Secondary or historical source; never the sole support for a current mechanic
[unverified]	An open question retained explicitly, not written as fact

The distinction that matters most in this tree is [paper-claim] versus [artifact-verified]. A speedup from an abstract is a claim about the authors' hardware and their choice of baseline; it becomes a fact about this problem only when someone rebuilds it.

Sources

• Benchmarking & profiling guidelines — the repository's existing harness, perf workflow and statistical rules
• fuzzy/benchmarks.md — the snapshot and comparison policy this page generalises
• sparkles-baseline.md — the hosts and harness actually available