Benchmarks
Every figure on this page was produced by a named command whose raw output is kept with the measurement. A cell without a measurement reads "not measured"; every cell of the 8 GB laptop column reads "not measured on 8 GB". Numbers from the Apple-silicon development machine are placeholders until the reference 8 GB Windows laptop without a GPU has been measured.
A figure appears here only when a named command produced it, with the command and raw output saved; the row's evidence names that output. Apple-silicon numbers are placeholders until the 8 GB Windows no-GPU laptop has been measured. The 8 GB laptop column reads not measured on 8 GB in every row; every other cell without a figure reads not measured.
| Benchmark | Repo | Command | Apple-silicon development machine (placeholder) | 8 GB Windows no-GPU laptop | Notes |
|---|---|---|---|---|---|
| 700 MB asset download (wall time, MB/s, peak JS heap) | runtime | node test/browser/range-server.mjs --asset big=734003200; Playwright demo ensure('big') | not measured | not measured on 8 GB | Sprint 1 done-when test |
| Resume after reload at 40 percent (one Range request) | runtime | Playwright reload-at-40 spec | not measured | not measured on 8 GB | |
| Eviction under a 1.5 GiB origin quota | runtime | CDP Storage.overrideQuotaForOrigin; 700 + 400 then 500 MB | not measured | not measured on 8 GB | |
| detectCapabilities() latency | runtime | Playwright capability spec | not measured | not measured on 8 GB | must resolve within 2 s |
| Same prompt, WebGPU versus WASM, SmolLM2-135M q4 test model (load, prefill, decode, tokens per second) | runtime | RUNTIME_MODEL=1 RUNTIME_REQUIRE_WEBGPU=1 npm run test:model | WebGPU: load 2,667 ms, prefill 66 ms, decode 51 ms, 155.9 tok/s. WASM: load 2,592 ms, prefill 368 ms, decode 732 ms, 10.9 tok/s. Same answer on both. | not measured on 8 GB | Sprint 2 done-when; test model, not the Hindi production model; filled only from this command's output Evidence: evidence/sprint-2/D1/model-run.json. |
| runtime model-load timing per phase (acquire, verify, hand-off, session, first token): SmolLM2-135M, WebAssembly engine in a worker | runtime | RUNTIME_MODEL=1 npm run test:devtools (headed) | Apple silicon, Chrome 157.0.8103.79, WebAssembly engine in a worker (WebGPU was forced off in these runs; WebGPU not measured here). D1 run: acquire 1,260.9 ms, verify 663 ms, hand-off 7.6 ms, session create 481.2 ms, first token 655.1 ms. T1 run: acquire 1,254 ms, verify 664.9 ms, hand-off 7.4 ms, session create 479.7 ms, first token 655.5 ms. | not measured on 8 GB | Sprint 3 slot, filled in Sprint 4 from the saved Airplane Mode DevTools-panel output of two headed runs (D1, then T1 independently). The figures are the panel's own phase durations; 'acquire' is the panel's 'Acquire (download, 0 when cached)' phase. SmolLM2-135M is the test model, not the production model. Apple-silicon figures are a placeholder until the 8 GB laptop is measured; not measured on 8 GB. Evidence: evidence/sprint-3/D1/devtools/devtools-figures.json and evidence/sprint-3/T1/r3/devtools/devtools-figures-t1.json. |
| runtime memory at model load (page JS heap where Chrome exposes it; worker where exposed) | runtime | RUNTIME_MODEL=1 npm run test:devtools (headed) | Apple silicon, Chrome 157.0.8103.79, page JS heap used (performance.memory, coarse). D1 run: 2,747,591 B before load, 79,554,898 B after session create, 79,611,170 B after first token. T1 run: 3,587,929 B, 33,865,195 B and 33,954,615 B at the same three points. The worker heap and WebAssembly memory were not exposed (source none), so there is no figure for them. | not measured on 8 GB | Sprint 3 slot, filled in Sprint 4 from the saved DevTools-panel output of the same two headed runs; the figures are read from the browser, never estimated. The two runs differ after session create and the difference is not explained here. Apple-silicon figures are a placeholder until the 8 GB laptop is measured; not measured on 8 GB. Evidence: evidence/sprint-3/D1/devtools/devtools-figures.json and evidence/sprint-3/T1/r3/devtools/devtools-figures-t1.json. |
| pagetriage page report (ms per page, native vs wasm) | pagetriage | cargo bench; wasm demo | not measured | not measured on 8 GB | |
| glyphsift validation of about 1 MiB of synthetic Devanagari (12,888 lines) | glyphsift | target/release/glyphsift check corpus-1mib.txt (five runs); WASM validate() and score_ppm() in Node, best of five | native CLI: 0.05 to 0.06 s (one cold run 0.17 s); WASM validate(): 232 ms; WASM score_ppm(): 75 ms | not measured on 8 GB | Browser figures not measured. Evidence: evidence/sprint-2/D3/throughput.log. |
| glyphsift score versus noise (mixed-family mean and per-family curves, seeds 1, 2, 3) | glyphsift | cargo run -p glyphsift-cli --release -- bench --corpus bench/corpus/clean.txt --seeds 1,2,3 --csv docs/score-vs-noise.csv --svg docs/score-vs-noise.svg | Mixed-family mean, ppm, at p = 0, 0.01, 0.02, 0.05, 0.1, 0.2, 0.4: 1,000,000; 998,476; 996,985; 991,696; 983,977; 967,538; 933,440. Falls at every step. | not measured on 8 GB | Sprint 2 done-when; regenerated in Sprint 3 after two rule changes (the Sprint 2 write-up keeps the Sprint 2 figures); filled only from this command's output. The score is deterministic (integer arithmetic), so it does not depend on the machine; the laptop column stays not measured until it is run there. Evidence: evidence/sprint-3/D3/bench-s3.log and docs/score-vs-noise.csv. |
| localdata set/get throughput, OPFS vs IndexedDB | localdata | Playwright behavioural suite timing | not measured | not measured on 8 GB | |
| localdata flat index, 100,000 x 384 + 256-byte synthetic chunks, OPFS, cold open + first query (median / worst of 5) | localdata | npm run bench:vectors | Apple M5 Pro, 48 GB: median 186 ms / worst 195 ms of 5 cold runs; browser-restart run 182 ms | not measured on 8 GB | Sprint 4 done-when row: cold open plus the first query (k = 10) must stay under 2,000 ms on an 8 GB laptop; this machine has more memory, so the figure is not an 8 GB figure. Measured on Apple M5 Pro, 48 GB (machine column), uncommitted working tree based on 0390d65 (commit column: 0390d65-dirty), by npm run bench:vectors in localdata; match is true on every row. JS heap was read with precise memory info enabled. The row is not measured on 8 GB: the 8 GB Windows laptop is the owner's step. Evidence: localdata/docs/benchmarks/vector-index.csv. |
| localdata flat index, 100,000 x 384 + 2,048-byte synthetic chunks, OPFS, cold open + first query (median / worst of 5) | localdata | npm run bench:vectors | Apple M5 Pro, 48 GB: median 181 ms / worst 183 ms of 5 cold runs; browser-restart run 182 ms | not measured on 8 GB | Sprint 4 done-when row: cold open plus the first query (k = 10) must stay under 2,000 ms on an 8 GB laptop; this machine has more memory, so the figure is not an 8 GB figure. Measured on Apple M5 Pro, 48 GB (machine column), uncommitted working tree based on 0390d65 (commit column: 0390d65-dirty), by npm run bench:vectors in localdata; match is true on every row. JS heap was read with precise memory info enabled. The row is not measured on 8 GB: the 8 GB Windows laptop is the owner's step. Evidence: localdata/docs/benchmarks/vector-index.csv. |
| dockit cold start to interactive, offline | dockit | Playwright pwa.spec.ts timing | not measured | not measured on 8 GB | skeleton only |
dockit: an answer over a PDF, in the browser, offline
dockit A PDF goes through pagetriage (what is on each page), text extraction, the glyphsift gate, localdata chunks and vectors, and a
small local model, in installed Chrome with the network blocked after the assets are cached. The command for every row is npm run bench:e2e;
the raw rows are in dockit's docs/benchmarks/e2e.csv (sha256 1bbe066f768889ca) and
docs/benchmarks/e2e-questions.csv (sha256 502b8aa50903c1de), taken at commit 0282d19da192.
Each time is the median and the worst of the runs in its row; no run was left out.
| Machine | Run kind | Runs | Time to first token (median / worst) | Total, first question (median / worst) | Peak memory: Chrome process tree, summed resident size (median / worst) | JS heap peak, main thread (median / worst) | WebAssembly memory peak, workers (median / worst) | Storage used (median / worst) | Requests after offline | Model and backend | Command | Commit, Chrome | 8 GB laptop |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Apple M5 Pro 48 GB | one Chrome session, model cached | 5 | 7,731 ms / 7,861 ms | 9,280 ms / 9,435 ms | 6,283 MiB / 6,377 MiB | 11 MiB / 31 MiB | 2,574 MiB / 2,574 MiB | 762 MiB / 762 MiB | 0 | Qwen3-0.6B q8 + multilingual-e5-small q8; transformers-js wasm (single-threaded) | npm run bench:e2e | 0282d19da192, Chrome/157.0.8103.79 | not measured on 8 GB |
| Apple M5 Pro 48 GB | after a full Chrome restart, same profile | 1 | 7,861 ms / 7,861 ms | 9,440 ms / 9,440 ms | 6,147 MiB / 6,147 MiB | 33 MiB / 33 MiB | 2,574 MiB / 2,574 MiB | 762 MiB / 762 MiB | 0 | Qwen3-0.6B q8 + multilingual-e5-small q8; transformers-js wasm (single-threaded) | npm run bench:e2e | 0282d19da192, Chrome/157.0.8103.79 | not measured on 8 GB |
Peak memory is the peak summed resident size of the test Chrome's processes (browser, GPU, utility and renderer); a page shared between processes is counted in each. The JS heap and the workers' WebAssembly memory are different figures and are shown under their own names.
Requests after offline counts application requests: those made by the page, its workers and its service worker, as the largest count of network requests attempted after the network was blocked. The browser itself may still check /dockit/sw.js for an update and fetch the favicon; this count does not include those two requests, and dockit's code makes neither of them.
Question set
Seven fixed questions over a synthetic three-page policy, six with an answer on a known page and one with no answer in the document. Correct means the answer holds the expected text and a source the answer cites names the page that holds it. First run: 4 of 6 correct with the correct source page. Over all 6 runs, counting every run of every question with none selected: 24 of 36. The question with no answer was refused in 6 of 6 runs.
Two page columns are kept apart. A cited source on the right page was shown in 24 of 36 answerable rows: a refused answer cites nothing, so it never counts there. The top-ranked retrieved passage was on the right page in 36 of 36 (retrieval quality, reported on its own).
| Question | Runs | Answer correct | Cited source page correct | Top-ranked retrieved passage on the right page | Time to first token (median / worst) | 8 GB laptop |
|---|---|---|---|---|---|---|
| q1 | 6 | 6 | 6 | 6 | 7,732 ms / 7,861 ms | not measured on 8 GB |
| q2 | 6 | 0 | 0 | 6 | 7,819 ms / 7,833 ms | not measured on 8 GB |
| q3 | 6 | 6 | 6 | 6 | 7,736 ms / 7,743 ms | not measured on 8 GB |
| q4 | 6 | 6 | 6 | 6 | 7,823 ms / 7,827 ms | not measured on 8 GB |
| q5 | 6 | 6 | 6 | 6 | 7,803 ms / 7,815 ms | not measured on 8 GB |
| q6 | 6 | 0 | 0 | 6 | 7,840 ms / 7,852 ms | not measured on 8 GB |
Done-when: build done on Apple M5 Pro 48 GB; 8 GB Windows done-when NOT met (owner runbook step 5).
Limitations
- Text-layer PDFs only. OCR is not yet supported: a scanned PDF is refused, and image-only pages are skipped.
- The model is Qwen3-0.6B, a small demo model, and not the pilot model (Qwen3-1.7B, hosted by the customer). Its answers can be wrong.
- Single-threaded WebAssembly: the page is not cross-origin isolated (GitHub Pages cannot send the headers), so the engine runs on one thread.
- The origin is shared: the demos of this lab all live on airplanemode-labs.github.io, so storage and quota are shared between them.
- The test document is a synthetic English policy, and the question set is seven fixed questions over it. A result here says nothing about real documents.
- Hindi answer quality is not measured: only the gate that tells clean from damaged Devanagari text layers is tested.
- Not measured on 8 GB. Every figure here is from the machine named in its row; the 8 GB laptop column reads not measured on 8 GB.
Reference machine
The reference machine is an 8 GB Windows laptop with no discrete GPU, Wi-Fi off, running the tools in Chrome. It has not been measured yet; the column stays empty until it has.