Bonsai 27B field guide
How prompt caching changes Bonsai 27B benchmarks
Why repeated prompts can make local Bonsai 27B tests look faster and how to design a trace that exposes cache effects.
Can prompt caching change a Ternary Bonsai 27B speed test?
Yes. A runtime can reuse work from a prior prompt when the next request shares the same beginning. That can reduce time to first token and can change the memory state around generation. Use different prompt prefixes, record warmup separately, and reload the model when you need a clean comparison.
What we observed
Earlier local reruns produced median decode rates from 16.30 to 22.58 tokens per second. Prompt reuse was one cause, but context size and memory pressure also changed between those runs.
The reported controlled run followed an explicit 4K reload. Each measured prompt began with different text, so the server could not reuse the full prompt prefix.
A safer test pattern
- Run one warmup and exclude it from the median.
- Change the first words of every measured prompt.
- Record the context limit and model load event.
- Keep other applications idle during the run.
Questions people ask
Is prompt caching bad?
No. It is useful in applications. It only becomes a problem when a benchmark hides that it reused work.
Does a different prompt remove every cache effect?
No. The runtime can still retain model and memory state. A reload gives a stronger boundary.
Primary sources
Vendor claims on this page are labeled as PrismML claims. Local results are added only after a saved trace completes.