Quick Answer: To improve local LLM inference speed, first calculate the theoretical ceiling from memory bandwidth and FLOPs. Then change one lever at a time and measure it against interleaved control runs. Accept a change only if it beats both neighbouring controls, use medians over at least six samples, and confirm long-context retrieval before calling a configuration the best.

Most local LLM setups are slower than they should be, and most owners never find out by how much. A "fine" 40 tokens per second might be half of what your hardware allows, or it might already be near the limit. You can't tell which without a number for the maximum. The open-source inference-optimisation skill is built around that single idea, and its workflow is worth copying even if you never install it.

Start With the Ceiling, Not the Flags

Most tuning guides start with a list of flags. This workflow starts with physics. Decode speed on a memory-bound GPU is roughly bytes read per token divided by memory bandwidth, so the theoretical maximum is set before you touch a config file.

The skill's reference, ceilings.md, covers five maxima:

  • Decode: bytes per token divided by bandwidth
  • Prefill: FLOPs per token divided by a realistic share of peak compute (the skill assumes about 50%)
  • Time to first token (TTFT): prefill cost for the prompt plus fixed overhead
  • Startup: weight loading time against storage and memory throughput
  • KV cache: how many tokens fit in the memory left after weights

The clearest example comes from a laptop with an Intel Arc B390 iGPU, 64 GB of unified LPDDR5X at about 115 GB/s, running a 35B mixture-of-experts model at 4 bits per weight. The ceiling calculation showed that 100 tokens per second for one user was only reachable with two useful speculative drafts. The model's draft head accepts about 14% of drafts at depth 2, so the practical target was roughly 88–97 tokens per second.

Here's why that matters. The team reached about 55% of that practical target, and the gap was mostly GEMV kernel efficiency, not a missing flag. Without the ceiling, they would have kept hunting for a setting that didn't exist.

That said, a ceiling is a bound, not a forecast. Real kernels, thermal limits and driver behaviour will always cost you something.

Change One Thing at a Time, Against Controls

Single-run benchmarks lie. The skill's own guidance says noise is ±5–10%, which is larger than most "wins" people report. A change that looks like 8% faster is often drift from thermal state, background load or a warm cache.

The loop the skill uses looks like this:

  1. Record a baseline and profile where the time goes.
  2. Pick the next lever by weighting the gap to the ceiling against how likely the lever is to help.
  3. Write the candidate as a config file, changing one variable only.
  4. Run control, candidate, control in a serialized queue, so both controls bracket the candidate.
  5. Accept the change only if it beats both neighbouring controls and regresses nothing by more than 3%.

The reason for the bracket is simple. If the first control and the second control disagree, your environment moved, and the candidate result means nothing. The queue script in the repo is AC-only and has a memory watchdog, so a laptop unplugged mid-run doesn't quietly corrupt your ledger.

This is slower than you want. It's also the only way to know that the number you're keeping is real.

Speculative Decoding Is About Per-Step Cost, Not Acceptance

Here's the counter-intuitive part. Most guides treat acceptance rate as the headline metric for speculative decoding. The higher the fraction of drafted tokens the target model accepts, the faster you go. That's incomplete.

The skill's position is that speculative decoding is about per-step cost. Every verification step has overhead, and a draft that is accepted more often can still make the whole system slower if drafting costs more than it saves. The practical moves are:

  • Prune the draft head's vocabulary so each drafting step does less work.
  • Sweep draft length per concurrency and context. The best depth for one user is often wrong for four users.
  • Keep sampling on the GPU so tokens don't bounce through the CPU on every step.

Multi-token prediction (MTP) setups are a good example. On an RTX 3090 running an EXL3 3-bpw Qwen3.8-27B with MTP, the repo's recipes report 96.2 tokens per second on prose and 141.1 on code for one user. Those numbers swing with the content, because acceptance swings with the content. A single acceptance figure can't describe the speed you'll see.

Most people stop here, and they shouldn't. Acceptance counters are useful for diagnosis, but they can mislead you if you read them as a speed score.

Prefill, KV Cache and Startup Are Separate Levers

Decode gets all the attention, but prefill (processing your prompt) decides how long users wait before the first token appears. Long prompts make this worse, and so does a small KV cache that forces smaller batches.

The B390 recipe shows how much these can move independently of decode:

  • Prefill at 32K went from 1,037 to 1,660 tokens per second.
  • Prefill at 256K went from 297 to 628 tokens per second, a 2.1× gain. The ceiling for that context was about 670–684, so the team got close to it.
  • KV cache went from 264K tokens to 4 × 256K.
  • Startup dropped from 113 seconds to 43.

On the Intel Arc Pro B70, prefill at 4K, 32K, 128K and 254K went from 1,654, 1,434, 928 and 650 tokens per second to 2,421, 2,259, 1,415 and 911. Here's what actually matters for you: a larger KV cache is only useful if you need the context. Every gigabyte given to KV is one not available for weights or batch headroom, so choose the context you'll really serve.

The same card also shows why engine choice needs its own test. On the same card and day, SGLang reached 82.6 tokens per second for one user against 75.7 on vLLM. Those are different engines on one machine, so the gap is real for that setup, but it doesn't generalize to every model.

Measurement Traps That Fake a Win

Once you trust your controls, the next danger is the benchmark itself. The skill's methodology notes list traps that show up again and again:

  • Chunk-counted tokens. Streamed chunks aren't tokens. Count tokens from the server's usage statistics.
  • Mis-sized prompts. A "32K prompt" built from characters or words may be nowhere near 32K tokens. Measure lengths with the model's own tokenizer.
  • Acceptance counters read as speed. Covered above; they explain speed, they don't measure it.
  • Drift. Thermal throttling and background load can shift results by more than your candidate change.
  • Unified-memory accounting. Container tools like cgroups don't see GPU driver residency on unified-memory machines. The repo's mem.sh reports the footprint including that residency.

Here's the uncomfortable question: how much of your last "speedup" was the change, and how much was the afternoon? If you can't answer that, the number doesn't belong in your notes yet.

Gate on Long-Context Retrieval Before Calling Anything "Best"

A configuration can be fast and broken. The skill won't call anything "best" until it passes three checks: an exact-match needle test at long context, a concurrent soak with the KV cache full, and a check that gains have plateaued. Only then does it move to a pinned, clean build and a set of acceptance gates.

This matters because long-context failures are quiet. A model can answer short prompts well and still lose a needle buried at 128K. The B390 run used these gates as part of its progress tracking, and the team logged failures alongside wins in a PLAN.md ledger. The list of things that didn't work is often the most useful artifact.

The H200 result shows the stakes. A DeepSeek-V4.1-Flash deployment on 4× H200 NVL crashed under load before tuning. After the work, it reached 229.1 tokens per second decode and 17,546 tokens per second prefill, with all six acceptance gates passing. Speed you can't sustain under load is not speed.

Results at a Glance

HardwareModel, quant and engineBefore → after
Intel Arc B390 iGPU (64 GB unified)Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTPDecode, 1 user: 43.6 → 53 tok/s; prefill 256K: 297 → 628 tok/s
Intel Arc Pro B70Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernelsDecode, 1 user: 91.2 tok/s; llama.cpp Q4_K_M on same card: 25.0 tok/s
RTX 3090Qwen3.6-35B-A3B, EXL3 3 bpw + MTP251.7 tok/s prose / 357.9 tok/s code, 1 user
4× H200 NVLDeepSeek-V4.1-Flash, vLLM + speculative decodingCrashed under load → 229.1 tok/s decode, all six gates pass

Treat the llama.cpp row with care. It uses a different quantization format from the EXL3 result, so it shows an engine-and-format gap, not a clean engine comparison. Those figures come from the repo's local-ai-registry and trellis-serve recipes, and the raw logs are linked from there.

Frequently Asked Questions

What is a local LLM inference optimisation skill?

It is a structured procedure, written for AI coding agents, for measuring and improving the serving speed of a local model. It covers decode, prefill, KV cache, startup and TTFT under a memory cap. It works with any OpenAI-compatible server and includes a benchmark harness and ledger.

How do I improve local LLM inference speed without guessing?

Compute the theoretical ceiling first, so you know how much headroom exists. Then change one variable at a time and compare each run against control runs before and after it. Keep a change only if it beats both controls and regresses nothing by more than 3%.

Why doesn't a higher speculative decoding acceptance rate make my model faster?

Speculative decoding adds drafting and verification work to every step. If the drafting cost per step rises faster than the accepted tokens save, total throughput can fall even as acceptance climbs. Tune draft length and vocabulary size against measured tokens per second, not acceptance alone.

How many samples do I need to trust a tokens-per-second benchmark?

Use at least six samples and compare medians, not single runs. Run-to-run noise is typically ±5–10%, so smaller differences are usually drift. Interleave control runs so environmental changes affect both sides of the comparison.

Pick one bottleneck on your own machine this week, compute its ceiling, and run a control-candidate-control test before changing anything else. Then write down what didn't work, because that log will save you the most time on your next tuning session. Pass the skill's workflow to someone whose local model feels slow but whose numbers are unknown, and let the ceiling tell you whether they've got headroom left.