Quick Answer: Realtime LLM steering means influencing a language model's output while you watch it generate, rather than editing a prompt and waiting for a full reply. Chuanqi Sun's open-source Spaceship IDE demonstrates this with speculative generation, cache invalidation on every control change, and four interaction ideas designed for models that reach 60 FPS interactivity.
Most people still steer an LLM the way they'd send a letter: write a prompt, wait, read the reply, then try again. Chuanqi Sun's Spaceship IDE asks what happens when that loop shrinks to frame-rate speed. The project starts from one conjecture: LLM and decision models will reach 60 FPS interactivity. If that happens, realtime LLM steering stops being a novelty and becomes a new way of working. This is an experiment, not a product, and that's exactly why it's worth studying.
Why 60 FPS Is the Real Conjecture
The project's central claim is simple: "In the near future, LLM and Decision Models will reach 60 FPS interactivity." Everything else follows from that assumption. At today's latencies, a language model is a request-response service. You can't steer it mid-sentence, so you steer it between sentences.
Frame-rate interactivity changes the unit of control. A slider, a knob, or a highlighted phrase becomes an input that reshapes text as it's being produced. That's a different interface problem from the chat box. It's closer to a musical instrument or a flight control stick, which is why the project borrows space-pilot language.
Here's the catch: the conjecture is about the future, but the engineering problems are present now. Spaceship IDE works within today's speeds by predicting ahead and throwing away predictions that no longer match the user's intent.
The Four Interaction Paradigms
The author lays out four candidate paradigms. Each one is a question the project tries to answer through a working demo, not a finished feature.
- Focus monitor: Can you visualize what the model is attending to, and use that display to guide your own attention toward what matters?
- Attention lock: Can you force the model to weight specific phrases more heavily, biasing its future behavior?
- Future vector steering: Can you decompose a task into eigenvector-like components, tune each one independently, and recompose them to influence incoming text?
- Realtime re-generation: When generation is cheap and fast, can you explore alternative futures by scrubbing back and forth through them?
The four are not equally mature. Focus monitoring and realtime re-generation are the most concrete, because they map onto things you can see on screen. Future vector steering is the most speculative. It assumes the task can be cleanly decomposed into independent components, and that assumption is unproven. Treat it as a research direction, not a working technique.
Notice what's missing from the list: "make the model smarter." The project is about control and perception, not raw capability. That framing is the most useful thing to take from it.
How Speculative Generation Makes It Feel Instant
Here's where the engineering gets interesting. To feel realtime today, the project uses aggressive speculative generation. The system starts producing likely future text before the user asks for it, so something is already waiting when a control moves.
The trade-off is that speculation is wasted work whenever it's wrong. Speculative decoding, a broader technique in which a cheaper draft produces candidate tokens that a larger model verifies, works on the same principle: spend compute on guesses and keep the ones that hold up. Spaceship IDE applies that instinct at the interface level.
The hard part is cache invalidation. The author describes a specific case: when the main knob moves backward, the system must empty the future text and cancel in-flight requests. Otherwise the user sees stale text that no longer matches the control position. This is the kind of detail that's easy to miss on a whiteboard and painful in production.
Here's what most people get wrong about this: stale speculation is worse than no speculation. A slow response that's correct feels more trustworthy than a fast one that quietly contradicts the controls. That's why the author says the system took "a lot of human judgement," and that Claude Code didn't one-shot these decisions.
Realtime Steering vs. Traditional Prompting
The table below compares common ways of steering a model. The rows are interaction styles rather than products, and the cells reflect the trade-offs the project's design implies.
| Steering approach | Control granularity | Latency requirement | Main risk |
|---|---|---|---|
| Prompt rewrite and resubmit | Whole request | Tolerates seconds per turn | Slow iteration; you lose context on each retry |
| Regenerate from a fixed point | One output branch | Needs fast re-generation to feel usable | Many near-duplicate outputs to compare |
| Realtime scrubbing (Spaceship IDE) | Continuous, via knobs | Near-frame-rate; depends on speculation | Stale text if cache invalidation is wrong |
| Attention biasing (attention lock) | Specific phrases | Must update during generation | Over-weighting one phrase can distort the whole output |
| Vector-style steering (future steering) | Independent task components | Requires fast recomposition | Assumes the task decomposes cleanly |
The comparison makes one trade-off clear. Finer control costs more latency budget and more opportunities for inconsistency. Coarse prompting is slow but predictable. Realtime steering sits at the other end of that spectrum.
Where the Popular Advice Goes Wrong
The common assumption is that faster generation automatically makes a model easier to control. Spaceship IDE suggests the opposite can happen. Cheap generation floods you with options. If you can regenerate a thousand times a minute, the bottleneck moves from producing text to judging it.
This is why the realtime regeneration idea is framed as scrubbing rather than generating. Scrubbing gives you a timeline you can navigate, which is a finite interface. A feed of endless alternatives is not. The design challenge is to keep the number of decisions a human has to make small enough to stay in flow.
That said, the counterargument has merit. Speed does help when you know what you're looking for. A designer iterating on a layout or a writer testing tone may benefit from seeing five variants in a second. The honest answer is that realtime steering helps most when the user already has a direction and needs to explore its neighborhood quickly.
Risks, Caveats, and the Privacy Question
If you try the hosted demo, pay attention to how it handles credentials. The project lets you run the simulator with your own API keys for Cerebras and Gemini. According to the README, keys stay in your browser's local storage. The author adds a blunt warning: "Do it at your own risk."
Local storage is convenient, but it's not a secrets vault. Any script running on the same origin can read it, so a compromised dependency or a malicious browser extension could expose your keys. For a demo, that's an acceptable trade-off if you use keys with low spending limits and rotate them afterward. For anything sensitive, route calls through a server you control.
The project is also a demonstration of plausible interactions, not a benchmarked system. Nothing in the source establishes how well these paradigms perform for real tasks, so treat the demos as sketches of possibility.
What Comes Next for Human-Centered Steering
The project sits inside a broader research agenda. The author is pursuing a master's thesis at the MIT Media Lab on how semantic and symbolic systems can be coupled to enhance human creativity, agency, and "joy of craft." Spaceship IDE is presented as a small step in that direction.
That context matters. The goal isn't to automate writing. It's to give people better handles on a process that's currently opaque. The focus monitor, for example, is an attempt to make the model's attention legible so the human can make better decisions. Interpretability in service of user agency is a more interesting target than raw speed.
Did the project find the right interaction primitives? Nobody knows yet, and the author doesn't claim to. But the design questions it raises, about what to show, what to let people control, and when speed helps versus hurts, will matter as models get faster regardless of whether this particular interface wins.
Frequently Asked Questions
What is realtime LLM steering?
Realtime LLM steering is influencing a language model's output while it generates, using continuous controls rather than prompt-and-wait cycles. It depends on fast inference and on managing speculative text so that controls and output stay in sync. Spaceship IDE is an open-source demo of the idea.
How to steer LLM output while it generates?
Use a control that maps to a model-side parameter, then update the generation as the control moves. Spaceship IDE predicts likely future text in advance and discards it when the control changes. Cancel in-flight requests and clear stale text immediately, or the interface will show output that contradicts the user's input.
Why does speculative generation need cache invalidation?
Speculative generation produces text before the user asks for it, so that text can become wrong when the user changes a control. Invalidation removes outdated predictions and cancels requests tied to the old state. Without it, the interface shows confident-looking text that no longer matches the current settings.
Is it safe to use my API keys in the Spaceship IDE demo?
The project stores keys in browser local storage, which keeps them off the author's servers but exposes them to any script on the same origin. Use keys with spending limits, avoid sensitive accounts, and rotate keys after testing. For production use, proxy requests through a backend you control.
Realtime LLM steering is still an open research question, but the engineering lessons are concrete today. If you want to see how speculation, invalidation, and control design fit together, clone the Spaceship IDE repository and try the live demo this week. Pick one knob, scrub it backward, and watch what happens to the future text. Then pass the repo to someone building an interface around a slow model, because the cache problems they hit will look a lot like these.