Quick Answer: A local AI harness is a small program that connects your terminal to language models, tools, and reusable skills without a desktop app or background service. Shim is an MIT-licensed example. It ships as one static binary with zero dependencies and supports OpenAI-compatible and Anthropic APIs, MCP servers, Agent Skills, and plain-language commands.
Setting up a coding agent shouldn't take longer than using it. Yet most setups still involve provider IDs, endpoint URLs, MCP JSON files, skills folders, and a new interface for every tool. Shim, a project published on GitHub by Balanov, tries to be the thin layer that handles all of that. The maintainer reports a 7.29 MiB stripped binary and a 9.96 ms median startup for shim --version over 30 runs on Windows. That's a local measurement, not a promise for every machine, but it shows what "tiny" looks like in practice.
What a Local AI Harness Actually Does
A harness sits between you and the model. It builds the prompt, exposes tools, asks permission before risky actions, and saves the session. Your model does the reasoning. The harness does everything around it.
Shim keeps that layer small on purpose. It doesn't need Electron, Node, Python, Docker, a database, or a daemon. Config is plain JSON, sessions are append-only JSONL, and network traffic happens only when you make an explicit model call, download a skill, run local discovery, or talk to an MCP server you configured. That last point matters if you're auditing where your prompts go.
Here's the misconception most guides skip: "any model" means any model behind an HTTP API. Shim doesn't load GGUF weights. You run those through Ollama, LM Studio, llama.cpp, or another inference server, and Shim talks to the server's endpoint. Once you see the harness as a client rather than an engine, the setup makes much more sense.
Connecting Models Without Hand-Editing Config
The common friction point is provider configuration. Shim's presets cover Ollama, LM Studio, llama.cpp, OpenAI, OpenRouter, Anthropic, Gemini, GitHub Models, Hugging Face, and custom endpoints. Local presets need no key. Hosted ones read credentials from environment variables such as OPENAI_API_KEY or ANTHROPIC_API_KEY.
A typical local-first setup looks like this:
- Start your local inference server, such as Ollama.
- Run
shim setupto create the config directory. - Run
shim model discover --addto find running local servers and register them. - Set a default with
--use, then verify it withshim model test.
For a custom endpoint, you point Shim at any OpenAI-compatible base URL, for example http://127.0.0.1:8000/v1. Keeping the endpoint on loopback means inference never leaves your machine.
Here's where people trip up. Shim never writes API keys into its config. If you set a key in your shell profile, it's still in plaintext on disk, just somewhere else. Use your OS or shell's secret mechanism, and keep project config free of credentials.
Plugging In Tools: MCP and Agent Skills
Shim supports two extension paths, and they solve different problems.
Model Context Protocol (MCP) gives the model live tools. Shim speaks MCP over stdio and Streamable HTTP. An MCP server exposes functions such as "list directory" or "query a database," and Shim namespaces each one as mcp_<server>_<tool>. The namespace prevents collisions when two servers both offer a search tool.
Agent Skills are instruction bundles defined in SKILL.md files. Shim scans four locations: the global skills/ directory beside its config, .shim/skills/, .agents/skills/, and .claude/skills/. Only each skill's name and description go into the system prompt. The full instructions load on demand through load_skill. That lazy loading keeps the context window from filling up with instructions you may never need.
The trade-off is worth stating plainly. Skills are instructions, not code, but they can still steer the agent toward shell commands or network calls. The source is explicit that skills are untrusted. Read them before you install them, especially if they ask for shell or network access.
Permissions: Why "Local" Doesn't Mean "Sandboxed"
Most people assume a local tool is a safe tool. That's the most dangerous assumption in this category.
Shim's default ask mode allows reading, listing, and searching workspace files without prompts. Writing files, running shell commands, and calling MCP tools all require confirmation. File tools reject lexical traversal and symlink escapes from the workspace. That's useful, but it isn't isolation.
The README is direct about this: Shim is not an OS sandbox. Shell commands and MCP servers run with your user's permissions, and workspace path checks won't contain a malicious process. Here's the cause-and-effect chain. If a repository file contains a prompt injection, the model might request a shell command. You approve it because it looks reasonable. The command runs with your full user rights. The confirmation prompt protects you from careless actions, not from a determined one. For untrusted code, use a container, a VM, or an OS-level sandbox.
For one-off runs, shim run --read-only "explain this repository" is safer than changing your global default. Use --yes only when you've scoped the task tightly.
Natural Language vs. the Full CLI
Shim offers two interaction styles, and choosing between them is a real design decision.
Plain-language commands are convenient. Shim has deterministic offline phrases for common English and Russian management requests, so "show my tools" maps directly to /tools without calling a model. For other languages, you can assign a local router model with shim router use ollama/qwen3:4b. The router distinguishes configuration requests from agent tasks, and every mutating command still needs confirmation.
The full CLI is better when you need repeatability. Scripts, CI jobs, and documented runbooks should call explicit commands. The table below summarizes the trade-off.
| Approach | Best for | Main risk | Confirmation behavior |
|---|---|---|---|
| Deterministic phrases (EN/RU) | Quick config changes, offline use | Limited phrase coverage | Mutating commands prompt |
| Local router model | Other languages, mixed requests | Router misreads intent | Mutating commands prompt |
Full CLI (shim model add, shim run) | Scripts, reproducible setup | Typos apply immediately | Depends on --yes or --read-only |
Interactive terminal (shim) | Exploratory work | Long sessions drift | Mode set by shim permissions |
The router detail deserves a caution. A router that runs on a remote model sends your management requests to that provider. Use a loopback model if you want the routing step to stay local. Your primary model handles normal conversation in whatever language it supports.
Where Shim Fits Among Terminal Agents
Here's the counter-intuitive part: a more capable agent isn't always the right tool. Heavier harnesses bring TUIs, servers, and ecosystems that add setup time and a larger surface to audit. Shim makes a different bet, and the README compares it openly with OpenCode, Pi, Goose, and Aider.
| Project | Core strength | Shim's different bet |
|---|---|---|
| OpenCode | Full-featured provider-agnostic terminal workspace | No TUI or server surface; conversational setup and a tiny static binary |
| Pi | Minimal, extensible coding-agent toolkit | Zero runtime dependencies and an allowlisted natural-language control layer |
| Goose | Broad MCP-powered automation ecosystem | Smaller local-first control plane with no daemon or desktop app |
| Aider | Git-native pair programming | General agent harness with MCP, lazy skills, and conversational management |
If you mostly want git-aware pair programming, Aider is the more direct fit. If you want a single binary that rewires models and tools without a daemon, Shim is the better bet. The project is also young. Its v0.1.0 roadmap lists streaming output polish, OS-level sandbox profiles, and OAuth helpers for remote MCP as unfinished work. Check those gaps against your needs before committing.
Frequently Asked Questions
What is a local AI harness?
A local AI harness is a client program that runs on your machine and connects a model to tools, skills, permissions, and saved sessions. It doesn't run the model itself. Shim is one example: a static binary that talks to OpenAI-compatible and Anthropic APIs, runs MCP servers, and loads Agent Skills from local directories.
How do I connect a local model to a terminal AI agent?
Start a local server such as Ollama, LM Studio, or llama.cpp. Then run shim model discover --add to register running servers, or use shim model add custom with your server's /v1 base URL. Verify the connection with shim model test before you start an agent run.
Can I run GGUF model files directly with Shim?
No. Shim is a harness, not an inference engine. Load GGUF weights in Ollama, LM Studio, or llama.cpp, then point Shim at the HTTP endpoint that server exposes. This keeps Shim tiny and lets you swap inference backends without changing your agent setup.
Is a local AI harness safe to run on untrusted repositories?
Not by itself. Shim runs shell commands and MCP servers with your user permissions and isn't an OS sandbox. Use --read-only for exploratory runs, review any skills before use, and run untrusted projects inside a container or VM.
Shim's real advantage is simple: fewer moving parts between you and your models. Install the binary, run shim setup, then connect one local model with shim model discover --add. Ask it to explain a repository in --read-only mode and note how the permission prompts feel. Then pass this to a teammate who's still hand-editing MCP JSON, and let them see what a local AI harness can do with one command.