The default answer in 2026 to “should this action be allowed?” is to ask an LLM. It’s the path of least resistance: send the context to a model, get back a yes/no, move on. It works well enough. And it’s quietly, cumulatively wasteful.
What does an LLM call actually cost?
A single GPT-4 inference draws roughly 0.3–1.5 watt-hours depending on context length and output tokens. That’s comparable to a Google search or an LED bulb running for an hour. One call is negligible.
But production AI agents don’t make one call. An agent loop typically invokes the model before, during, and after each tool execution. A coding agent working through a feature might make 50–200 tool calls in a session. Each one preceded by an LLM deciding whether the tool call is safe.
Let’s do the math for a mid-size team:
- 20 developers using AI coding agents
- 100 tool calls per developer per day
- Each tool call checked by an LLM guardrail
- 2,000 LLM guardrail checks per day
- ~1.5 kWh per day just for safety checks
- ~400 kWh per year
That’s roughly the annual electricity consumption of a refrigerator. For safety checks. That could run on a Raspberry Pi at 0.004 watts.
Where the energy goes
An LLM inference pass involves:
- Tokenization and embedding lookup — negligible
- Forward pass through N transformer layers — each layer is a matrix multiply across the full model dimension. For a 70B model, that’s billions of floating-point operations.
- Attention computation — quadratic in sequence length
- Sampling — negligible
The forward pass dominates. And it runs on GPUs — A100s or H100s — that draw 300–700 watts at full load. A single GPU-hour costs $1–3 in cloud credits and produces roughly 0.3–0.5 kg of CO2 (depending on the grid mix).
The alternative
Our Rete network runs on CPU. A single core. 124 microseconds per check. At 8,000 checks per second, the power draw is roughly 4 watts — the power consumption of the CPU core itself. Per check: 0.0005 watt-seconds. Or about 0.14 milliwatt-hours.
Compared to an LLM guardrail at 0.5 watt-hours per check, that’s a factor of roughly 3,500x less energy. For the same decision. With better reliability.
This isn’t about LLMs being bad
LLMs are remarkable tools. They’re genuinely useful for code generation, summarization, reasoning about ambiguous situations. We use them in our own work.
The point is that not every decision requires a 70-billion-parameter model. “Is this command destructive?” is a question with a finite, knowable answer space. You can enumerate the dangerous commands. You can parse the AST. You can match against a policy. You don’t need to sample from a probability distribution to decide whether rm -rf / should be allowed.
Using an LLM for policy enforcement is like using a jet engine to toast bread. It works. It’s just not the right tool.
What we’re doing about it
ellm-guardrail is our first product built on this conviction. It replaces LLM-based safety checks with deterministic Rete inference. It runs on a single core. It uses 50 MB of memory. It’s free for small teams.
We’re exploring other places where deterministic inference can replace expensive model calls: CI/CD policy checks, compliance verification, configuration validation. If you’re working on a problem where a rule engine could replace a model call, we’d like to hear about it.