I keep seeing teams apply their DevOps model to AI systems, then wonder why it doesn't quite fit.
DevOps was built around deterministic software: the same input produces the same output, tests can verify behaviour, and staging tells you something useful about production. Our CI/CD, monitoring, and incident response practices rely on those properties.
LLMs don't have them. Agents built on LLMs have them even less.
What breaks when you treat an LLM pipeline like code
The obvious difference is determinism. You can test a function. You cannot test whether an LLM response is "correct" in quite the same way. Its output depends on model versions, prompts, settings, and context. Correctness is often a judgment call.
This breaks several things teams rely on:
Testing: Traditional test suites don't work well for LLM outputs. You end up with either overly brittle tests that break on minor phrasing changes, or tests so loose they don't catch actual regressions. Evaluating LLM quality requires different methods — sampling, human review pipelines, scoring models.
Deployment confidence: In a standard CI/CD pipeline, passing tests means you can deploy with reasonable confidence. With an LLM pipeline, passing tests means you've verified the plumbing works; whether the outputs are actually good is a separate question.
Incident response: When an LLM-powered feature behaves unexpectedly, the root cause is often not a code change. It might be a shift in user input distribution, model drift after a provider update, a prompt that worked in testing but degrades on real traffic, or retrieved context that's silently stale. None of these show up in standard error logs.
LLMOps: the operational layer that had to be invented
The term is awkward, but the work is real. LLMOps covers the parts of running an LLM in production that DevOps doesn't.
In practice, that includes:
Prompt versioning: Prompts are code, but most teams don't treat them that way. They live in config files, environment variables, or worse, hard-coded in functions. When a prompt change causes a regression, you want to know what changed, when, and why — the same as any other code change.
Token cost monitoring: LLM calls have variable cost based on input/output length, model choice, and caching behaviour. Without monitoring, costs can spike unexpectedly. This is especially true when retrieval-augmented generation (RAG) is involved and the retrieved context is longer than expected.
Evaluation pipelines: Some teams use a second LLM for automated scoring. Others sample outputs for human review. The right approach depends on the risk of the use case.
Hallucination guardrails: For use cases where factual accuracy matters, output validation is worth investing in — whether that's retrieval verification, structured output schemas, or downstream checks that catch responses before they reach users.
AgentOps: the next layer
LLMOps deals with individual model interactions. AgentOps deals with chains of them, plus tools, memory, and branching logic.
This introduces new failure modes that are harder to observe and debug:
Multi-step failures: An error in step three of a five-step agent may only manifest in the final output. The cause and effect are separated in time and in the trace.
Tool permission boundaries: Agents with tool access can take actions with real-world effects — sending emails, writing files, calling APIs. Getting the permission model right (what can the agent do, under what conditions, with what confirmation requirements) is a safety concern, not just an operational one.
Memory and context accumulation: Long-running agents accumulate context. How that context is managed — what gets summarized, what gets dropped, what gets stored and retrieved — affects both output quality and cost.
Observability gaps: Standard APM tools weren't designed for multi-step LLM traces. You need tooling that can show you the reasoning path, the retrieved context, the tool calls made, and the cost of each step in a single view. This tooling exists now, but adoption is still uneven.
The practical bit
None of this should be intimidating. It is learnable, and the tooling has improved.
I would push back on the idea that you can adopt agents without matching operational investment. Without observability, prompt versioning, and a coherent evaluation approach, systems work in demos and then degrade in production for reasons that are hard to trace.
The teams I've seen do this well treat operations as engineering work, invest in evaluation early, and define which actions require a human.
That doesn't require a new team or new job titles. It requires the same engineering discipline as any other production system, adjusted where AI behaves differently.
If you're building in this space, I'm happy to compare notes on the operational model.