Observability, security, and cost control are all critical for LLM applications. Here are a few things you could do to enhance their efficiency.
After the article on Ollama and Docker carried in the July 2026 issue of OSFY, a few readers wrote in with the same question. A gentleman from Coimbatore put it best: “Sir, model has been running for two months and everybody is happy, but now my CFO is asking me questions I cannot answer.”
I got a call in April that said the same thing, but from the finance side. A friend who runs engineering at a logistics company in Pune, someone I have known since our Wipro days, rang on a Thursday to tell me their AI spend had gone from Rs 42,000 in February to Rs 1.1 million by mid-April. He wanted to know why and how that happened.
I asked which service was responsible. Nobody knew. One of his leads said they could grep the application logs and try to work it out, which is a reasonable instinct but would not have helped. Three teams had shipped LLM features in six weeks. None had instrumented anything beyond what Django gave them. In an access log, a request that makes fourteen model calls looks the same as one that makes a single call.
It was the agent. Someone had built one that could call five internal APIs and keep looping until it liked its own answer, with no ceiling on the retry. We found the flaw in two days once traces existed. Before that, we were guessing while the meter ran.
I have had that conversation about fifteen times now. This article is about what I usually end up explaining.
Most of your platform is fine
Some consultants will tell you AI changes everything and your stack needs rethinking. Usually they have a proposal ready. Kubernetes is still Kubernetes. ArgoCD does not care what runs in the pod. Prometheus scrapes as it always did. Most of what you built stays.
But three things break.
They fail without noise
A retail client of ours had a support assistant that spent eleven days describing a returns policy withdrawn in 2024. Dashboards were green throughout, uptime was fine, and the error rate was zero. Standard monitoring read a well-formed 200 as success. But here it only meant the service had replied.
The cost signal gets delayed by a month
On a normal service, wasteful code shows up as load within minutes. Someone sends 40,000 tokens of context in every request because that is easier than writing a retrieval filter, and nothing happens. No spike, no alert. Finance forwards a PDF four weeks later. Thirty-day feedback loops are bad for engineering discipline.
They take instructions from text you do not control
System prompt, retrieved documents and user input get glued into one string and handed to a model that cannot tell which part carries your authority. A PDF uploaded by a customer becomes executable in a limited but real sense. Not shellcode, but it can change what your software does next.
Everything goes through one door
Most teams end up building the same component and putting it in the middle. Model gateway, AI proxy, inference router — the name varies. Applications stop calling models directly and call the gateway instead.
This looks like unnecessary indirection until you try to enforce a spending limit across four teams that do not have one. It is the only place in the architecture where enforcement is possible.
We use LiteLLM on most engagements. It speaks the OpenAI dialect that every SDK understands and routes to whatever sits behind it, including your own vLLM boxes or an Ollama instance. Envoy AI Gateway and the Kong AI plugins work well if you already run those at the edge. Writing your own thin proxy is possible, though the open source options are mature enough that it mostly buys you maintenance work.
yaml model_list: - model_name: fast-tier litellm_params: model: openai/qwen2.5-7b-instruct api_base: http://vllm-qwen.inference.svc.cluster.local:8000/v1 api_key: os.environ/INTERNAL_KEY - model_name: heavy-tier litellm_params: model: anthropic/claude-sonnet-4-6 api_key: os.environ/ANTHROPIC_KEY litellm_settings: callbacks: [“otel”, “langfuse”] cache: true cache_params: type: redis host: redis.platform.svc.cluster.local ttl: 3600 general_settings: master_key: os.environ/LITELLM_MASTER_KEY max_budget: 250000 budget_duration: 30d alerting: [“slack”]
Routing, caching, tracing hooks, budget and alerts are in one file. The platform team owns it, and product teams do not think about it.
The value shows up later. When a better open model lands next quarter, you edit a mapping. We did that for a client in Ahmedabad on a Tuesday morning: 40 minutes, no downtime, and nobody outside the platform team noticed. Without a gateway, fourteen services would each have carried their own client library and hardcoded model string in a config file nobody had opened since the engineer who wrote it left in 2024.
Traces first
This is the cheapest fix and it makes everything else visible.
OpenTelemetry now carries semantic conventions for generative AI. Model name, token counts, temperature, finish reason, and tool calls all have agreed attribute names. LLM spans land in the same Tempo or Jaeger you already run, on the same trace ID as the HTTP request that started things. An LLM call is a span, nested in the request span, next to your Postgres spans. Not a separate system with its own dashboard.
Instrumentation is cheap with OpenLLMetry or the native OTel libraries.
python
from traceloop.sdk import Traceloop
from traceloop.sdk.decorators import workflow
Traceloop.init(
app_name=”claims-assistant”,
api_endpoint=”http://otel-collector.platform:4318”,
)
@workflow(name=”summarise_claim”)
def summarise_claim(claim_id: str, document: str):
context = vector_store.search(document, top_k=5)
return llm.invoke(
system=SUMMARY_PROMPT,
user=f”{context}\n\n{document}”,
)
Spans are not the whole picture, because the unit you want to reason about is a conversation. Langfuse handles that for us. Open source — self-hosts on Postgres and ClickHouse — gives session grouping, prompt versioning and cost per user. It also gives a human somewhere to mark bad outputs as bad, which is how you build a test set that reflects reality.
Arize Phoenix is worth knowing for retrieval problems. If you are trying to work out why a RAG pipeline keeps pulling the wrong chunk, it will get you there faster. I use both.
yaml
services:
langfuse:
image: langfuse/langfuse:latest
depends_on: [postgres, clickhouse]
ports:
- “3000:3000”
environment:
DATABASE_URL: postgresql://postgres:${PG_PASS}@postgres:5432/langfuse
CLICKHOUSE_URL: http://clickhouse:8123
NEXTAUTH_URL: https://llm-obs.internal.company.in
SALT: ${LANGFUSE_SALT}
TELEMETRY_ENABLED: “false”
Check that last setting on every tool you install. If you self-host for data residency reasons, it matters. I have seen two air-gapped deployments where three components were still sending usage statistics to a vendor endpoint. Nobody was careless. They had not read that far into the documentation.
On metrics, keep the list short. Track time to first token separately from total latency, because nine seconds to first token feels broken even when the full response arrives at twelve. Token counts as a distribution, not a mean, since a mean hides the one path sending 50,000. Cache hit rate at the gateway warns you before the bill moves. Check tool call depth for agentic workloads, with an alert on it. If you self-host, watch GPU memory and KV cache utilisation rather than CPU.
I gave bad advice about this for a year
Through 2025 I told clients evaluation could wait. Plumbing first: tracing, costs, guardrails, then quality once the base was steady. That was wrong and I have stopped saying it.
What happened was that teams got everything else right, shipped it, and it all degraded over three or four months. Prompts collect small edits from five or six people, and each edit is defensible alone. By the time someone notices the assistant has gone mediocre, there is no baseline and no way to identify the change that caused it. A team in Bengaluru lost almost a month rebuilding a prompt from memory and old Slack threads.
Run evals in CI: a fixed dataset against every prompt change, merge blocked when the score drops. Sample in production too, because your fixed dataset does not contain the questions users actually ask. Promptfoo is the most CI-friendly option and drops into GitHub Actions like a linter. Ragas is built for retrieval, and gives faithfulness and context relevance scores. DeepEval is pytest-shaped.
yaml - name: Run prompt evaluations run: | npx promptfoo eval \ --config evals/support-assistant.yaml \ --output results.json npx promptfoo eval --assert-threshold 0.85
Thresholds take a few weeks to tune and the team will be annoyed with you for a fortnight. Sit through it.
The document tells your model what to do
Direct injection is a user typing something clever into your chat box. Indirect injection is that text sitting inside a document, web page, email or database row that your retrieval step fetched. The second is the real danger: the attacker never touches your application, and your retrieval layer was built by people thinking about relevance rather than trust.
There is no clean fix, whatever a vendor tells you. Keep system instructions and untrusted content in separate message roles rather than concatenating everything into one string. Run a classifier at the gateway, whether Llama Guard, Prompt Guard or something small you tune yourself. Then assume something gets through and limit what it can do.
That last part matters most. Real damage rarely comes from the model saying something rude. It comes from the model calling a tool it should not have called, with parameters it chose. The defence is authorisation, not a better prompt. Tool calls should run with the end user’s permissions rather than the service account’s, which means threading identity through the whole chain. Reads can be automatic, writes need confirmation, deletes should be hard. Code execution goes in a sandbox: gVisor, Firecracker, or at minimum a stripped container with a hard timeout and no network.
Deny egress by default from inference workloads.
yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: agent-egress-restricted namespace: llm-apps spec: podSelector: matchLabels: app: agent-runtime policyTypes: [“Egress”] egress: - to: - namespaceSelector: matchLabels: name: platform ports: - port: 4000 protocol: TCP
Twenty lines. It will not stop an attacker who already has execution and time, but it stops the common case.
Then there is data handling, which people forget. Prompts and responses are among the most sensitive material your company holds, and by default all of it goes into trace storage in plaintext, indefinitely. Redact at the collector, before storage. Presidio is the standard open source answer and drops into an OTel processor. Set retention on the observability store. Keeping traces forever is defensible engineering and indefensible privacy, and under the DPDP Act that distinction is legal rather than philosophical. If a customer asks where their prompts go and for how long, someone should be able to answer in a paragraph.
Supply chain is the same discipline you already apply elsewhere. Models are dependencies you did not write. Pin by digest, not a floating tag. Generate an SBOM with Syft, scan with Trivy or Grype, sign with Cosign so admission control can verify what runs. Use Safetensors instead of pickle, since pickle deserialisation executes arbitrary code. Mirror weights into your own registry, because depending on an external service staying up and a repository staying public is a risk somebody should have signed off.
Use less money
The Pune bill was a shock because nothing reported anything for thirty days. Shortening that loop is most of the work.
Tag every gateway request with team, service, and environment. Record input tokens, output tokens, cached tokens and computed cost. Send it to Prometheus and Langfuse; then put it on a Grafana board so that engineers see without looking for it. If you self-host, OpenCost or Kubecost gives GPU node cost per namespace. Together, these let you talk in unit terms: cost per conversation, per resolved ticket, per document. Those numbers survive a board meeting. Raw monthly spend only causes panic.
Set budgets with alerts at 70 percent rather than 100 percent, with hard caps on non-production.
What brings the number down, in order of impact:
Route by difficulty
Most requests do not need your best model. Classification, extraction, routing and reformatting run well on a 7B or 8B open model you host yourself at fixed cost, leaving the expensive tier for hard reasoning. A decent router routinely halves the bill.
Cache
Exact-match caching at the gateway is free money on repeated queries. Semantic caching goes further but needs a carefully chosen threshold, since a loose one serves confident answers to questions nobody asked.
Fix your context
I audited a RAG pipeline last year that pulled twenty chunks where five sufficed and resent the full conversation history every turn. Trimming cut tokens by around 60 percent with no measurable quality drop. I check this first on every engagement now.
Batch properly if you self-host
vLLM’s continuous batching and PagedAttention give several times the throughput on the same card. Quantised builds, as I wrote in July, usually deliver most of the quality at a fraction of the memory.
Cap output tokens per endpoint
A model will write nine paragraphs where two were requested.
None of this is clever. It is dull platform work, which is why it waits for a bill.
Self-host or not
Regular readers know I am not a purist here.
Nearly everyone seriously ends up hybrid, and the platform layer is what makes that workable. Self-host the high-volume, predictable, privacy-sensitive work: embeddings, classification, extraction, internal knowledge assistants. That belongs to your own GPUs, where fixed cost beats per token pricing once volume settles. Keep frontier commercial models for low-volume work where the extra reasoning quality changes the outcome. The gateway lets you make that decision per workload rather than per company and revisit it each quarter as open models close the gap.
Starting from nothing, the order that works is: gateway and tracing first, even if you change nothing in the applications, because a week of visibility teaches more than six months of meetings. Cost attribution and caching second. Guardrails and redaction third. Evals fourth, and not much later than that.
Things I would not skip:
- Gateway from day one. Retrofitting is painful and turns political.
- Prompts in Git as versioned artefacts, not strings in code.
- Someone reads twenty real production outputs weekly. A human finds what metrics miss.
- Hard caps on dev spend, not alerts.
Treat every retrieved document as hostile until proven otherwise.Keep a rollback path for prompt and model changes. They are deployments.
Where this leaves us
Pune is fine now. They capped the agent loop at four calls, moved summarisation to a self-hosted Qwen model and turned on caching. They run at about 150,000 a month. The answers improved too, because the traces exposed a retrieval bug that had been there since the first release.
None of those fixes were difficult. All of them were visible the moment the team could see what their software was doing.
Getting a model to answer is solved by tools you can install in an afternoon. Running twenty of them without losing track of what they cost, what they tell customers and what they can touch is not solved. That is where the work has moved.
Nearly every piece you need is open source, self-hostable and already running in production somewhere. You need no budget and nobody’s permission. A gateway, somewhere to put traces, and a willingness to look at what your systems are doing will take you a long way.
Put the layer in before you need it.
