29. Reliability Engineering for LLM Apps

What this section is for

Section 29 teaches you how to make LLM apps reliable in the real world.

LLM calls introduce new failure modes:

• variable latency,
• rate limits and quotas,
• provider outages,
• non-deterministic outputs,
• schema drift and partial responses,
• long prompts that slow everything down.

The goal is to keep your system usable even when the model is slow, wrong, or unavailable.

The reliability problem in one sentence

LLM calls are expensive and variable. You need budgets, timeouts, retries, fallbacks, and instrumentation so the rest of your app stays stable.

Reliability principles for LLM apps

• Timeout everything: no unbounded waits.
• Retry safely: only when idempotent and with backoff.
• Fail gracefully: fallbacks and degraded modes are normal.
• Cache deliberately: cache what’s safe and stable; avoid caching secrets.
• Stream when helpful: improve perceived latency with partial rendering.
• Observe everything: logs, traces, and metrics that tie output to inputs and costs.

Reliability by layer (network → model → UX)

Reliability is a stack:

• Network layer: timeouts, retries, circuit breakers.
• Model layer: determinism settings, validation, structured output constraints.
• Pipeline layer: retrieval timeouts, context budgets, caching.
• UX layer: streaming, progress indicators, degraded mode messaging.
• Ops layer: monitoring, alerting, incident runbooks.

Section 29 map (29.1–29.5)

• 29.1 Timeouts, retries, and idempotency
• 29.2 Circuit breakers and fallback modes
• 29.3 Streaming responses and partial rendering
• 29.4 Caching strategies (prompt+context caching)
• 29.5 Observability: traces, metrics, and prompt logs

Where to start

• Start with 29.1
• Reference: cost and tokens (Part XI)