Telemetry Design: What to Log and What Not to Log
AI systems fail in unfamiliar ways because the “code path” is not only code. A single user request can trigger a chain of events that includes policy checks, retrieval, reranking, tool calls, and a final response that is shaped by model randomness and latency pressure. When something goes wrong, teams either have enough telemetry to reconstruct that chain, or they guess. Guessing is expensive: it burns engineering time, leaks trust, and often leads to fixes that do not actually target the problem.
Telemetry is the discipline of turning invisible behavior into evidence. In practice, it is a set of decisions about what signals to capture, how to structure them, how to protect users, and how to make those signals usable under pressure. Good telemetry makes the rest of the operational stack possible: canaries, regressions, incident response, cost control, and security review.
Flagship Router PickQuad-Band WiFi 7 Gaming RouterASUS ROG Rapture GT-BE98 PRO Quad-Band WiFi 7 Gaming Router
ASUS ROG Rapture GT-BE98 PRO Quad-Band WiFi 7 Gaming Router
A flagship gaming router angle for pages about latency, wired priority, and high-end home networking for gaming setups.
- Quad-band WiFi 7
- 320MHz channel support
- Dual 10G ports
- Quad 2.5G ports
- Game acceleration features
Why it stands out
- Very strong wired and wireless spec sheet
- Premium port selection
- Useful for enthusiast gaming networks
Things to know
- Expensive
- Overkill for simpler home networks
A useful starting point is to treat every request as an “episode” with a stable identity, a timeline, and a small number of facts that must be true for the system to be considered healthy. Those facts vary by product, but the method stays the same.
The Three Layers: Metrics, Logs, and Traces
Telemetry is often described as three complementary layers.
**Metrics** answer “how often” and “how much.” They are aggregated counts and distributions: latency percentiles, error rates, token totals, GPU utilization, cache hit rate, tool invocation frequency. Metrics are the fastest way to see that something changed.
**Logs** answer “what happened.” They are structured event records: a tool call succeeded, a retrieval query returned no results, a policy blocked an action, a response was truncated. Logs are where investigations move from suspicion to proof.
**Traces** answer “where time went.” A trace is a request timeline with spans: policy check, embedding, vector search, rerank, model generation, tool call, post-processing. Traces are how teams understand the shape of latency and where to place optimization effort.
AI workloads stretch all three layers because a single request may traverse more subsystems than a traditional API. If traces are missing, teams over-index on model latency. If logs are missing, teams blame “model behavior” for failures that are actually retrieval gaps or tool timeouts.
What Makes AI Telemetry Different
Traditional web services can often treat “input” and “output” as opaque. AI systems cannot. The difference is not philosophical. It is operational:
- A model’s behavior depends on the prompt, system policies, and retrieved context.
- A model’s cost depends on tokens, which depend on prompt length and output length.
- A model’s reliability depends on the tool and retrieval chain, not only on the model.
That is why operational maturity in AI tends to converge on “version everything that influences behavior” and “log the minimum evidence needed to reconstruct a decision.”
Change discipline is the foundation here. When prompts, tools, and policies shift without a stable identity, telemetry cannot tell you whether a regression came from a model update or a policy edit. That is why teams treat prompts and policies as deployable artifacts with versioning, as described in Change Control for Prompts, Tools, and Policies: Versioning the Invisible Code.
The Minimal Event Schema That Pays for Itself
A telemetry system becomes usable when a small set of fields is present everywhere. These fields do not need to be perfect. They need to be consistent.
A practical minimal schema for AI requests looks like this:
- **request_id**: a unique identifier for the request
- **session_id**: stable across a user session (not necessarily a user identity)
- **timestamp**: event time in UTC with sufficient precision
- **route**: which serving route handled the request (model, router, cascade)
- **model_id**: model name plus version or checkpoint
- **prompt_version**: identifier for the system prompt, tool policies, and templates
- **retrieval_profile**: which retrieval pipeline and index were used
- **tool_policy_version**: identifier for tool permissions and routing rules
- **tokens_in / tokens_out**: measured tokens for prompt and generation
- **latency_ms**: per-span latency where possible, not only total
- **outcome**: success, soft-fail, hard-fail, blocked, timeout
This schema is intentionally small. It avoids storing raw content by default while still enabling correlation. If an incident is reported, the schema makes it possible to pull the trace and the key events for that request without searching through unstructured text.
Logging Content Without Becoming a Data Liability
Raw prompts and outputs are often the most tempting things to log, and also the most dangerous. They can contain personal data, proprietary information, secrets pasted into chat boxes, or confidential business context. That does not mean content should never be captured. It means content capture must be treated as a controlled capability rather than a default behavior.
A workable policy tends to include these rules:
- **Default to structured summaries**, not raw content. Log lengths, token counts, safety classifications, tool selections, and retrieval result identifiers.
- **Capture raw content only for explicit workflows**, such as opt-in user reports, debugging sessions, or regulated audit requirements.
- **Use redaction before storage.** Redaction is not a one-time regex sweep. It is a pipeline with evolving rules and test coverage, as explored in Redaction Pipelines for Sensitive Logs.
- **Treat retention as a first-class variable.** Short retention with strict access often beats long retention with weak boundaries.
Content is also duplicated across systems. If raw prompts are stored in the feedback database, they do not need to be copied into analytics and logs. The simplest way to reduce risk is to not create extra copies.
The content problem connects directly to corpus hygiene. If your system later trains or fine-tunes on captured conversations, content storage becomes a training data pipeline. That is where practices like PII Handling and Redaction in Corpora move from compliance concerns to core infrastructure.
Tracing the AI Chain: Retrieval, Reranking, Tools, and Output
For AI systems, the chain between input and output often includes two high-variance modules: retrieval and tools.
For retrieval, the key is to log identifiers and scores rather than entire documents. A trace should include:
- retrieval query string or its normalized form
- embedding model identifier
- index identifier and filter parameters
- top-k document ids returned
- reranker model identifier and scores
- final context budget used
For tools, the trace should include:
- tool name and policy decision
- arguments (redacted) or argument hashes
- tool call duration and outcome
- retries and fallback paths
This is where cross-category alignment matters. Agents, in particular, can turn tool use into a multi-step action chain. If the system cannot produce an audit trail of agent actions, it becomes hard to answer basic questions about correctness and user trust. That is why teams build structured records similar to Logging and Audit Trails for Agent Actions even when they are not legally required.
Sampling, Cardinality, and the Cost of Observability
Telemetry itself consumes budget. High-cardinality labels can explode metrics costs and degrade performance. Excess logging can flood storage and slow services. The discipline is to decide which signals are “always on,” which are sampled, and which are activated only during incidents.
A robust approach often looks like:
- **Always-on metrics** for latency, error rates, token totals, cache hit rates, and tool usage counts.
- **Sampled traces** for end-to-end timing, with higher sampling for error cases.
- **Event logs** for state transitions (blocked, timed out, retried, degraded), stored as structured JSON.
Sampling policies should be explicit. Many teams sample traces at a low rate for healthy traffic and at a high rate for unhealthy traffic. That is not only cheaper; it is more useful.
Capacity planning and observability are closely linked because telemetry volume scales with traffic. If you do not model that volume, monitoring costs can become a hidden tax. Capacity discipline for tokens and queues is treated directly in Capacity Planning and Load Testing for AI Services: Tokens, Concurrency, and Queues.
Telemetry for Quality: Making “Good” Measurable
Telemetry is not only for failures. It is how “quality” becomes measurable enough to manage. That means defining proxy measures that correlate with user satisfaction and safety.
Common quality signals include:
- **refusal rate** and **policy block rate** by route and user segment
- **citation coverage** and **retrieval success rate** for grounded answering systems
- **tool success rate** and **tool latency** distributions
- **response truncation rate** and **timeout rate**
- **user correction rate** and follow-up patterns that indicate dissatisfaction
The point is not to reduce quality to a single number. The point is to make regressions visible, then make improvements provable.
When a regression is detected, telemetry should support root cause analysis. If it does not, the system will develop a habit of shipping fixes based on anecdotes. The kind of structured investigation needed is described in Root Cause Analysis for Quality Regressions.
Privacy Boundaries and Access Controls
Telemetry is sensitive because it is closer to “what users do” than many other datasets. A mature system defines boundaries along at least three dimensions:
- **who can access the data**
- **what fields are visible to each role**
- **how long the data persists**
A common pattern is tiered access:
- aggregated metrics are broadly visible
- traces are visible to on-call and platform teams
- raw content, when stored, is restricted to a small set of responders with audit logging
Access boundaries should be enforced technically. A policy document is not enough when pressure hits during an incident.
Turning Telemetry Into Action
Telemetry is only useful if it changes decisions. The operational loop often looks like:
- Telemetry detects a deviation.
- Synthetic tests confirm it is real and repeatable.
- The system degrades safely or rolls back.
- The team diagnoses root cause and ships a fix.
- The fix becomes a test, a monitor, and a new guardrail.
Telemetry is the first step in that loop, but it is not the whole loop. A practical next step is proactive validation via Synthetic Monitoring and Golden Prompts, and the final step is learning discipline through Blameless Postmortems for AI Incidents: From Symptoms to Systemic Fixes.
Related reading on AI-RNG
- MLOps, Observability, and Reliability Overview
- In-category: Synthetic Monitoring and Golden Prompts, Change Control for Prompts, Tools, and Policies: Versioning the Invisible Code, Redaction Pipelines for Sensitive Logs, Root Cause Analysis for Quality Regressions
- Cross-category: Logging and Audit Trails for Agent Actions, PII Handling and Redaction in Corpora
- Series routes: Infrastructure Shift Briefs, Deployment Playbooks
- Site navigation: AI Topics Index, Glossary
More Study Resources
- Category hub
- MLOps, Observability, and Reliability Overview
- Related
- Synthetic Monitoring and Golden Prompts
- Change Control for Prompts, Tools, and Policies: Versioning the Invisible Code
- Redaction Pipelines for Sensitive Logs
- Root Cause Analysis for Quality Regressions
- Logging and Audit Trails for Agent Actions
- PII Handling and Redaction in Corpora
- Infrastructure Shift Briefs
- Deployment Playbooks
- AI Topics Index
- Glossary
Books by Drew Higgins
Bible Study / Spiritual Warfare
Ephesians 6 Field Guide: Spiritual Warfare and the Full Armor of God
Spiritual warfare is real—but it was never meant to turn your life into panic, obsession, or…
