AI RNG: Practical Systems That Ship
A bug that only appears sometimes feels like a ghost. You stare at dashboards, you skim logs, you try three small changes, and the system keeps misbehaving on its own schedule. The fastest way out is to treat the ghost like a witness case: collect facts, build a controlled reproduction, and force the system to repeat the failure until it becomes ordinary.
Premium Gaming TV65-Inch OLED Gaming PickLG 65-Inch Class OLED evo AI 4K C5 Series Smart TV (OLED65C5PUA, 2025)
LG 65-Inch Class OLED evo AI 4K C5 Series Smart TV (OLED65C5PUA, 2025)
A premium gaming-and-entertainment TV option for console pages, living-room gaming roundups, and OLED recommendation articles.
- 65-inch 4K OLED display
- Up to 144Hz refresh support
- Dolby Vision and Dolby Atmos
- Four HDMI 2.1 inputs
- G-Sync, FreeSync, and VRR support
Why it stands out
- Great gaming feature set
- Strong OLED picture quality
- Works well in premium console or PC-over-TV setups
Things to know
- Premium purchase
- Large-screen price moves often
This is a realistic case study you can copy. The details are generic on purpose, but the decisions, experiments, and guardrails are the same moves that turn confusion into a fix you can trust.
The call
The alert is simple: elevated 500s on a checkout endpoint. Not a full outage, not a constant failure. The graph looks like a sawtooth: bursts of errors, then calm, then bursts again. Users report that retrying sometimes works.
That pattern is the first clue. Intermittent failures often mean at least one of these is involved:
- A race or timing window
- A dependency that fails under a specific input shape
- A cache or state that flips between good and bad
- A resource limit that is hit only under load
You do not pick a winner yet. You turn the story into a falsifiable claim.
Stabilize the signal
The failure statement is kept narrow:
- Expected: checkout returns 200 with an order ID.
- Observed: checkout returns 500 with an internal error.
- Context: bursts under peak traffic; retries often succeed.
- Signal: a specific error signature in logs tied to the same code path.
Before touching code, you capture a small evidence bundle:
- A few failing request IDs
- The exact error message and stack trace for each
- A count of how often it happens per minute
- The deployment and configuration version currently running
This bundle is the line between engineering and guessing. It becomes your ground truth when someone suggests a fix that only sounds right.
Build a reproduction harness
The fastest reproduction is rarely local. It is usually a controlled replay.
You pull a sanitized request payload from logs, strip sensitive fields, and build a script that can hit a staging environment with the same endpoint. You add two things:
- A single command that runs 200 requests with controlled concurrency
- A pass or fail output that counts error signatures
The initial run is disappointing: staging shows no failures. That is still progress. It means the failure depends on something outside the code path alone.
You broaden the harness by adding environment variables and toggles that let you swap one factor at a time: dependency endpoints, feature flags, connection pool sizes, cache settings. Each run is still one command.
Reduce the world until the system confesses
Now the work is isolation. The goal is to make the bug happen more often by removing uncertainty, not by adding more complexity.
You run the same harness against production in a safe read-only mode, or you replay traffic through a production-like sandbox that mirrors the dependency path. The failure appears again. Good. You have a loop.
Then you begin controlled cuts:
- Reduce the request payload to the smallest shape that still fails
- Reduce the endpoint logic to the smallest branch that still fails
- Reduce concurrency to see whether the bug depends on load
- Remove caches and see whether the burst pattern changes
- Swap dependency versions while keeping everything else fixed
At this point, you can track progress with a simple table.
| Clue | What it suggested | How it was tested | What happened |
|---|---|---|---|
| Bursty failures | State flips or resource is exhausted | Record pool metrics and cache hit rates | Connection pool shows spikes near limits |
| Retries succeed | Transient failure, not deterministic input bug | Re-run the same payload repeatedly | Same payload succeeds most of the time |
| Failures cluster under concurrency | Race window or lock contention | Sweep concurrency levels | Failure rate rises sharply past a threshold |
| Stack trace points to JSON parse | Bad input or partial read | Capture raw bytes and lengths | Payload sometimes truncated |
The last line is the turning point. The raw bytes reveal that a request body is occasionally being read partially, producing invalid JSON. That should never happen in a correct request pipeline, which means the bug is in the boundary between transport and parsing.
The falsifying experiment
Two hypotheses are now competing:
- The client sometimes sends malformed bodies.
- The server sometimes reads incomplete bodies under load.
A falsifying experiment separates them.
You capture the failing raw body bytes from the server side and write them to disk. Then you replay them through the parsing code path locally. The parse fails consistently. That confirms the bytes are invalid, but it does not tell you where the invalidity is introduced.
Next, you capture raw body bytes at two points:
- As received from the transport layer
- As passed into the JSON parser
If the bytes differ, the server path is corrupting or truncating them. If they match, the client sent bad data.
The result shows a mismatch: the bytes passed to the parser are shorter than the bytes recorded at the earliest possible interception point. The server is truncating. That falsifies the client hypothesis for this incident.
Fix the cause, not the symptom
A common bad fix here is to catch the parse error and return a nicer error. That is a good user experience upgrade, but it does not address the truncation.
The real fix depends on what you find next:
- A misconfigured max body size causing early termination
- A stream that is read without consuming the full content length
- A timeout or cancellation path that returns partial bytes
- A middleware component that buffers incorrectly under concurrency
In this case study, the culprit is a custom middleware that reads the request stream for logging and then rewinds incorrectly under certain compression settings. Under load, the rewind fails silently, and downstream code reads a partial stream.
The fix is direct:
- Replace the stream reading with a safe buffered approach
- Ensure the stream is rewound only when buffering is complete
- Add an explicit invariant: downstream must see the same content length that transport received
Verify with regression protection
The harness that found the bug becomes the regression test strategy.
- A unit test validates the middleware stream behavior for compressed and uncompressed bodies.
- An integration test validates that the endpoint can parse bodies at the boundary sizes and under concurrency.
- A load-oriented test runs in CI nightly to detect regressions in buffering behavior.
Verification is not only tests. You also add observability that would have made this faster:
- Log content length at transport reception and at parser input, sampled
- Track request body truncation as a distinct metric
- Alert when truncation rises above a tiny threshold
Prevention that compounds
The final step is the day-after prevention work. The goal is not to write a long document. The goal is to add small guardrails that make the next similar incident cheaper.
A practical prevention list for this case:
- Remove or harden middleware that reads request bodies
- Standardize safe request logging patterns across services
- Add a regression pack entry for stream truncation
- Add a runbook note: how to detect partial-body failures quickly
The point of the story is not that this exact bug will happen to you. The point is that the workflow works because it forces proof:
- The failure is measurable.
- The reproduction is repeatable.
- The hypothesis is falsifiable.
- The fix is verified.
- The prevention is permanent.
Keep Exploring AI Systems for Engineering Outcomes
• AI Debugging Workflow for Real Bugs
https://ai-rng.com/ai-debugging-workflow-for-real-bugs/
• How to Turn a Bug Report into a Minimal Reproduction
https://ai-rng.com/how-to-turn-a-bug-report-into-a-minimal-reproduction/
• Root Cause Analysis with AI: Evidence, Not Guessing
https://ai-rng.com/root-cause-analysis-with-ai-evidence-not-guessing/
• AI for Logging Improvements That Reduce Debug Time
https://ai-rng.com/ai-for-logging-improvements-that-reduce-debug-time/
• AI for Building Regression Packs from Past Incidents
https://ai-rng.com/ai-for-building-regression-packs-from-past-incidents/
Books by Drew Higgins
Prophecy and Its Meaning for Today
New Testament Prophecies and Their Meaning for Today
A focused study of New Testament prophecy and why it still matters for believers now.
