What Actually Breaks When an LLM Analyzes Stocks

Published Jul 28, 2026 · llmmarket-datareliability

I run a pipeline that pulls Korean and US market data, computes indicators, and hands the result to an LLM to write a scheduled briefing. It has been in production long enough to fail in interesting ways.

The failures people expect are the model inventing a price out of thin air. That is not what happens. What happens is worse and quieter: the model faithfully describes inputs that are wrong, and invents infrastructure that does not exist. Both produce output that reads as competent.

Here are five that actually happened, with what fixed them.

1. The API endpoint that never existed

The discovery job called this on every other round:

GET /uapi/domestic-stock/v1/quotations/capture-uplmt   (TR: FHPST01830000)

It returned 404 on every call. The path looks right. The TR code follows the broker’s naming convention exactly — FHPST prefix, eight digits, the same shape as the dozen endpoints that do work. Nothing about it looks invented.

It was invented. I checked it against the broker’s full API specification workbook — every documented endpoint, exported 2026-04-17 — and searched for the path and the TR code. Zero matches. Neither existed.

This is the purest form of hallucination in this domain, and it is easy to miss because the failure is silent by design: the job caught the 404, logged a warning, and continued with a degraded score. The feature it fed (an “online interest” bonus) had been dead for weeks while the other five indicators carried the score.

The fix was not to guess a replacement. I disabled the function body — return [] — kept the signature and call sites intact, and wrote down two candidate real endpoints from the spec for whenever it matters enough to restore.

The check that generalizes: every external path the model produces gets matched against the provider’s spec file before it ships. Not “does it look right.” Does it appear in the document, verbatim.

2. A buy signal with no price

A scheduled message went out reading, in effect:

📗 [construction stock] 0 KRW  BUY (67%)

The analysis function’s return dict was missing the current_price key. The caller did analysis.get("current_price", 0). So the price became zero, and the sentence generator wrote a fluent, confident sentence around a zero.

The part worth sitting with is the 67%. That confidence number was computed from the indicator scores, which were fine. Nothing in the pipeline connected data completeness to confidence. A record could be missing its price entirely and still be reported at 67% conviction, because the two things had never been wired together.

The check that generalizes: a completeness gate before narration. If a required field is absent, the record does not get a sentence written about it — it gets dropped and counted. Defaulting to 0 turns a missing value into a plausible one, which is the worst available outcome.

3. Yesterday’s close that was actually right now

The intraday report computed the opening gap using:

prev_close = cur_price   # ← 09:10 price, not yesterday's close

Every gap direction was computed against the current price instead of the prior session’s close. A stock that opened up was reported as opening down.

This is the dangerous class. There is no error, no 404, no zero. The output is internally consistent, correctly formatted, and inverted. A reader — including me — has no signal that anything is wrong, because the only tell would be knowing the right answer already.

The fix pulled the actual prior close from the daily chart endpoint (stck_clpr), with the first intraday bar’s open as fallback.

The check that generalizes: when two values can be confused for each other, match on an explicit key, never on position or convenience. In the daily-bar arrays this pipeline reads, I now match rows by explicit date string. Reading the array one index off — taking yesterday’s bar as today’s — produces exactly this failure mode, and I have misdiagnosed it as a code bug more than once when the code was fine and my indexing was not.

4. The test run that wrote to production

The discovery job had a --dry-run flag. It skipped sending the message. It did not skip state.update() or the log write.

So every QA run mutated the state file and appended to the discovery log. One record with a timestamp of 10:13 — impossible for a job scheduled at 09:05 — sat in the log until someone noticed the number was odd. The closing report reads that log for its statistics. Test runs were quietly editing the dataset that later analysis would summarize.

The fix threaded the flag through all eight round handlers and guarded every write, with a stderr line on each skip so a dry run is visibly a dry run.

The check that generalizes: in an analysis pipeline, “dry run” has to mean read-only, verified by diffing the state file before and after. It does not mean “skip the last step.” Contamination here is worse than a crash, because it survives and gets averaged in.

5. My own confident misjudgment

Reviewing scheduled jobs, an extra entry showed up next to the numbered ones:

com.aigeenya.stockreport.discovery      ← no number
com.aigeenya.stockreport.discovery5..8  ← numbered

The unnumbered one was flagged as a leftover duplicate and queued for deletion. It was not. It was the legitimate job for a different nightly task, and the names simply looked alike. The call was reversed a day later, before anything was removed.

Nothing hallucinated a fact here. Two things that shared a naming pattern were assumed to share a purpose. That is the same failure as the fabricated endpoint, one level up: a confident conclusion drawn from surface similarity rather than from the spec.

The check that generalizes: before deleting or disabling anything, find the thing that creates it. If you cannot point to the definition, you do not know what it is.

What this says about “which LLM is best for stock analysis”

Four of these five had nothing to do with which model was used. Swapping in a stronger model would have fixed exactly one — the fabricated endpoint — and even there, a spec-matching step catches it regardless of model.

The rest were context problems. Wrong field, missing field, contaminated history, surface pattern-matching. The model’s job was to turn a structured record into a sentence, and it did that correctly every time. It was correct about wrong inputs.

So the question that actually moves reliability is not which model. It is:

What is in the context, and where did each field come from? Every value should trace to a source and a timestamp.
What happens when a field is missing? If the answer is “a default,” you have converted an absence into a claim.
Can the analysis step write anything? If yes, your test runs are training data for tomorrow’s summary.
Does every external reference exist? Check paths against the spec, not against plausibility.

None of that is model selection. It is the contract around the model. In my experience that contract is where essentially all of the reliability lives — and it is also the part that no benchmark measures.

Tickers are omitted deliberately. This describes engineering failures in a data pipeline; it is not analysis of any security and not a recommendation to buy or sell anything.