The ten ways LLM telemetry goes wrong

Your cost dashboard, your token budget and your error rate all read the numbers that an instrumentation library writes on a span. When one of those numbers is wrong, nothing fails. The dashboard is just wrong, with full confidence. These are the classes of bug that spantruth measures, each one checked against what the provider actually sent.

To see which of them your own versions have, run uvx spantruth doctor in your project.

spantruth 0.4.2 · 2026-11-07T20:45:58+00:00 · the full report · source

ST01Ghost span

The app made the call, and no finished span ever reaches a backend.

The call is not there. Its cost, its latency and its error are all absent, and nothing says so. Streams that the app leaves early or that fail mid-way are the usual victims.

In this run: 25 cells in 3 libraries
Largest cost error: -100%
Example: anthropic/async-stream-dropped: never ended: chat claude-sonnet-5-5
In public: openllmetry#4536 (reported by armanunix), openllmetry#4643 (reported by chrikrah), openinference#3790, fixed (reported by feiiiiii5), openinference#3930, fixed (reported by armanunix), openlit#1828 (reported by chrikrah)

ST02Double span

One call reports its token usage on two spans.

A backend that sums token attributes across spans counts the call twice. The Anthropic SDK traces itself since 1.14.0, so a library on top causes this unless it handles the SDK's span, as OpenInference does since openinference-instrumentation-anthropic 3.1.1.

In this run: 15 cells in 1 library
Largest cost error: +100%
Example: anthropic/async-basic: expected 1, reported 2: usage on 2 spans: anthropic.messages.create, chat claude-sonnet-5-5
What to do today: Let one side report the call: set ANTHROPIC_OPEN_TELEMETRY=false, or pass open_telemetry=False to the client.
In public: langfuse#18978: dashboards show about 2x (reported by rvndbalaji), opentelemetry-python-genai#971 (reported by lmolkova), openinference#4119, fixed (reported by PabloPardoGarcia)

ST03Cache-blind tokens

Cached tokens are left out of the input count, or their own counts are wrong.

The cost of a cached call is wrong, because a cache read costs a fraction of an input token and a cache write costs more than one. Your cache hit rate is wrong too.

In this run: 8 cells in 2 libraries
Largest cost error: -100%
Example: anthropic/cache-hit: expected 4096, reported nothing
In public: openllmetry#4583 (reported by armanunix)

ST04Phantom tokens

The span reports more tokens than the provider sent, or invents a count.

The cost is too high. When a stream carries no usage and the library estimates from the text, the number looks real and is not.

In this run: 8 cells in 3 libraries
Largest cost error: +0.2%
Example: openai/stream-no-usage: expected nothing, reported 7: estimated: the stream carried no usage
In public: openllmetry#4377 (reported by arthi-arumugam-git)

ST05Vanishing tokens

The span reports fewer tokens than the provider sent, or none.

The cost is too low, and a token budget or a rate alert fires late or never.

In this run: 9 cells in 1 library
Largest cost error: -100%
Example: anthropic/async-stream: expected 2311, reported nothing
In public: sentry-python#7948 (reported by chrikrah), logfire#2549 (reported by chrikrah)

ST06Hidden reasoning

The reasoning tokens that the provider reported do not appear on the span.

Reasoning is billed as output, so the total can be right while the reason for it is gone: you cannot see how much of your bill is thinking.

In this run: 15 cells in 5 libraries
Example: anthropic/stream-thinking: expected 1103, reported nothing
In public: openllmetry#4660 (reported by chrikrah), inspect_ai#5815 (reported by chrikrah)

ST07False OK

A call failed, and its span does not say so, or does not say how.

Your error rate is too low. A stream that dropped its connection counts as a success in every panel that filters on status or on `error.type`.

In this run: 19 cells in 3 libraries
Example: anthropic/async-stream-dropped: expected ERROR, reported OK
In public: openllmetry#4535 (reported by feiiiiii5), langfuse-python#2039 (reported by chrikrah)

ST08False alarm

A call succeeded, and its span says it failed.

Your error rate is too high. A 429 that the SDK retried and recovered from pages somebody for an outage that the user never saw.

In this run: Not seen

ST09Silent stop

The finish reason is missing or wrong.

You cannot tell a complete answer from one that hit `max_tokens` or a content filter, so truncated answers look the same as finished ones.

In this run: 60 cells in 4 libraries
Example: anthropic/async-basic: expected end_turn, reported nothing
In public: anthropic-sdk-python#2055 (reported by chrikrah)

ST10Observer effect

The instrumentation changes what the application receives.

The telemetry is not the worst part: your app behaves differently with tracing on. It gets other chunks, another exception, or it hangs.

In this run: Not seen

A cost figure is for the token mix of one scenario, at list price from developers.openai.com and platform.claude.com on 2026-11-06. Input tokens include cache reads and writes, so the uncached part is input minus reads minus writes. A span that never ends reports nothing, and usage on two spans counts twice only in a backend that sums across spans. A backend with another formula shows another number.