wiretruth
Does the telemetry an LLM instrumentation library reports match what the provider actually sent? Each cell replays one provider response through one library and checks every reported value against the wire.
wiretruth 0.1.0 · latest versions · 2026-10-11T13:06:29+00:00 · Python 3.12 · semantic-conventions-genai 6fd0d763 · source
Pages: pinned versions · latest releases. A fix that ships upstream turns its cell green on the latest-releases page first.
pass every graded value matches the wiremissing the span ended without a value the wire carriedwrong a reported value does not match the wirelost the span never ended, so no backend ever receives itbroken the application saw something different from an uninstrumented runinconclusive the harness could not run the cell; no verdict on the library
| Scenario | openinference | openllmetry | otel-official | control-faulty | control-honest |
|---|---|---|---|---|---|
anthropic/async-basicOne call (async) | pass | pass | pass | pass | pass |
anthropic/async-streamStream read to the end (async) | pass | wrong | pass | pass | pass |
anthropic/async-stream-droppedConnection drops mid-stream (async) | pass | missing | pass | wrong | pass |
anthropic/async-stream-early-exitApp closes the stream after two events (async) | lost | lost | pass | lost | pass |
anthropic/async-stream-error-eventOverloaded error event mid-stream (async) | pass | missing | pass | wrong | pass |
anthropic/basicOne call | pass | pass | pass | pass | pass |
anthropic/cache-hitPrompt cache read and write | pass | pass | pass | wrong | pass |
anthropic/rate-limitedHTTP 429 | pass | pass | pass | wrong | pass |
anthropic/streamStream read to the end | pass | wrong | pass | pass | pass |
anthropic/stream-cache-cumulativeCache counts repeated in message_delta | pass | wrong | pass | wrong | pass |
anthropic/stream-cache-hitPrompt cache read and write, streamed | pass | wrong | pass | wrong | pass |
anthropic/stream-droppedConnection drops mid-stream | pass | lost | pass | wrong | pass |
anthropic/stream-early-exitApp closes the stream after two events | lost | lost | pass | lost | pass |
anthropic/stream-early-exit-withApp leaves the stream after two events with a with block | pass | lost | pass | lost | pass |
anthropic/stream-error-eventOverloaded error event mid-stream | pass | lost | pass | wrong | pass |
anthropic/tool-callTwo parallel tool uses | pass | pass | pass | pass | pass |
openai/async-basicOne call (async) | pass | pass | pass | pass | pass |
openai/async-streamStream read to the end (async) | pass | pass | pass | pass | pass |
openai/async-stream-droppedConnection drops mid-stream (async) | pass | wrong | pass | wrong | pass |
openai/async-stream-early-exitApp closes the stream after two chunks (async) | lost | pass | pass | lost | pass |
openai/basicOne call | pass | pass | pass | pass | pass |
openai/cache-hitPrompt cache hit | pass | pass | pass | pass | pass |
openai/rate-limitedHTTP 429 | pass | pass | pass | wrong | pass |
openai/reasoningReasoning tokens | pass | pass | pass | pass | pass |
openai/streamStream read to the end | pass | pass | pass | pass | pass |
openai/stream-cache-hitPrompt cache hit, streamed | pass | pass | pass | pass | pass |
openai/stream-droppedConnection drops mid-stream | pass | wrong | pass | wrong | pass |
openai/stream-early-exitApp closes the stream after two chunks | lost | pass | pass | lost | pass |
openai/stream-early-exit-withApp leaves the stream after two chunks with a with block | lost | pass | pass | lost | pass |
openai/stream-no-usageStream without include_usage | pass | pass | pass | wrong | pass |
openai/tool-callTwo parallel tool calls | pass | pass | pass | pass | pass |
The controls run the suite's own reference instrumentation.
control-honest has to pass every check, and control-faulty has to fail
exactly its planted faults. If either moves, the suite is wrong, not a library.
Findings
Every finding cites the rule it applies. To dispute one, open an issue: a disputed finding is marked as such in known_issues.toml.
lost openinference · anthropic/async-stream-early-exit · span.ended
- Note
never ended: messages.create- Upstream
- reported
- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
lost openinference · anthropic/stream-early-exit · span.ended
- Note
never ended: messages.create- Upstream
- reported
- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
lost openinference · openai/async-stream-early-exit · span.ended
- Note
never ended: ChatCompletion- Upstream
- reported Fix proposed in https://github.com/Arize-ai/openinference/pull/3796.
- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
lost openinference · openai/stream-early-exit · span.ended
- Note
never ended: ChatCompletion- Upstream
- reported Fix proposed in https://github.com/Arize-ai/openinference/pull/3796.
- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
lost openinference · openai/stream-early-exit-with · span.ended
- Note
never ended: ChatCompletion- Upstream
- reported Fix proposed in https://github.com/Arize-ai/openinference/pull/3796.
- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
wrong openllmetry · anthropic/async-stream · output_tokens
- Expected
419- Reported
420- Read from
gen_ai.usage.output_tokens- Upstream
- reported message_delta carries the cumulative output count, and the wrapper adds it to message_start's placeholder of 1.
- Rule
- Output tokens, with reasoning tokens included.
missing openllmetry · anthropic/async-stream-dropped · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
lost openllmetry · anthropic/async-stream-early-exit · span.ended
- Note
never ended: anthropic.chat- Upstream
- reported Fix proposed in https://github.com/traceloop/openllmetry/pull/4539.
- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
missing openllmetry · anthropic/async-stream-error-event · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
wrong openllmetry · anthropic/stream · output_tokens
- Expected
419- Reported
420- Read from
gen_ai.usage.output_tokens- Upstream
- reported message_delta carries the cumulative output count, and the wrapper adds it to message_start's placeholder of 1.
- Rule
- Output tokens, with reasoning tokens included.
wrong openllmetry · anthropic/stream-cache-cumulative · output_tokens
- Expected
467- Reported
468- Read from
gen_ai.usage.output_tokens- Upstream
- reported message_delta carries the cumulative output count, and the wrapper adds it to message_start's placeholder of 1.
- Rule
- Output tokens, with reasoning tokens included.
wrong openllmetry · anthropic/stream-cache-hit · output_tokens
- Expected
511- Reported
512- Read from
gen_ai.usage.output_tokens- Upstream
- reported message_delta carries the cumulative output count, and the wrapper adds it to message_start's placeholder of 1.
- Rule
- Output tokens, with reasoning tokens included.
lost openllmetry · anthropic/stream-dropped · span.ended
- Note
never ended: anthropic.chat- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
lost openllmetry · anthropic/stream-early-exit · span.ended
- Note
never ended: anthropic.chat- Upstream
- reported Fix proposed in https://github.com/traceloop/openllmetry/pull/4539.
- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
lost openllmetry · anthropic/stream-early-exit-with · span.ended
- Note
never ended: anthropic.chat- Upstream
- reported Fix proposed in https://github.com/traceloop/openllmetry/pull/4539.
- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
lost openllmetry · anthropic/stream-error-event · span.ended
- Note
never ended: anthropic.chat- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
wrong openllmetry · openai/async-stream-dropped · error.status
- Expected
ERROR- Reported
['OK']- Upstream
- reported _ensure_cleanup() sets OK and ends the span before the ERROR status is set, so ERROR is dropped.
- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing openllmetry · openai/async-stream-dropped · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
wrong openllmetry · openai/stream-dropped · error.status
- Expected
ERROR- Reported
['OK']- Upstream
- reported _ensure_cleanup() sets OK and ends the span before the ERROR status is set, so ERROR is dropped.
- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing openllmetry · openai/stream-dropped · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
wrong control-faulty · anthropic/async-stream-dropped · error.status
- Expected
ERROR- Reported
['UNSET']- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing control-faulty · anthropic/async-stream-dropped · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
lost control-faulty · anthropic/async-stream-early-exit · span.ended
- Note
never ended: chat claude-sonnet-5-5- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
wrong control-faulty · anthropic/async-stream-error-event · error.status
- Expected
ERROR- Reported
['UNSET']- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing control-faulty · anthropic/async-stream-error-event · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
wrong control-faulty · anthropic/cache-hit · input_tokens
- Expected
5802- Reported
1187- Read from
gen_ai.usage.input_tokens- Note
cache reads and writes left out- Rule
- Input tokens SHOULD include all types of input tokens, including cached tokens. For Anthropic, whose `input_tokens` excludes them, cache reads and cache writes MUST be added. OpenInference states the same rule for `llm.token_count.prompt`.
wrong control-faulty · anthropic/rate-limited · error.status
- Expected
ERROR- Reported
['UNSET']- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing control-faulty · anthropic/rate-limited · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
wrong control-faulty · anthropic/stream-cache-cumulative · input_tokens
- Expected
5673- Reported
1279- Read from
gen_ai.usage.input_tokens- Note
cache reads and writes left out- Rule
- Input tokens SHOULD include all types of input tokens, including cached tokens. For Anthropic, whose `input_tokens` excludes them, cache reads and cache writes MUST be added. OpenInference states the same rule for `llm.token_count.prompt`.
wrong control-faulty · anthropic/stream-cache-hit · input_tokens
- Expected
8424- Reported
1543- Read from
gen_ai.usage.input_tokens- Note
cache reads and writes left out- Rule
- Input tokens SHOULD include all types of input tokens, including cached tokens. For Anthropic, whose `input_tokens` excludes them, cache reads and cache writes MUST be added. OpenInference states the same rule for `llm.token_count.prompt`.
wrong control-faulty · anthropic/stream-dropped · error.status
- Expected
ERROR- Reported
['UNSET']- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing control-faulty · anthropic/stream-dropped · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
lost control-faulty · anthropic/stream-early-exit · span.ended
- Note
never ended: chat claude-sonnet-5-5- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
lost control-faulty · anthropic/stream-early-exit-with · span.ended
- Note
never ended: chat claude-sonnet-5-5- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
wrong control-faulty · anthropic/stream-error-event · error.status
- Expected
ERROR- Reported
['UNSET']- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing control-faulty · anthropic/stream-error-event · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
wrong control-faulty · openai/async-stream-dropped · error.status
- Expected
ERROR- Reported
['UNSET']- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing control-faulty · openai/async-stream-dropped · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
lost control-faulty · openai/async-stream-early-exit · span.ended
- Note
never ended: chat gpt-5-mini- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
wrong control-faulty · openai/rate-limited · error.status
- Expected
ERROR- Reported
['UNSET']- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing control-faulty · openai/rate-limited · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
wrong control-faulty · openai/stream-dropped · error.status
- Expected
ERROR- Reported
['UNSET']- Rule
- Span status SHOULD follow Recording Errors: an operation that ended in an error sets the status to Error.
missing control-faulty · openai/stream-dropped · error.type
- Rule
- `error.type` is conditionally required if the operation ended in an error. Any low-cardinality identifier is allowed, so only its presence is graded.
lost control-faulty · openai/stream-early-exit · span.ended
- Note
never ended: chat gpt-5-mini- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
lost control-faulty · openai/stream-early-exit-with · span.ended
- Note
never ended: chat gpt-5-mini- Rule
- GenAI spans SHOULD end when the response is fully received or the operation is terminated due to an error or cancellation.
wrong control-faulty · openai/stream-no-usage · output_tokens
- Reported
7- Read from
gen_ai.usage.output_tokens- Note
estimated: the stream carried no usage- Rule
- Output tokens, with reasoning tokens included.
Subjects and versions
- openinference: OpenInference. anthropic 1.13.0, openai 3.28.0, openinference-instrumentation-anthropic 3.0.3, openinference-instrumentation-openai 0.1.65, opentelemetry-exporter-otlp-proto-http 1.45.1, opentelemetry-sdk 1.45.1
- openllmetry: OpenLLMetry. anthropic 1.13.0, openai 3.28.0, opentelemetry-exporter-otlp-proto-http 1.45.1, opentelemetry-instrumentation-anthropic 0.62.4, opentelemetry-instrumentation-openai 0.62.4, opentelemetry-sdk 1.45.1
- otel-official: OpenTelemetry GenAI instrumentation. anthropic 1.13.0, openai 3.28.0, opentelemetry-exporter-otlp-proto-http 1.45.1, opentelemetry-instrumentation-genai-anthropic 1.2b0, opentelemetry-instrumentation-genai-openai 1.2b0, opentelemetry-sdk 1.45.1
- control-faulty: Control: reference instrumentation with four planted faults. anthropic 1.13.0, openai 3.28.0, opentelemetry-exporter-otlp-proto-http 1.45.1, opentelemetry-sdk 1.45.1
- control-honest: Control: reference instrumentation. anthropic 1.13.0, openai 3.28.0, opentelemetry-exporter-otlp-proto-http 1.45.1, opentelemetry-sdk 1.45.1