Skip to content

Telemetry outfile writes "[Circular]" for shared metric endTime and histogram boundaries #29406

Description

@yupengtang

What happened?

With telemetry written to a file (GEMINI_TELEMETRY_TARGET=local plus GEMINI_TELEMETRY_OUTFILE), most metric data points in the file have "endTime": "[Circular]" instead of a timestamp, and histogram bucket boundaries are written as "[Circular]" too.

The cause is safeJsonStringify in packages/core/src/utils/safeJsonStringify.ts. It adds every object it visits to a WeakSet and never removes it, so any object that is referenced a second time is replaced with "[Circular]", even when there is no cycle:

const shared = { x: 1 };
safeJsonStringify({ a: shared, b: shared });
// {"a":{"x":1},"b":"[Circular]"}

The OpenTelemetry SDK reuses one endTime array for all data points in a collection, and one boundaries array per histogram, so every data point after the first loses them. The file exporters started using safeJsonStringify in #19244 to get past the real span processor cycle from #19228.

Other callers have the same problem whenever an object is shared, for example RecordingContentGenerator (--record-responses) and the gen_ai tool definition attributes in LoggingContentGenerator.

Repro on main (09e048f), headless, no account needed:

cat > fake.responses <<'EOF'
{"method":"generateContentStream","response":[{"candidates":[{"content":{"parts":[{"text":"Hello there."}],"role":"model"},"finishReason":"STOP","index":0}],"usageMetadata":{"promptTokenCount":10,"candidatesTokenCount":3,"totalTokenCount":13}}]}
{"method":"generateContent","response":{"candidates":[{"content":{"parts":[{"text":"{}"}],"role":"model"},"finishReason":"STOP","index":0}]}}
{"method":"countTokens","response":{"totalTokens":10}}
EOF
GEMINI_CLI_TRUST_WORKSPACE=true GEMINI_TELEMETRY_ENABLED=true GEMINI_TELEMETRY_TARGET=local \
  GEMINI_TELEMETRY_OUTFILE=$PWD/telemetry.log GEMINI_API_KEY=fake \
  node packages/cli/dist/index.js -p "say hello" --fake-responses-non-strict $PWD/fake.responses
grep -c '"endTime": \[' telemetry.log              # 3
grep -c '"endTime": "\[Circular\]"' telemetry.log  # 30
grep -c '"boundaries": "\[Circular\]"' telemetry.log  # 17

What did you expect to happen?

Only real cycles should become "[Circular]". In that run those are the 2 _that entries from the span processor. Every endTime and boundaries value should be written out.

Client information

Client Information

Built from source at 09e048f (0.62.0-nightly.20260918), Node 20.19.5, Linux (Ubuntu on WSL2). I didn't run /about because the repro uses --fake-responses and doesn't sign in.

Login information

Not needed for the repro (--fake-responses-non-strict, dummy API key).

Anything else we need to know?

I have a fix ready: track the objects on the current path from the root, not every object seen, so only an object that contains itself is marked. With it, the same run writes all 33 endTime arrays and all histogram boundaries, and the two _that cycles are still marked [Circular]. The existing cycle tests in safeJsonStringify.test.ts and file-exporters.test.ts pass unchanged. I added tests for shared references, and they fail on main. Happy to open the PR if that works for you.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/enterpriseIssues related to Telemetry, Policy, Quota / Licensingkind/bugpriority/p2Important but can be addressed in a future release.status/bot-triaged

    Type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions