Skip to content

Trace genai dashboard - #398

Merged
thorrester merged 14 commits into
mainfrom
trace-genai-dashboard
May 10, 2026
Merged

thorrester merged 14 commits into
mainfrom
trace-genai-dashboard

Conversation

@thorrester

@thorrester thorrester commented May 3, 2026 •

Copy link
Copy Markdown
Member

Pull Request

Short Summary

Adds a GenAI tab to the trace detail view — a dedicated panel showing LLM-specific metrics (token usage, cost, operation breakdown, model usage, eval results, tool calls) pulled from Scouter's GenAI span data. Also expands the trace filter sidebar to support namespace, version, and instance_id facets alongside the existing service/status filters.

Context

GenAI trace panel (TraceDetailContent, SpanGenAiPanel, TraceGenAiPanel)

The trace waterfall had no way to surface LLM-specific span attributes. When a trace contains GenAI spans (has_genai_spans from the Scouter response), a "GenAI" tab now appears next to the existing Waterfall and Map tabs. Selecting it renders TraceGenAiPanel with aggregate metrics across the trace, and SpanGenAiPanel in the detail pane for per-span breakdowns (messages, eval results, tool call blocks, params, raw attributes).

TraceDetailContent receives genai: GenAiTraceMetricsResponse | null and genAiBySpanId: Record<string, GenAiSpanRecord> — when null or when has_genai_spans is false, the tab is hidden entirely.

Facet sidebar expansion (FacetSidebar, TraceDashboard)

Before: facet sidebar offered Service and Status filters only.
After: Service, Namespace, Version, Instance ID, and Status.

TraceDashboard now fires four parallel facet requests instead of two (service_namespace, service_version, service_instance_id filters each resolved independently). FacetSidebar accepts the new facet arrays and their set/clear callbacks. When Scouter returns no facet values for a dimension (e.g. namespace isn't indexed yet), the sidebar falls back to a free-text input with an Apply button.

Agent observability charts (charts.ts)

Added buildOperationBarChart — a horizontal bar Chart.js config ranked by span count — consumed by the new TraceGenAiPanel operation breakdown card.

Rust genai route (scouter/genai/route.rs)

GenAiDashboardRequest was defined locally and is now imported from opsml-types. is_valid_trace_id previously rejected UUID-format trace IDs (containing hyphens) because it checked raw chars against hex; fixed by stripping hyphens before validation. Unit test added for the boundary cases including UUID format.

Python tracing (opsml/scouter/tracing/__init__.py)

Replaced set_entity + add_queue_item on ScouterSpan with attach_eval — a structured method that attaches an eval record to the trace via profile_uid and a dict/Pydantic context payload. Baggage propagation refactored from _current_context (which only fetched the ambient OTel context) to _apply_baggage_to_context (which builds a new context with all provided baggage items applied). This fixes cases where baggage was silently dropped when a non-None context was passed.

init_tracer is replaced by configure_tracing + get_tracer / reset_tracer_provider to align with the updated Scouter client API. ScouterResourceConfig and TraceFilters added to exports.

AppState / service loading (opsml_app/src/app.rs)

Added normalize_drift_paths to resolve relative drift profile paths before passing them to ScouterQueue::from_path_rs. Previously, drift paths stored in card_map.json as relative paths (e.g. ./profiles/drift.json) would fail at runtime when the working directory differed from the service artifact directory. from_spec now checks app_state().is_offline() and skips registration when offline (instead of always registering).

File Change
crates/opsml_server/opsml_ui/src/lib/components/trace/genai/SpanGenAiPanel.svelte New: per-span LLM detail panel with tabbed view (messages, eval, params, tool, server, raw)
crates/opsml_server/opsml_ui/src/lib/components/trace/genai/TraceGenAiPanel.svelte New: aggregate GenAI metrics panel (tokens, cost, ops, model usage, evals)
crates/opsml_server/opsml_ui/src/lib/components/trace/TraceDetailContent.svelte Adds GenAI tab; wires genai and genAiBySpanId props
crates/opsml_server/opsml_ui/src/lib/components/trace/TraceDashboard.svelte Fetches namespace/version/instance_id facets; passes to sidebar
crates/opsml_server/opsml_ui/src/lib/components/trace/filters/FacetSidebar.svelte Adds namespace, version, instance_id filter sections
crates/opsml_server/opsml_ui/src/lib/components/card/agent/observability/charts.ts Adds buildOperationBarChart
crates/opsml_server/src/core/scouter/genai/route.rs Moves GenAiDashboardRequest to types; fixes UUID trace ID validation
crates/opsml_server/tests/api/scouter/trace.rs New tests for paginated traces, facets, spans-by-id routes
py-opsml/python/opsml/scouter/tracing/__init__.py Replaces set_entity/add_queue_item with attach_eval; fixes baggage propagation
crates/opsml_app/src/app.rs Adds normalize_drift_paths; offline-aware from_spec
crates/opsml_app/src/types.rs New ScouterResourceConfig, ServiceInfo, and instance ID resolution

Is this a Breaking Change?

Yes — ScouterSpan.set_entity and ScouterSpan.add_queue_item are removed from the Python tracing API. Callers using those methods must migrate to attach_eval. The init_tracer function is also removed; callers must switch to configure_tracing + get_tracer. All other Rust HTTP contracts, SvelteKit routes, and server handler signatures are unchanged.

@github-actions

github-actions Bot commented May 3, 2026 •

Copy link
Copy Markdown

Dependency Review

The following issues were found:
  • ✅ 0 vulnerable package(s)
  • ✅ 0 package(s) with incompatible licenses
  • ✅ 0 package(s) with invalid SPDX license definitions
  • ⚠️ 16 package(s) with unknown licenses.
See the Details below.

License Issues

Cargo.lock

PackageVersionLicenseIssue Type
scouter-client0.26.0NullUnknown License
scouter-dataframe0.26.0NullUnknown License
scouter-dispatch0.26.0NullUnknown License
scouter-drift0.26.0NullUnknown License
scouter-evaluate0.26.0NullUnknown License
scouter-events0.26.0NullUnknown License
scouter-http0.26.0NullUnknown License
scouter-macro0.26.0NullUnknown License
scouter-observability0.26.0NullUnknown License
scouter-profile0.26.0NullUnknown License
scouter-semver0.26.0NullUnknown License
scouter-settings0.26.0NullUnknown License
scouter-state0.26.0NullUnknown License
scouter-tonic0.26.0NullUnknown License
scouter-tracing0.26.0NullUnknown License
scouter-types0.26.0NullUnknown License

Scanned Manifest Files

Cargo.lock
  • openssl@0.10.78
  • rpassword@7.4.0
  • arrow@58.2.0
  • arrow-arith@58.2.0
  • arrow-array@58.2.0
  • arrow-buffer@58.2.0
  • arrow-cast@58.2.0
  • arrow-csv@58.2.0
  • arrow-data@58.2.0
  • arrow-ipc@58.2.0
  • arrow-json@58.2.0
  • arrow-ord@58.2.0
  • arrow-row@58.2.0
  • arrow-schema@58.2.0
  • arrow-select@58.2.0
  • arrow-string@58.2.0
  • baked-potato@0.25.0
  • cc@1.2.62
  • const-hex@1.19.0
  • deltalake@0.32.1
  • deltalake-core@0.32.1
  • digest@0.11.3
  • dispatch2@0.3.1
  • evmap@11.0.0
  • fancy-regex@0.18.0
  • filetime@0.2.28
  • generator@0.8.8
  • h2@0.4.14
  • hashbag@0.1.13
  • hashbrown@0.17.1
  • hostname@0.4.2
  • hybrid-array@0.4.12
  • idna_adapter@1.2.2
  • jni@0.22.4
  • jni-macros@0.22.4
  • js-sys@0.3.98
  • jsonschema@0.46.4
  • left-right@0.11.7
  • loom@0.7.2
  • lz4_flex@0.13.1
  • matchers@0.2.0
  • metrics@0.24.5
  • metrics-exporter-prometheus@0.18.3
  • metrics-util@0.20.3
  • objc2@0.6.4
  • objc2-encode@4.1.0
  • objc2-foundation@0.3.2
  • objc2-open-directory@0.3.2
  • openssl@0.10.79
  • openssl-sys@0.9.115
  • parquet@58.2.0
  • pin-project@1.1.12
  • pin-project-internal@1.1.12
  • potato-agent@0.25.0
  • potato-head@0.25.0
  • potato-provider@0.25.0
  • potato-spec@0.25.0
  • potato-state@0.25.0
  • potato-type@0.25.0
  • potato-util@0.25.0
  • potato-workflow@0.25.0
  • potatohead-macro@0.25.0
  • quick-xml@0.39.4
  • rapidhash@4.4.1
  • redox_syscall@0.7.5
  • referencing@0.46.4
  • reqwest@0.13.3
  • rpassword@7.5.2
  • rustls@0.23.40
  • rustls-platform-verifier@0.7.0
  • scoped-tls@1.0.1
  • scouter-client@0.26.0
  • scouter-dataframe@0.26.0
  • scouter-dispatch@0.26.0
  • scouter-drift@0.26.0
  • scouter-evaluate@0.26.0
  • scouter-events@0.26.0
  • scouter-http@0.26.0
  • scouter-macro@0.26.0
  • scouter-observability@0.26.0
  • scouter-profile@0.26.0
  • scouter-semver@0.26.0
  • scouter-settings@0.26.0
  • scouter-state@0.26.0
  • scouter-tonic@0.26.0
  • scouter-tracing@0.26.0
  • scouter-types@0.26.0
  • simd_cesu8@1.1.1
  • siphasher@1.0.3
  • strum@0.28.0
  • strum_macros@0.28.0
  • sysinfo@0.39.0
  • tokio@1.52.3
  • tonic@0.14.6
  • tonic-build@0.14.6
  • tonic-health@0.14.6
  • tonic-prost@0.14.6
  • tonic-prost-build@0.14.6
  • tower-http@0.6.10
  • utoipa@5.5.0
  • utoipa-gen@5.5.0
  • wasm-bindgen@0.2.121
  • wasm-bindgen-futures@0.4.71
  • wasm-bindgen-macro@0.2.121
  • wasm-bindgen-macro-support@0.2.121
  • wasm-bindgen-shared@0.2.121
  • web-sys@0.3.98
  • windows-sys@0.60.2
  • windows-targets@0.53.5
  • windows_aarch64_gnullvm@0.53.1
  • windows_aarch64_msvc@0.53.1
  • windows_i686_gnu@0.53.1
  • windows_i686_gnullvm@0.53.1
  • windows_i686_msvc@0.53.1
  • windows_x86_64_gnu@0.53.1
  • windows_x86_64_gnullvm@0.53.1
  • windows_x86_64_msvc@0.53.1
  • arrow@58.1.0
  • arrow-arith@58.1.0
  • arrow-array@58.1.0
  • arrow-buffer@58.1.0
  • arrow-cast@58.1.0
  • arrow-csv@58.1.0
  • arrow-data@58.1.0
  • arrow-ipc@58.1.0
  • arrow-json@58.1.0
  • arrow-ord@58.1.0
  • arrow-row@58.1.0
  • arrow-schema@58.1.0
  • arrow-select@58.1.0
  • arrow-string@58.1.0
  • baked-potato@0.23.0
  • cc@1.2.61
  • cesu8@1.1.0
  • const-hex@1.18.1
  • deltalake@0.32.0
  • deltalake-core@0.32.0
  • digest@0.11.2
  • fancy-regex@0.17.0
  • filetime@0.2.27
  • h2@0.4.13
  • hashbrown@0.17.0
  • hybrid-array@0.4.11
  • idna_adapter@1.2.1
  • iri-string@0.7.12
  • jni@0.21.1
  • jni-sys@0.3.1
  • js-sys@0.3.95
  • jsonschema@0.46.2
  • lz4_flex@0.13.0
  • metrics@0.24.3
  • metrics-exporter-prometheus@0.18.1
  • metrics-util@0.20.1
  • openssl-sys@0.9.114
  • parquet@58.1.0
  • pin-project@1.1.11
  • pin-project-internal@1.1.11
  • potato-agent@0.23.0
  • potato-agent@0.24.0
  • potato-head@0.23.0
  • potato-head@0.24.0
  • potato-provider@0.23.0
  • potato-provider@0.24.0
  • potato-spec@0.23.0
  • potato-spec@0.24.0
  • potato-state@0.23.0
  • potato-state@0.24.0
  • potato-type@0.23.0
  • potato-type@0.24.0
  • potato-util@0.23.0
  • potato-util@0.24.0
  • potato-workflow@0.23.0
  • potato-workflow@0.24.0
  • potatohead-macro@0.23.0
  • potatohead-macro@0.24.0
  • quick-xml@0.39.2
  • redox_syscall@0.7.4
  • referencing@0.46.2
  • reqwest@0.13.2
  • rustls@0.23.39
  • rustls-platform-verifier@0.6.2
  • scouter-client@0.25.0
  • scouter-dataframe@0.25.0
  • scouter-dispatch@0.25.0
  • scouter-drift@0.25.0
  • scouter-evaluate@0.25.0
  • scouter-events@0.25.0
  • scouter-http@0.25.0
  • scouter-macro@0.25.0
  • scouter-observability@0.25.0
  • scouter-profile@0.25.0
  • scouter-semver@0.25.0
  • scouter-settings@0.25.0
  • scouter-state@0.25.0
  • scouter-tonic@0.25.0
  • scouter-tracing@0.25.0
  • scouter-types@0.25.0
  • siphasher@1.0.2
  • sysinfo@0.38.4
  • tokio@1.52.1
  • tonic@0.14.5
  • tonic-build@0.14.5
  • tonic-health@0.14.5
  • tonic-prost@0.14.5
  • tonic-prost-build@0.14.5
  • tower-http@0.6.8
  • utoipa@5.4.0
  • utoipa-gen@5.4.0
  • wasm-bindgen@0.2.118
  • wasm-bindgen-futures@0.4.68
  • wasm-bindgen-macro@0.2.118
  • wasm-bindgen-macro-support@0.2.118
  • wasm-bindgen-shared@0.2.118
  • web-sys@0.3.95
  • windows-sys@0.45.0
  • windows-targets@0.42.2
  • windows_aarch64_gnullvm@0.42.2
  • windows_aarch64_msvc@0.42.2
  • windows_i686_gnu@0.42.2
  • windows_i686_msvc@0.42.2
  • windows_x86_64_gnu@0.42.2
  • windows_x86_64_gnullvm@0.42.2
  • windows_x86_64_msvc@0.42.2

Fixes

format

refactor tracing
@thorrester
thorrester force-pushed the trace-genai-dashboard branch from d170a66 to 63c522e Compare May 3, 2026 23:28
@thorrester
thorrester marked this pull request as ready for review May 10, 2026 15:49
@thorrester
thorrester merged commit 0967537 into main May 10, 2026
29 checks passed
@thorrester
thorrester deleted the trace-genai-dashboard branch May 10, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant