[Feat] Add UCM metrics for SGLang HiCache and Direct Linker by wz-cloud · Pull Request #1635 · ModelEngine-Group/unified-cache-management

wz-cloud · 2026-11-06T08:39:48Z

Purpose

Expose UCM storage and scheduling metrics for SGLang's HiCache and Direct Linker integrations through SGLang's existing Prometheus /metrics endpoint. With SGLang metrics enabled, UCM uses built-in metric definitions unless an explicit configuration is supplied.

Fixes #1617.

Modifications

Reuse UCM's accumulators, dispatcher and background Prometheus logger with SGLang's PROMETHEUS_MULTIPROC_DIR. Export ucm: metrics with model_name and per-worker rank labels.
Instrument HiCache storage transfers and Direct Linker asynchronous submit/check/wait operations, including successful bytes/pages, duration, bandwidth and submission/completion errors.
Track Direct Linker lookup, queueing, load/offload completion, cancellation and failure paths. Preserve task draining and develop's per-task distributed backend load options.
Add metrics lifecycle/configuration handling, built-in defaults, optional YAML overrides and an independent UCM disable switch. Extend PrometheusStatsLogger to accept an in-memory configuration.
Add four Grafana dashboards and CPU-only regression coverage for metric export, configuration, worker lifecycle, asynchronous I/O and dashboard definitions.

Test

At head 2952f801:

110 passed, 53 subtests passed across the SGLang metrics, linker metrics, direct preparation, load/offload workers, dump partition, HiCache configuration, patch selection, load-task options and metrics collector tests:

python -m pytest --noconftest -q -o addopts= -o log_cli=false \
  test/test_sglang_direct_preparation.py test/test_sglang_load_workers.py \
  test/test_sglang_offload_workers.py test/test_sglang_linker_metrics.py \
  test/test_sglang_dump_partition.py test/test_sglang_metrics.py \
  test/test_sglang_hicache_config.py test/test_sglang_patch_selection.py \
  test/test_load_task_options.py test/test_collect_ucm_metrics.py

test/test_ucm_connector_metrics.py: 433 passed, 6 failed. The same six failures reproduce on unmodified develop a03985fa: default configuration/example consistency, KV metric-count expectation, repeated layer-load hooks (two cases), and hybrid layerwise load/save byte tests.
An additional ad hoc comparison covered 96 combinations of Store implementations, replication/event options, metrics enablement and success/failure paths. Store arguments, call order and outcomes matched develop; metric counts were also checked.
git diff --check passed. Black and isort checks passed for the resolved adapter and related test files.

Hardware validation reported by the author: multi-device testing on A2 verified service startup through both the HiCache and Direct Linker integration paths, with no material performance impact observed. The automated results above use CPU tests with simulated storage dependencies; no quantitative hardware benchmark results are included here.

Documentation

Usage and metric semantics are described in #1617. This PR adds examples/metrics/metrics_configs_sglang.yaml and the SGLang, HiCache, Direct Linker and UCM Store Grafana dashboards. The explicit SGLang configuration is synchronized with develop's updated KV metric definitions.

No Markdown paths under the bilingual documentation trees are changed.

…ation

Preserve upstream transfer events, nonblocking layer submission and dump partitioning. Observe batched dumps per native task and retain address buffers through completion.

Preserve SGLang I/O metrics while forwarding distributed backend load options. Synchronize the SGLang example configuration with the updated KV metrics and cover successful and failed distributed load submissions.

student-jhz

整体结论:comment(建议按意见修改后合入,无阻塞性问题)。

理由:本 PR 为 SGLang HiCache 与 Direct Linker 暴露 UCM 指标,实现与测试质量高:禁用时零开销(不读时钟、不算字节,有专门测试锁定);记录失败只告警、不影响存储操作结果;submit/wait/check 失败分相统计且与 YAML 文档语义一致;生命周期处理严谨(指标注册先于 Store 创建、初始化失败按序关闭资源、close 幂等且全量关闭后抛首个错误、Linker 关闭时先 drain 队列再最终 flush);内置默认配置与示例 YAML 由测试锁定一致且不污染 vLLM 默认配置;CI 51 项检查(py-linter、cpp-linter、gtest、Docs·Bilingual、e2e sglang-a2/pc-a2)全部 success。主要问题:1 个 [major] 文档同步缺失(见行内意见),2 个 [minor](HiCache 关闭时丢最后一个导出窗口的增量;两条集成路径的 enable_metrics/metrics_config_path 配置入口不一致),其余为 [nit]。无可靠性 blocker。

PR 规范性:标题带 [Feat] 前缀、一句话意图清晰;描述 Purpose/Modifications/Test/Documentation 四段齐全,Test 段给出完整命令、develop 基线失败对照(6 个失败属本 PR 未触碰的 test_ucm_connector_metrics.py)与 96 组合对照说明,Documentation 段明示双语文档未改动;commit message(feat(sglang)/test(sglang))与 PR 内容一致;改动单一主题,约 19k 行为 dashboard JSON 与示例 YAML 配套文件,范围聚焦。

亮点(已核实):

measure_io/AsyncIoObservation/LinkerOperation 在未启用时不读时钟、不算字节,test_disabled_and_empty_observations_do_not_read_clock 以 perf_counter 抛 AssertionError 验证。
指标故障不影响存储操作结果(test_io_v2_pool_sizes_flattened_and_recording_failure 覆盖);观测 finish 幂等,错误分相计数与文档一致。
test_registration_precedes_store_setup_and_failure_closes_logger 断言初始化失败时 [logger, store, store.close, logger.close] 的事件顺序;close() 从 develop 的首个异常中断改为全量关闭后抛首个错误,消除句柄泄漏。
子进程真实多进程 Prometheus 导出测试:双 worker 按 rank 聚合、与 sglang: 序列共存、重复采集不重放、非法显式配置不静默回退、dashboard 指标名与配置一致性校验。
get_default_sglang_metrics_config() 与示例 YAML 由测试锁定完全一致;基于 deepcopy,DEFAULT_METRICS_CONFIG 不被污染(测试断言)。

student-jhz · 2026-11-07T07:08:29Z

@@ -0,0 +1,1095 @@
+# SGLang UCM metrics: shared by HiCache and direct Linker integrations.


[major][文档同步] 本 PR 引入用户可见的 SGLang 指标配置面:HiCache 经 SGLang enable_storage_metrics 开启,Direct Linker 经 SGLang enable_metrics + UCM YAML enable_metrics/metrics_config_path 控制,并新增本示例文件与 4 个 dashboard;但 metrics 用户指南完全未同步:docs/source/user-guide/metrics/metrics.md 与 docs/docs-site/docs/{en,zh}/user-guide/observability/metrics.md 仅覆盖 vLLM 路径(engine/worker_rank 标签、vLLM /metrics),全文无 SGLang 内容,PR 描述指向 issue #1617 而非仓内文档。用户无法得知 SGLang 下如何开启、指标以 ucm: 前缀出现在 SGLang /metrics、标签为 model_name/worker_id、以及两条路径的配置入口差异。
建议:在 docs/docs-site/docs/{en,zh}/user-guide/observability/metrics.md(双语成对)及 docs/source/user-guide/metrics/metrics.md 增加 SGLang 小节,说明开启方式、配置项位置、标签语义、本示例文件与 4 个 dashboard 的用法。

student-jhz · 2026-11-07T07:08:29Z

+                        logger.exception("Failed to close UCM store")
+        finally:
+            if self._metrics_logger is not None:
+                self._metrics_logger.shutdown()


[minor][可靠性] HiCache close() 只调用 self._metrics_logger.shutdown(),而 shutdown() 仅停止轮询线程、不做最终 drain(见 ucm/observability.py 的 shutdown 实现);最后一次轮询之后累积的计数增量(最多一个 log_interval,示例配置为 5s)在 worker 退出时丢失。同一 PR 为 Linker 路径专门实现了 shutdown_metrics_logger()(停线程 → drain → 最终导出,且有 test_shutdown_stops_poller_before_final_export 覆盖),两条路径行为不一致;from_hicache 失败分支(本文件 237 行)同样只调 shutdown()。
建议:close() 的 finally 与 from_hicache 失败分支改用 ucm.integration.sglang.metrics.shutdown_metrics_logger(self._metrics_logger)(内部已做 None 判断与异常兜底)。

student-jhz · 2026-11-07T07:08:29Z


-        return UnifiedCacheStoreConfig(module_path=module_path, name=name, config=cfg)
+        return UnifiedCacheStoreConfig(
+            module_path=module_path, name=name, config=cfg, launch_config=dict(kvc)


[minor][易用性] 同名选项 enable_metrics/metrics_config_path 在两条集成路径的读取位置不一致:HiCache 的 launch_config=dict(kvc) 取自 kv_connector_extra_config(或 ucm_connectors[0] 条目)内部;Direct Linker 的 adapter.launch_config 取自 UCM YAML 文档顶层键或 linker_extra_config(见 ucm_direct_store_adapter.py from_config)。同一份 UCM YAML 无法同时为两条路径配置指标开关/路径,用户容易配错位置。
建议:让 load_from_config 在解包 UCM_CONFIG_FILE 后同样接受文档顶层 enable_metrics/metrics_config_path(条目内显式值优先),统一两条路径的入口;至少在用户指南中明确说明两条路径的配置位置差异。

student-jhz · 2026-11-07T07:08:29Z

+  - name: "drampool_resource_log_read_errors_total"
+    documentation: "Number of failures opening, reading, or parsing the DramPool resource log"
+
+# Gauge metrics configuration


[nit][可读性] '# Gauge metrics configuration' 注释位于 Direct Linker counter 条目(391-441 行,均属 counter: 列表)之前,而真正的 gauge: 键在 443 行,注释位置与内容不符、易误导。
建议:将该注释移至 443 行 gauge: 上方,391 行处保留 '# Direct linker: ...' 分段注释即可。

student-jhz · 2026-11-07T07:08:29Z

+
+# Gauge metrics configuration
+  # Direct linker: per-worker counters and asynchronous operation timing.
+  - name: sglang_linker_lookup_requests_total


[nit][代码规范] Linker 段(391-441 行 counter、1023-1101 行 histogram)与文件其余部分 YAML 风格不一致:该段 name/documentation 不加双引号、documentation 用单引号折行、buckets 用块式列表;其余部分统一为双引号字符串 + 行内 bucket 数组,疑似不同工具生成后直接拼接。
建议:统一为文件其余部分的风格(双引号 + 行内数组),便于后续维护与 diff 审读。

student-jhz · 2026-11-07T07:08:29Z

        sequence = 0

-        def submit(task: tuple[list[PoolTransfer], list[object]]) -> None:
+        def submit(task) -> None:


[nit][可读性] 队列元素扩为三元组时,def submit(task) -> None: 直接丢掉了原有类型注解(原为 task: tuple[list[PoolTransfer], list[object]])。
建议:更新为 def submit(task: tuple[list[PoolTransfer], list[object], LinkerOperation | None]) -> None:,与文件内其余带注解签名保持一致。

student-jhz · 2026-11-07T07:08:29Z

+            # rates exist before the first completion, without fabricating samples.
+            record_metrics(
+                {
+                    item["name"]: 0


[nit][可靠性] 零值初始化用 item["name"] 直接取键:若用户 YAML 的 counter: 里混入缺 name 的条目,此处会 KeyError 使 linker 初始化失败;而注册侧(_register_metrics_by_type)与 get_metric_definitions 对同类条目均为静默跳过,两处容错不一致。
建议:改为 item.get("name", ""),保持与注册侧一致的宽容行为(与下方 startswith 判断天然配合)。

wangzheng added 4 commits November 5, 2026 19:42

feat(sglang): add Prometheus metrics lifecycle and HiCache instrument…

c3cb82b

…ation

feat(sglang): instrument direct linker scheduling and batched store I/O

65ed15f

Preserve upstream transfer events, nonblocking layer submission and dump partitioning. Observe batched dumps per native task and retain address buffers through completion.

test(sglang): add metrics dashboards and export regression coverage

600feb8

Merge origin/develop into feat/sglang-metrics-clean

2952f80

Preserve SGLang I/O metrics while forwarding distributed backend load options. Synchronize the SGLang example configuration with the updated KV metrics and cover successful and failed distributed load submissions.

wz-cloud requested review from FangRun2, Infinite666, SuperMarioYL, Tarrei, Wwwzff, harrisonyhq, mag1c-h, nrj868, qyh111 and ygwpz as code owners November 6, 2026 08:39

wz-cloud closed this Nov 6, 2026

wz-cloud deleted the feat/sglang-metrics-clean branch November 6, 2026 08:41

wz-cloud restored the feat/sglang-metrics-clean branch November 6, 2026 08:44

student-jhz reviewed Nov 7, 2026

View reviewed changes

student-jhz mentioned this pull request Nov 7, 2026

feat(sglang): add request-level cache match and load logs #1634

Open

Navigation Menu

Repository navigation

[Feat] Add UCM metrics for SGLang HiCache and Direct Linker - #1635

[Feat] Add UCM metrics for SGLang HiCache and Direct Linker#1635
wz-cloud wants to merge 4 commits into
ModelEngine-Group:developfrom
wz-cloud:feat/sglang-metrics-clean

wz-cloud commented Nov 6, 2026

Uh oh!

student-jhz left a comment

Uh oh!

student-jhz Nov 7, 2026

Uh oh!

student-jhz Nov 7, 2026

Uh oh!

student-jhz Nov 7, 2026

Uh oh!

student-jhz Nov 7, 2026

Uh oh!

student-jhz Nov 7, 2026

Uh oh!

student-jhz Nov 7, 2026

Uh oh!

student-jhz Nov 7, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,1095 @@
		# SGLang UCM metrics: shared by HiCache and direct Linker integrations.

Conversation

wz-cloud commented Nov 6, 2026

Purpose

Modifications

Test

Documentation

Uh oh!

student-jhz left a comment

Choose a reason for hiding this comment

Uh oh!

student-jhz Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

student-jhz Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

student-jhz Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

student-jhz Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

student-jhz Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

student-jhz Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

student-jhz Nov 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants