Skip to content

cel: every evaluation site discards EvalDetails — actual cost is measured on every expression and reported nowhere, and OptTrackState would give flow policy explain, DAP evaluate and check witnesses a real per-node trace instead of a re-evaluation #1485

Open

Description

@picatz

Observed behavior

Every Flowstate evaluation runs with cost tracking on and throws the measurement away: evalProgram (pkg/flowstate/v1/celenv.go:243) reads out, _, err := prg.ContextEval(...), and the five policy sites do the same (taskpolicy.go:437, netpolicy/rules.go:103, auth/assume.go:229, auth/secretpolicy.go, runfilter.go:217). EvalDetails.ActualCost() is read only in tests (celcost_test.go:277-366, celenv_optional_test.go:98). No production code uses cel.OptTrackState, interpreter.EvalState or EvalStateObserver (grep at 40cc365).

What the discarded data would answer, on surfaces that exist and ask the question by other means:

Desired outcome

Evaluator keeps what cel-go measured and offers it in two forms: a per-evaluation outcome (actual cost, peak memory once available, cancellation cause) that the step executor aggregates into the task span and the task metrics on both drivers; and an opt-in explain evaluation that plans with OptTrackState, joins the state map to source_info positions, and renders position → value through the same redaction seams flowdebug uses (flowdebug/inspect.go:172-233; celcomplete/doc.go:23-30 states why values never leave without them). flow policy explain, DAP evaluate on failure and flowtest witnesses read the one trace.

Acceptance criteria

  • A task span carries the summed expression cost for the step and the run metrics carry a histogram of per-evaluation cost, on both drivers, pinned by the observe conformance case.
  • -o json for flow run local does not carry cost: it is interpreter-version-dependent and would make a run document differ across builds; the run document stays a projection of durable state.
  • flow policy explain (cli: no CEL policy surface can be tested against cases — a flow policy test and flow policy explain over the five activations that already exist #1468) reports the deciding rule and the value of each sub-expression of that rule, redacted, from one evaluation rather than a re-evaluation.
  • A failing DAP evaluate and a failing expect.check: show the same position → value rendering, produced by one function.
  • ErrorKindExpression failures name the cause (cost, memory, cancelled) read from EvalCancelledError.Cause, with a conformance case per cause.

Constraints and dependencies

  • Cost and state tracking are per-program options; the program cache (celenv.go:372) keys on env and expression, so an explain program must be a distinct cache entry or uncached (it is a debugging path; uncached is fine).
  • OptTrackState retains every intermediate value for the life of the evaluation, so explain is bounded by MaxCommandBytes-class limits and never enabled on the hot path.
  • Nothing measured here may enter RunState or the run document (invariant 10 and the version-pinning argument in docs/ARCHITECTURE.md); spans and metrics are the only sinks.
  • cel-go v0.32.0's "observers report every evaluation step" (Report every evaluation step to every observer cel-expr/cel-go#1419) changes what a state observer sees; the trace format should be pinned after the bump, not before.

Activity

  1. added
    enhancementNew feature or request
    engineExecution engine, schema, runtime internals
    cliThe flow CLI: flags, output, ergonomics
    observabilityTraces, metrics, logs, profiles
    on Sep 2, 2026
  2. picatz commented on Sep 7, 2026

    @picatz
    OwnerAuthor

    One attribute to carry on the cost metric when this lands: the run's tenant (the identity's namespace, which the memo already records). At 7530b29 every evaluation site goes through DefaultEvaluator() with one process-wide Limits (call.go:59, wait.go:393, concurrency.go:131, value.go:53, loop.go:260, switch.go:204, eval_task_http.go:658 and the rest), so on a shared worker (#236's Tier 1a) there is no per-tenant budget and no per-tenant number. #236's gap 6 deliberately declines an app-layer limiter and points at Temporal namespace rate limits for volume; expression cost is the one resource those limits cannot see, because it is spent inside a workflow task. A histogram keyed by tenant makes a noisy neighbour visible without building a limiter, and is the evidence a later decision about per-tenant Limits (Tier 1b) would need. Nothing here changes what enters RunState.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cliThe flow CLI: flags, output, ergonomicsengineExecution engine, schema, runtime internalsenhancementNew feature or requestobservabilityTraces, metrics, logs, profiles

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions