Skip to content

Batch Kyo requests across composed relations - #400

Open
yoohaemin wants to merge 2 commits into
refactor/typed-interpreter-zipfrom
fix/kyo-composed-batching
Open

yoohaemin wants to merge 2 commits into
refactor/typed-interpreter-zipfrom
fix/kyo-composed-batching

Conversation

@yoohaemin

Copy link
Copy Markdown
Owner

Kyo datasource implementations ran Batch.run separately for each relation invocation. (Lookup & Lookup) therefore fetched the same keys twice, and derived many/optional relations issued separate batches for each outer input.

Build an immutable KyoBatchQuery through composition and submit all ready requests in each dependency stage to Kyo Batch together. Requests remain associated with their output positions even when Kyo groups different sources in a different order. Overlapping groups such as [[1, 2], [2, 3]] now produce one callback with [1, 2, 3] while preserving both groups in the result.

This PR is stacked on #396 and uses its typed zipWith primitive. The diff contains only the Kyo batching change and its tests/documentation.

Compatibility: startingFrom, startingFromMany, and datasource callback signatures are unchanged. Low-level reify.apply / applyMultiple now return a reusable query plan; call .run for the Kyo effect. Custom reified implementations can use KyoBatchQuery.lift. The module README explains the migration. Deduplication is per dependency stage; this does not add a persistent cache or guarantee concurrent datasource effects.

Validation: 14 regression tests on JVM and Scala.js cover zipped branches, overlapping groups, duplicate/empty inputs, optional and many composition, mixed sources, dependent stages, custom datasources, typed failures, rerunning plans, and 4,096-input bulk traversal. Formatting and generated workflow checks pass.

A separate pre-existing core proof-derivation issue was reproduced with the original Kyo interpreter: a composed relation with identical input/output types can resolve to an identity proof and bypass its datasources. This PR leaves proof derivation unchanged; the dependent-stage regression uses distinct input/output types.

Copilot AI balanced review requested due to automatic review settings October 17, 2026 05:42

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Long public query composition chains can overflow the stack during execution.

1 open issue
What changed in this PR

Introduces reusable Kyo query plans to batch requests across composed relations while preserving output positions.

Changes:

  • Groups independent requests by dependency stage.
  • Migrates interpreter execution to query plans and documents compatibility.
  • Adds batching regression tests and test dependencies.
File Description
kyo-batch/​src/​test/​scala/​decrel/​reify/​KyoBatchSpec.scala Tests batching, ordering, composition, and plan reuse.
kyo-batch/​src/​main/​scala/​decrel/​reify/​KyoBatchQuery.scala Implements query composition and staged execution.
kyo-batch/​src/​main/​scala/​decrel/​reify/​kyoBatch.scala Integrates query plans into the interpreter.
kyo-batch/​README.md Documents batching semantics and migration.
build.sbt Adds Kyo module test dependencies.

Review effort: Balanced

Comment on lines +23 to +24
def step: Step[S, B] = KyoBatchQuery.this.step match {
case Done(value) => f(value).step

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants