[NathanG-TD]

Portable design standards: one specification, three engines

Summary

A build now has exactly three inputs: the design specification (the design agent's output, which names no platform, container or organisation), an organisation profile (one per organisation per platform), and the platform binding. tooling/build/build_context.py resolves them into one build context, and every binding's Jinja templates read only that context. They derive no names and hold no product literals.

PostgreSQL, DuckDB and Teradata are all on the build context. Both reference specifications (IT Service Desk and Customer Orders) deploy on all three engines, and every check the build renders passes. One specification gives every binding the same relations and the same checks, and a test holds them to that.

This PR includes PR #74 (Jinja-only bindings), which was merged into this branch rather than main.

What changes

Standards (design/)

• Platform Implementation Authoring Standard: the three build inputs; the build context; binding declarations (Binding:, Types:, Functions:) that bring a platform's SQL into the context as data; a new Gaps field for gaps no check can detect; and conformance status that follows generated evidence (§8.1).
• Design Specification Standard and Organisation Profile Standard: what each input carries, in tables the linters and the resolver read as the rule.
• Settled where the bindings disagreed, each now enforced by a test:
  • Semantic §5 bounds path discovery at four hops (PostgreSQL and DuckDB walked four, Teradata three). The resolver passes the bound to every binding.
  • Validation §10: a legacy run's derived PRODUCT trust entry is graded by §4.3 and capped at partial. PostgreSQL and DuckDB had shown partial even with ERROR failures.
  • Validation defines ValidationRun and ValidationArea as entities. The standard-owned discovery and trust-map views are consumer views placed by the profile. base_view is an optional role that a binding declares it emits.
• No platform names in design/: a new platform-name lint rule, with no ignore-file escape. The Design Language's Appendix A, a hand-kept copy of the bindings' physical types that had drifted (native VECTOR on Teradata, pgvector, vss), now points at the bindings' declarations.

Tooling

• tooling/build/: the profile checker, the binding-declaration loader and the resolver. tooling/evals/expressions.py renders the expression vocabulary through each binding's declared SQL fragments, and reports any function a binding lacks rather than approximating it.
• tooling/bindings/render.py renders a binding's declared phases, deploy script and checks into a manifest that lists every gap. validate.py runs the checks and publishes wire-schema 2.1 evidence through a driver per engine (engines.py), with no platform == ... branches.
• tooling/bindings/conformance.py generates each binding's CONFORMANCE.md from the tests, the reference builds and the binding's declared gaps, replacing the hand-written files. --check fails on an out-of-date report and names any gap it adds.
• tooling/skill/verify_skill.py --package checks what an install actually receives (git archive HEAD). {platform} routes are now checked on every binding; they were hard-coded to Teradata.

Bindings

• PostgreSQL and DuckDB: rewritten as templates over the build context; the old per-platform inputs (TEMPLATE_INPUTS.md, entities.json, context.json, placement.json) are gone.

• Teradata: moved onto the build context and engine-tested on a ClearScape system. The real engine found and fixed:

  • no VECTOR type on 17.20 or 20.00.24, so a Vector[n] is VARBYTE(n × 8) of little-endian 8-byte floats, checked to that length. TD_VECTORDISTANCE reads VARBYTE as 8-byte floats; with 4-byte floats every distance was wrong, and no error was raised;
  • GRANT ALL cannot go to a role;
  • grants ran before the objects they grant on existed;
  • interval literals take no precision;
  • an untyped NULL in a union branch fails only when the view is queried;
  • hours_between and days_between are now exact.

  Similarity search on Teradata is now rendered: one stored procedure per Embedding: block runs TD_VECTORDISTANCE over the store's current vectors, as dynamic SQL because the operator takes no parameters. It is engine-tested for ranking, threshold, limit and a wrong-dimension query. A new Deploying user setting grants CREATE PROCEDURE, which a database's creator does not get, and the read and agent tiers get EXECUTE PROCEDURE. Types may be sized as {dim*n}, and the deploy splitter keeps a procedure as one statement.

  test_teradata_dialect.py holds every rendered statement to PLATFORM_PROFILE §7. Teradata now grants its governed history views to the read and agent tiers, as PostgreSQL grants its point-in-time functions.

• Adoption, on every binding: an engine test deploys over an existing table. It found every binding treating an adopted container as its own: changing its grants, flagging the organisation's other tables as unregistered, and publishing their metadata. All three are fixed.

Examples

• examples/it-service-desk-data-product/profiles/ holds the organisation profiles: PostgreSQL, DuckDB and two contrasting Teradata ones.
• The generated customer360-duckdb example and the binding-specific inputs under the ITSD example are removed: examples hold inputs, not generated output (Authoring §7).

CI

.github/workflows/validate.yml runs the validation suite, then conformance.py --check on DuckDB, PostgreSQL (service container) and Teradata (TERADATA_TEST_DSN secret), and fails on any skip. docs/ci-setup.md covers the setup.

Before merging

1. Decide on the graph-lineage and catalogue-interface templates (port or retire). They still read the old flat context, and no phase renders them. Two test_binding_package tests fail on them alone, so CI cannot go green until this is settled. They are left untouched pending that decision.
2. Add the TERADATA_TEST_DSN repository secret (docs/ci-setup.md). Until then the Teradata tests skip and CI fails, by design: a missing engine is a gap, never a pass. Then make validate a required check.
3. Delete REMAINING_WORK.md. It is the handoff note for this branch, not part of the corpus.
4. PR feat: declare product layout so evaluators stop assuming names #75 (layout standard): update it or withdraw it. The organisation profile now covers layout and naming.

For review

• History for consumers. Every binding now grants full history to the read and agent tiers, including the past values of personal-data columns. The access-layer grant matrix (§5) never names the governed full-contract surface. Confirm the intent and add the row.
• Tables apart from views. Teradata best practice keeps them in separate databases, so grants can be database-level. That is planned as a separate PR, applied to PostgreSQL and DuckDB too, so every binding has the same layout.
• In-database encoding. mldb.ONNXEmbeddings returns VARBYTE as 4-byte floats, so vectors encoded in Teradata must be converted to 8-byte floats before they are stored. Whether the function can emit 8-byte floats directly is untested.
• Adoption modes. Only the reference mode is implemented. Authoring §4.3 says the modes are "defined separately", but they are not defined yet.

Evidence

	PostgreSQL	DuckDB	Teradata
Engine	16.4 (psycopg 3.3.6)	1.5.6 (Python driver)	20.00.24.60 (teradatasql 20.0.0.71)
Tests passed	36	36	39
Tests failed	0	0	2 (the untouched templates above)
Tests skipped	0	0	0
Gaps the builds report	13	14	17
Gaps the binding declares	8	11	13

From the last full conformance.py run, on all three engines; each binding's CONFORMANCE.md has the detail.

• Compositions: both reference specifications, each with all six modules. Smaller compositions are not shown to deploy (a declared gap).
• Portability: the IT Service Desk specification builds under four example profiles (PostgreSQL, DuckDB and two contrasting Teradata ones), and under a renamed variant of each platform's profile that changes every container name and no template. Customer Orders builds under each platform's profiles. No output carries another case's names.
• Profile settings: Teradata declares Container space; the profiles' mask protection and object-storage placement are not rendered by any binding and are reported as gaps.
• Validation suite: 253 of 255 pass, with the same two failures. Design lint, catalogue and skill-package checks are clean.
• Every binding's status is draft. None is gap-free, so none may be standard (§8.1), and a package test now enforces that.

How to verify

python -m unittest discover -s tooling/validation/tests
pip install -r tooling/bindings/requirements.txt duckdb pytz "psycopg[binary]" teradatasql
POSTGRES_TEST_DSN="..." TERADATA_TEST_DSN='{"host": "...", "user": "...", "password": "..."}' \
    python tooling/bindings/conformance.py --check
python tooling/skill/verify_skill.py --package

A full Teradata run takes 17 to 55 minutes, depending on the system's load.

[earthshiner]

Heads-up from #76, which is based on this PR.

The two failing test_binding_package tests are the catalogue-interface and graph-lineage templates listed under "Before merging" item 1. #76 does not change those templates or the tests, so it inherits exactly these two failures and does not mask them.

We would rather not delete the templates, and we would not exempt them from the gate. Our suggestion is a follow-up PR after this one merges that settles where the governance, stage and graph containers are placed (nothing in the profile standard places them today) and then ports both sets onto the build context. A tracking issue is open for it: #78.

Question for you: do you want to settle port-or-retire in this PR, or are you happy for it to be handled separately?

[earthshiner]

See my latest comments about the 2 Teradata test failures and how we might deal with them. I raised an issue #78