Skip to content

feat: publish the 2.0.0 metadata contract (based on #77) - #76

Merged
earthshiner merged 27 commits into
mainfrom
feat/metadata-contract-2-0
Oct 21, 2026
Merged

earthshiner merged 27 commits into
mainfrom
feat/metadata-contract-2-0

Conversation

@earthshiner

@earthshiner earthshiner commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

What

Release 2.0.0 of the consumer contract, prepared and not tagged. This PR is based on #77 (feat/portable-design-standards) and must merge after it; its diff against that branch is the whole of this change.

It does two things.

  1. Publishes the contract. A versioned release bundle (release/v2.0.0/: contract, bindings, examples and conformance-check indexes), a compatibility policy and a migration note, and a deployed StandardDeclaration so a deployment says which version of the standard it implements. A 1.1.0 was prepared earlier and never tagged; its content is part of this release.
  2. Closes four gaps in the metadata an agent reads, on Postgres, DuckDB and Teradata:
    • Stable identities. Logical and placement-independent (domain.ticket, domain.ticket:current); physical location sits in separate columns.

    • Composite-key grouping. Keys and relationship column pairs are ordered rows (KeyColumn, RelationshipColumn); a relationship is the join group. A composite natural key, which was a comma-joined string, is now its ordered rows.

    • Ordered paths. RelationshipPaths returns one row per step and key position; the concatenated join_path string is removed.

    • Access-column mappings. AccessColumn gives one mapping row per output column of a consumer view.

    • Composite joins can be declared. A design author writes [-> Target.attribute] on each source attribute; attributes of one entity pointing at one target form one join in declaration order, and [join: name] separates two joins to the same target. The lint requires the named attributes to be exactly one key of the target, of matching types. The standard declares its own: ValidationArea to ValidationRun on three columns. The resolver now also registers the joins of standard-owned relations, so every product's relationship catalogue and path view are larger than before (23 relationships in the IT Service Desk reference, up from 7).

It also adds the comment-coverage check (every container, object and column carries a comment) and the invariants INV-SEMANTIC-016 to 020.

How it sits on #77

The Semantic relations are defined in design/modules/semantic.md and placed by the resolver, so the design change creates the new relations on every platform. The resolver derives identities, keys, relationship pairs and one consumer-view column list once; each binding registers them, creates the views and declares the checks. binding.py and render.py gain two check scopes (contract, product).

This is a breaking change

Removed attributes, renamed identities, and a rows-based path view. Business data tables and the Domain module's keying and temporal mechanics are unchanged. There is no migration script: a 1.x deployment re-registers. release/migrations/v2.0.0.md gives the identity rule and the steps.

How to Test

python -m pytest tooling -q
python tooling/release/build_release.py --check
python tooling/validation/design_lint.py

About three minutes. Result here: everything passes except 2 failures, with the Postgres and Teradata engine tests skipped. The two failures are present on #77's own head (two Teradata template checks on catalogue-interface), unchanged by this PR. The 60 skips are the Postgres and Teradata engine tests.

Verification, stated plainly

Platform Evidence
DuckDB Reference products deployed; every rendered check run. A test breaks each of the five contract checks and the coverage check, and others cover a composite natural key, a composite join, the mappings and the declaration. Conformance report regenerated: 46 passed, 0 failed, 0 skipped
Postgres Renders and shares registration with DuckDB; a static audit finds every schema, table, view, function and view column commented. The engine tests are written but not run (no POSTGRES_TEST_DSN)
Teradata Renders under strict undefined variables and passes the dialect tests. No DDL, view or check has been run on a Teradata system. Risks I could not check: REGEXP_SIMILAR semantics; REPLACE VIEW with a column list and LOCKING; COMMENT ON DATABASE/PROCEDURE; and the size of the unrolled RelationshipPaths view (about 54,000 characters, ten branches)

Review points

  • Merge order. After Feat/portable design standards #77. The CONFORMANCE.md for Postgres and Teradata are out of date (this PR adds tests) and need regenerating on a machine with those engines; I regenerated only DuckDB's, to avoid recording their engine tests as skipped.
  • Teradata path view. It follows Feat/portable design standards #77's own approach (one branch per path length) but is much larger. If Teradata rejects its size, a small helper view for the edges would shrink it, at the cost of one more standard-owned view.
  • Teradata registration. The registration template is now shared with the other bindings, so the new rows are populated at deployment.
  • Composite references. A reference attribute carries one source column, so a reference to a composite identity pairs the first column and reports the rest as a gap.
  • Not tagged. Tag v2.0.0 after the Teradata live run.

Checklist

  • Commit messages follow discipline
  • No secrets committed
  • Tests pass, apart from the two failures already on Feat/portable design standards #77
  • Documentation updated (design, platform READMEs, release notes, DuckDB conformance report)
  • No new file types needing .gitignore changes

🤖 Generated with Claude Code

earthshiner and others added 11 commits October 14, 2026 10:38
MetadataCoverageCheck previously covered only declared entity columns.
Add a catalogue sweep per platform that returns every schema, table,
view, routine and column with no comment, including objects no
declared entity mentions.

The sweep exposed gaps in the rendered Postgres product: 10 views and
13 functions were uncommented and no view column carried a comment.
Comment the entity views, history and search functions, discovery and
trust-map views, and the product schemas.

The Postgres check is not yet executed against a database; the
database-backed tests need POSTGRES_TEST_DSN.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The registry defines product_version as the published contract version
of the data product. The manifest view comment wrongly described it as
the version of the standard the deployment implements.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Specifies the versioned release bundle and the deployed standard
version declaration for agents reading a deployed product. Sub-project
A of two; the metadata gaps are sub-project B.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Add standard_declaration to the Semantic module: one row per
implemented component (standard, modules, patterns) carrying the
platform and standard version, read from the root VERSION file at
render time. The manifest view gains standard_version so the bootstrap
read answers which contract a deployment implements.

Additive, so this prepares release 1.1.0. Only the Postgres binding
declares it; the render output refuses the reserved release directory.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Add tooling/release, which generates release/vX.Y.Z/ (contract,
bindings, examples and checks indexes) from the repository sources,
and a test that fails when the committed bundle is stale or the current
version lacks its migration note or compatibility row.

Authored COMPATIBILITY.md sets the versioning policy and lists the
known limitations the next release addresses. The trust-map comments
become explicit statements so the access surface can be enumerated.
The bundle is not a module and has no design or binding directory.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Brings in the Jinja-only bindings (PR 74), including the Jinja DuckDB
binding. Convert the Teradata comment-coverage sweep to a Jinja
template that reads the build context's containers, register its
render case, regenerate the 1.1.0 bundle, and record that the DuckDB
binding has not yet adopted the declaration or comment parity.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Specifies logical identities, key definitions, relationship column
groups, path rows and access-column mappings across all three
platforms, folded with the prepared 1.1.0 content into one release.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Logical identities, key_definition/key_column, relationship_column, access_column,
standard_declaration, relationship_edges/relationship_paths as rows, via_relationship_id
composition, INV-SEMANTIC-016..020; dependants (export, VAL-14, examples) rewritten.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Logical, placement-independent identities (module.entity); key
definitions with ordered key columns; relationships as join groups of
ordered column pairs; path rows instead of concatenated strings; and
one mapping row per output column of an access object. Adds the five
conformance checks INV-SEMANTIC-016 to 020 with a mutation test each.

The DuckDB Jinja binding also adopts what 1.1.0 prepared: the
standard declaration, comment parity (view columns, macros,
sequences) and the comment-coverage sweep. The Semantic design and
both reference specifications acknowledge the new invariants.

The release bundle is regenerated in a following commit, after the
Teradata binding is merged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
VERSION becomes 2.0.0. The release bundle now publishes the identity
grammar, the ordered-group tables and the row-based path surface, and
all three bindings declare the standard version. COMPATIBILITY.md sets
the stable surfaces, the known limitations and the platform
differences; migrations/v2.0.0.md gives the identity rule, the steps a
deployment takes and the execution evidence per platform.

The Teradata identity check now tests the module qualifier instead of
matching container names by substring, which could fail valid
identities.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
earthshiner and others added 8 commits October 14, 2026 16:32
This work is based on PR 77: every binding reads one build context and
the Semantic relations come from the design. Take PR 77's tree and keep
only the release specs, the release bundle and its generator, and the
VERSION file from the earlier implementation. The 2.0.0 metadata
contract is re-applied on this structure in the commits that follow.

The release generator and its tests still target the earlier renderer
and are ported in a later commit.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…cture (Postgres, DuckDB)

Re-express the contract on PR 77's architecture. The design defines
the new Semantic relations (KeyDefinition, KeyColumn,
RelationshipColumn, AccessColumn, StandardDeclaration); the resolver
adds logical identities, key definitions, relationship column pairs
and one shared consumer-view column list; registration writes them;
RelationshipPaths becomes rows; the manifest declares the standard
version.

Adds the five contract checks and the comment-coverage sweep as
declared checks (new contract and product scopes), comment parity for
views, view columns, functions, schemas and sequences, and engine
tests including a mutation test per check. Executed on DuckDB; the
Postgres templates render and share the registration.

Teradata and the release generator follow.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Registration writes logical identities, key and relationship column rows,
access identities, access columns and standard declarations. RelationshipPaths
is rows (unrolled branches, as Teradata views take no recursion or WITH);
the manifest ends with standard_version. Consumer views select the context
view_columns. Adds the five contract checks and the comment-coverage sweep,
and comments on databases, views, view columns, the trust map and the
similarity procedures.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ext structure

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…nd publish the bundle

Merge the Teradata port (registration shared with the other bindings,
rows-based RelationshipPaths unrolled per path length, the five
contract checks and comment coverage, comment parity). Port the release
generator to the build-context pipeline, regenerate the 2.0.0 bundle
across all three platforms and refresh DuckDB's conformance report.

Teradata has not been executed; see release/migrations/v2.0.0.md.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@earthshiner earthshiner changed the title feat: publish the 2.0.0 metadata contract and release bundle feat: publish the 2.0.0 metadata contract (based on #77) Oct 14, 2026
@earthshiner
earthshiner changed the base branch from feat/merge-portable-design-standards to feat/portable-design-standards October 14, 2026 06:15
earthshiner and others added 3 commits October 14, 2026 21:11
200 attributes across the Semantic, Memory, Observability and Prediction
modules and the catalogue-interface pattern had no description, so the
column comments generated from the design repeated the column name
(180 of 478 catalogue columns). Each now states what it holds.

Also list all four entity surfaces in the identity grammar table.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A design author can now state a multi-column join: [-> Target.attribute]
on each source attribute pairs it with the named target attribute, the
named attributes of one entity that point at one target form one join
in declaration order, and [join: name] separates two joins to the same
target. The lint requires the named attributes to be exactly one key of
the target, of matching types.

The resolver builds one relationship per join group and now scans the
standard-owned relations as well as the specification's entities, so the
standard's own joins are registered, starting with ValidationArea to
ValidationRun on three columns. The relationship reference check joins
on every pair on all three platforms.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@earthshiner
earthshiner changed the base branch from feat/portable-design-standards to main October 14, 2026 11:48
Record the decisions, open design questions, and proposed approach for porting the Teradata graph-lineage and catalogue-interface templates. Remove the completed release-contract and metadata-contract design documents.

@NathanG-TD NathanG-TD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the metadata contract is a great addition.

earthshiner and others added 3 commits October 21, 2026 16:29
A Reference column was always typed as the surrogate BIGINT, even when its
target is keyed only by a natural key. DesignDecision.superseded_by and
ChangeLog.related_decision_id hold a decision_id, which is text, so the
generated reference check compared a bigint with a varchar and PostgreSQL
rejected it. DuckDB coerced the types and hid the error.

Relationships now cover the memory relations too, which exposed it. Take the
type from the target's single natural key when it has no surrogate.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The contract-identity check holds NOT LIKE ... || '.%'. The test helper passed
an empty parameter list, which made psycopg read the percent sign as a
placeholder and fail on PostgreSQL. DuckDB and the Teradata wrapper already
accept None.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@earthshiner
earthshiner merged commit 889b6ff into main Oct 21, 2026
1 check failed
NathanG-TD added a commit that referenced this pull request Oct 22, 2026
Taken from the conformance-reports artifact of PR #82's CI run, where
both engines ran: 49 passed, none failed or skipped on each. DuckDB adds
the README-section package test; PostgreSQL adds it and the engine tests
from #76, recorded for the first time. The Teradata report is not
included: its engine tests skipped in that run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NathanG-TD added a commit that referenced this pull request Oct 22, 2026
Run locally against the Teradata test system: 60 passed, none failed or
skipped. It clears the two package failures #81 fixed and records, for
the first time on Teradata, the rendering and engine tests of the 2.0.0
metadata contract (#76) and the README-section package test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants