Feat/portable design standards

.github/workflows/validate.yml

@@ -0,0 +1,93 @@
+name: validate
+
+# Required check for merging to main (see design/core/IMPLEMENTATION_AUTHORING.md
+# section 8). Runs every repository validation suite: design and specification
+# lint, template hygiene (Jinja only), template rendering for every binding,
+# and native engine tests for DuckDB, PostgreSQL and Teradata.
+#
+# Teradata has no container image, so its engine tests run against a disposable
+# system named by the TERADATA_TEST_DSN repository secret (teradatasql connection
+# parameters as JSON). Until the secret is set those tests skip, and the job
+# fails: missing engine coverage is a gap, never a pass. The tests own every
+# database and role prefixed `anptest_` on that system, so runs that use it are
+# serialised across branches.
+#
+# A skipped test is a gap, not a pass (standard section 8). The suites are run
+# with engines available, and the job fails if any test reports a skip.
+
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+permissions:
+  contents: read
+
+concurrency:
+  group: validate-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  validate:
+    name: validate
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    concurrency:
+      group: teradata-engine
+      cancel-in-progress: false
+
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_PASSWORD: ci-only-disposable
+          POSTGRES_DB: ci
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 5s
+          --health-timeout 5s
+          --health-retries 10
+
+    env:
+      POSTGRES_TEST_DSN: postgresql://postgres:ci-only-disposable@localhost:5432/ci
+      TERADATA_TEST_DSN: ${{ secrets.TERADATA_TEST_DSN }}
+
+    defaults:
+      run:
+        shell: bash
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install dependencies
+        run: python -m pip install -r tooling/bindings/requirements.txt duckdb pytz "psycopg[binary]" teradatasql
+
+      - name: Validation suite (design lint, template hygiene, rendering)
+        run: python -m unittest discover -s tooling/validation/tests -v 2>&1 | tee validation.log
+
+      # Runs the binding and package tests once and regenerates each binding's
+      # CONFORMANCE.md from them. Fails on any failing or skipped test, and when a
+      # committed report differs from the regenerated one (Authoring Standard 8.1);
+      # the regenerated reports are uploaded, to be committed when they change.
+      - name: Binding suite and conformance reports (DuckDB, PostgreSQL and Teradata engine tests)
+        run: python tooling/bindings/conformance.py --check --output build/conformance 2>&1 | tee bindings.log
+
+      - name: Upload the regenerated conformance reports
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: conformance-reports
+          path: build/conformance/
+
+      - name: Fail on skipped tests
+        run: |
+          if grep -E "skipped|\.\.\. skipped" validation.log bindings.log; then
+            echo "A skipped test is a conformance gap, not a pass." >&2
+            exit 1
+          fi

README.md

@@ -24,7 +24,7 @@ ai-native-data-products/
 │                               physical-storage · validation · access-layer
 ├── implementation/
 │   ├── teradata/               PLATFORM_PROFILE + modules/ and patterns/ bindings
-│   ├── duckdb/                 portable file binding and executable conformance
+│   ├── duckdb/                 reusable Jinja bindings for a portable file engine
 │   └── postgres/               reusable Jinja bindings, native grants and history constraints
 ├── tooling/
 │   ├── validation/             the design linter (+ tests)

SKILL.md

@@ -45,7 +45,7 @@ design/            platform-agnostic standards: the source of truth for what and
                    validation · access-layer
 implementation/
   teradata/        the concrete binding: DDL templates, queries, grants, platform profile
-  duckdb/          portable file binding and executable Customer360 example
+  duckdb/          reusable Jinja bindings for a portable single-file engine (no native roles)
   postgres/        reusable Jinja bindings with native roles and temporal constraints
 tooling/
   validation/      design_lint.py - checks the standards are well formed

design/core/ADVOCATED_STANDARDS.md

@@ -411,8 +411,8 @@ Three properties follow, and they are the reason this material is expressed as d
 | Version | Change |
 |---|---|
 | 2.1 | Added `DEC-GRAPH-SESSION-NODES` and `DEC-GRAPH-LOAD-CADENCE`; placement, naming and physical design deferred to the organisation profile and binding settings; zone-naive and external-allocator requirements given structured homes. |
-| 2.0 | Restructured from advisory prose into a decision catalogue expressed in the design language. Physical-design guidance moved to the Teradata platform profile; access-layer and object-placement material moved to their patterns; temporal column contracts moved to the temporal-lifecycle-metadata pattern. |
+| 2.0 | Restructured from advisory prose into a decision catalogue expressed in the design language. Physical-design guidance moved to the platform profiles; access-layer and object-placement material moved to their patterns; temporal column contracts moved to the temporal-lifecycle-metadata pattern. |
 | 1.6 | Added the access-layer section, deferring placement and naming to the Object Placement Standard. |
-| 1.5 | Established platform neutrality; Teradata physical design identified as a platform profile. |
+| 1.5 | Established platform neutrality; platform physical design identified as a platform profile. |
 | 1.4 | Surrogate key strategy rewritten to address instability in versioned entities. |
 | 1.0-1.3 | Initial advocated standards and naming alignment. |

design/core/DESIGN_LANGUAGE.md

@@ -6,7 +6,7 @@ status: standard
 version: 2.0
 normative: true
 lint: ignore-file
-lint_reason: meta document - necessarily names SQL keywords and shows platform bindings to define the boundary
+lint_reason: meta document - necessarily names SQL keywords to define the boundary
 ---
 
 # Design Language
@@ -49,7 +49,7 @@ Design documents are written as **interface specifications**: they declare what
 
 Every sentence in a design document must pass one test:
 
-> **Would this sentence change if the target platform changed from Teradata to Postgres to DuckDB?**
+> **Would this sentence change if the target platform changed?**
 >
 > - **No**: it stays true on every platform → it belongs in `design/`.
 > - **Yes**: it changes with the platform → it belongs in `implementation/{platform}/`.
@@ -113,7 +113,7 @@ status: standard          # draft | standard | deprecated
 version: 2.0
 normative: true           # false marks the document advisory, not conformance-binding
 implements: search        # implementation documents only: the design anchor satisfied
-platform: teradata        # implementation and platform-profile documents only
+platform: {platform}      # implementation and platform-profile documents only: a directory under implementation/
 supersedes: []            # anchors this document replaces, when deprecating
 ---
 ```
@@ -380,6 +380,8 @@ A design document **fails** the linter if any of the following appear:
 2. **SQL statements inside any fenced block.** Within any code block, a line beginning with a SQL statement keyword, `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `CREATE`, `ALTER`, `DROP`, `GRANT`, `REVOKE`, is a violation. (The pseudo-notation in the Entity Notation section uses none of these, so it passes.)
 3. **Platform data types or vendor tokens, anywhere.** High-precision tokens that are never ordinary English and only ever appear in SQL: `VARCHAR`, `BIGINT`, `BYTEINT`, `SMALLINT`, `DECIMAL(`, `FLOAT32`, `TIMESTAMP(`, `PRIMARY INDEX`, `UNIQUE PRIMARY INDEX`, `GENERATED ALWAYS AS IDENTITY`, `NOT NULL`, `DEFAULT `, `COMMENT ON`, and any `TD_`-prefixed function. Use the logical types of the Logical Type Vocabulary section instead.
 
+4. **A platform's name, anywhere.** Any platform with a binding under `implementation/`, or one the linter lists as known. A design sentence that names a platform fails the boundary test of The Design / Implementation Boundary by construction. Unlike the three rules above, this one has no escape hatch.
+
 **Escape hatch.** A core/meta document that must legitimately name SQL (this document; the linter's own README) carries an ignore directive on its first line:
 
 ```
@@ -409,29 +411,9 @@ Before a design document is considered conforming:
 
 ---
 
-## Appendix A: Illustrative Platform Bindings
-
-This appendix is illustrative only. Authoritative bindings live in each platform's implementation. It is here to make the boundary concrete; it is why this document carries the `ignore-file` directive.
+## Appendix A: Where the Bindings Live
 
-| Logical type | Teradata | Postgres | DuckDB |
-|---|---|---|---|
-| `Identifier` | `BIGINT` via keymap (stable across versions) | `BIGINT` + identity/keymap | `BIGINT` + sequence |
-| `NaturalKey` | `VARCHAR(n)` | `VARCHAR(n)` / `TEXT` | `VARCHAR` |
-| `Reference -> E` | `BIGINT` | `BIGINT` | `BIGINT` |
-| `ShortText` / `Text` / `LongText` | `VARCHAR(n)` / `CLOB` | `VARCHAR(n)` / `TEXT` | `VARCHAR` |
-| `Json` | `JSON` | `JSONB` | `JSON` / `STRUCT` |
-| `Decimal(p,s)` | `DECIMAL(p,s)` | `NUMERIC(p,s)` | `DECIMAL(p,s)` |
-| `Timestamp` | `TIMESTAMP(6) WITH TIME ZONE` | `TIMESTAMPTZ` | `TIMESTAMPTZ` |
-| `Date` | `DATE` | `DATE` | `DATE` |
-| `Flag` | `BYTEINT` | `BOOLEAN` | `BOOLEAN` |
-| `Vector[dim]` | native `VECTOR` (`FLOAT32(dim)`) | `vector(dim)` (pgvector) | `FLOAT[dim]` (vss) |
-
-| Capability | Teradata | Postgres | DuckDB |
-|---|---|---|---|
-| `NearestNeighbors` | `TD_VectorDistance` | `ORDER BY emb <=> q LIMIT k` (pgvector) | `array_cosine_similarity` / vss |
-| `ApproxIndex` | KMEANS / HNSW via Vector Store | `ivfflat` / `hnsw` index | vss HNSW |
-| `Embed(text, model)` | in-database ONNX embedding | external API | external API |
-| `CurrentStateFilter` | flag predicate | flag predicate | flag predicate |
+How each logical type and capability is realised is the binding's, declared once as data in each `implementation/{platform}/PLATFORM_PROFILE.md`: its `Types:` block gives the physical type of every logical type, its `Functions:` block the SQL for every expression function, and its module and pattern directories the capability bindings. This document deliberately carries no copy of them, which would drift from the bindings it describes.
 
 ---
 

design/core/DESIGN_SPECIFICATION.md

@@ -102,10 +102,11 @@ Every reference target must resolve to an entity declared in the specification o
 
 ### 3.5 Entity sections
 
-`Synonyms:` is a single line of terms. `Volume:` and `Features:` hold indented fields.
+`Description:` and `Synonyms:` are single lines. `Volume:` and `Features:` hold indented fields. Every attribute also carries its business meaning as a trailing `// description`; the build records both as the objects' metadata (`RichMetadata`).
 
 | Section | Required on |
 |---|---|
+| `Description` | every entity |
 | `Volume` | every entity of kind `History`, `Reference` or `Relationship` |
 | `Features` | every feature group: an entity a `Model:` reads |
 | `Synonyms` | none |
@@ -148,15 +149,15 @@ Every specification also requires a `Product` block. A product with the `memory:
 
 #### Standard-owned relations
 
-Relations a build creates for an included module without the specification declaring them, placed and named through the organisation profile. A specification entity of the same name replaces the standard's.
+Relations a build creates for an included module without the specification declaring them, placed and named through the organisation profile and shaped by the entity definitions of the module or pattern that owns them. A specification entity of the same name replaces the standard's. Every relation outside the Memory runtime facet has a consumer view; the views listed are consumer views a binding defines over the relations.
 
 | Owner | Relations |
 |---|---|
 | `domain` | `none`: the specification declares every Domain entity. |
 | `search` | `none`: the specification declares its embedding entities. |
 | `prediction` | `none`: the specification declares its feature groups and prediction entities; `FeatureValue` is created when a feature group declares `storage: tall`. |
-| `semantic` | `all` the module's entities. |
-| `observability` | `all` the module's entities; the validation pattern's `ValidationRun` and `ValidationArea`; and the discovery views `LineageGraph` and `LineageRunLatest`. |
+| `semantic` | `all` the module's entities, and the discovery views `ColumnCatalogue`, `DataProductManifest` and `RelationshipPaths`. |
+| `observability` | `all` the module's entities; the validation pattern's `ValidationRun` and `ValidationArea`; and the discovery views `LineageGraph`, `LineageRunLatest` and `TrustMap`. |
 | `memory` | `by facet`: the entities of each enabled facet. |
 
 ### 4.2 `Embedding:`
@@ -268,7 +269,7 @@ A product design decision, recorded once here and captured into Memory's `Design
 |---|---|---|---|
 | `Title` | yes | text | One line. |
 | `Category` | yes | enum | `ARCHITECTURE`, `SCHEMA`, `NAMING`, `PERFORMANCE`, `SECURITY`, `INTEGRATION` or `OPERATIONAL`, as Memory's `DesignDecision` defines them. |
-| `Module` | yes | module | The module or pattern the decision belongs to. |
+| `Module` | yes | module | The module the decision is recorded against, one of the product's `modules`. A pattern's decision names the module whose behaviour it governs, and a decision governing several modules is one block per module ([Memory](../modules/memory.md) §5.2). |
 | `Applies to` | no | text | The entity or block the decision concerns. |
 | `Settles` | no | text | The catalogued decision this record settles for its module, when it records one. |
 | `Context` | yes | text | What prompted it. |
@@ -359,7 +360,7 @@ Expressions are platform-neutral. Each binding maps every function below to its
 `spec_lint` reports, in addition to its composition, decision and invariant checks:
 
 - frontmatter keys outside section 3.1, a missing or malformed `product_code`, and any `platform` key;
-- a platform named anywhere in the specification: any directory under `implementation/`, or one of Teradata, Snowflake, Databricks, BigQuery, Redshift, Synapse, Oracle, SQL Server, Postgres or DuckDB;
+- a platform named anywhere in the specification: any directory under `implementation/`, or a platform the design linter lists (`tooling/validation/design_lint.py`), as for every document in `design/`;
 - an entity without a valid `profile`, an invalid `allocation`, or a departure from either default with no covering `Decision:`;
 - a reference whose target does not resolve, or a multi-target reference without a discriminator;
 - a missing entity section, a feature group with no derived attribute, or a section field missing, unknown or invalid for its type;