[NathanG-TD]

Title: Python code generation and TOML configuration (#80)

Summary

Every platform binding now generates its SQL from a Python dialect instead of Jinja templates, and an organisation configures the standard in two TOML files instead of one markdown profile per platform. The point is issue #80: the repository is the skill, a skill upload stops at 200 files, and each platform used to cost 35 to 45 files. A platform now costs five, so ten platforms fit with room to spare. Rendered SQL is unchanged.

Net change in shipped files

• Shipped package (git archive, after export-ignore): 194 -> 89 (-105)
• Tracked files in the repository: 215 -> 110 (-105)
• Files per platform binding: 35 to 38 -> 5
• 94 templates (3,434 lines) become 4 dialect modules (2,419 lines)
• At 10 platforms the package would be about 124 files, against a new CI budget of 180 (the upload limit is 200)

What changed

1. Python dialects replace the Jinja templates

• Each implementation/{platform}/dialect.py subclasses the shared implementation/portable/dialect.py and overrides only what that platform does differently.
• PHASES lists the deployment phases in order; each phase comes from a method phase_ and is written to NN-.sql. CHECKS lists the validation checks; each comes from a method check__. key_allocation() and maintenance() generate the statements a product's own pipelines run.
• render.py asks the dialect for each phase, the deploy script and each check. All SQL lives in the dialects; tooling holds none.
• All templates and the Jinja2 dependency are removed (tooling/bindings/requirements.txt is deleted). All tooling is now standard-library Python 3.11+.
• A binding is now README.md, PLATFORM_PROFILE.md, CONFORMANCE.md, binding.toml and dialect.py.

2. binding.toml declares each binding

• Quoting, emitted roles, settings, gaps, types and expression functions moved out of PLATFORM_PROFILE.md into binding.toml, read with tomllib.
• Phases, checks and the deploy template are no longer declared there; the dialect owns them.
• A platform is now any directory with a binding.toml. implementation/portable/ has none and is not a platform.

3. An organisation configures the standard in two TOML files

• design.toml: the organisation, its classification and retention. It names no platform, so the design role may read it.
• implementation.toml: tables shared by every target, then one [targets.] per deployment target naming its platform. A target can extend another.
• The skill ships config/design.example.toml and config/implementation.example.toml (duckdb, postgres and teradata targets) for the worked example's organisation. The contrasting organisation is in examples/it-service-desk-data-product/contrasting/. Organisations keep their own copies in their own repositories.
• The field tables in ORGANISATION_PROFILE.md remain the schema, so no resolution rule changed.

4. Retired Teradata code

• The graph-lineage exposure and the catalogue-interface binding (with the Alation example) were written for the pre-build-context templates and were never rendered by any phase. They are retired rather than ported; their last versions are at 5fe4803. A build that asks for either still reports it as a gap.

5. Routing fix

• roles/build.md sends every build to PLATFORM_PROFILE.md section 7, SQL Idioms and Driver Constraints. On DuckDB, section 7 was the access boundary and the SQL idioms were section 10. DuckDB's profile is renumbered, the authoring standard now requires section 7 on every binding, and a test enforces it.

Breaking changes for users of the tooling

• render.py and build_context.py now take: <implementation.toml> --target [--design <design.toml>]
• Markdown organisation profiles are no longer read. Convert to the two TOML files; config/*.example.toml shows the shape.
• Jinja2 is no longer needed or used.

New CI checks (so the file count can't creep back)

• Package budget (verify_skill.py): fails above 180 shipped files, warns above 160. A failure lists the largest areas.
• Document size (verify_skill.py): no design or binding document over 600 lines.
• Per-platform limit (test_binding_package.py): at most 6 files in a binding.
• Dialect purity (test_binding_package.py): an AST check that a dialect imports only the dialect it extends plus json and re, makes no file, network or clock calls, and contains no product names.
• No unused code (test_binding_package.py): every dialect method must be reached by a phase, a check or a pipeline statement in the reference builds. A deliberately added unused method was caught.
• The temporal naming rule (TLM-04) now also checks the SQL inside dialect.py strings.

Evidence

• Rendered SQL, deploy scripts and manifests are byte-identical to the templates' output for both reference specifications under all four example targets (8 builds). The only output difference is that context.json no longer lists template paths.
• Every test passes live on DuckDB 2.0.0, PostgreSQL 16.11 and Teradata 20.00.24.60: 126 tests, none skipped. All three CONFORMANCE.md reports are regenerated from that run.
• design_lint and verify_skill --package are clean, and the release bundle is unchanged.
• CI has no Teradata system, so it trusts the committed Teradata report's live run, as merged in Simplify/80 file count #82.

Documentation

• The authoring standard is rewritten for dialects: section 5 (binding structure), 5.1 (declarations without phases, checks or deploy), 6 (template rules become dialect rules), 7, 8 (checks and packaging budget) and 10.
• Also updated: roles/build.md, the design documents' Implementations rows, design/README, DESIGN_LANGUAGE, GLOSSARY, SKILL.md, README, the tooling READMEs, and every binding README section (template files become dialect methods).

Notes for reviewers

• 27d9e45 is a WIP commit that d293a7f completes. Squash-merging gives a clean history.
• Not in this PR: organisation positions on DEC-* decisions in design.toml (deferred). Further optional cuts are available (about 19 files: maintainer-only tools, release/, prompts/, tooling READMEs) but not needed until the package nears 160 files, which would take roughly 17 platforms.

🤖 Generated with Claude Code

[earthshiner]

@NathanG-TD, this is a good piece of work.
Taking a platform from 35–45 files to 5 fixes the skill-upload limit, and the dialect approach is easier to reason about than the Jinja templates were. I like the guardrails: the file budget, the purity test and the “every method must be reached” check should stop the file count creeping back.

I read the tooling (render.py, the dialect loader, binding.py, org_profile.py, build_context.py, build_release.py, verify_skill.py), the new tests and the workflow. I did not go through portable/dialect.py or the three platform dialects line by line, so I’m relying on your byte-identical result for the generated SQL. I also didn’t run the suite. CI is green.

One thing I’d fix before merging: the Teradata evidence

CI can’t reach a Teradata system. The Teradata engine tests skip and the job trusts the live run recorded in the committed CONFORMANCE.md. That was reasonable while the SQL came from templates that rarely changed. This PR rewrites all the Teradata generation, though, and nothing ties that report to the current dialect.py. The test only checks the report’s header and status.

The byte-identical comparison is also a one-off. The old templates are deleted, so I couldn’t find anything that would catch a later edit changing the rendered Teradata SQL while CI stays green.

Could we add a golden-file test for the rendered output, or store a hash of the generated SQL in CONFORMANCE.md and compare it in CI? Either would show the live run still matches the code.

Smaller points, none blocking

reached() in test_binding_package.py uses sys.settrace. If anyone runs the suite under coverage or a debugger, the tracer gets replaced and the test could pass when it shouldn’t. The finally: sys.settrace(None) would also switch off a tracer that was already running. An AST or call-graph approach would be sturdier, but I’m fine with this if you note the limitation.
The purity test is a bit weaker than the standard’s wording. It blocks bare calls like open and relies on the import allowlist for the rest. It wouldn’t catch getattr(builtins, ...). It’s a good guardrail, but “makes no file, network, clock or environment call” reads like a guarantee. I’d soften the wording or tighten the check.
Configuration.design_source() finds the design file by replacing “implementation” with “design” in the filename. If the file is called anything else, the design path ends up equal to the implementation path. The user then gets confusing “does not belong here” errors. A clear message, or requiring --design in that case, would be kinder.
In org_profile._block, a TOML list of tables is turned into Python {'a': 1} text with no error. Nested mappings are already rejected, so I’d reject this case too.
Phase filenames now come from position (f"{i:02d}-{phase}.sql"). Inserting a phase renames every later file. Fine if intended, but worth a line in the authoring standard.
dialect_class and dialect_for ignore a repo argument. build_release.py passes repo= around, but dialects always load from the repo this checkout lives in. That matches the old load_binding behaviour, so only a problem if you ever build a release from another checkout.
portable/dialect.py is 1,303 lines. The 600-line budget only covers Markdown, but this is the file people and agents will read most. Splitting it by module could help later.

Please squash on merge, as you said, to fold in the WIP commit.

Overall I’m happy with the direction. If the Teradata evidence gets tied to the code, I’m comfortable approving.

[NathanG-TD]

Thanks @earthshiner, all addressed in 177fd5a and aa1437f.

Teradata evidence tied to the code

Each CONFORMANCE.md now records a SHA-256 digest of every file the reference builds render.
--trust-committed stands the committed live run in for the skipped engine tests only when its recorded digest matches the current output. Any change to the generated SQL fails CI until the engine tests are run live again.
All three reports are regenerated from fresh live runs:
DuckDB 2.0.0: 50 passed
PostgreSQL 16.9: 50 passed
Teradata 20.00.24.60: 61 passed
None failed or skipped.
The Teradata digest matches the one I calculated locally.

Smaller points

reached(): restores the tracer that was running instead of clearing it. The comment notes that it displaces coverage or debugger tracers while it runs.
Purity test: now also rejects getattr/setattr/delattr/vars and the interpreter’s own names such as builtins, import and subclasses. The authoring standard now calls it a guardrail, not a sandbox.
design_source(): if the implementation file’s name doesn’t contain “implementation”, the build asks for --design instead of reusing the implementation path.
_block: a TOML list holding tables or lists is rejected, like nested mappings.
Phase filenames: the authoring standard notes that inserting a phase renames every later file.
repo argument: build_bindings refuses a repo other than the checkout the dialects load from, and the dialect_class docstring says so.
portable/dialect.py size: agreed it’s worth splitting by module. I’ve left it as a follow-up rather than growing this PR.