Skip to content

Simplify/80 file count - #82

Merged
NathanG-TD merged 11 commits into
mainfrom
simplify/80-file-count
Oct 23, 2026
Merged

NathanG-TD merged 11 commits into
mainfrom
simplify/80-file-count

Conversation

@NathanG-TD

Copy link
Copy Markdown
Collaborator

Brings the skill package back under the 200-file upload limit: 250 → 194 files. Addresses #80.

This is the first part of the simplification pass. The design for the whole pass (options, CI checks against file creep, TOML configuration files, phases) is in docs/simplification-design.md on the docs/80-simplification-design branch, which is being reviewed. The two changes here are the ones that are common sense, A and B1. The rest waits for that review.

A. Keep files an agent never reads out of the package

An install is git archive of a commit, which is also what GitHub's "Download ZIP" gives you. A new .gitattributes marks these export-ignore, so they stay in the repository but leave the package:

  • tooling/**/tests/
  • .github/
  • docs/
  • REMAINING_WORK.md
  • .gitignore, .gitattributes

The three .gitkeep files in design/ are deleted, since those directories already hold files. The README's install section now says to install from the ZIP (or git archive HEAD) where a file limit applies, because a clone carries everything.

B1. One README per binding, with a section per module and pattern

Each binding kept a README in every module and pattern directory. That was 34 files, a quarter of implementation/, all with the same layout. They are now numbered sections of the platform's README.md:

  • The section title includes the design anchor, e.g. ## 4. Semantic module (semantic).
  • The sections are in deployment order, and each module or pattern has the same number on every binding: §1 object placement, §2 temporal lifecycle metadata, §3–§8 the modules, §9 validation, §10 access layer, §11 physical storage. Teradata adds §12, the catalogue interface.
  • Each section names the directory its templates are in.
  • Templates are unchanged and stay where they were. A module or pattern with nothing to render keeps its section and loses its directory: DuckDB's access-layer and every binding's physical-storage.
  • The content was moved, not rewritten: headings are one level lower, and relative links are re-pointed to work from the new location.

What changed to match:

  • Authoring standard §5: one numbered section per module and pattern, numbered alike on every binding. A template directory exists only where something renders. A new module or pattern adds a section, not a file.
  • Design documents: design/README.md, DESIGN_LANGUAGE.md and every module and pattern document's "Implementations" line point at the README section plus the template directory.
  • Role files: routes cite sections by number, e.g. implementation/{platform}/README.md §4. verify_skill already checks §N citations on every platform.
  • Release index: build_release.py read binding coverage from directories, which would have dropped DuckDB's access layer and physical storage from bindings.json. It now reads the README sections, and release/v2.0.0/bindings.json is unchanged.
  • Package test: the per-directory README checks are replaced by a test that requires every module and pattern section on every binding, numbered alike and naming a real design anchor. It also checks that no README remains below the platform root.
  • Catalogue: regenerated. The platform catalogues now list only the platform profile, since the per-directory binding documents are gone.

Also fixed: six broken relative links

These were already broken on main:

  • examples/README.md linked per-platform ITSD directories that no longer exist. It now links the shared example with each platform's profile.
  • Two Teradata example profiles linked design/core/ one level short.
  • The Teradata catalogue-interface section and the Alation example linked the amended modules/semantic/03-registry.sql.j2, removed in d183770. They now link its last version in history and say that the registry 2.0 changes are not yet in the build context's DataProductRegistry, which is part of the open port-or-retire decision.

How to test

python -m unittest discover -s tooling/validation/tests
python -m unittest discover -s tooling/bindings/tests
python -m unittest discover -s tooling/release/tests
python tooling/validation/design_lint.py design implementation
python tooling/catalogue/build_catalogue.py --check
python tooling/release/build_release.py --check
python tooling/skill/verify_skill.py --package

Verification

Check Result
Package (verify_skill.py --package) 194 files, conforms. Before: 250
Validation tests 272 run, pass
Bindings tests 118 run, pass, 96 skipped: the DuckDB, Postgres and Teradata engine tests, with no engine installed locally
Release tests 18 run, pass; release bundle current
Design lint, catalogue check Clean, current
Relative links Every link in every tracked .md file resolves, including #section anchors

No template changed (git diff main --stat -- '*.j2' is empty), so rendered SQL is unaffected. CI runs the DuckDB engine tests.

Review points

  • Headroom is small. 194 leaves 6 files. Consolidating templates (B2 in the design) is the next step and takes a binding from about 45 files to about 14. Until then, adding files should be avoided.
  • A clone still carries everything. export-ignore affects only the ZIP and git archive. If Claude Code installs from a clone and has its own limit, that is open question 1 in the design doc.
  • Section numbers are now part of the contract. Role files cite them, and the new package test fails if a section's number differs between bindings.
  • Bigger files. The merged READMEs are about 480 lines for DuckDB and Postgres and about 640 for Teradata. Routes cite sections, so an agent can go straight to the one it needs.

Checklist

  • Commit messages follow discipline (three commits: package filter, README sections, link fixes)
  • No secrets committed
  • Tests pass (engine tests skipped locally, as above)
  • Documentation updated (authoring standard, design README and documents, role files, platform READMEs, install section)
  • No new file types needing .gitignore changes

NathanG-TD and others added 3 commits October 22, 2026 12:57
An install is `git archive` of a commit, which is also GitHub's ZIP, and
an upload stops at 200 files. Mark the tests, CI configuration, docs/,
the work log and the git metadata files export-ignore: they stay in the
repository and leave the package. Delete the three .gitkeep files, whose
directories are not empty. The README says to install from the ZIP where
a file limit applies, since a clone carries everything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#80)

Each binding kept a README per module and pattern directory: 34 files,
a quarter of implementation/, all of one shape. Fold them into the
platform README as numbered sections, headed with the design anchor and
numbered alike on every binding, in deployment order, so the role files
can cite them. Templates stay in their directories; a module or pattern
with nothing to render keeps its section and loses its directory.

The authoring standard, design README and module and pattern documents
point at the sections; the role routes cite them by number; the release
index reads coverage from the sections, so bindings.json is unchanged.
A package test now requires the sections and their common numbering in
place of the per-directory READMEs. No template changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The examples index linked per-platform ITSD directories that no longer
exist; it now links the shared example with each platform's profile.
Two Teradata profiles linked design/ one level short. The Teradata
catalogue-interface section and the Alation example linked the amended
registry template, removed when the binding moved onto the build
context; they now link its last version and say it is not yet ported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NathanG-TD and others added 3 commits October 23, 2026 09:21
Taken from the conformance-reports artifact of PR #82's CI run, where
both engines ran: 49 passed, none failed or skipped on each. DuckDB adds
the README-section package test; PostgreSQL adds it and the engine tests
from #76, recorded for the first time. The Teradata report is not
included: its engine tests skipped in that run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each cached deployment was reset by deleting every table and sending all
of the build's row writes again: 1,837 INSERTs for the IT Service Desk
product, 42.6 s a reset, about 30 resets a run. After a product's first
deployment, its tables are now copied into volatile tables, and a reset
is a DELETE and an INSERT ... SELECT per table, sent as multi-statement
requests: 4.0 s. Volatile tables sit in session spool, outside every
product container and the dictionary, so no check sees them.

They are declared from the context's columns with the binding's own
character set, since CREATE TABLE AS refuses a source with an identity
column. Larger multi-statement batches were measured and did not help.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run locally against the Teradata test system: 60 passed, none failed or
skipped. It clears the two package failures #81 fixed and records, for
the first time on Teradata, the rendering and engine tests of the 2.0.0
metadata contract (#76) and the README-section package test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@earthshiner earthshiner left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a really good initiative.
We must add check to prevent the file count growing larger than the limit again, though.

One other thing, if the readmes become fewer but larger, the agent now has more to read to get the information = more tokens. So we need to add an indexing strategy for the readmes so that the agent can go straight to the content of value without reading the whole readme file.

NathanG-TD and others added 5 commits October 23, 2026 10:49
There is no permanent Teradata system for CI, so its engine tests skip
and the check could never pass. With TERADATA_TEST_DSN unset, the
workflow now passes `--trust-committed teradata` to conformance.py: the
skipped engine tests take their outcomes from the live run the committed
report records, and the log says so, naming that run's engine version.

The committed report is trusted only if it records a clean live run.
Everything else is still regenerated and compared, and an engine test
the report does not carry still fails, so a new one needs a new live run.
DuckDB and PostgreSQL skips still fail. The CI guide says how to commit a
Teradata report from a live run, and the authoring standard's merge rule
allows engine evidence from such a report, stated as such.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The loader sent one request per row: a key allocation MERGE, a SELECT of
the key and an INSERT, about 576 requests per load and five loads a run.
Each entity's rows now go in one batched insert (executemany), and its
keys are allocated through the binding's own template with every MERGE
sent together, then read back in one query. A category whose parent is
not yet loaded waits for the next batch.

On Teradata a load drops from about 85 s to about 8 s, and the full
engine run from 21 to 16 minutes, every test passing, the conformance
report unchanged. The remaining per-row statements are the key
allocation MERGEs: the binding's template allocates one natural value
at a time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent may read only a file's first 100 lines before deciding whether
to read on, and the platform READMEs now run to 470-630 lines. Each
README longer than 100 lines now has an index near the top: one row per
section, with the line it starts on, a link, and what it covers.

build_catalogue.py keeps the rows and line numbers, as it keeps the
catalogue tables; the "Covers" text is written by hand and kept across
regenerations. CI fails a README over 100 lines without an index, an
index ending past line 100, or stale line numbers. In the platform
READMEs the index replaces the hand-written section table, keeping its
descriptions. The authoring standard asks for the index.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate grepped for any "skipped", and a validation test named for
skipped engine tests tripped it on a run where nothing skipped. It now
matches unittest's own markers, "... skipped" per test and "skipped=" in
the summary, and the test is renamed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each binding's key allocation template took one natural value, so a
loader allocating a batch sent one statement per value. The template
now also takes `staged`, a table the caller has written natural values
to, and allocates the whole set in one statement: a MERGE on Teradata,
still matched on the keymap's primary index, and on DuckDB and
PostgreSQL the upsert their single form already uses, as one INSERT ...
SELECT ... ON CONFLICT DO NOTHING. NULLs and duplicates are ignored,
existing identifiers kept, and a SELECT returns each staged value with
its identifier. The single-value form renders as before.

The engine tests' loader stages each keymap's values in a temporary
table and uses the set form; the key allocation test also checks the set
form against the single one. On Teradata, key allocation in a run drops
from 910 statements to 21, and the full run to 14.8 minutes, every test
passing, the conformance report unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@NathanG-TD

Copy link
Copy Markdown
Collaborator Author

We will add a file check/limit test in the next PR. Index changes to large README files (over 100 lines) has been added. Teradata tests have been optimised and now run roughly in 50% of original time.

@NathanG-TD
NathanG-TD merged commit 5fe4803 into main Oct 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants