Repository navigation
Simplify/80 file count - #82
Merged
Merged
Conversation
An install is `git archive` of a commit, which is also GitHub's ZIP, and an upload stops at 200 files. Mark the tests, CI configuration, docs/, the work log and the git metadata files export-ignore: they stay in the repository and leave the package. Delete the three .gitkeep files, whose directories are not empty. The README says to install from the ZIP where a file limit applies, since a clone carries everything. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#80) Each binding kept a README per module and pattern directory: 34 files, a quarter of implementation/, all of one shape. Fold them into the platform README as numbered sections, headed with the design anchor and numbered alike on every binding, in deployment order, so the role files can cite them. Templates stay in their directories; a module or pattern with nothing to render keeps its section and loses its directory. The authoring standard, design README and module and pattern documents point at the sections; the role routes cite them by number; the release index reads coverage from the sections, so bindings.json is unchanged. A package test now requires the sections and their common numbering in place of the per-directory READMEs. No template changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The examples index linked per-platform ITSD directories that no longer exist; it now links the shared example with each platform's profile. Two Teradata profiles linked design/ one level short. The Teradata catalogue-interface section and the Alation example linked the amended registry template, removed when the binding moved onto the build context; they now link its last version and say it is not yet ported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Taken from the conformance-reports artifact of PR #82's CI run, where both engines ran: 49 passed, none failed or skipped on each. DuckDB adds the README-section package test; PostgreSQL adds it and the engine tests from #76, recorded for the first time. The Teradata report is not included: its engine tests skipped in that run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each cached deployment was reset by deleting every table and sending all of the build's row writes again: 1,837 INSERTs for the IT Service Desk product, 42.6 s a reset, about 30 resets a run. After a product's first deployment, its tables are now copied into volatile tables, and a reset is a DELETE and an INSERT ... SELECT per table, sent as multi-statement requests: 4.0 s. Volatile tables sit in session spool, outside every product container and the dictionary, so no check sees them. They are declared from the context's columns with the binding's own character set, since CREATE TABLE AS refuses a source with an identity column. Larger multi-statement batches were measured and did not help. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run locally against the Teradata test system: 60 passed, none failed or skipped. It clears the two package failures #81 fixed and records, for the first time on Teradata, the rendering and engine tests of the 2.0.0 metadata contract (#76) and the README-section package test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earthshiner
approved these changes
Oct 22, 2026
earthshiner
left a comment
Collaborator
There was a problem hiding this comment.
This is a really good initiative.
We must add check to prevent the file count growing larger than the limit again, though.
One other thing, if the readmes become fewer but larger, the agent now has more to read to get the information = more tokens. So we need to add an indexing strategy for the readmes so that the agent can go straight to the content of value without reading the whole readme file.
There is no permanent Teradata system for CI, so its engine tests skip and the check could never pass. With TERADATA_TEST_DSN unset, the workflow now passes `--trust-committed teradata` to conformance.py: the skipped engine tests take their outcomes from the live run the committed report records, and the log says so, naming that run's engine version. The committed report is trusted only if it records a clean live run. Everything else is still regenerated and compared, and an engine test the report does not carry still fails, so a new one needs a new live run. DuckDB and PostgreSQL skips still fail. The CI guide says how to commit a Teradata report from a live run, and the authoring standard's merge rule allows engine evidence from such a report, stated as such. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The loader sent one request per row: a key allocation MERGE, a SELECT of the key and an INSERT, about 576 requests per load and five loads a run. Each entity's rows now go in one batched insert (executemany), and its keys are allocated through the binding's own template with every MERGE sent together, then read back in one query. A category whose parent is not yet loaded waits for the next batch. On Teradata a load drops from about 85 s to about 8 s, and the full engine run from 21 to 16 minutes, every test passing, the conformance report unchanged. The remaining per-row statements are the key allocation MERGEs: the binding's template allocates one natural value at a time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent may read only a file's first 100 lines before deciding whether to read on, and the platform READMEs now run to 470-630 lines. Each README longer than 100 lines now has an index near the top: one row per section, with the line it starts on, a link, and what it covers. build_catalogue.py keeps the rows and line numbers, as it keeps the catalogue tables; the "Covers" text is written by hand and kept across regenerations. CI fails a README over 100 lines without an index, an index ending past line 100, or stale line numbers. In the platform READMEs the index replaces the hand-written section table, keeping its descriptions. The authoring standard asks for the index. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The gate grepped for any "skipped", and a validation test named for skipped engine tests tripped it on a run where nothing skipped. It now matches unittest's own markers, "... skipped" per test and "skipped=" in the summary, and the test is renamed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each binding's key allocation template took one natural value, so a loader allocating a batch sent one statement per value. The template now also takes `staged`, a table the caller has written natural values to, and allocates the whole set in one statement: a MERGE on Teradata, still matched on the keymap's primary index, and on DuckDB and PostgreSQL the upsert their single form already uses, as one INSERT ... SELECT ... ON CONFLICT DO NOTHING. NULLs and duplicates are ignored, existing identifiers kept, and a SELECT returns each staged value with its identifier. The single-value form renders as before. The engine tests' loader stages each keymap's values in a temporary table and uses the set form; the key allocation test also checks the set form against the single one. On Teradata, key allocation in a run drops from 910 statements to 21, and the full run to 14.8 minutes, every test passing, the conformance report unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
We will add a file check/limit test in the next PR. Index changes to large README files (over 100 lines) has been added. Teradata tests have been optimised and now run roughly in 50% of original time. |
This was referenced Oct 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the skill package back under the 200-file upload limit: 250 → 194 files. Addresses #80.
This is the first part of the simplification pass. The design for the whole pass (options, CI checks against file creep, TOML configuration files, phases) is in
docs/simplification-design.mdon thedocs/80-simplification-designbranch, which is being reviewed. The two changes here are the ones that are common sense, A and B1. The rest waits for that review.A. Keep files an agent never reads out of the package
An install is
git archiveof a commit, which is also what GitHub's "Download ZIP" gives you. A new.gitattributesmarks theseexport-ignore, so they stay in the repository but leave the package:tooling/**/tests/.github/docs/REMAINING_WORK.md.gitignore,.gitattributesThe three
.gitkeepfiles indesign/are deleted, since those directories already hold files. The README's install section now says to install from the ZIP (orgit archive HEAD) where a file limit applies, because a clone carries everything.B1. One README per binding, with a section per module and pattern
Each binding kept a README in every module and pattern directory. That was 34 files, a quarter of
implementation/, all with the same layout. They are now numbered sections of the platform'sREADME.md:## 4. Semantic module (semantic).access-layerand every binding'sphysical-storage.What changed to match:
design/README.md,DESIGN_LANGUAGE.mdand every module and pattern document's "Implementations" line point at the README section plus the template directory.implementation/{platform}/README.md §4.verify_skillalready checks§Ncitations on every platform.build_release.pyread binding coverage from directories, which would have dropped DuckDB's access layer and physical storage frombindings.json. It now reads the README sections, andrelease/v2.0.0/bindings.jsonis unchanged.Also fixed: six broken relative links
These were already broken on
main:examples/README.mdlinked per-platform ITSD directories that no longer exist. It now links the shared example with each platform's profile.design/core/one level short.modules/semantic/03-registry.sql.j2, removed in d183770. They now link its last version in history and say that the registry 2.0 changes are not yet in the build context'sDataProductRegistry, which is part of the open port-or-retire decision.How to test
Verification
verify_skill.py --package).mdfile resolves, including#sectionanchorsNo template changed (
git diff main --stat -- '*.j2'is empty), so rendered SQL is unaffected. CI runs the DuckDB engine tests.Review points
export-ignoreaffects only the ZIP andgit archive. If Claude Code installs from a clone and has its own limit, that is open question 1 in the design doc.Checklist
.gitignorechanges