Skip to content

fix(csv-generate): consume fixed-size buffered remainders once - #555

Merged
wdavidw merged 1 commit into
adaltas:masterfrom
tanveerchana:fix-generator-buffered-remainders
Nov 3, 2026
Merged

wdavidw merged 1 commit into
adaltas:masterfrom
tanveerchana:fix-generator-buffered-remainders

Conversation

@tanveerchana

@tanveerchana tanveerchana commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

fixed_size can duplicate or drop generated text when a record leaves a remainder larger than the next read size. Even a single record is affected:

import { generate } from "csv-generate/sync";

generate({
  columns: [() => "abc"],
  length: 1,
  fixed_size: true,
  highWaterMark: 1,
});
// Before: "abcbc"
// After:  "abc"

The same options with three records currently produce "abcc\nabc\nabc" instead of "abc\nabc\nabc". This also affects the stream and callback APIs, including custom columns longer than ordinary chunk sizes.

Drain oversized buffered remainders before generating another record, and clear a remainder when transferring it into the current output. This prevents negative split offsets and repeated final suffixes. The patch keeps the existing character-based sizing and encoding behavior. Generated distributions were rebuilt and checked, and are left to the release build process.

Found by code inspection and independently reproduced on master 9fabcbe, csv-generate 4.6.2. No matching issue or PR found in searches for fixed-size generation, fixedSize, fixed_size and fixed_size_buffer. This concerns text-mode remainder consumption; #522 concerns object-mode generation.

Validation on Linux / Node.js v24.22.1:

  • Four added regression tests fail before the fix; all five tests in test/options.fixed_size.js pass afterward. Covers sync, async stream iteration, callback, several chunk sizes and custom delimiter/EOF output.
  • npm ci: passed, including the normal Husky setup.
  • npm test: passed for all 14 workspaces; 1,101 tests passing and five existing tests pending. TypeScript checks pass.
  • npm run build: passed for all seven build projects.
  • npm run lint:check: zero errors, five existing pending-test warnings. npm run lint:staged and commitlint passed. The test file passes Prettier. The existing while(true){ formatting in read.js fails a standalone Prettier check on both the untouched base and this patch; left unchanged.
  • An additional 480 ASCII output checks passed across record lengths, chunk sizes, delimiters and EOF settings. Four rebuilt CommonJS/browser-ESM sync assertions passed in Node. Other Node versions and real-browser execution were not run locally.
  • git diff --cached --check: passed.

Drain oversized remainders before generating more records and clear consumed
buffered text. Add sync, stream and callback regression tests.

AI assistance: implemented and tested with OpenAI Codex.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants