Skip to content

fix: set resource limits on filers and executors - #164

Merged
uniqueg merged 4 commits into
devfrom
fix/filer-executor-limits
Nov 21, 2026
Merged

uniqueg merged 4 commits into
devfrom
fix/filer-executor-limits

Conversation

@uniqueg

@uniqueg uniqueg commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Closes #160
Closes #163

Summary

Lets TESK run in namespaces that require resource limits, e.g. CERIT-SC (a ResourceQuota on limits.cpu, and a default memory limit of 512 MB); see elixir-cloud-aai/generic-issues#122. Together with tesk-api#209.

  • fix: give executors a memory limit when they set CPUs #160, executors: the taskmaster now fills in the limit of each resource that an executor requests without one (limits_from_requests), not only when there are no limits at all. The TESK API sets the CPU limit of cpu_cores only, so an executor with cpu_cores and ram_gb used to get no memory limit; a LimitRange default below its request made its pod invalid, and the task ran forever. Limits that are set are kept; executors without requests get no limits.
  • feat: set resource limits on filer pods #163, filers: filer pods now request 100m CPU and 128Mi memory and are limited to 1 CPU and 512Mi by default (DEFAULT_RESOURCES). The existing setting FILER_RESOURCES replaces the default, {} leaves it out (the old behaviour), and invalid values log an error and keep the default, as FILER_POD_SECURITY_CONTEXT/FILER_SECURITY_CONTEXT do. The check now also refuses quantities that aren't in Kubernetes' notation, numbers, and requests above their limits, also with clients without pydantic models.
  • End-to-end test: a task in a new namespace tesk-e2e-quota (tests/e2e/manifests/quota/: a ResourceQuota on limits.cpu, a default memory limit of 512Mi, and copies of the taskmaster's service account and role, which a unit test keeps equal to rbac.yaml). The executor requests 100m CPU and 600Mi memory with a CPU limit only, as the TESK API sets them. CI applies the new manifests.
  • Docs: taskmaster.md (what it does, FILER_RESOURCES), filer.md (new section "Resources"), upgrading.md (new "Unreleased" section, including that ram_gb is now a hard memory limit for tasks that also set cpu_cores), development.md (the e2e setup).

The filer's memory: its peak resident memory was about 60 MiB for a 2 GiB HTTP input (measured locally); it streams its transfers, and boto3's S3 transfers buffer roughly 10 × 8 MiB. 512Mi leaves plenty of headroom.

Verification

  • uv run coverage run -m pytest: 901 passed, 8 skipped (e2e, which needs a cluster); coverage 100%.
  • pre-commit run --all-files (ruff, ruff format, mypy, actionlint, gitleaks, file checks): all passed.
  • mkdocs build --strict: passed.
  • The regression test for fix: give executors a memory limit when they set CPUs #160 (test_executor_with_cpu_and_memory_gets_memory_limit) fails on origin/dev and passes with the fix.
  • The new e2e test didn't run locally (no Docker access in the agent sandbox); CI's End-to-end test job ran it on kind and passed.
  • Sourcery's review: fixed the resource-name check and a guard for arithmetic errors of the quantity parser (ff2eccf); answered the rest in the threads (keeping the default instead of failing follows the sibling settings; executors without requests get no invented limits, now documented; deployment.md is in the TESK repo).
  • CI at ff2eccf: all checks passed (CI passed, End-to-end test, Test (Python 3.10-3.13), Lint, Docs, image scans, CodeQL, Codecov patch/project).
  • Code scanning: 22 open alerts, all medium Trivy findings in the base images that the scheduled scan of dev already reports (the same 22); none is critical or high, and none comes from this PR.

Declined review findings

  • Make ram_gb a soft request again for tasks with cpu_cores (fill limits only for resources that a LimitRange or quota constrains): fix: give executors a memory limit when they set CPUs #160 asks for request = limit for memory, as for CPU and as for tasks that set only ram_gb. The upgrade notes now say plainly that tasks that go over ram_gb are OOM-killed.
  • Declare pyyaml in the dev extra for the RBAC copy test: kubernetes, a runtime dependency, requires it.
  • Separate settings per resource instead of the JSON FILER_RESOURCES: it already exists (0.12.0) and matches the other filer settings.

🤖 Generated with Claude Code

uniqueg and others added 3 commits November 21, 2026 17:40
The TESK API sets an executor's CPU limit from cpu_cores, and its
requests from cpu_cores and ram_gb. The taskmaster copied the requests
to the limits only if there were no limits at all, so an executor with
both got no memory limit; a LimitRange default below its request made
its pod invalid, and the task ran forever.

Fill in the limit of each requested resource that has none, and keep
the limits that are set.

Refs #160

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Namespaces with a ResourceQuota on limits.cpu (e.g. CERIT-SC) refuse
containers without a CPU limit, so every filer pod was refused. The
filer now requests 100m CPU and 128Mi memory and is limited to 1 CPU
and 512Mi; FILER_RESOURCES replaces the default, and {} leaves it out.
FILER_RESOURCES also refuses invalid quantities and requests above
their limits, logs an error and keeps the default.

An end-to-end test runs a task in a namespace with such a quota and a
default memory limit below the executor's request, which also covers
the executor limits of the previous commit. Document both changes.

Refs #160, #163

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refuse requests or limits that aren't objects also with clients
without pydantic models, test more quantity notations and the RBAC
copy of the quota namespace, check both filers' limits end to end,
and say in the upgrade notes that ram_gb is now a hard memory limit
for tasks that also set cpu_cores.

Refs #160, #163

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Implementation

Add default filer resources and validate configured quantities and resource names before applying FILER_RESOURCES.

  • src/tesk_core/filer_class.py L10-14, L54-130, L406-411 (+4 small edits)

Fill each executor resource limit missing from its request while preserving limits already supplied.

  • src/tesk_core/taskmaster.py L158-159, L202-222

Configuration

Add a quota-constrained e2e namespace with its quota, default memory limit, and taskmaster RBAC, and apply it in CI.

  • .github/workflows/ci.yml L187-189, L240
  • tests/e2e/manifests/quota/quota.yaml L1-70

Documentation

Explain the e2e quota namespace setup and how its manifests and RBAC relate to the default test namespace.

  • docs/development.md L93-97, L115, L124-129

Document filer resource defaults, overrides, and the memory headroom rationale.

  • docs/filer.md L361-380

Describe per-resource executor limits and update the FILER_RESOURCES setting documentation.

  • docs/taskmaster.md L29-36, L298

Record upgrade implications of filer defaults and the newly enforced executor memory limits.

  • docs/upgrading.md L7-48

Tests

Extend the e2e task harness to target arbitrary namespaces and assert task completion and resource limits in the quota namespace.

  • tests/e2e/cluster.py L116-122 (+12 small edits)
  • tests/e2e/test_taskmaster_e2e.py L14-21, L185-228

Test default filer resources, safe fallback for invalid settings, valid Kubernetes quantities, and quota-manifest RBAC consistency.

  • tests/unit/test_filer_hardening.py L79-95, L195-250, L439-507 (+6 small edits)
  • tests/unit/test_manifests.py L18, L117-142

Other changes

Files not grouped.

  • tests/unit/test_taskmaster_resources.py L1-187

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@codecov

codecov Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (af37fba) to head (ff2eccf).

Additional details and impacted files
@@            Coverage Diff            @@
##               dev      #164   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           10        10           
  Lines         1799      1838   +39     
=========================================
+ Hits          1799      1838   +39     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@uniqueg
uniqueg marked this pull request as ready for review November 21, 2026 18:01
@sourcery-ai

sourcery-ai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Linked issues

Checked at commit ff2eccf

#160 fix: give executors a memory limit when they set CPUs · 2 of 3 met

# Requirement Status Notes
1 Executors with both cpu_cores and ram_gb get matching CPU and memory requests and limits. ✅ Met The taskmaster fills each missing resource limit from its request, and unit tests cover the combined CPU-and-memory case.
2 Keep limits set by the TESK API. ✅ Met Existing limits are preserved; tests cover limits that differ from their requests.
3 Test the four reported cases and keep deployment.md and upgrading.md consistent with the behavior. 🟡 Partly met The unit tests cover the four reported scenarios and upgrading.md describes the change, but this repository has no deployment.md and no deployment guidance was updated.

#163 feat: set resource limits on filer pods · 2 of 3 met

# Requirement Status Notes
1 Filer pods have default resource requests and limits when no setting is provided. ✅ Met The filer uses DEFAULT_RESOURCES unless FILER_RESOURCES overrides them.
2 Configured resources are used; invalid values fail at startup with a clear message. 🟡 Partly met Valid overrides are applied, and invalid values are checked and logged, but the setting falls back to the default instead of failing startup.
3 Document the filer resource settings. ✅ Met The filer and taskmaster documentation describes the defaults, overrides, and validation behavior.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 5 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="src/tesk_core/filer_class.py" line_range="98" />
<code_context>
+                raise ValueError(
+                    f"resources.{kind}.{resource} isn't a quantity: {quantity!r}"
+                )
+            quantities[kind][resource] = parse_quantity(quantity)
+    for resource, request in quantities["requests"].items():
+        limit = quantities["limits"].get(resource)
</code_context>
<issue_to_address>
**Extreme quantities crash taskmaster**

When `FILER_RESOURCES` contains a syntactically matched quantity with an exponent too large for the parser's decimal arithmetic, `parse_quantity` receives the exponent without a magnitude bound. An exponent too large for its decimal arithmetic raises an arithmetic exception that `_object_setting` does not catch, so constructing the filer terminates the taskmaster instead of logging the invalid setting and using the default.

Bound quantity magnitudes before parsing and convert parser arithmetic failures into `ValueError` so `_object_setting` falls back to the default.
</issue_to_address>

### Comment 2
<location path="src/tesk_core/filer_class.py" line_range="94-98" />
<code_context>
+            raise ValueError(f"resources.{kind} isn't an object")
+        quantities[kind] = {}
+        for resource, quantity in values.items():
+            if not isinstance(quantity, str) or not _QUANTITY.fullmatch(quantity):
+                raise ValueError(
+                    f"resources.{kind}.{resource} isn't a quantity: {quantity!r}"
+                )
+            quantities[kind][resource] = parse_quantity(quantity)
+    for resource, request in quantities["requests"].items():
+        limit = quantities["limits"].get(resource)
</code_context>
<issue_to_address>
**Invalid resource names reach Kubernetes**

When FILER_RESOURCES contains an unrecognized resource key with a valid quantity and client-side model validation does not reject it, `_check_resources` validates quantities but not resource names, so `_object_setting` accepts the setting and Kubernetes rejects the filer workload instead of using the default; tasks needing the filer fail.

Validate resource names in `_check_resources` and route invalid settings through the existing default fallback.
</issue_to_address>

### Comment 3
<location path="src/tesk_core/taskmaster.py" line_range="215-220" />
<code_context>
+        resources: The `resources` of the executor's container; changed in
+            place.
+    """
+    requests = resources.get("requests")
+    if requests:
+        limits = dict(resources.get("limits") or {})
+        for name, quantity in requests.items():
+            limits.setdefault(name, quantity)
+        resources["limits"] = limits
+
+
</code_context>
<issue_to_address>
**Unrequested executors violate quota**

When an executor has no CPU request or limit and runs in a namespace requiring `limits.cpu` without a default CPU limit, `limits_from_requests` adds limits only for requested resources, so Kubernetes rejects an executor pod without a CPU request or limit in a namespace whose `ResourceQuota` requires `limits.cpu` and whose `LimitRange` does not default a CPU limit; the executor cannot start.

Ensure requestless executors receive a suitable CPU limit, or explicitly require a namespace `LimitRange` that defaults one when using a quota on `limits.cpu`.
</issue_to_address>

### Comment 4
<location path="src/tesk_core/filer_class.py" line_range="149-150" />
<code_context>
             value = validate(value).model_dump(
                 by_alias=True, exclude_none=True, mode="json"
             )
+        if check is not None:
+            check(value)
     except ValueError as error:
         # `json.JSONDecodeError` and pydantic's `ValidationError` are
</code_context>
<issue_to_address>
**Invalid setting falls back**

The issue asks invalid FILER_RESOURCES values to fail at start with a clear message. The check is added here, but the setting handler catches its ValueError, logs it, and uses the default instead of failing.
</issue_to_address>

### Comment 5
<location path="docs/upgrading.md" line_range="30" />
<code_context>
+
+### Executors get a limit for each resource they request
+
+An executor gets a limit for each resource that it requests without one,
+also when other resources have limits; before, only an executor without
+any limits got its requests as limits. The TESK API sets the CPU limit of
</code_context>
<issue_to_address>
**Deployment docs not updated**

The issue asks both deployment.md and upgrading.md to match the executor behavior. The upgrading guide is updated, but deployment.md is not updated in this diff.
</issue_to_address>

Sourcery assessment

Approval pending. 4 findings to address first.

Blocking findings: src/tesk_core/filer_class.py:98, src/tesk_core/filer_class.py:98, src/tesk_core/taskmaster.py:220, src/tesk_core/filer_class.py:150


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread src/tesk_core/filer_class.py Resolved
Comment thread src/tesk_core/filer_class.py Resolved
Comment thread src/tesk_core/taskmaster.py Resolved
Comment thread src/tesk_core/filer_class.py Resolved
Comment thread docs/upgrading.md Resolved
FILER_RESOURCES now also refuses resources that a container can't
have (e.g. a typo like cpus) and turns arithmetic errors of the
quantity parser into ValueError, so both keep the default instead of
failing every filer pod or stopping the taskmaster. Document that
tasks without cpu_cores need a LimitRange default CPU limit in
namespaces with a quota on limits.cpu.

Refs #163

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New gaps against the linked issues

Comment thread src/tesk_core/filer_class.py Resolved
@uniqueg
uniqueg merged commit d8702ef into dev Nov 21, 2026
59 checks passed
@uniqueg
uniqueg deleted the fix/filer-executor-limits branch November 21, 2026 19:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: set resource limits on filer pods fix: give executors a memory limit when they set CPUs

1 participant