Repository navigation
fix: set resource limits on filers and executors - #164
Conversation
The TESK API sets an executor's CPU limit from cpu_cores, and its requests from cpu_cores and ram_gb. The taskmaster copied the requests to the limits only if there were no limits at all, so an executor with both got no memory limit; a LimitRange default below its request made its pod invalid, and the task ran forever. Fill in the limit of each requested resource that has none, and keep the limits that are set. Refs #160 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Namespaces with a ResourceQuota on limits.cpu (e.g. CERIT-SC) refuse
containers without a CPU limit, so every filer pod was refused. The
filer now requests 100m CPU and 128Mi memory and is limited to 1 CPU
and 512Mi; FILER_RESOURCES replaces the default, and {} leaves it out.
FILER_RESOURCES also refuses invalid quantities and requests above
their limits, logs an error and keeps the default.
An end-to-end test runs a task in a namespace with such a quota and a
default memory limit below the executor's request, which also covers
the executor limits of the previous commit. Document both changes.
Refs #160, #163
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refuse requests or limits that aren't objects also with clients without pydantic models, test more quantity notations and the RBAC copy of the quota namespace, check both filers' limits end to end, and say in the upgrade notes that ram_gb is now a hard memory limit for tasks that also set cpu_cores. Refs #160, #163 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer's GuideImplementationAdd default filer resources and validate configured quantities and resource names before applying
Fill each executor resource limit missing from its request while preserving limits already supplied.
ConfigurationAdd a quota-constrained e2e namespace with its quota, default memory limit, and taskmaster RBAC, and apply it in CI.
DocumentationExplain the e2e quota namespace setup and how its manifests and RBAC relate to the default test namespace.
Document filer resource defaults, overrides, and the memory headroom rationale.
Describe per-resource executor limits and update the
Record upgrade implications of filer defaults and the newly enforced executor memory limits.
TestsExtend the e2e task harness to target arbitrary namespaces and assert task completion and resource limits in the quota namespace.
Test default filer resources, safe fallback for invalid settings, valid Kubernetes quantities, and quota-manifest RBAC consistency.
Other changesFiles not grouped.
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #164 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 10 10
Lines 1799 1838 +39
=========================================
+ Hits 1799 1838 +39 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Linked issuesChecked at commit ff2eccf #160 fix: give executors a memory limit when they set CPUs · 2 of 3 met
#163 feat: set resource limits on filer pods · 2 of 3 met
|
There was a problem hiding this comment.
Hey - I've found 5 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="src/tesk_core/filer_class.py" line_range="98" />
<code_context>
+ raise ValueError(
+ f"resources.{kind}.{resource} isn't a quantity: {quantity!r}"
+ )
+ quantities[kind][resource] = parse_quantity(quantity)
+ for resource, request in quantities["requests"].items():
+ limit = quantities["limits"].get(resource)
</code_context>
<issue_to_address>
**Extreme quantities crash taskmaster**
When `FILER_RESOURCES` contains a syntactically matched quantity with an exponent too large for the parser's decimal arithmetic, `parse_quantity` receives the exponent without a magnitude bound. An exponent too large for its decimal arithmetic raises an arithmetic exception that `_object_setting` does not catch, so constructing the filer terminates the taskmaster instead of logging the invalid setting and using the default.
Bound quantity magnitudes before parsing and convert parser arithmetic failures into `ValueError` so `_object_setting` falls back to the default.
</issue_to_address>
### Comment 2
<location path="src/tesk_core/filer_class.py" line_range="94-98" />
<code_context>
+ raise ValueError(f"resources.{kind} isn't an object")
+ quantities[kind] = {}
+ for resource, quantity in values.items():
+ if not isinstance(quantity, str) or not _QUANTITY.fullmatch(quantity):
+ raise ValueError(
+ f"resources.{kind}.{resource} isn't a quantity: {quantity!r}"
+ )
+ quantities[kind][resource] = parse_quantity(quantity)
+ for resource, request in quantities["requests"].items():
+ limit = quantities["limits"].get(resource)
</code_context>
<issue_to_address>
**Invalid resource names reach Kubernetes**
When FILER_RESOURCES contains an unrecognized resource key with a valid quantity and client-side model validation does not reject it, `_check_resources` validates quantities but not resource names, so `_object_setting` accepts the setting and Kubernetes rejects the filer workload instead of using the default; tasks needing the filer fail.
Validate resource names in `_check_resources` and route invalid settings through the existing default fallback.
</issue_to_address>
### Comment 3
<location path="src/tesk_core/taskmaster.py" line_range="215-220" />
<code_context>
+ resources: The `resources` of the executor's container; changed in
+ place.
+ """
+ requests = resources.get("requests")
+ if requests:
+ limits = dict(resources.get("limits") or {})
+ for name, quantity in requests.items():
+ limits.setdefault(name, quantity)
+ resources["limits"] = limits
+
+
</code_context>
<issue_to_address>
**Unrequested executors violate quota**
When an executor has no CPU request or limit and runs in a namespace requiring `limits.cpu` without a default CPU limit, `limits_from_requests` adds limits only for requested resources, so Kubernetes rejects an executor pod without a CPU request or limit in a namespace whose `ResourceQuota` requires `limits.cpu` and whose `LimitRange` does not default a CPU limit; the executor cannot start.
Ensure requestless executors receive a suitable CPU limit, or explicitly require a namespace `LimitRange` that defaults one when using a quota on `limits.cpu`.
</issue_to_address>
### Comment 4
<location path="src/tesk_core/filer_class.py" line_range="149-150" />
<code_context>
value = validate(value).model_dump(
by_alias=True, exclude_none=True, mode="json"
)
+ if check is not None:
+ check(value)
except ValueError as error:
# `json.JSONDecodeError` and pydantic's `ValidationError` are
</code_context>
<issue_to_address>
**Invalid setting falls back**
The issue asks invalid FILER_RESOURCES values to fail at start with a clear message. The check is added here, but the setting handler catches its ValueError, logs it, and uses the default instead of failing.
</issue_to_address>
### Comment 5
<location path="docs/upgrading.md" line_range="30" />
<code_context>
+
+### Executors get a limit for each resource they request
+
+An executor gets a limit for each resource that it requests without one,
+also when other resources have limits; before, only an executor without
+any limits got its requests as limits. The TESK API sets the CPU limit of
</code_context>
<issue_to_address>
**Deployment docs not updated**
The issue asks both deployment.md and upgrading.md to match the executor behavior. The upgrading guide is updated, but deployment.md is not updated in this diff.
</issue_to_address>Sourcery assessment
Approval pending. 4 findings to address first.
Blocking findings: src/tesk_core/filer_class.py:98, src/tesk_core/filer_class.py:98, src/tesk_core/taskmaster.py:220, src/tesk_core/filer_class.py:150
FILER_RESOURCES now also refuses resources that a container can't have (e.g. a typo like cpus) and turns arithmetic errors of the quantity parser into ValueError, so both keep the default instead of failing every filer pod or stopping the taskmaster. Document that tasks without cpu_cores need a LimitRange default CPU limit in namespaces with a quota on limits.cpu. Refs #163 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #160
Closes #163
Summary
Lets TESK run in namespaces that require resource limits, e.g. CERIT-SC (a
ResourceQuotaonlimits.cpu, and a default memory limit of 512 MB); see elixir-cloud-aai/generic-issues#122. Together with tesk-api#209.limits_from_requests), not only when there are no limits at all. The TESK API sets the CPU limit ofcpu_coresonly, so an executor withcpu_coresandram_gbused to get no memory limit; aLimitRangedefault below its request made its pod invalid, and the task ran forever. Limits that are set are kept; executors without requests get no limits.DEFAULT_RESOURCES). The existing settingFILER_RESOURCESreplaces the default,{}leaves it out (the old behaviour), and invalid values log an error and keep the default, asFILER_POD_SECURITY_CONTEXT/FILER_SECURITY_CONTEXTdo. The check now also refuses quantities that aren't in Kubernetes' notation, numbers, and requests above their limits, also with clients without pydantic models.tesk-e2e-quota(tests/e2e/manifests/quota/: aResourceQuotaonlimits.cpu, a default memory limit of 512Mi, and copies of the taskmaster's service account and role, which a unit test keeps equal torbac.yaml). The executor requests 100m CPU and 600Mi memory with a CPU limit only, as the TESK API sets them. CI applies the new manifests.taskmaster.md(what it does,FILER_RESOURCES),filer.md(new section "Resources"),upgrading.md(new "Unreleased" section, including thatram_gbis now a hard memory limit for tasks that also setcpu_cores),development.md(the e2e setup).The filer's memory: its peak resident memory was about 60 MiB for a 2 GiB HTTP input (measured locally); it streams its transfers, and boto3's S3 transfers buffer roughly 10 × 8 MiB. 512Mi leaves plenty of headroom.
Verification
uv run coverage run -m pytest: 901 passed, 8 skipped (e2e, which needs a cluster); coverage 100%.pre-commit run --all-files(ruff, ruff format, mypy, actionlint, gitleaks, file checks): all passed.mkdocs build --strict: passed.test_executor_with_cpu_and_memory_gets_memory_limit) fails onorigin/devand passes with the fix.End-to-end testjob ran it on kind and passed.deployment.mdis in the TESK repo).CI passed,End-to-end test,Test (Python 3.10-3.13),Lint,Docs, image scans, CodeQL, Codecov patch/project).devalready reports (the same 22); none is critical or high, and none comes from this PR.Declined review findings
ram_gba soft request again for tasks withcpu_cores(fill limits only for resources that aLimitRangeor quota constrains): fix: give executors a memory limit when they set CPUs #160 asks for request = limit for memory, as for CPU and as for tasks that set onlyram_gb. The upgrade notes now say plainly that tasks that go overram_gbare OOM-killed.pyyamlin thedevextra for the RBAC copy test:kubernetes, a runtime dependency, requires it.FILER_RESOURCES: it already exists (0.12.0) and matches the other filer settings.🤖 Generated with Claude Code