Skip to content

Migrate to pyproject.yaml and use uv in CI/CD - #1396

Merged
guillett merged 95 commits into
mainfrom
xp_uv
Oct 19, 2026
Merged

guillett merged 95 commits into
mainfrom
xp_uv

Conversation

@guillett

@guillett guillett commented Sep 21, 2026 •

Copy link
Copy Markdown
Member

Breaking changes

  • Remove Python 3.10 support
  • Remove extras
    • dev is moved to [dependency-groups]/dev in pyproject.toml
    • tracker as it is not maintained nor used

Technical changes

  • Move from pip to uv for package management
  • Rely on uv --resolution=[highest, lowest-direct] in CI to ensure a large test coverage on multiple numpy versions
  • Remove setup.py and setup.cfg and move logic to pyproject.toml
  • Reduce Makefile tasks, CI and CD complexity
    • Lint is only run with a single version of Python

Questions

  • Should tracker really be removed?
    • How could be fund maintaining this dependancy?

Comment

I already set up https://pypi.org/manage/project/OpenFisca-Core/settings/publishing/

@benoit-cty

Copy link
Copy Markdown
Contributor

Hello, don't know if you have finish, here is some pre-review:

  • uv.lock The lockfile isn't excluded from codespell. Bare codespell in make check-style reports astroid ==> asteroid at lines 13 and 2101 and exits 65 — deterministic lint failure.
  • pyproject.toml skip = "./venv" was dropped from [tool.codespell]. The lint job restores venv/ into the repo root before make check-style, so codespell now scans the whole virtualenv (confirmed: a typo under venv/ is reported on this branch, skipped on master).
  • pyproject.toml setuptools >= 61.0 is too low for the PEP 639 license = "AGPL-3.0-or-later" string plus license-files. With setuptools 70 it's a hard ValueError: invalid pyproject.toml config: 'project.license'. Needs >= 77.0.3.
  • pyproject.toml Stray [dependency-groups] dev = ["ruff>=0.16.7"] shadows the dev extra: uv sync installs ruff only, not the lint/type toolchain, and declares a second conflicting ruff constraint.
  • .github/workflows/_before-pip.yaml Cache keys still hash setup.py, but dependencies now live in pyproject.toml, so dependency changes no longer bust the cache. Same in _lint-pip.yaml and merge.yaml.
  • pyproject.toml New template extra has uncapped versions, against the file's own capping rule (and the templates depend back on openfisca-core[web-api]).

@guillett
guillett marked this pull request as draft September 21, 2026 16:13
@guillett

Copy link
Copy Markdown
Member Author

Sorry, I forgot to set this PR as draft. It is not ready at all and I don't know if I'm tacking this the right way.

@benoit-cty

Copy link
Copy Markdown
Contributor

Ok, you may have a look to openfisca/openfisca-france#2547

@guillett

guillett commented Oct 8, 2026

Copy link
Copy Markdown
Member Author

Oh many thanks and sorry @benjello I did not meant to use you to run agent AI on the failing job but more globally to have your feedback on the documented changes.

@benjello

benjello commented Oct 8, 2026

Copy link
Copy Markdown
Member

No pb

@guillett

guillett commented Oct 8, 2026

Copy link
Copy Markdown
Member Author

@benjello, @benoit-cty and @openfisca/international-maintainers eager to get your feedback and reviews

@benoit-cty

Copy link
Copy Markdown
Contributor

Nice work.
Is there a reason you kept setup.py ?
In OpenFisca-France it has been removed when we switch to UV.

@guillett

guillett commented Oct 8, 2026 •

Copy link
Copy Markdown
Member Author

From memory I kept setup.py because initials attempts were failures. I gave it another try and I managed to move logic to pyproject.toml. Thanks for the heads up!

@guillett
guillett requested a review from a team October 12, 2026 12:41

@benjello benjello left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work to have uv and pyproject

@benoit-cty benoit-cty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not merge, Comming soon...

@benoit-cty

Copy link
Copy Markdown
Contributor

Fixes to PR #1396 (pip → uv migration) after code review

A review of #1396 found 10 issues. Four of them would have stopped the release pipeline (nothing published to PyPI, no tag pushed), and one silently disabled every doctest. This document lists each fix and why it was needed.

Release pipeline

1. Invalid reusable workflow path — .github/workflows/merge.yaml

uses: $/.github/workflows/push.yaml → uses: ./.github/workflows/push.yaml.

Why: GitHub rejects $/… as an invalid workflow reference. With it, merge.yaml would fail on every push to main, so publish-to-pypi and push-tag would never run.

2. has-functional-changes output had a trailing newline — .github/workflows/push.yaml

The job output switched from a literal block (|) to a folded, stripped block (>-).

Why: | keeps the final newline, so the output was "yes\n". merge.yaml tests == 'yes', which was therefore always false, and publishing was always skipped. Parsing the YAML now gives the expression with no trailing newline.

4. Windows post-release check never triggered — windows-test.yaml, merge.yaml

  • windows-test.yaml can now also be triggered by workflow_call (with a version input) and workflow_dispatch. It still runs on push: tags, for tags pushed by a maintainer.
  • merge.yaml has a new test-on-windows job. It runs after push-tag and calls the workflow with the version from push-jobs.
  • The version to install is inputs.version || github.ref_name, so every trigger works.

Why: push-tag pushes the tag with GITHUB_TOKEN. GitHub does not start new workflow runs from events created with that token, so the push: tags trigger would never fire after a release. Calling the workflow directly brings back the check that merge.yaml used to run.

7. Broken Authorization header when dispatching the doc deploy — merge.yaml

  • The token is passed through env: and used as "Authorization: Bearer ${OPENFISCADOC_BOT_ACCESS_TOKEN}" on a single line.
  • curl now uses --fail-with-body.

Why: the backslash-newline inside double quotes left the next line's indentation in the header (Bearer <token>). Without --fail, curl exits 0 on a 401, so a failed dispatch went unnoticed. Passing secrets through env: instead of putting ${{ }} directly in the script is also what GitHub recommends.

6. PyPI trusted publishing — no code change, setup needed

uv publish uses OIDC trusted publishing (environment: pypi, id-token: write) in place of the PYPI_TOKEN_OPENFISCA_BOT secret. This is the right direction, but before merging, a PyPI maintainer of openfisca-core must:

  1. On PyPI → openfisca-core → Publishing, add a GitHub trusted publisher with:
    • owner openfisca
    • repository openfisca-core
    • workflow merge.yaml
    • environment pypi
  2. Optionally, create the pypi environment under GitHub → Settings → Environments and add protection rules. GitHub creates it automatically the first time it is used.

Without step 1, the first merge to main fails at uv publish with an authentication error. Once trusted publishing works, the old PYPI_TOKEN_OPENFISCA_BOT secret can be revoked.

Tests and tooling

3. pytest settings ignored — pyproject.toml

[tool.pytest] → [tool.pytest.ini_options].

Why: pytest reads the [tool.pytest] table only from version 9. With the locked pytest 8.4.2, addopts (including --doctest-modules), python_files, testpaths and doctest_optionflags were all ignored, and every doctest in openfisca_core stopped running without any error. [tool.pytest.ini_options] works on pytest 8 and 9.

Check: pytest --co openfisca_core/commons now collects 19 tests, against 10 before.

pytest 9 allowed — pyproject.toml, openfisca_core/tools/test_runner.py

  • The constraint is now pytest >=8.3.3, <10.0. uv.lock was refreshed and still resolves to 8.4.2.
  • OpenFiscaPlugin.pytest_collect_file now takes the file_path: pathlib.Path argument instead of the old path: py.path.local argument.

Why the code change: pytest 9 removed the path hook argument, which had been deprecated since pytest 7. With it, any openfisca test run under pytest 9 crashed with PluginValidationError. file_path has existed since pytest 7, so the new code works on the whole supported range.

5. mypy target set to Python 3.10 — pyproject.toml

python_version = 3.10 is removed from [tool.mypy], so mypy checks against the interpreter it runs on (≥ 3.11).

Why: the package now requires Python ≥ 3.11 and uses typing.Self and typing.Required. Also, recent numpy stubs use the 3.12 type statement, so mypy stopped right away with a syntax error, even with python_version = "3.11". Without a pinned version, mypy matches the Python that CI and contributors actually use.

Still open (not caused by this PR, and check-types is not run in CI): make check-types now gets past the stubs but reports existing type errors in indexed_enums (10) and experimental (1). mypy 1.20 also crashes internally on openfisca_core/types.py (Cannot serialize PlaceholderType). These need a separate PR.

8. Broken make install — tasks/test_code.mk

install now runs uv sync. The install-test target, which still used pip, is removed.

Why: install depended on install-deps and install-edit, which were deleted with tasks/install.mk. uv sync installs the project in editable mode with the dev group, and that group already pulls in OpenFisca-Core[ci] (the country and extension templates). So install-test is no longer needed.

9. Dependabot did not update uv.lock — .github/dependabot.yml

package-ecosystem: pip → package-ecosystem: uv.

Why: with the pip ecosystem, Dependabot never updates uv.lock. Its PRs would leave the lock out of date, and uv sync --locked in the integration job would fail.

10. Fragile version parsing — .github/is-version-number-acceptable.sh

grep '^version =' pyproject.toml | cut -d '"' -f 2 → uv version --short.

Why: the grep matched any top-level version = "…" line and failed on single quotes. uv version --short reads [project].version properly and is already used in push.yaml and merge.yaml. The script already runs with uv in CI.

Verification

Run locally:

Check Result
make test-code with pytest 8.3.3 (lowest allowed) ✅ pass (core doctests included)
make test-code with pytest 9.1.1 ✅ pass
make test-core openfisca_args="--in-parallel" with pytest 9 ✅ pass
make lint ✅ pass
make install ✅ pass
is-version-number-acceptable.sh ✅ reads 46.0.0, exits 0
All workflow and Dependabot YAML files parse; has-functional-changes has no trailing newline ✅
make check-types ⚠️ still fails on existing type errors (see #5)

The GitHub Actions changes (1, 2, 4, 7) can only be fully checked on GitHub: by a merge to main, or by running windows-test.yaml by hand through workflow_dispatch.

@benoit-cty benoit-cty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've checked that the PyPI trusted publishing was already setup.

@guillett

Copy link
Copy Markdown
Member Author

@benoit-cty from your comment I understand you pushed a commit on the branch with fixes but it is not the case. Am I understanding it wrongly?

Comment thread .github/workflows/merge.yaml Resolved
Comment thread .github/workflows/windows-test.yaml Resolved
@guillett
guillett merged commit fda2f9c into main Oct 19, 2026
13 checks passed
@guillett
guillett deleted the xp_uv branch October 19, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants