Skip to content

Fix worker deserialization of interactive class methods - #1942

Open
harshitdeora wants to merge 1 commit into
joblib:mainfrom
harshitdeora:fix/pydantic-validator-serialization
Open

harshitdeora wants to merge 1 commit into
joblib:mainfrom
harshitdeora:fix/pydantic-validator-serialization

Conversation

@harshitdeora

Copy link
Copy Markdown

Closes #1732.

When a class defined in __main__ keeps a bound class method in its state, Loky's getattr based method reducer can try to find that method while the class is still being reconstructed in a worker. Deserialization fails with AttributeError, surfaced by Parallel as BrokenProcessPool. Pydantic models with @model_validator(mode='before') hit this path.

Keep Cloudpickle's own method reducer when Cloudpickle is the selected Loky pickler. It reconstructs the method from its function and bound instance, so it does not require the method to be discoverable on a partially restored class. Other Loky reducers and the non Cloudpickle pickler path stay as before.

The new subprocess regression uses a small class with a bound method in class state, without requiring Pydantic as a test dependency. On unchanged main, the exact test fails with AttributeError: validate; with this commit it passes. The issue's Pydantic reproducer likewise failed on base and returned [1001001] with the fix on Windows / Python 3.13 / Pydantic 2.14.3.

Validation: 4 related interactive serialization tests passed (1 skipped), plus test_parallel_pickling and test_cloudpickle_wrapper passed. All applicable pre-commit hooks and git diff --check passed. The broader test_parallel.py run was stopped after initial passing cases because of its runtime; the full suite was not completed locally.

@codecov

codecov Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.55%. Comparing base (436cada) to head (624ac9d).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1942      +/-   ##
==========================================
+ Coverage   93.54%   93.55%   +0.01%     
==========================================
  Files          46       46              
  Lines        8655     8660       +5     
==========================================
+ Hits         8096     8102       +6     
+ Misses        559      558       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pydantic models not (un)pickled correctly with _LokyPickler

1 participant