Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #891
Problem
MongoObserver.save_sources()looks up existing source files by filename and md5:but uploads them without an
md5field:PyMongo 3 added
md5to GridFS files automatically. PyMongo 4 no longer does(see the PyMongo 4 migration guide),
so the lookup never matches and every run stores another copy of each unchanged source file.
Fix
Store the digest that
save_sources()already has:This is the only code change. It also works with PyMongo 3, where the value matches the md5 PyMongo computes itself.
Tests
Added to
tests/test_observers/test_mongo_observer.py:test_mongo_observer_save_sources_reuses_unchanged_file: saving an unchanged file twice returns the same GridFS id, stores only one copy, and the stored file has the correctmd5. This test fails without the fix.test_mongo_observer_save_sources_uploads_changed_file: after the file is edited, saving it returns a new id.Locally (Python 3.9, deps from
dev-requirements.txtplussetuptools<81): full suite passes (749 passed), and black and flake8 pass.Out of scope
resource_event()has a similar md5 lookup, but resources are recorded rather than uploaded, so it needs its own analysis. I've left it unchanged and can follow up separately.Note on CI
The test jobs are expected to fail for reasons unrelated to this diff. They fail the same way on
master:pkg_resources.sacred/dependencies.pyimports it, so pytest stops at startup before any test runs. Replace deprecated pkg_resources usage in mongo observer #943 and replace pkg_resource with importlib{_,.}{metadata,resources} #945 address this by moving toimportlib.test_tinydb_observer.py::test_serialisation_of_pandas_dataframefails on Python 3.11 with newer pandas.With
setuptools<81installed, the suite passes locally. If you'd prefer CI to be green on this PR in the meantime, I'm happy to addsetuptools<81todev-requirements.txthere, or to wait until #943/#945 is merged and rebase.