Skip to content

Store md5 when uploading sources to GridFS (fixes #891) - #949

Open
adriennie wants to merge 2 commits into
IDSIA:masterfrom
adriennie:issue-891
Open

adriennie wants to merge 2 commits into
IDSIA:masterfrom
adriennie:issue-891

Conversation

@adriennie

Copy link
Copy Markdown

Fixes #891

Problem

MongoObserver.save_sources() looks up existing source files by filename and md5:

self.fs.find_one({"filename": abs_path, "md5": md5})

but uploads them without an md5 field:

self.fs.put(f, filename=abs_path)

PyMongo 3 added md5 to GridFS files automatically. PyMongo 4 no longer does
(see the PyMongo 4 migration guide),
so the lookup never matches and every run stores another copy of each unchanged source file.

Fix

Store the digest that save_sources() already has:

_id = self.fs.put(f, filename=abs_path, md5=md5)

This is the only code change. It also works with PyMongo 3, where the value matches the md5 PyMongo computes itself.

Tests

Added to tests/test_observers/test_mongo_observer.py:

  • test_mongo_observer_save_sources_reuses_unchanged_file: saving an unchanged file twice returns the same GridFS id, stores only one copy, and the stored file has the correct md5. This test fails without the fix.
  • test_mongo_observer_save_sources_uploads_changed_file: after the file is edited, saving it returns a new id.

Locally (Python 3.9, deps from dev-requirements.txt plus setuptools<81): full suite passes (749 passed), and black and flake8 pass.

Out of scope

  • resource_event() has a similar md5 lookup, but resources are recorded rather than uploaded, so it needs its own analysis. I've left it unchanged and can follow up separately.
  • Duplicates already stored in a database are not removed.

Note on CI

The test jobs are expected to fail for reasons unrelated to this diff. They fail the same way on master:

  1. CI installs setuptools >= 81, which no longer includes pkg_resources. sacred/dependencies.py imports it, so pytest stops at startup before any test runs. Replace deprecated pkg_resources usage in mongo observer #943 and replace pkg_resource with importlib{_,.}{metadata,resources} #945 address this by moving to importlib.
  2. test_tinydb_observer.py::test_serialisation_of_pandas_dataframe fails on Python 3.11 with newer pandas.

With setuptools<81 installed, the suite passes locally. If you'd prefer CI to be green on this PR in the meantime, I'm happy to add setuptools<81 to dev-requirements.txt here, or to wait until #943/#945 is merged and rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MD5 hash for gridfs is deprecated

1 participant