Skip to content

Separate Cal-Adapt WRF downloads from CF conversion - #4173

Draft
dlebauer wants to merge 1 commit into
PecanProject:developfrom
dlebauer:codex/caladapt-stages
Draft

dlebauer wants to merge 1 commit into
PecanProject:developfrom
dlebauer:codex/caladapt-stages

Conversation

@dlebauer

Copy link
Copy Markdown
Member

Cal-Adapt WRF ingestion currently lives in draft #3967 as a downloader that also extracts, converts units, and writes CF files, with a source-specific bypass in met.process(). This replacement keeps download and CF conversion separate and uses the normal site-level dispatch.

  • download.CalAdaptWRF() stages native Zarr chunks through caladaptaer::cae_stage_zarr() and writes annual JSON manifests. It does not convert weather values.
  • met2CF.CalAdaptWRF() reads the manifests and delegates extraction, conversion, provenance, and validated output reuse to cae_build_met_drivers(). It publishes annual files with PEcAn prefixes that retain model, scenario, and resolution.
  • Registration metadata supplies defaults and a product-prefix template, identifies the raw JSON format, and disables statistical gap filling. There is no CalAdaptWRF branch or skip-list entry in met.process().
  • General ingestion changes forward climate options for site downloads, distinguish products during database lookup, allow explicit creation of a named raw format, and carry the CF result forward when standardization is disabled.

The source defaults to CESM2 / ssp370 / d03. d01 and d02 remain selectable. Source calendars and missing values are preserved. The JSON manifests reference local cache paths, so keep the cache until conversion finishes. Shared-cache downloads should run serially; this PR does not add cache locking.

Dependency / merge order: keep this PR in draft until the caladaptaer stack #25–30 lands. The adapter requires the runtime APIs in #25–28; it was tested against stack head f5d50dacb71fbe88052f7b30b7b3b39ba29b85cf. Dependencies point to the normal upstream repository, assuming those changes have merged, rather than pinning a temporary stack branch. The functions fail explicitly when the source-staging API is absent. Remote Zarr access requires R-linked GDAL >=3.9.

Validation:

  • 45 assertions passed across the new ingestion tests and nearby dispatcher/registration tests, using the actual proposed CF builder with a local eight-variable fixture and mocked remote staging.
  • Checks cover native 365-day calendars, retained missing values, precipitation totals, humidity conversion, wind speed, reuse and corrupted-output repair, preceding precipitation records, character site IDs, product identity, normal CF dispatch, and handoff to model conversion.
  • Roxygen documentation regenerated; package build and git diff --check passed.
  • R CMD check --no-manual --no-vignettes --no-tests --no-examples completed with no errors, 4 warnings and 3 notes. Reports concern unchanged datasets/compression, unbuilt vignettes, the license stub, existing global bindings, and a local PROJ diagnostic. The focused tests above were run separately; the full package test suite and live downloads were not run.

This PR does not fix SIPNET's assumption of complete Gregorian years or connect the direct-file UCLA reanalysis APIs. The documentation identifies the downstream calendar limitation.

Supersedes #3967; retains credit for Akash B V's original contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant