goldfinch

Climate service processes defined through CWL.

• Setup and Development
• Project Structure
  • Processes Layout
  • Testing Layout
• Usage Commands
• Examples
  • CWL CommandLineTool Generation from Python Click CLI
• License

Setup and Development

For quick setup only with specific CWL dependencies, install the package as follows.

pip install -e .

Install the specific dependencies of the relevant CLI to be generated. Otherwise, the following command can be used to install all dependencies across all locally defined CLI.

Warning

Because the CLI tool attempts to load the other Python file to identify its CWL definition from click decorators, any packages or dependencies this Python script imports has to be installed in the environment where the tool is run.

pip install -e ".[processes]"

Project Structure

A minimal overview of the main directories:

goldfinch/
├── src/goldfinch/processes/
│   ├── chain/
│   │   ├── chain.py
│   │   └── test.py
│   ├── indicator/
│   │   ├── hdd.py
│   │   └── test.py
│   └── subset/
│       ├── poly_subset.py
│       └── test.py
├── tests/
│   └── ... generic/shared unit tests
├── docker/
│   └── Dockerfile
└── ... project configuration and tooling files

Processes Layout

Each process is a standalone Click CLI module within src/goldfinch/processes/<process-name>/. Processes are directly convertible to CWL definitions using click2cwl (see Usage Commands below).

Docker Packaging

The project includes a docker/Dockerfile for containerized deployment. Process CLIs can specify Docker container requirements when generating CWL definitions using the --docker option with click2cwl, ensuring reproducible execution environments.

![NOTE]

All Python dependencies are defined centrally in pyproject.toml under dependencies and processes extras. The docker/Dockerfile installs the full project via make install, making all process dependencies available in the container.

Per-process environment.yml files (if present) are documentation only and are not used by the Docker build or deployment. They serve as reference for developers about which dependencies are relevant to each process.

Testing Layout

The tests/ directory contains generic/shared tests for common functionality.

Each process subdirectory includes a local test.py that demonstrates contextual CLI usage of that process. Those local test.py files are also executed as part of validation (i.e.: make test and GitHub CI/CD), not only documentation and examples. These tests can help users and AI figure out how to employ them.

Usage Commands

All options can be obtained by running the following command:

click2cwl [-h|--help]

The typical command to run the CWL convertion is as follows for the desired Python CLI process:

click2cwl --process <path/to/python.py> [--command <command>] [--output-cwl <path/to/package.cwl>] [--output-format <yaml|json>]

By default, the output CWL is produced as YAML on the standard output with only the CommandLineTool definition. A file can be generated in YAML or JSON format by specifying the --output-cwl option.

If the target Python module defines multiple Click commands, use --command to pick the entrypoint to convert. For example, if the module defines a cli chain group with multiple subcommands (see example chain process), the --command cli command can be used to generate that "main" endtrypoint. By convention, the main entrypoint of a Click CLI is often defined as cli, but this is entirely up to the function name wrapped by the Click decorators. Obviously, the subcommands can also be invoked respectively with the same --command option.

An embedded CommandLineTool within a fully defined CWL Workflow graph can be generated by specifying the -w|--workflow shortcut option or explicitly using the --cwl <clt|cwl> options. This can also be generated as JSON or YAML, and either written to a file or standard output.

Alternatively, the Job Parameters (which would be used to call the CWL process, see below) can be generated as follows. Note that to avoid parameter mangling between the CWL converter and the underlying CLI process arguments, they should be distinguished by an explicit -- separator. This can also use similar output file and format options as above.

click2cwl --process <path/to/python.py> --job [<path/to/job-params.yml>] -- [CLI ARGUMENTS ...]

It should then be possible to test the result as follows if everything was invoked correctly and with all input requirements met:

cwltool <path/to/output.cwl> <path/to/job-params.yml>

Examples

CWL CommandLineTool Generation from Python Click CLI

Using the example CLI for poly_subset.

Generate the CWL CommandLineTool definition in YAML format to standard output. Note that additional metadata can be provided using -m options, which will be mapped to corresponding CWL schema fields if available.

click2cwl \
  -p "src/goldfinch/processes/subset/poly_subset.py" \
  -m "id=poly_subset" \
  -m "author=fmigneault" \
  -m "version=1.0" \
  --cwl-version "v1.2"

` Will produce the following output:

$namespaces:
  s: https://schema.org/
$schemas:
- http://schema.org/version/9.0/schemaorg-current-http.rdf
baseCommand: python /home/chamigfr/dev/goldfinch/src/goldfinch/processes/subset/poly_subset.py
class: CommandLineTool
cwlVersion: v1.2
id: poly_subset
inputs:
  buffer:
    inputBinding:
      position: 5
      prefix: -b
    type: None?
  end:
    inputBinding:
      position: 7
      prefix: -e
    type: string?
  first_level:
    inputBinding:
      position: 8
      prefix: -f
    type: string?
  help:
    inputBinding:
      position: 1
      prefix: -h
    type: boolean?
  input:
    inputBinding:
      position: 2
      prefix: -i
    type: string?
  last_level:
    inputBinding:
      position: 9
      prefix: -l
    type: string?
  output:
    inputBinding:
      position: 3
      prefix: -o
    type: File?
  poly:
    inputBinding:
      position: 4
      prefix: -p
    type: File?
  start:
    inputBinding:
      position: 6
      prefix: -s
    type: string?
  verbose:
    inputBinding:
      position: 10
      prefix: -v
    type: None?
outputs:
  results:
    outputBinding:
      glob: .
    type: Directory
requirements:
  EnvVarRequirement:
    envDef: {}
  ResourceRequirement: {}
s:author:
- class: s:Person
  s:name: fmigneault
s:softwareVersion: '1.0'
stderr: std.err
stdout: std.out

Reusing a Click interface for multiple CommandLineTool CWL definitions

Using the example CLI indicator, which can employ heating_degree_days based on xclim.indicators.atmos.heating_degree_days, amongst multiple indicators based on xclim's registry. The click definition is defined as a generic interface that takes any available indicator parameter. However, we can define a specific CWL CommandLineTool for heating_degree_days by enforcing the indicator value, which is necessary to provide distinct CWL metadata and process definitions for corresponding indicators.

The following demonstrates the generic definition, which will contain all indicator options as the CWL enum symbols.

click2cwl \
    -p "src/goldfinch/processes/indicator/hdd.py" \
    --exclude-help \
    --exclude-version \
    --cwl-version "v1.2" \
    --docker "birdhouse/goldfinch:0.1.0" \
    --env "TEST=VALUE"

Will produce the following output:

$namespaces:
  s: https://schema.org/
$schemas:
- http://schema.org/version/9.0/schemaorg-current-http.rdf
baseCommand: python -m goldfinch.processes.indicator.hdd
class: CommandLineTool
cwlVersion: v1.2
hints:
  DockerRequirement:
    dockerPull: birdhouse/goldfinch:0.1.0
id: clt  # NOTE: default ID; can be customized with -m option
inputs:
  chunks:
    inputBinding:
      position: 7
      prefix: --chunks
    type: string?
  dask_maxmem:
    inputBinding:
      position: 6
      prefix: --dask-maxmem
    type: string?
  dask_nthreads:
    inputBinding:
      position: 5
      prefix: --dask-nthreads
    type: None?
  engine:
    inputBinding:
      position: 8
      prefix: --engine
    type: string?
  indicator:
    inputBinding:
      position: 1
      prefix: --indicator
    type:
    - symbols: [humidex, heat_index, mean_temperature_from_max_and_min, ...]  # note truncated for brevity
      type: enum
  input:
    type:
    - 'null'
    - inputBinding:
        position: 2
        prefix: -i
      items: string
      type: array
  output:
    inputBinding:
      position: 3
      prefix: -o
    type: string?
  verbose:
    inputBinding:
      position: 4
      prefix: -v
    type: None?
outputs:
  results:
    outputBinding:
      glob: .
    type: Directory
requirements:
  EnvVarRequirement:
    envDef:
      TEST: VALUE
  ResourceRequirement: {}
stderr: std.err
stdout: std.out

Instead, the following command can be used to generate a specific CWL CommandLineTool for the heating_degree_days indicator, by passing it explicitly to the underlying CLI definition. To avoid argument ambiguity, the -- separator is used to distinguish between click2cwl options and those of the underlying CLI process.

Note that the -m id=heating_degree_days option is used for consistency in the generated CWL id field.

click2cwl \
    -p "src/goldfinch/processes/indicator/hdd.py" \
    --exclude-help \
    --exclude-version \
    --cwl-version "v1.2" \
    --docker "birdhouse/goldfinch:0.1.0" \
    --env "TEST=VALUE" \
    -m "id=heating_degree_days" \
    -- \
    --indicator "heating_degree_days"