Proposal for Deliverables-focused Repository Organization

[dlebauer]

Background

Moving toward wrapping up phase 1b, we have two distinct approaches to organizing this work in the workflows repository. One approach uses a directory structure based on project phases (#1, #7) and the other is structured around a workflows that will be iteratively developed over the course of the project (#2).

This is a brief outline of the rationale behind organizing code around final deliverables, and why this has advantages over organizing code around project phases. It also illustrates an example of how this could be done.

This approach aims to:

• Limit the scope of our version controlled software repositories to actively maintained workflows and documentation.
• Enable external collaborators to find and use actively maintained workflows.
• Leverage Git tags and GitHub releases to archive snapshots of deliverables from intermediate phases, allowing us to include reports, data, and other assets alongside the associated version controlled code.

Features of the Proposed Structure

1. Open Science & Collaboration

   • Workflows are accessible and easy to discover, reuse and adapt.
   • Avoids copy-pasting code from one phase to the next - treating each piece of code as an iterative improvement on the previous version. This maintains provenance of the code and its development history.
   • Git and GitHub have features (branches, tags, releases) designed to track stages along the iterative development process. Using directory structure seems redundant.
   • These features are consistent with Wilson et al 2014 Best Practices in Scientific Computing.

2. Clarity & Maintainability

   • Straightforward top-level directories, each with a single purpose (e.g. “monitoring” or “downscaling).
   • Intermediate deliverables at each phase are clearly marked as done when they are tagged and released.
   • The repository does not contain code that is not intended to be further developed, maintained, or tested.

Example Structure for Repositories Focused on Software Deliverables

Below is a multi-repository layout that organizes sub-workflows by purpose rather than phase.

Each repository has a README describing usage, a sequence of scripts, and a report that summarizes workflow outputs.

workflow1/
├── LICENSE
├── README.md
├── _targets.R
├── scripts/
│   ├── 01_step1.R
│   ├── 02_step2.R
│   ├── 0(n)_step(n).R
├── report/report.qmd 
workflow2/
...

Specific example of how CCMMF workflows could be divided:

workflows/       # generic tools that apply to many repositories
ensembles/       # currently the focus of e.g. workflows/2a*/
site_selection/  # currently part of downscaling; but could be migrated to ensembles repo
downscaling/
calibration/
uncertainty/
training/

Summary

The project’s conceptual framework emphasizes agile and iterative development and open science, and is focused on producing a specific set of deliverables including that include software, workflows, and documentation. The project itself has been organized into a timeline with enumerated "tasks" and "phases" for the purposes of planning, these phases weren't intended as a way of structuring the software and other deliverables.

Discussion Points

The primary goal here is to gain consensus on the core idea of organizing our code around the software we are developing. If not, I'd like to better understand what this approach lacks, and how organizing code around the structure of the project plan fits into the approach and goals of the CCMMF project.

How did you envision organizing the code? Do you think that organizing code around deliverables is a good idea?

Related topics that we can discuss here but do not require near-term resolution include:

1. How to identify code that should be in PEcAn and efficiently migrate the code to PEcAn.
2. How to organize GitHub actions and CI tests.
3. (How) Should we divide workflows into separate directories and repositories?
4. How to handle large external datasets that will remain outside the repo; we store only small test data plus scripts to fetch. Right now we are curating data in a ccmmf directory on geo.bu.edu that can be shared via rsync, but this needs to be documented.

[hdpriest-ui]

(some stream-of-consciousness here)

My thoughts around the git repo/workflow organzation are basically to focus the repos and the methods on a single deliverable for each repo. That repo obviously depends on pecan/sipnet, and so the idea would be that for a person on CARB's side to replicate the deliverable, they should have to:

1. clone the appropriate workflow repo
2. run the workflow's demo/test method (uses small data included in the repo). This is to both give the user an example of how to run the thing, but also to test that their environment is correct and able to access resources needed by the container/code
3. run the workflow for real

The workflow repo should also have the methods needed to obtain the real data from whatever source it needs to get it from.

Some advantages of this are:

• There is one repo for each workflow/deliverable. That repo supports the deliverable, and when the deliverable is done, the repo stops moving.
• We won't have workflows running into each other w/r/t software/data science ops. If workflows share a repo, and workflow Define architecture and features required for operational pipeline using PEcAn. #3 has PRs at the same time as workflow Define SIPNET: structure, inputs, outputs, states #4, one workflows commits could be blocked from progressing while another workflow's issues are resolved.
• Workflows in single-workflow-repos can be tested, confirmed to pass, and left in a green state. With multi-workflow repos, in which we're sort of always working on the-next-thing, a repo could have failing tests and appear very in-progress (which it might be), but a particular workflow is in fact perfectly healthy.
• The purpose of a single-workflow repo is clear. A common feeling I have when getting into PEcAn's codebase is "where do I even start?"

Disadvantages:

• Code duplication. If we have single-workflow repos, data logistics code will likely be duplicated. Similarly, certain aspects of interaction with pecan/sipnet might also be re-used. To an extent this could be rolled into PEcAn proper, but it would also be an anti-pattern to include project-specific methodology in the main PEcAn codebase.
• Conflicting designs across workflows: each workflow will begin to represent an interface with PEcAn, an interface which must be supported in an ongoing fashion such that the deliverable to CARB continues to function. This could be solved by creating a single interface that supports all workflows, but that implies additional work.

[dlebauer]

To update - in recent discussions we've agreed to structure the repository around deliverables (e.g. data prep, monitoring, model runs, scenarios, downscaling, etc) rather than project phases. As CI lead, @hdpriest-ui owns the architecture and will guide decisions on how shared tooling is centralized and reused, with others providing feedback as we iterate existing workflows and new workflows come online.

Summaries to specific questions above:

1. How to identify code that should be in PEcAn and efficiently migrate the code to PEcAn.

Henry owns this, will be an ongoing part of PR reviews and discussions

2. How to organize GitHub actions and CI tests.

Henry is actively working on this
To support GHA, tests, and interoperability more generally - will prioritize identifying specific inputs/outputs from each stage of the workflow

3. (How) Should we divide workflows into separate directories and repositories?

Yes

4. How to handle large external datasets that will remain outside the repo; we store only small test data plus scripts to fetch. Right now we are curating data in a ccmmf directory on geo.bu.edu that can be shared via rsync, but this needs to be documented.

Will use AWS S3 buckets + rclone

If there are any lingering details to discuss, please re-open or create a new thread.

[hdpriest-ui]

minor note that we are not storing it in AWS, but a NCSA-hosted location supported by the S3 protocol

[dlebauer]

Updated idea, each in its own repository

github.com:ccmmf/
  ├─ workflows/              # General tools for managing workflows in other repositories, i.e. targets infrastructure & docs        
  ├─ data_preparation/       # Tools for external data (weather, soil, remote sensing).
  ├─ site_selection/         # Or keep as distinct workflow w/ downscaling
  ├─ monitoring/             # Process remote-sensing data into agronomic events
  │    ├─ event1/            # e.g. Tillage detection code
  │    ├─ eventn/            # ...
  ├─ scenarios/              # Tools for creating future climate & management scenarios
  ├─ model_runs/             # ensembles             
  ├─ model_calibration/      # 
  ├─ model_analysis/         # sensitivity, validation, uncertainty analysis; or these could be separate folders
  ├─ downscaling/            # Downscaling + aggregation of model outputs to county or state
  └─ documentation/          # general docs? Most repo-specific docs should go in their repos alongside code. Perhaps this could be the 'training' aspect

[hdpriest-ui]

for my own clarity on this - in this design is 'workflows' a git repo, or a directory within a git repo?

[dlebauer]

Workflows and others at that level are repositories

[dlebauer]

I've put together a very rough architecture diagram that focuses on 1) organizing workflow components into separate repositories and 2) key data flows based on this thread and internal discussions. It shows repositories organized around discrete workflow components (with an overarching workflows orchestrator).

It is a draft, intended as a working model for the team to facilitate discussion and and make the workflow and repository intentions more explicit.

Feedback appreciated - please discuss, ask ?s, suggest changes here / Slack / Google Drive.

• Google Drive: CCMMF Software & Data Architecture (Repository-Centric View)
• PDF CCMMF Software & Data Architecture (Repository-Centric View).pdf

[mdietze]

What I've long found unclear in all of this is what lives in ccmmf and what lives in pecan, and I worry there's more being put in ccmmf than I'd prefer.

[dlebauer]

I agree with this concern. There has been a history of project-specific code not being integrated into PEcAn, and it’s important to recognize that risk.

We’ve discussed this boundary before: putting generic, reusable functionality in PEcAn and project-specific workflow and integration code that in CCMMF. It’s worth revisiting and clarifying that line as the workflows and code mature. Part of @hdpriest-ui's refactoring of the targets workflow has been to have targets directly call PEcAn functions. I think this helps to define the line between integrated and composable workflows (in ccmmf) and modular functionality (in PEcAn).

One practical complication is that the bar for merging code into PEcAn is (appropriately) higher than for iterating on workflow code. Relying too early on changes to PEcAn when we are still stabilizing production workflows is its own risk. My intent has been to use the CCMMF repositories as a staging and refactoring space to develop and exercise functionality in real workflows, iteratively pulling out functions with tests and documentation so that they are ready to migrate into PEcAn as they mature and stabilize.

The goal is not to duplicate PEcAn, but to have CCMMF workflows increasingly call into* PEcAn as functionality hardens, while avoiding premature coupling that could slow delivery of operational code.

[hdpriest-ui]

Echoing some of David's thoughts here; i would envision/hope that as workflows stabilize and particular steps become re-used, they would eventually be subsumed into PEcAn proper where appropriate.

Certainly, we have to avoid the anti-pattern of putting methods into the workflow layer that are minor alterations of existing pecan methods for the sake of convenience.

The workflow layer should remain a helpful tool to standardize common paths through PEcAn libraries, but, in order to remain transparent and reproducible, it must avoid introducing novel scientific/statistical methods not present in PEcAn. The best way to leverage pre-production methods within a workflow would be to have that workflow depend on a feature branch of the PEcAn stack.

[dlebauer]

The best way to leverage pre-production methods within a workflow would be to have that workflow depend on a feature branch of the PEcAn stack.

I agree that this sounds like a good idea in context. Perhaps there is a straightforward way to do this, but it can be complicated and risk resulting in diverging and conflicting code.

Perhaps it is mainly a matter of communication with the community about the state of specific feature branches, and agreement on how to manage the conflict risk.

The approach I've used is to either install a library (install_github('org/repo@branch')) or source individual files (source('raw.github..../org/repo/branch/path/file.R')) from the feature branch, but especially for new functionality, I found it easier to do the initial prototyping in ccmmf repositories, with the intent of later migration of the feature.

I think such an approach would be most helpful when modifying existing code than when developing new code.