Skip to content

Add branch: key to dynamically generate workflow steps #455

@crkrenn

Description

@crkrenn

Disclaimer/Intro:
Maestro is powerful, but does not support dynamic workflows like optimization. @jsemler has written a private package called encorewf that supports limited iteration. I asked ChatGPT o4-mini for a more general solution, and they (gender-neutral?) suggested the following. My full chat with ChatGPT is here.

@jwhite242, @jsemler, @FrankD412, @daub1, @antimatterhorn, Please let me know what you think!

Problem:
Maestro’s current study spec requires all steps to be defined statically (or via parameter expansion). There is no built-in way to run a script at runtime that emits new step definitions based on completed tasks.

Proposal:
Introduce a new branch: field under run: that executes a user‐supplied command after its depends: finish. That command should print valid YAML step definitions to stdout, which Maestro will parse and inject into the DAG on the fly.

Example:

- name: run-lulesh
  run:
    cmd: |
      $(LULESH)/lulesh2.0 -s $(SIZE) -i $(ITERATIONS) -p > $(outfile)
    depends: [make-lulesh]

- name: branch-postprocess
  run:
    # After all run-lulesh_* instances complete, run this script.
    branch: |
      python scripts/generate_postprocess_steps.py --input-dir $(OUTPUT_PATH)
    depends: [run-lulesh_*]

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions