Disclaimer/Intro:
Maestro is powerful, but does not support dynamic workflows like optimization. @jsemler has written a private package called encorewf that supports limited iteration. I asked ChatGPT o4-mini for a more general solution, and they (gender-neutral?) suggested the following. My full chat with ChatGPT is here.
@jwhite242, @jsemler, @FrankD412, @daub1, @antimatterhorn, Please let me know what you think!
Problem:
Maestro’s current study spec requires all steps to be defined statically (or via parameter expansion). There is no built-in way to run a script at runtime that emits new step definitions based on completed tasks.
Proposal:
Introduce a new branch: field under run: that executes a user‐supplied command after its depends: finish. That command should print valid YAML step definitions to stdout, which Maestro will parse and inject into the DAG on the fly.
Example:
- name: run-lulesh
run:
cmd: |
$(LULESH)/lulesh2.0 -s $(SIZE) -i $(ITERATIONS) -p > $(outfile)
depends: [make-lulesh]
- name: branch-postprocess
run:
# After all run-lulesh_* instances complete, run this script.
branch: |
python scripts/generate_postprocess_steps.py --input-dir $(OUTPUT_PATH)
depends: [run-lulesh_*]
Disclaimer/Intro:
Maestro is powerful, but does not support dynamic workflows like optimization. @jsemler has written a private package called
encorewfthat supports limited iteration. I asked ChatGPT o4-mini for a more general solution, and they (gender-neutral?) suggested the following. My full chat with ChatGPT is here.@jwhite242, @jsemler, @FrankD412, @daub1, @antimatterhorn, Please let me know what you think!
Problem:
Maestro’s current study spec requires all steps to be defined statically (or via parameter expansion). There is no built-in way to run a script at runtime that emits new step definitions based on completed tasks.
Proposal:
Introduce a new
branch:field underrun:that executes a user‐supplied command after itsdepends:finish. That command should print valid YAML step definitions to stdout, which Maestro will parse and inject into the DAG on the fly.Example: