@fredericpoitevin @apeck12
Maybe we should provide a simplified interface to create the "driver" files needed to produce the small data hdf5 files. (See some limited documentation here: https://confluence.slac.stanford.edu/display/PSDM/1.+Generation+of+the+small+data+hdf5+files)
A rough example driver can be seen here: /cds/group/psdm/sw/tools/smalldata_tools/smalldata_tools/producers/smd_producer.py . This Python file defines what is saved by smalldata_tools.
One idea could be a script to populate the producer/driver from the btx configuration yaml?
We should also explore how to simplify the overall setup process for the smalldata generation, since if I understand correctly, the workflow right now seems to be:
- The code is cloned/copied for the experiment at hand
- The driver/producer file is edited for saving the appropriate data
- The driver is used by ARP to save this data into hdf5 files every run.
- Different data can be saved on a run-by-run basis, but that logic needs to be coded into the driver file, or a new driver needs to be created, and then submitted through the pipeline again.
This may be error-prone for people who are less familiar with the set of tools.
I could also be wrong on the workflow, since I've just started looking into it. In any case, we probably want to limit the total number of configurations needed to be done. (E.g. don't want to expect people to configure btx, smalldata etc)
@fredericpoitevin @apeck12
Maybe we should provide a simplified interface to create the "driver" files needed to produce the small data hdf5 files. (See some limited documentation here: https://confluence.slac.stanford.edu/display/PSDM/1.+Generation+of+the+small+data+hdf5+files)
A rough example driver can be seen here:
/cds/group/psdm/sw/tools/smalldata_tools/smalldata_tools/producers/smd_producer.py. This Python file defines what is saved by smalldata_tools.One idea could be a script to populate the producer/driver from the btx configuration yaml?
We should also explore how to simplify the overall setup process for the smalldata generation, since if I understand correctly, the workflow right now seems to be:
This may be error-prone for people who are less familiar with the set of tools.
I could also be wrong on the workflow, since I've just started looking into it. In any case, we probably want to limit the total number of configurations needed to be done. (E.g. don't want to expect people to configure btx, smalldata etc)