slahes in paramter value lead to error

[doutriaux1]

I am trying to pass path to a file as values to a aparameter that seems to trip maestro completely.

Example yaml;

description:
  name: I_dont_like_slashes
  description: Fails to run if params have slashes

env:
  labels:
      HDF5_DISABLE_VERSION_CHECK: 1

global.parameters:
  SOURCE_FILE:
    values: [ '/some/long/path/that/would/be/here/an/file/name.txt', ]
    label: pth_%%

study:
      - name: fail
        description: Won't be generated
        run:
          cmd: echo $(SOURCE_FILE)
          depends: []

The study is generated by never ran, the log says:

[2021-02-08 14:58:23,005: INFO] Calling execute for StepRecord 'fail_pth_/some/long/path/that/would/be/here/an/file/name.txt'
[2021-02-08 14:58:23,006: INFO] Directory does not exist. Creating directories to /g/g19/cdoutrix/maestro/I_dont_like_slashes_20210208-145820/fail/pth_somelongpaththatwouldbehereanfilename.txt
[2021-02-08 14:58:23,008: INFO] Generating script for fail_pth_/some/long/path/that/would/be/here/an/file/name.txt into /g/g19/cdoutrix/maestro/I_dont_like_slashes_20210208-145820/fail/pth_somelongpaththatwouldbehereanfilename.txt
[2021-02-08 14:58:23,009: ERROR] (2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/WS2/cdoutrix/pipenvs/kosh_spack/lib/python3.8/site-packages/maestrowf/conductor.py", line 381, in main
    completion_status = conductor.monitor_study()
  File "/usr/WS2/cdoutrix/pipenvs/kosh_spack/lib/python3.8/site-packages/maestrowf/conductor.py", line 351, in monitor_study
    completion_status = dag.execute_ready_steps()
  File "/usr/WS2/cdoutrix/pipenvs/kosh_spack/lib/python3.8/site-packages/maestrowf/datastructures/core/executiongraph.py", line 870, in execute_ready_steps
    self._execute_record(_record, adapter)
  File "/usr/WS2/cdoutrix/pipenvs/kosh_spack/lib/python3.8/site-packages/maestrowf/datastructures/core/executiongraph.py", line 548, in _execute_record
    record.generate_script(adapter, self._tmp_dir)
  File "/usr/WS2/cdoutrix/pipenvs/kosh_spack/lib/python3.8/site-packages/maestrowf/datastructures/core/executiongraph.py", line 89, in generate_script
    adapter.write_script(scr_dir, self.step)
  File "/usr/WS2/cdoutrix/pipenvs/kosh_spack/lib/python3.8/site-packages/maestrowf/abstracts/interfaces/scriptadapter.py", line 116, in write_script
    self._write_script(ws_path, step)
  File "/usr/WS2/cdoutrix/pipenvs/kosh_spack/lib/python3.8/site-packages/maestrowf/interfaces/script/localscriptadapter.py", line 84, in _write_script
    with open(script_path, "w") as script:
FileNotFoundError: [Errno 2] No such file or directory: '/g/g19/cdoutrix/maestro/I_dont_like_slashes_20210208-145820/fail/pth_somelongpaththatwouldbehereanfilename.txt/fail_pth_/some/long/path/that/would/be/here/an/file/name.txt.sh'
[2021-02-08 14:58:23,013: INFO] Study exiting, cleaning up...
[2021-02-08 14:58:23,013: INFO] Squeaky clean!

[doutriaux1]

I spoke previously with @FrankD412 his suggestion were:

You have two options here. One is to name each of the paths by using the “name” key. This one can get tricky as I believe there is a uniqueness requirement that must be met. That would look something like the following:

PARAM:
    values: [p1, p2, p3, p4]
    label: PARAM.%%
    name: [a, b, c, d]

The other option is to hash your workspaces. I’ve seen users use this one more regularly when they are wanting reduce the length of their paths or include paths as a parameter. You simply have to add “--hashws" to the “maestro run” sub-command.

The hash workspace option faild with a message stating unicode object must be encoded. Could it possibly be because I'm sending a dictionary as an input value for a paramter?

I'm trying to implement the first option but I'm using a custom generator, I tried to add a 4th argument to the add_parameter function of my ParameterGenerator object but that seems to be ignored.

[FrankD412]

@doutriaux1 -- Are you using the develop version of Maestro? The current develop version fixes a Unicode error that was encountered when using hashws. For the add_parameter, the key may actually be names; I'll have to do some digging.

[jsemler]

@doutriaux1 -- Another way to support this use case in maestro is to organize the directory structure so all files that will be parameterized are located in the same directory. Set the directory as a variable, label, or dependency. Set the filenames as parameters and construct the path in the step.

description:
    name: my_study
    description: Iterate through a list of files.

env:
    dependencies:
        paths:
            - name: DIRECTORY
              path: /path/to/files

global.parameters:
    SOURCE_FILE:
        values: [ 'name1.txt', 'name2.txt']
        label: file_%%

study:
    - name: echo_file_path
      description: Echo the filepath.
      run:
          cmd: echo $(DIRECTORY)/$(SOURCE_FILE)
          depends: []

This should also work with pgen.

[doutriaux1]

@jsemler i thought of this. But unfortunately, that does not work in my case has the path changes with each value.

[FrankD412]

@doutriaux1 Here's another thought I had. Does each file have an absolutely unique path? You can do something like the following if the files are collected in sets. It's a little more hardcoded, but is absolutely explicit on where the file is being pulled from.

Do you mind if I convert this to a discussion? This is useful conversation, but isn't related to code issues/features.

env:
    dependencies:
        paths:
            - name: DIR1
              path: /path/to/files1
           - name: DIR2
              path: /path/to/files2
           ...

global.parameters:
    SOURCE_FILE:
        values: [ 'name1.txt', 'name2.txt']
        label: file_%%
    SOURCE_DIR:
        values: [1, 2, 1, 2]
        label: dir_%%

study:
    - name: echo_file_path
      description: Echo the filepath.
      run:
          cmd: |
              dir=$(SOURCE_DIR)
              if [ "$dir" -eq "1" ]
              then
                  DIRECTORY=$(DIR1)
              if [ "$dir" -eq "2" ]
              then
                  DIRECTORY=$(DIR2)
              ...
              fi
              echo $(DIRECTORY)/$(SOURCE_FILE)
          depends: []