Exec2 tasks are started with a cwd in a veiled directory

[lbt]

The exec2 task is starting inside the 'private' 'alloc/' dir instead of the exec2 'alloc_mounts/' dir.

As root on nomad test machine with nomad and nomad-driver-exec2 installed via apt. The
exec2 driver is in /opt/nomad/data/plugins via symlink and /usr/bin/ as installed.

data_dir is /opt/nomad-test/data
opt, nomad-test are root:root 755
data is root:root 700
as per: https://developer.hashicorp.com/nomad/docs/deploy/production/requirements#hardening-nomadnomad.hcl:

There must be a config mentioning the plugin so
nomad.hcl:

plugin "nomad-driver-exec2" {}

Start the nomad agent:

nomad agent -dev-connect -plugin-dir=/opt/nomad/data/plugins/ -config=nomad.hcl -data-dir=/opt/nomad-test/data

The job is being used is
exec2test.nomad:

exec2test.nomad:
job "e2job" {
  reschedule { # keep the logs cleaner
    attempts  = 0
    unlimited = false
  }
  type = "service"
  group "e2g" {
    count = 1
    task "e2t" {
      restart { # keep the logs cleaner
        attempts = 0
      }
      driver = "exec2"
      config {
        command = "bash"
        args    = ["local/run.sh"]
      }
      template {
        destination = "local/run.sh"
	perms = "0755"
        data = <<EOD
#!/bin/bash
set -x
# Keep all output interleaved :)
exec 2>&1
whoami
env | grep NOMAD_TASK_DIR
env | grep NOMAD_ALLOC_DIR

# This is a veiled directory that we should not (and can't) cd to.
pwd

cd local
pwd

cd ($pwd)

# Traverse a veiled directory (which we don't even have linux permission to enter when traversing from root in a hardened system which is odd) and arrive at an unveiled directory:
cd ../secrets
pwd

# Hang around
sleep 1d
EOD
      }
    }
  }
}

Run the job (purge any existing runs and ensure hardened):

nomad job stop -purge e2job
chmod 700 /opt/nomad-test/data/
nomad job run exec2test.nomad

This succeeds and the nomad logs contain:

[INFO]  client.driver_mgr.nomad-driver-exec2: exec2 runner: driver=exec2 oom_score_adj=0 unveil_defaults=true unveil_paths=[rwxc:/opt/nomad-test/alloc_mounts/0a53001c-1d6b-58cf-7556-7d237410a1cd-e2t/local, rwxc:/opt/nomad-test/alloc_mounts/0a53001c-1d6b-58cf-7556-7d237410a1cd-e2t/alloc, rwxc:/opt/nomad-test/alloc_mounts/0a53001c-1d6b-58cf-7556-7d237410a1cd-e2t/secrets, rwxc:/opt/nomad-test/alloc_mounts/0a53001c-1d6b-58cf-7556-7d237410a1cd-e2t/tmp] @module=exec2 args=[local/run.sh] cmd=bash timestamp=2026-03-19T12:46:46.338Z

and the job logs have:

+ whoami
whoami: cannot find name for user ID 80785
+ env
+ grep NOMAD_TASK_DIR
NOMAD_TASK_DIR=/opt/nomad-test/alloc_mounts/0a53001c-1d6b-58cf-7556-7d237410a1cd-e2t/local
+ env
+ grep NOMAD_ALLOC_DIR
NOMAD_ALLOC_DIR=/opt/nomad-test/alloc_mounts/0a53001c-1d6b-58cf-7556-7d237410a1cd-e2t/alloc
+ pwd
/opt/nomad-test/data/alloc/0a53001c-1d6b-58cf-7556-7d237410a1cd/e2t
+ cd local
+ pwd
/opt/nomad-test/data/alloc/0a53001c-1d6b-58cf-7556-7d237410a1cd/e2t/local
++ pwd
+ cd /opt/nomad-test/data/alloc/0a53001c-1d6b-58cf-7556-7d237410a1cd/e2t/local
local/run.sh: line 15: cd: /opt/nomad-test/data/alloc/0a53001c-1d6b-58cf-7556-7d237410a1cd/e2t/local: Permission denied
+ cd ../secrets
+ pwd
/opt/nomad-test/data/alloc/0a53001c-1d6b-58cf-7556-7d237410a1cd/e2t/secrets
+ sleep 1d

I recognise these are bind mounts to inodes that are unveiled and as such there is no security failing.

However having the working directory be a veiled directory feels wrong.

This file: https://github.com/hashicorp/nomad-driver-exec2/blob/main/pkg/shim/shim.go#L357

Says:
cmd.Dir = e.env.TaskDir
which is not the same as:
cmd.Dir = filepath.Dir(e.env.Env["NOMAD_TASK_DIR"])

Which is used at https://github.com/hashicorp/nomad-driver-exec2/blob/main/pkg/shim/shim.go#L265 to find the directory to make the relative tmp/ dir

https://developer.hashicorp.com/nomad/docs/reference/runtime-environment-settings#task-directories
Says:
"The allocation working directory contains a task working directory for each task".
Now I understood this to be the linux "current working directory" (and for the other drivers this is the case).
Then it says: "relative to the task working directory" ... "alloc/" "local/" "secrets/"
Then it says that the path of "alloc/" is in NOMAD_ALLOC_DIR which finally pins us down to the exec2 task's value which we see in the output above.
Working back from NOMAD_ALLOC_DIR suggests the "task working directory" should be filepath.Dir(e.env.Env["NOMAD_TASK_DIR"]) and that it would be both consistent with the docs and less surprising.

I do note that:
hashicorp/nomad#8919 says
"The NOMAD_TASK_DIR is the local/ directory provided to each task, whereas the "task directory" is the directory that Nomad uses to set up the task, and may not be the actual filesystem of the task, "

If I patch the driver the output is:

+ whoami
whoami: cannot find name for user ID 89368
+ env
+ grep NOMAD_TASK_DIR
NOMAD_TASK_DIR=/opt/nomad-test/alloc_mounts/05762172-b640-dd44-0dbb-5d7e8678e6ed-e2t/local
+ env
+ grep NOMAD_ALLOC_DIR
NOMAD_ALLOC_DIR=/opt/nomad-test/alloc_mounts/05762172-b640-dd44-0dbb-5d7e8678e6ed-e2t/alloc
+ pwd
/opt/nomad-test/alloc_mounts/05762172-b640-dd44-0dbb-5d7e8678e6ed-e2t
+ cd local
+ pwd
/opt/nomad-test/alloc_mounts/05762172-b640-dd44-0dbb-5d7e8678e6ed-e2t/local
++ pwd
+ cd /opt/nomad-test/alloc_mounts/05762172-b640-dd44-0dbb-5d7e8678e6ed-e2t/local
+ cd ../secrets
+ pwd
/opt/nomad-test/alloc_mounts/05762172-b640-dd44-0dbb-5d7e8678e6ed-e2t/secrets
+ sleep 1d

and cd $(pwd) is no longer an error :)

I'd be happy to submit the tiny PR if this is deemed correct.

[tgross]

"The allocation working directory contains a task working directory for each task".

Now I understood this to be the linux "current working directory" (and for the other drivers this is the case).
...
Working back from NOMAD_ALLOC_DIR suggests the "task working directory" should be filepath.Dir(e.env.Env["NOMAD_TASK_DIR"]) and that it would be both consistent with the docs and less surprising.

The allocation/task "working directory" in those docs is referring only to the Nomad client assembling the workload's filesystem. As an easy counterexample, a docker task has a CWD that's going to be set by the Dockerfile or by the task config. The exec2 driver is the "weirdo" here. 😁

Being consistent with other task drivers is a good goal though. If I run a raw_exec job and exec into it to find the process working directory, I see:

$ nomad alloc exec 3904cbaf /bin/sh
sh-5.3# pwd
/var/nomad/dev/data/alloc/3904cbaf-4aaf-26b6-04a6-95e3bcdd9594/http

Which would be the equivalent of the NOMAD_TASK_DIR for sure. So yeah, let's go ahead and change this driver to do the same. If you'd like to open a PR I'd be happy to review it. The changelog will want to mention this because it breaks backwards compatibility for existing jobs.

(Internal ref https://hashicorp.atlassian.net/browse/NMD-1317)