WIP: Non-Interactive Docker Run #5
Conversation
This needs to be properly refactored and probably be exposed to the outside so the user is in control. Maybe default to interactive runs while introducing some flag to turn this off?
…-part-map Add V80 to Alveo part_map
|
Additionally to this change here, we also have a new feature branch feature/poetry_setup which aims to remove the Docker based setup completely or to at least make it optional. Maybe this would also fix your issue here? |
|
Yes, getting rid of the whole docker setup would be ideal in the long term. This is more like a workaround... This also sometimes messes up the logging/console output, that is why it is a draft... |
|
This doesn't affect the Singularity setup, so it is fine for me if you implement a user-controllable DOCKER_INTERACTIVE that defaults to the old behavior (letting people use the Container interactively). |
Tests, notebooks and build flows now default to non-interactive docker run while while the container-only option defaults to the interactive bash prompt just as before.
|
I have just updated the defaults: Tests, notebooks (hosting the server) and build flows now default to non-interactive docker run while the container-only option defaults to the interactive bash prompt just as before. Is this fine for everyone? |
|
Doesn't this break the integrated interactive pdb debugging that launches by default when the builder fails? |
|
Hm, yes, probably, so we should change the default to always be interactive? Or disable the pdb by default? What actually happens when some CI or experiment job (maybe even on some cluster node which you are not even observing with some interactive terminal) drops you into this pdb session? Does it wait and hang-up your job until timeout or does it crash immediately as there is no input attached? |
|
Jobs on the cluster will directly end the pdb automatically and exit the job |
|
Yes, but I would still default to the old behavior as in finn/dev for maximum compatibility. Disabling DOCKER_INTERACTIVE is only relevant for your dvc setup, so you can simply overwrite it with an env var on your end. |
When running some automated finn builds by some experiment manager doing parameter sweeps, the jobs failed with some TTY issues from within docker, which I have tracked down to the container being connected to a non-TTY output device. So I think there should be an option to disable DOCKER_INTERACTIVE, when it is not absolutely needed. This PR is certainly not ready (it just disables DOCKER_INTERACTIVE...). This is more of reminder, to myself that I rely on this being fixed...
This needs to be properly refactored and probably be exposed to the outside so the user is in control. Maybe default to interactive runs while introducing some flag to turn this off?
See Xilinx#967