gcr.io/cos-cloud/cos-gpu-installer:latest fails to install GPU drivers on compute instances

[DanClarkeBuilds]

Using gcr.io/cos-cloud/cos-gpu-installer:latest to install Nvidia GPU drivers on compute instances on start up fails to install drivers, as a result Tensorflow operations only recognize CPU devices. we are forced to downgrade to a previous stable image version as a workaround as GPU support is required, but ideally latest should be used.

Looking at the gcr.io container registry for this image gcr.io/cos-cloud/cos-gpu-installer:latest there has been a new release of a latest and a v2.0.0 version release as of 22/07/2020, neither of these images install graphics drivers. downgrading to v20200701 is installing drivers as expected.

Our compute instances are using 2x Tesla P100-PCIE-16GB FYI

Using latest

Results in the below log on instance start up:

Tensor flow only is able recognise CPUs using latest:

Using v20200701

We are able to see GPU drivers installed and GPUs are visible using Nvidia-smi

Tensorflow is then able recognise GPUs using v20200701: