Cog/Docker Issue

[kevinelliott]

Trying to run the example:

cog run python pixray.py --drawer=pixel --prompt=sunrise --output myfile.png

and am getting errors saying that the docker base images do not exist or I do not have access.

=> [stage-0 4/4] WORKDIR /src                                                                                                                                                               0.1s
 => exporting cache                                                                                                                                                                          4.2s 
 => => preparing build cache for export                                                                                                                                                      4.2s 
                                                                                                                                                                                                  
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them                                                                                              
                                                                                                                                                                                                  
Running 'python pixray.py --drawer=pixel --prompt=sunrise --output myfile.png' in Docker with the current directory mounted as a volume...
Unable to find image 'cog-pixray-base:latest' locally
docker: Error response from daemon: pull access denied for cog-pixray-base, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
ⅹ exit status 125

How to resolve?

[bfirsh]

Hey @kevinelliott! It looks like something is failing further up in the log output. Would you mind pasting the full output?

If I recall correctly, this sometimes happens if Docker has run out of disk space. docker system prune might help you out here. (Warning: it'll delete old containers, in case you have any data in there.)

Cog should also show you a better error. We'll fix that...

[dribnet]

Just checking in - this this still an issue for you @kevinelliott ?

[kevinelliott]

Hey! I'm trying again right now and will give you more details. Yes it is still an issue @dribnet thanks for your patience.

[kevinelliott]

@dribnet @bfirsh

Now getting this:

 > [stage-0  7/11] RUN --mount=type=cache,target=/root/.cache/pip pip install -r /tmp/requirements.txt && rm /tmp/requirements.txt:                                                                                                                        
#13 0.669 Looking in links: https://download.pytorch.org/whl/torch_stable.html                                                                                                                                                                             
#13 0.677 Collecting git+https://github.com/bfirsh/taming-transformers.git@7a6e64ee (from -r /tmp/requirements.txt (line 28))                                                                                                                              
#13 0.678   Cloning https://github.com/bfirsh/taming-transformers.git (to revision 7a6e64ee) to /tmp/pip-req-build-yggmbwwx                                                                                                                                
#13 0.679   Running command git clone -q https://github.com/bfirsh/taming-transformers.git /tmp/pip-req-build-yggmbwwx                                                                                                                                     
#13 19.31   WARNING: Did not find branch or tag '7a6e64ee', assuming revision or ref.
#13 19.32   Running command git checkout -q 7a6e64ee
#13 19.95 Collecting git+https://github.com/openai/CLIP@573315e (from -r /tmp/requirements.txt (line 29))
#13 19.95   Cloning https://github.com/openai/CLIP (to revision 573315e) to /tmp/pip-req-build-gn0d2n3x
#13 19.95   Running command git clone -q https://github.com/openai/CLIP /tmp/pip-req-build-gn0d2n3x
#13 21.00   WARNING: Did not find branch or tag '573315e', assuming revision or ref.
#13 21.01   Running command git checkout -q 573315e
#13 21.25 Collecting git+https://github.com/pvigier/perlin-numpy@6f077f8 (from -r /tmp/requirements.txt (line 30))
#13 21.25   Cloning https://github.com/pvigier/perlin-numpy (to revision 6f077f8) to /tmp/pip-req-build-sp57krxc
#13 21.25   Running command git clone -q https://github.com/pvigier/perlin-numpy /tmp/pip-req-build-sp57krxc
#13 23.19   WARNING: Did not find branch or tag '6f077f8', assuming revision or ref.
#13 23.20   Running command git checkout -q 6f077f8
#13 23.44 Collecting git+https://github.com/fbcotter/pytorch_wavelets (from -r /tmp/requirements.txt (line 46))
#13 23.44   Cloning https://github.com/fbcotter/pytorch_wavelets to /tmp/pip-req-build-svg06k15
#13 23.44   Running command git clone -q https://github.com/fbcotter/pytorch_wavelets /tmp/pip-req-build-svg06k15
#13 24.91 Collecting git+https://github.com/pixray/aphantasia@7e6b3bb (from -r /tmp/requirements.txt (line 49))
#13 24.91   Cloning https://github.com/pixray/aphantasia (to revision 7e6b3bb) to /tmp/pip-req-build-ewkp8nee
#13 24.91   Running command git clone -q https://github.com/pixray/aphantasia /tmp/pip-req-build-ewkp8nee
#13 29.03   WARNING: Did not find branch or tag '7e6b3bb', assuming revision or ref.
#13 29.03   Running command git checkout -q 7e6b3bb
#13 30.70 Collecting torch==1.9.0+cu102
#13 30.72   Downloading https://download.pytorch.org/whl/cu102/torch-1.9.0%2Bcu102-cp38-cp38-linux_x86_64.whl (831.4 MB)
#13 60.91 /root/.pyenv/pyenv.d/exec/pip-rehash/pip: line 20:    69 Killed                  "$PYENV_COMMAND_PATH" "$@"
------
executor failed running [/bin/sh -c pip install -r /tmp/requirements.txt && rm /tmp/requirements.txt]: exit code: 137
ⅹ Failed to build Docker image: exit status 1

[kevinelliott]

This was a new system build, so I needed to raise the memory allocated to the new Docker installation and managed to pass that now.

[kevinelliott]

 => exporting to image                                                                                                                                                                                                                                0.1s 
 => => exporting layers                                                                                                                                                                                                                               0.0s 
 => => writing image sha256:3566e44d22a44c29b12c81bdb9ab030282bff434897a5383325514cd72178b4d                                                                                                                                                          0.0s 
 => => naming to docker.io/library/cog-pixray-base                                                                                                                                                                                                    0.0s 
 => exporting cache                                                                                                                                                                                                                                   0.0s 
 => => preparing build cache for export                                                                                                                                                                                                               0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

Running 'python pixray.py --drawer=pixel --prompt=sunrise --output myfile.png' in Docker with the current directory mounted as a volume...
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: driver error: failed to process request: unknown.

[kevinelliott]

I had to disable the GPU and CUDA in cog.yaml because my system has an AMD card and thus doesn't support CUDA.

[kevinelliott]

Almost had success there!

Running 'python pixray.py --drawer=pixel --prompt=sunrise --output myfile.png' in Docker with the current directory mounted as a volume...
Using seed: 16034035987680541322
Running pixeldrawer with 80x45 grid
100%|███████████████████████████████████████| 338M/338M [02:26<00:00, 2.42MiB/s]
100%|███████████████████████████████████████| 335M/335M [02:34<00:00, 2.27MiB/s]
Using device: cpu
Optimising using: Adam
Using text prompts: ['sunrise']
0it [00:00, ?it/s]/usr/local/lib/python3.8/site-packages/torch/nn/functional.py:3609: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  warnings.warn(
ⅹ exit status 137

[okanji]

I had to disable the GPU and CUDA in cog.yaml because my system has an AMD card and thus doesn't support CUDA.

This saved me! Seems so obvious now. Thank you!