'docker': 'ewatercycle/wflow-grpc4bmi:2020.1.3', 'apptainer': 'ewatercycle-wflow-grpc4bmi_2020.1.3.sif' fails

[MarkMelotto]

failed: [localhost] (item={'docker': 'ewatercycle/wflow-grpc4bmi:2020.1.3', 'apptainer': 'ewatercycle-wflow-grpc4bmi_2020.1.3.sif'}) => {"ansible_loop_var": "item", "changed": true, "cmd": ["apptainer", "build", "/data/volume_2/samba-share/singularity-images/ewatercycle-wflow-grpc4bmi_2020.1.3.sif", "docker://ewatercycle/wflow-grpc4bmi:2020.1.3"], "delta": "0:00:50.716577", "end": "2025-03-11 14:14:43.176576", "item": {"apptainer": "ewatercycle-wflow-grpc4bmi_2020.1.3.sif", "docker": "ewatercycle/wflow-grpc4bmi:2020.1.3"}, "msg": "non-zero return code", "rc": 255, "start": "2025-03-11 14:13:52.459999", "stderr": "INFO: Starting build...\nGetting image source signatures\nCopying blob sha256:03c144e2d480848d4c5c10950fb2335456941825470a07ef472f98c5e00c7f2f\nCopying blob sha256:b0a33a599b146fb4578cea9bc9cc3918406aa92207c672053a3b6931db682c23\nCopying blob sha256:f8416d8bac72cefc0ce17bd2dc0c03aa43e123d309db92ee23be9382192cf2ed\nCopying blob sha256:8f41eb8f6886160d6245e7dadb68900c4a833343ad3edbd5263b97ad2314ba06\nCopying blob sha256:5c77dd766da0e29e5303f96d3388572d191804357a2ed25592e5bf293df02356\nCopying blob sha256:e184d2a7d23ed478850bb29b7b88fd0b780efbe9f21a84113a6d31b1e87d4d0e\nCopying blob sha256:58b1281d60224980c968439428d0ed1e3965f6b36b64494a3434b7f89b19f409\nFATAL: While performing build: conveyor failed to get: initializing source oci:/root/.apptainer/cache/blob:53b90fc597cbd269be18cb295cc1be6520e2e7bf67e14b3ba9c47e4d88cc5bf1: writing blob: happened during read: unexpected EOF", "stderr_lines": ["INFO: Starting build...", "Getting image source signatures", "Copying blob sha256:03c144e2d480848d4c5c10950fb2335456941825470a07ef472f98c5e00c7f2f", "Copying blob sha256:b0a33a599b146fb4578cea9bc9cc3918406aa92207c672053a3b6931db682c23", "Copying blob sha256:f8416d8bac72cefc0ce17bd2dc0c03aa43e123d309db92ee23be9382192cf2ed", "Copying blob sha256:8f41eb8f6886160d6245e7dadb68900c4a833343ad3edbd5263b97ad2314ba06", "Copying blob sha256:5c77dd766da0e29e5303f96d3388572d191804357a2ed25592e5bf293df02356", "Copying blob sha256:e184d2a7d23ed478850bb29b7b88fd0b780efbe9f21a84113a6d31b1e87d4d0e", "Copying blob sha256:58b1281d60224980c968439428d0ed1e3965f6b36b64494a3434b7f89b19f409", "FATAL: While performing build: conveyor failed to get: initializing source oci:/root/.apptainer/cache/blob:53b90fc597cbd269be18cb295cc1be6520e2e7bf67e14b3ba9c47e4d88cc5bf1: writing blob: happened during read: unexpected EOF"], "stdout": "", "stdout_lines": []}

[MarkMelotto]

line 16 & 17 fail in https://github.com/eWaterCycle/infra/blob/main/roles/prep_shared_data/defaults/main.yml

[BSchilperoort]

Hi Mark, does this error still occur? Did you encounter running this on SRC or locally?

[MarkMelotto]

@BSchilperoort I am not sure, I think so. It could also have been a problem with the samba server not having all the available data

[BSchilperoort]

maybe the Samba server didn't have the singularity images the workflow expected 🤔 So it then tries to build them from the docker images and something went wrong there.

See the task;

infra/roles/prep_shared_data/tasks/main.yml

Lines 4 to 5 in e67696d

	- name: Build apptainer image files (sif) for each model
	ansible.builtin.include_tasks: apptainer-images.yml

and

infra/roles/prep_shared_data/tasks/apptainer-images.yml

Lines 1 to 18 in e67696d

	---
	- name: Apptainer image dir
	file:
	path: '{{ apptainer_image_root }}'
	state: directory
	mode: 0755
	- name: Big temp dir
	file:
	path: /data/volume_2/tmp
	state: directory
	mode: 0755
	- name: grpc4bmi Apptainer images
	command: apptainer build {{ apptainer_image_root }}/{{ item.apptainer }} docker://{{ item.docker }}
	environment:
	TMPDIR: /data/volume_2/tmp
	args:
	creates: '{{ apptainer_image_root }}/{{ item.apptainer }}'
	loop: '{{ grpc4bmi_images }}'

[MarkMelotto]

yep, I figured it would be something like that. But I was not sure if this was a big issue, I dont think so

[sverhoeven]

To create the sif files you need a lot of space in temporary space, the apptainer build command seems to use /data/volume_2/tmp as tmp dir. Could that disk have been full?