Sidecar should be able to access files without permission

[matusdrobuliak66]

Current Behavior

Sometimes, it happens that a user uploads a file that the sidecar doesn't have permission to access. As a result, the closing process hangs because the upload to S3 fails. This also means that the user can no longer open the service.

Example of issue:

log_source=simcore_sdk.node_ports_common.filemanager:_upload_path(426) | log_uid=None | log_oec=None| log_msg=The upload failed with an unexpected error:
Traceback (most recent call last):
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_sdk/node_ports_common/filemanager.py"", line 410, in _upload_path
    e_tag, upload_links = await _upload_to_s3(
                          ^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_sdk/node_ports_common/filemanager.py"", line 468, in _upload_to_s3
    await r_clone.sync_local_to_s3(
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_sdk/node_ports_common/r_clone.py"", line 264, in sync_local_to_s3
    await _sync_sources(
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_sdk/node_ports_common/r_clone.py"", line 188, in _sync_sources
    folder_size = await _get_folder_size(
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_sdk/node_ports_common/r_clone.py"", line 164, in _get_folder_size
    result = await _async_r_clone_command(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ""/home/scu/.venv/lib/python3.11/site-packages/simcore_sdk/node_ports_common/r_clone.py"", line 102, in _async_r_clone_command
    raise RCloneFailedError(
simcore_sdk.node_ports_common.r_clone.RCloneFailedError: Command rclone --config /tmp/tmprezhzs0_ size /dy-volumes/home/smu/work/workspace --json --links finished with exit code=6:
2025/02/03 10:21:14 ERROR : .Trash-8005: failed to open directory "".Trash-8005"": open /dy-volumes/home/smu/work/workspace/.Trash-8005: permission denied
{""count"":37,""bytes"":16934799989,""sizeless"":0}

Potential solutions:

1. Ask the user whether it is okay to "delete" these files. If they say YES, these files can be ignored during upload to S3 via RClone.
2. The sidecar can have more privileges to access files that it previously couldn't. Therefore, the sidecar would be able to change the permissions so it can upload them via RClone.

[GitHK]

The issue is caused by related the version of Jupyter-Lab (they enabled trashes as some point)

When a file is deleted from the fronted of a JupyterLab version 4.x.x a trash folder is created which has no group and other read permission.

The only service that is currently using 4.x.x is s4l and @mguidon will look into removing this. The older version that are impacted should also be deprecated across all deployments.

[sanderegg]

here some proposals, to be discussed as well

sidecar actively changes the permission in the user service

1. sidecar can do something such as docker exec ${CONTAINER} chmod --recursive 777, prior to uploading

sidecar runs a separate container to upload

1. sidecar gets the userID inside the user service container docker exec %{CONTAINER} id to get the id of the running user
2. sidecar deploys a separate container which job is to run rclone with the SAME user as in the service container

sidecar manages to ensure the folder always has some specific group properties that cannot be changed by the user in teh container

1. maybe ACL rights?

sidecar runs as root or elevated privileges (I don't like that option)

1. sidecar has full access anyway --> security wise not recommended
2. review how the sidecar is connected to networks

[GitHK]

To address the issue with the .Trash-8005 folder this can simply be added to the ignore list. Will fix the current issue at hand which is a causing pain. But it's not a general approach. This buys us time for a more proper fix.

[mguidon]

can you do sth like find /<id>/work/worspace -readable -type f and then pipe that into the rclone command? Or is that potentially very slow?