Nexus store specialization: forge.attach() method fails for files larger than 2 GB

[jk-n]

The dataset was of 13.6 GB.

when asked about this to the backend team they mentioned it might be a Forge issue.

[annakristinkaufmann]

• The nexus-store specialization of Nexus Forge uses the nexussdk.files.create() method from the nexussdk package in the _upload_one() method (

  nexus-forge/kgforge/specializations/stores/bluebrain_nexus.py

  Line 171 in 7902a5f

  def _upload_one(self, path: Path, content_type: str) -> Dict:

  )
• the nexussdk.files.create() method makes use of the requests package (https://github.com/BlueBrain/nexus-python-sdk/blob/master/nexussdk/utils/http.py)
• for the requests package, the error reported on this issue was already described elsewhere: "OverflowError: string longer than 2147483647 bytes" when trying requests.put psf/requests#2717:

Requests can't handle a put call for very large data objects. However, it can accept the data as a file-like object instead, and the size issue will not show up

• See also: lfit/releng-lftools@1c04b58
• Since Nexus Delta uses PUT when there is no @id provided for the file and POST when an @id is being provided, the issue only occurs when not providing an @id for a file which is the case in the Nexus Forge implementation