Read files in chunks from remote storage

[jbentvelsen]

Currently, all workflows require the complete dataset to be available on the local disk. However, services like AWS may also support reading single frames from dataset at a time. With this 'streaming' approach the user can directly start training, without having the download the full file first.

[tkuenzmw]

I really like where you are going with this! This is exactly what I was brainstorming as well about the datastore/generator-like functionality From: Joris ***@***.***> Sent: Monday, 30 October 2023 13:58 To: MATLAB-Community-Toolboxes-at-INCF/DeepInterpolation-MATLAB ***@***.***> Cc: Subscribed ***@***.***> Subject: [MATLAB-Community-Toolboxes-at-INCF/DeepInterpolation-MATLAB] Read files in chunks from remote storage (Issue #36) Currently, all workflows require the complete dataset to be available on the local disk. However, services like AWS may also support reading single frames from dataset at a time. With this 'streaming' approach the user can directly start training, without having the download the full file first. — Reply to this email directly, view it on GitHub<#36>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A6JZROKKWFG3Z44UDKQPGM3YB6P4BAVCNFSM6AAAAAA6V7EXM6VHI2DSMVQWIX3LMV43ASLTON2WKOZRHE3DQMRZHAYTEOA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>