CDMI Queues were originally added to support storage systems that managed ETL processes. These were typically slower background batch processes. In today's AI applications, queues are still used for ETL, but the performance requirments have dramatically increased when used in newer AI workloads.
Originally we were assuming a third-party workflow manager to handle queue-to-queue transfers. Now we need a mechanism to specify how the queues can talk with each other. E.g. one queue requesting data from another. This is found in bulk data processing tasks, e.g. document OCR.
Given these changes, we should review if and what changes should be considered going forward.
CDMI Queues were originally added to support storage systems that managed ETL processes. These were typically slower background batch processes. In today's AI applications, queues are still used for ETL, but the performance requirments have dramatically increased when used in newer AI workloads.
Originally we were assuming a third-party workflow manager to handle queue-to-queue transfers. Now we need a mechanism to specify how the queues can talk with each other. E.g. one queue requesting data from another. This is found in bulk data processing tasks, e.g. document OCR.
Given these changes, we should review if and what changes should be considered going forward.