Problem
Right now, all Datadrip jobs are being enqueued to the same 24hr queue, causing both fast & slow jobs to have to wait unnecessarily for others to finish. "We" have long understood this problem and solved it with the SLO system introduced by @sizief .
We need to change the Datadrip API to allow each backfill to similarly to what we do with regular jobs use the SLO system and be able to specificy which SLO is better suited for it.
As an example, right now a job to toggle a feature flag a lightweight and fast task, and doing a company migration from Payroll to Compensations a long running task, is considered the same. They must both wait for N amount of time for other jobs to finish before they're enqueued and then some more time before they're executed.
Problem
Right now, all Datadrip jobs are being enqueued to the same 24hr queue, causing both fast & slow jobs to have to wait unnecessarily for others to finish. "We" have long understood this problem and solved it with the SLO system introduced by @sizief .
We need to change the Datadrip API to allow each backfill to similarly to what we do with regular jobs use the SLO system and be able to specificy which SLO is better suited for it.
As an example, right now a job to toggle a feature flag a lightweight and fast task, and doing a company migration from Payroll to Compensations a long running task, is considered the same. They must both wait for N amount of time for other jobs to finish before they're enqueued and then some more time before they're executed.