Maybe tone down or better scale annotation sharding?

[miraep8]

I recently kicked off a series of annotation apps for 29 samples and noticed that they spawned > 2900 jobs. Was just thinking it might be a bit excessive/maybe could scale better? I think by spawning so many jobs we eventually get diminishing returns on speed up due to having to wait on the scheduler.

Perhaps something to take into account for future versions as well/maybe for the functional annotation pipeline. 🤔 Just wanted to throw it out there as a potential concern.

[nickp60]

Its splitting them into 200-contig chunks, I suppose we could have that set as a config parameter. Base on your finding you have an average of ~20k contigs in your assembly. Are you running on the raw assembly or the bins? There's no logic in the annotation stage to filter out low-quality contigs so thats a possible improvement. Binning will remove the shortest contigs under I think 1kb but if running on assemblies that junk will still be in there

[miraep8]

Yep, I was running on the raw assembly! I am mostly running the app to get the Cazyme results, and at least for run_dbcan they do recommend running on the raw assembly actually (or at least they advise against quality filtering to generate MAGs). Guess it would be interesting to look at their claim and see how many of the annotated Cazymes are actually on sequences that don't bin well into MAGs. 🤔

I guess as a separate question - can the app actually be run on bins? I think we only have an app for annotating the assembly, but correct me if I am wrong as I have actually only ever run it on the assembly. If not - should we make a new app - annotate bins?

[miraep8]

As an alternative - maybe we can dynamically set the nseqs parameter using the results from the Quast report?

[nickp60]

That's good to know they recommend running it on the assembly. Having an app for annotating the bins might be nice; it isn't currently configured that way but it would just be a few lines of code to add similar to what we did for the classification app. And yes -- using the quast reported number of sequencing could be used to split it up.

I guess its a question too of what is more costly -- more jobs, or longer-running jobs. Right now they tend to finish in the 3.5hr * attempt runtime (afaik). If we increase from 200 contig chunks, we will likely have to scale that with it somehow.