ByteDance Lightning models / Feature Reactive LoRAs / MIDI Processing - Extracting .mid from audio and/or stem separations?

[GomuSkelly]

Hey Ryan :)

Audio/MIDI reactive LoRAs was something we'd been trying to figure out with your pack for months so thanks for releasing that wf a few weeks back, hope to finally get round to testing it tonight!

We've been using Stable Audio Open in a project that we've started to release some pieces of ( MozAIk ). One of the applications of the system is DJing with generated audio in realtime.

We use Lightning 4-step in our template wf for speed as we can have a small animated loop to accompany the generated audio. Wondering if you've had much success with the ByteDance Lightning models in any of your audio reactive workflows? (We haven't in any of our tests so far!)

Also, is there a way with your pack to create .mid's from the audio / separated stems yet for use in external applications?

Cheers

[ryanontheinside]

@GomuSkelly I potentially will be focusing a lot on real-time soon. I'll get back to you on this!

[GomuSkelly]

@ryanontheinside Thanks for the reply, and nice to hear!

Love the pack, but it's a lil bit of monster to set up for beginners and we're trying to move very slowly with minimal pro-user volumes of custom nodes (and the dependency conflicts that follow) in our upcoming workflows, to get non-ComfyUI folk up and running with our basic project templates. Keeping it modular as much as possible.

Amorano / Jovi has started to break out different elements from his pack, we've asked him to consider a Jovi_MIDI to accompany his Jovi_Spout and Jovi_GLSL (excited about these ones for realtime stuff), but I also hope that at some stage this might be something you consider doing also.

Is this something you'd be open to with your own node pack in the future, so we could pick and choose the core elements we need, or would it be too complex to try and seperate out discreet features / other distinct aspects of the functionality you have going with your pack without major refactoring?

[ryanontheinside]

@GomuSkelly I suppose it would depend on the components that need to be separated