Alternative quantization for models

[mindinthedocs]

@ServeurpersoCom
Hello. I've been wondering if it makes sense to try alternative quantization methods for models (especially for DiT model)? I'm new to this, but maybe something like HQQ would be better, have you considered it yet?

[ServeurpersoCom]

Hello! The flow matching DiT is so effective that even at Q4_K_M, the difference in rendering compared to BF16 is imperceptible! I could even have suggested higher quantization levels. But since the small LLM is sensitive with its 5Hz audio codes, I stopped at Q5_K_M because below that, the results start to hallucinate.

[ServeurpersoCom]

Random generation with a small visible difference on the amplitude graph in the final rendering at Q4, exactly like in diffusion for images :) it does not affect the quality, it is just a rounding error which produces a different choice of DiT.

BF16_3519225289.mp3
Q8_0_3519225289.mp3
Q4_K_M_3519225289.mp3

Request :

{
  "caption": "A progressive rock epic with complex time signatures, intricate guitar work, and dynamic male vocals. Features extended instrumental passages, synthesizer layers, and dramatic shifts between soft and heavy sections.",
  "lyrics": "[Intro]\n\n[Verse 1]\nWaking in a world that's not my own\nSeeds of doubt have long been sown\nReality is bending at the seams\nCan't tell the difference between life and dreams\n\n[Verse 2]\nClockwork minds and digital souls\nPlaying predetermined roles\nAre we the masters or the slaves\nOf the futures that we crave\n\n[Chorus]\nParallel dimensions collide\nNowhere left to run and hide\nThe truth is stranger than we know\nLet the revelation flow\n\n[Instrumental]\n\n[Verse 3]\nQuantum leaps through space and time\nSearching for the paradigm\nEvery answer breeds more questions still\nClimbing up the endless hill\n\n[Chorus]\nParallel dimensions collide\nNowhere left to run and hide\n\n[Outro]",
  "audio_codes": "3101,3717,63635,25586,48291,39979,37121,36788,11665,18311,12671,40839,11663,26855,53616,15305,14474,18675,27876,38462,14207,35308,25025,24450,24523,15188,13787,21252,51979,38301,15211,63857,4821,53569,49586,8635,63450,22443,7603,26307,30409,30394,35600,22914,14829,14064,23874,5312,7176,29736,51200,27728,17408,35332,24322,12159,10176,62412,6528,12664,61376,26968,31600,53624,46008,27928,13664,28048,42240,40872,58360,25568,44816,50961,1408,51152,16022,61504,14376,12424,25224,19008,13320,51240,49580,25433,30241,62249,51200,4991,53682,37927,12755,27870,8710,1060,18989,13356,23588,29700,37394,13354,7220,29686,10051,60349,29719,25731,53340,48695,40695,5628,38455,13547,25958,38311,44935,24453,11718,60414,59773,60374,20404,28119,13285,33135,24663,15390,20420,35309,10031,13327,13318,21055,58383,25743,63551,60423,12359,13871,14375,63655,23,13327,26686,53375,24647,15108,39934,62365,54542,43789,30908,35251,12068,12676,63956,62865,18758,10173,28099,12221,42309,48259,44292,3387,29628,5004,32725,1388,40436,27893,24004,12223,27028,14487,14135,52120,38382,13656,35770,22147,25382,46902,49644,12197,11172,38380,26308,26421,27012,29515,24406,21829,1469,11730,23854,21666,37209,17348,38035,11682,2435,52157,6850,57659,45312,40705,53399,50688,47616,7492,29192,1056,40480,36281,47104,37567,50383,42366,13584,49584,11192,63440,52120,27424,11256,61392,55104,9544,27912,26944,61360,19440,63984,43468,51067,50883,9987,52010,1330,50172,11301,10264,38240,14976,43136,48433,34848,28597,6792,25256,20992,12750,813,30534,26094,32790,53261,9788,63013,38930,52236,21523,51730,11293,54315,13346,14890,18475,14891,32198,14855,21041,52807,29335,13631,13742,23812,24325,38213,27973,32046,54543,44375,62420,13164,15168,13633,13249,25519,23095,10765,20238,21775,2063,14855,17983,4623,7175,2055,2055,9735,2055,63623,40495,53279,53295,53319,40463,21055,22143,53351,12487,53319,35367,18454,40975,16447,575,12359,10823,12675,15196,13074,13147,52185,9704,10664,10107,9107,53248,51518,5446,11607,50606,38253,49605,3591,52758,58382,14867,4733,24494,11158,26850,37157,24768,11280,19568,6768,6760,1672,2248,40520,50688,46088,43032,50824,40448,2560,17424,3600,54288,39520,63314,24897,4556,44998,56766,60894,1531,12602,18854,14863,40535,13919,17971,18365,26215,13511,3112,44112,27664,40464,36682,32991,20151,10879,13999,54391,11399,28191,12903,2191,58479,41015,41543,36159,2315,2314,2371,27457,53701,13890,40584,1664,9864,4369,6864,55120,20800,13800,8712,26280,1224,397,8533,12579,24175,24364,37383,1541,57868,57861,37930,25642,25635,12842,14891,27706,30259,28211,2094,39445,2060,32267,13115,39300,38772,13399,39941,52747,60930,21555,14849,43188,55815,63533,60943,7231,38053,25655,36863,59886,62600,47446,63939,29512,42942,40374,53054,53119,40255,47918,50031,50023,50614,50607,50622,53630,53565,2309,2310,2310,53573,40838,40766,40831,53638,40894,51646,49599,11607,11717,36805,25983,54062,52094,38300,25372,2916,14212,12109,11220,11742,36742,49535,36487,14103,13535,63423,12223,27583,11015,36103,25799,41735,27010,23941,26359,16103,52441,61416,38232,14664,13192,3529,31200,60920,60400,6608,57816,53200,53184,53712,55753,53712,39824,50576,57914,444,25596,36806,44487,46534,10206,11653,25597,52165,836,53716,24004,49149,56770,51722,49171,38930,2833,15031,24079,62039,50759,49287,33343,21007,25151,6159,45687,26159,61567,26175,10991,8199,16927,15911,49199,1675,14338,17019,15420,28219,5164,7204,25618,40483,39459,38442,40466,36353,33281,3172,3108,50112,13735,13764,13120,23488,58872,19416,24008,47552,47584,2440,1992,25568,17896,63976,47000,13248,18352,48592,52096,47356,23951,22007,11774,10239,6655,63479,40398,53198,37325,26581,3029,28118,14189,53661,27260,37500,27650,15019,14494,13406,53862,26655,25766,12838,12391,24199,63687,39023,559,639,11503,51583,48319,53311,48206,29703,2245,1661,14523,14509,26804,37514,22724,37971,12827,19,3,194,67,14525,14460,26056,30487,34816,10690,1016,63472,60880,27080,39232,9136,60881,29656,44490,45050,62968,60880,14304,12767,15199,2391,15207,14237,10631,23943,10751,21959,28038,27919,51079,61263,10135,35207,15167,52607,711,2185,3712,680,39080,37224,28928,49072,58240,27472,3392,38720,56264,43408,28032,1936,25936,2880,26976,13705,38832,7601,25044,34488,49646,36838,37221,37348,25061,50478,37300,21861,37230,22294,37084,23972,23837,23835,24021,21343,21276,5964,63443,53756,390,2891,13771,23956,21339,18836,21404,57747,36772,37860,25003,23523,23964,11099,18828,5381,15150,14906,27770,27763,12924,25724,25204,14850,14395,27717,27651,39939,41988,41477,57989,60582,6759,63605,38226,14996,25747,28277,44148,23229,61621,12422,12500,15123,14484,25683,31340,36020,59189,63284,60614,11981,27917,25739,41132,30907,41132,11453,11334,11926,14421,27741,28684,41012,46205,10429,8382,12548,11397,25229,14997,53444,51010,51010,4995,14297,26247,53671,12711,39247,11713,62926,26562,60415,54668,24892,24542,61924,15247,24414,2310,3019,3462,8132,13698,53189,21893,36811,836,62076,45445,4374,49645,49650,49539,51778,49026,12620,4866,63938,23036,17357,63544,5070,22850,32808,3624,42000,26256,2308,18951,40727,43151,42961,27585,2937,6504,9608,32032,11696,13736,12176,39848,23000,63441,58304,19937,7041,26073,1969,28000,6650,63456,35493,27752,34434,31232,36368,10668,2184,58400,15872,849,12416,26577,45640,26041,25477,63943,22159,28671,53230,8199,5164,18467,27691,27675,27658,40467,37898,27659,14851,55868,1541,52739,26633,62725,42365,31678,1486,11815,15399,2543,8519,13701,13157,15205,5991,13269,51541,27892,5103,25551,2127,14356,46636,37357,12647,53375,11783,2055,14855,27271,17431,28182,28215,23615,8775,11311,1695,14367,26199,38454,28215,10295,12351,11911,12807,17926,15935,14895,12311,55839,1559,25622,36039,50887,12798,62917,12228,50643,29634,57213,44476,12115,19358,25503,25455,12742,12791,15158,7247,53255,24287,58,31803,14799,38215,62850,46067,43153,16,2608,12848,15920,28200,25104,2248,4616,4656,10256,50688,4608,4624,5120,51712,53384,26128,29946,768,53571,55238,62974,41862,43006,42192,23983,44247,11839,10823,15406,19364,14351,39551,24481,22136,11320,19064,14187,4922,26215,7855,63183,13111,10879,8839,28694,38567,11975,25295,12967,11455,20231,15367,28351,12415,38039,15170,30064,53689,56185,2888,18769,35104,50600,63865,53616,53568,15672,54720,63992,45000,12736,54760,12736,55799,42438,41982,42989,16324,7149,26575,12223,36743,38215,17055,15135,14687,12679,12703,12791,12727,11589,12283,49581,49468,50668,50604,49980,49539,39926,52972,36803,36820,40309,39667,37317,11558,39595,37355,37333,38340,14182,2405,14156,12219,47555,11015,49607,25799,49599,41607,39223,41687,4903,2463,12799,62407,62847,37318,49919,46782,44407,30375,15134,17831,2535,25415,23494,13247,41655,44238,15191,15191,15150,10742,23943,49471,47495,29983,2407,14751,12279,49479,39167,44759,4847,17759,12727,24455,13631,49351,41663,58015,17639,22879,12727,24967,13631,49351,44223,44703,17639,20319,12791,23943,39167,41607,43703,41695,33807,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847,35847",
  "vocal_language": "en",
  "keyscale": "F# minor",
  "timesignature": "4",
  "bpm": 132,
  "duration": 239,
  "seed": 3519225289,
  "lm_temperature": 0.85,
  "lm_cfg_scale": 2,
  "lm_top_p": 0.9,
  "lm_top_k": 0,
  "lm_batch_size": 1,
  "synth_batch_size": 1,
  "synth_model": "acestep-v15-turbo-BF16.gguf",
  "lm_model": "acestep-5Hz-lm-4B-Q8_0.gguf"
}

[ServeurpersoCom]

The cover mode uses an FSQ roundtrip that compresses the source latents from 25Hz (the VAE codec rate) down to 5Hz (the LM audio code rate) and back up to 25Hz before feeding them to the DiT. This projects the source audio into the latent space the model was trained on, rather than feeding raw VAE latents the DiT has never seen in this context. The tradeoff is that 5x temporal compression destroys fine spectral detail (riffs, specific voicings) while preserving the rhythmic and melodic skeleton. The DiT's self-attention then uses this degraded source as structural guidance while the flow matching process recreates its own fine details, which is why the output stays in sync with the original but never reproduces specific riffs faithfully.
This is architectural, not a model size issue. The FSQ bottleneck is upstream of the DiT. The whole point is to let the denoising process hallucinate coherent detail rather than copying the source. We'll see how ACE-Step XL handles it!

[ServeurpersoCom]

I've actually experimented quite a bit with bypassing the FSQ roundtrip entirely, feeding the 25Hz VAE latents straight into the DiT without the 5Hz bottleneck, trying to find the sweet spot on personal commercial tracks that sit closest to the DiT's training distribution. It produces some really cool results, but it's fairly unstable and hard to tune since it pushes the model outside what it learned.

[ServeurpersoCom]

Here are a couple of examples where I took a simple commercial track with tones I liked but very little structural complexity, and fed it directly into the DiT as raw 25Hz VAE latents bypassing the FSQ roundtrip. The results I kept are the ones where the DiT went somewhere genuinely unexpected: almost electro-symphonic textures with dissonances that feel deliberate, as if the model found its own internal logic for resolving the input.
Cover-5Hz-FSQ-Bypass-A.mp3
Cover-5Hz-FSQ-Bypass-F.mp3
But it's not really controllable like a real music2music

[mindinthedocs]

Is "A" the original or one of the covers? If not, would be interesting to listen to the original which was covered to rate how much was preserved.
Yea I figured out that 5hz might be the issue, but surprisingly sometimes model got singing pattern quite well (no "ref" was ticked).
Wonder if a trick with stretching the original song would work (probably not).

I had good results with covers in Suno, in terms of preservation of melodies and song elements which "defined" the songs as we used to know it. But it's a walled garden with copyright issues (many tracks, even obscure, are marked as "existing work of art" or "lyrics are copyrighted"), even if you don't plan publishing it and doing it for yourself only. Having to manipulate the original record to overcome this and so on.
I'm mostly interested in cover/remix functionality, not in music generation from scratch. I'd test a build with FSQ bypass for sure.
Do you also know any other decent models?

[ServeurpersoCom]

It's not even a cover, it's audio conditioning, on a single step! The other 49 are pure hallucination. The source isn't here, it's copyrighted. It's very basic synthwave. But anyway it's absolutely not a remix, the source has nothing to do with it. When you do that, the balance point between a hallucination that has nothing to do with it and a copy of the original is very fine!

This is the very beginning in open-weight musical models. Luckily there's ACE-Step, I don't know of any others, there's nothing that currently rivals it.
In fact, this mode would be great if we could train a large DiT on all the music on the planet... Which is impossible for copyright reasons, not for technical reasons! I could add a "Bypass FSQ roundtrip" checkbox for the Cover, a simple if(), when I test the DiT XL in 4B

[ServeurpersoCom]

Another example cherry-picked for largest divergence, even on Q8_0 vs BF16.

BF16.json
BF16_1224283445.mp3
Q4_K_M_313355069.mp3
Q8_0_3528706179.mp3

[ServeurpersoCom]

ggml-org/llama.cpp#4782
ggml-org/llama.cpp#6368
ggml-org/llama.cpp#4806
It didn't seem to catch on (at least not for the LLMs), but I prefer to diverge as little as possible from GGML / llama.cpp

[ServeurpersoCom]

MagicBoxCarillon-Remix_289542998.mp3