Any evidence for Mythos using Loop Transformer(LT)?

[tabsanro]

Is there any concrete evidence supporting the speculation that Claude (Mythos) might be using a Loop Transformer (LT)?

I’ve been particularly interested in researching LT over the past few weeks, so this caught my attention. If anyone has insights or relevant information, I’d really appreciate it if you could share.

[DeepPytorch]

Claude Mythos would have taken too long to train with a loop transformer; it uses a recursive depth MoE model. However, it can be made more efficient by using a MoR model. I've coded loop transformers before; they take a ridiculously long amount of time to train, but they are extremely powerful for their size, unlike Claude Mythos, which is ridiculously powerful because of how large it is.
If you are interested in Loop transformers, then you should research MoRs.

[hamzadevcore]

Does it take long to train, or long to run, or both?

[vendortofreedom-alt]

Does it take long to train, or long to run, or both?

It would likely take long for both training and inference, depending on the recurrence depth and number of active experts. Training especially becomes expensive because each additional loop effectively increases compute per token, even if parameters are shared. Inference can also slow down significantly at higher loop counts, although techniques like adaptive halting or sparse expert routing could reduce the overhead. Could be days could be months, so is it really worth it?