This paper have many interesting takeaways about looped transformer: 1. Loop b… · @SuJinyan6 · TwKit

@SuJinyan6

This paper have many interesting takeaways about looped transformer: 1. Loop beats no loop (no matter you loop entire network stack, only the middle stack, loop each layer, only attention, only MLP, or both MLP and attention, loop generally beats no loop. With we add more loops, the gain would gradually saturate, reverse or destabilize training. 2. Add specilization helps: Later loops see representations that earlier loops have already transformed, so it is better to give each loop a small low-rank adapter. 3. The original parallel loop transformers (PLT) is not doing correctly and need to be fixed by giving RMS-normalize to both the embedding and the shifted state, then combine them with a linear gate.

曝光 771 · 评论 1 · 点赞 17 · 书签 7