A few days ago,@VoidAsuka shared a blog post with me about tito, meaning token… · @Chonghan_Liu · TwKit

@Chonghan_Liu

A few days ago,@VoidAsuka shared a blog post with me about tito, meaning token-in, token-out in reinforcement learning. https://srush.github.io/verified-tokenizer/ I didn't pay much attention to it at the time, mostly because tito issues have been around for a long time, and both the community and our internal teams had already solved them while training models. Then, one afternoon while casually looking through the vocabulary, I dug up that blog post and glanced at it. Suddenly, I realized something: wait, we haven't actually completely fixed this. Because everyone's designs vary so much, the community's support code for it is totally fragmented, and there's no real standard. That's how the PR I submitted to verl came about. https://github.com/verl-project/verl/pull/8463 emmm, I feel like tito is a well-worn topic at this point, but it's a really messy piece of work. It isn't a difficult concept, but it's one of the reasons you quickly discover a severe training-inference mismatch in reinforcement learning. There's no deep mathematical logic to it; it's just messy, unglamorous engineering work. Like Yao shunyu said, right now what we need more of is engineering care, which also involves fixing infrastructure bugs.

曝光 135 · 评论 0 · 点赞 2 · 书签 3