Training-inference mismatch has two sources: (1) Numerical discrepancy: Infere… · @SuJinyan6 · TwKit

@SuJinyan6

Training-inference mismatch has two sources: (1) Numerical discrepancy: Inference engine and trainer run their own kernels and precision (and an FP8 sampler widens the gap). (2) Policy lag: Asynchronous causes the lag between inference engine and the trainer weight. And there are three types of corrections: (1) Truncated importance sampling. (2) Score centering. (3) PPO clips. Score centering deals well with Numerical discrepancy but falls behind IS in terms of Policy lag, and PPO works well on Policy lag but collapses under Numerical discrepancy. While IS works for both sources. They find that: (1). The numerical discrepancy is around a constant while Policy lag increases with the steps. Thus, Score centering work well with numerical discrepancy. (2). After scoring centering, there is still a residual bias that can be expanded into 4 terms: prefix, continuation, token, tail. Score centering keeps the "token" term, which IS cancels on every token whose ratio stay under 2. (3). Importance sampling has an expiry date. As the lag spreads, the IS ratios get extreme and the effective sample size drops. (4) When doing PPO, whether the reference policy being the trainer before each batch or the sampler's weights matters. They thus propose discrepancy-lag correction(DLC) that (1) Center the numerical discrepancy(2) Clip the lag ratio with the PPO clip against the sampler's weights (3) When the effective sample size falls below half the batch, copy the trainer's weights to the sampler. In general, the best practical takeaway from this paper is that: If you run a quantized sampler synchronously, the error is a constant and centering or IS is enough. If you run asynchronously, watch the ESS of the lag ratio and resync when it crosses one half. The step at which that happens depends on the model (step 11 on 1.7B, never within 40 steps on 0.6B), so a fixed sync interval is either wasteful or late. Put PPO's reference at the weights that generated each token, not at the trainer before the batch.

曝光 458 · 评论 0 · 点赞 11 · 书签 5