Opus 5.5 and GLM-5.3 tie at 73.3% on FrontierMemory. But the more interesting … · @yeemio · TwKit

@yeemio

Opus 5.5 and GLM-5.3 tie at 73.3% on FrontierMemory. But the more interesting part is how Letta tests whether an AI’s memory actually helps it work. Not “can you remember what I said?” More like: “can you use what happened last time to do the next job properly?” There are a few useful ideas in the setup. 1. Make memory necessary. Some information needed for the task, such as user preferences, project conventions or quirks of a tool, is left in earlier work records rather than repeated in the new request. That helps test whether the agent uses past experience, rather than just solving everything from the current prompt. 2. Let the model work out what is worth keeping. The model turns those records into notes and skill files. This is memory stored outside the model, not experience trained into its weights. 3. Give those memories to an agent doing the task. This is where the harness, the software running the agent, matters too. The model has to extract useful lessons. The system has to make those lessons available, and the agent has to find and use them at the right moment. Saving a file is not the same as using it next time. One example in the article makes this very clear. One model learned to check both notes and email. Another tied that behavior to the exact file sizes it had seen before. When the file sizes changed, the rule no longer triggered. Both had written a memory. One captured a useful method. The other captured an accident of the previous task. That also matters when reading the leaderboard. The model comparison uses the same Letta setup to test how useful different models’ generated memories are for later tasks. It is not a comparison of the memory features in ChatGPT, Claude or other chat products. For me, this is the useful evaluation standard it points toward: don’t just check whether the AI saved information. Check whether that experience helps with the next job. There are only 15 tasks so far. And 73.3% is a task success rate, not the improvement caused by memory. Comparisons against no memory and the full original records would help show how much the memory-generation step actually contributes. The rankings are worth watching. But this way of testing memory is more interesting to me than who is ahead today.

曝光 73 · 评论 1 · 点赞 1 · 书签 1