From GPT-2 to Kimi Delta Attention: How Transformers Evolved into Modern AI Memory Systems
Twenty-two thousand five hundred and eighty. That is roughly how many GPT-2-scale models (≈124M parameters) fit inside Kimi K3 (≈2.8T parameters, 2026). Scale mattered—but it is not the whole story.
A parallel evolution happened inside the architecture itself:
AI models gradually changed from systems that repeatedly search every previous token into systems that maintain, update, erase, and selectively retrieve structured internal memories.
This article combines that memory-systems interpretation with the concrete architectural worklog 22580: From GPT2 to Kimi3, Explained by ali (@waterloo_intern), including the original diagrams, plus the research path from full softmax attention → linear attention → DeltaNet → Gated DeltaNet → Kimi Delta Attention → Kimi Linear / Kimi K3.