enwik8 test, bits per byte (lower is better): • 39.5M params: 1.018, and 0.980 with dynamic evaluation (learning while reading) • 86.3M params: 1.027 / 0.976 • 85.3M without the memory: 1.075 / 0.999
Honest caveat: the tables hold 82M count entries (a 1.1 GB file) on top of the parameters. The memory is built by counting, no gradients, but the model still runs a normal transformer.