Teaching a World Model to Play Pokémon: Learning Game Dynamics from Screenshots

Teaching a World Model to Play Pokemon

The author trained a LeWorldModel (a JEPA-style world model) on Pokémon Red to predict the next screen from a screenshot and a button press, then plan a sequence to select a starter Pokémon. The encoder turns screenshots into 192-dimensional embeddings; SIGReg prevents embedding collapse. Training used 1,040 (screenshot, action, next-frame) pairs. The first plan failed because the model spammed A without pressing B to dismiss dialogue; rollout fine-tuning fixed that, and the model successfully walked to the Poké Ball and selected Squirtle in 11 steps. The post doesn't disclose total training time or final success rate.