Two Hours of Atari

An agent learned each game in two hours of play, the same practice as the human tester.

Loading…

Score0
Average–
What it sees
Expected points for each move
Game
Practice

Info

About

The human scores in Atari benchmarks come from a professional game tester. DeepMind measured them for its DQN agent (Mnih et al., 2015). The tester practiced each game for 2 hours. Then the tester played about 20 games of each, up to 5 minutes each.

In 2019, the Atari 100k benchmark (Kaiser et al.) gave agents about the same quantity of play: 100,000 moves, or 1 hour and 51 minutes of game time. BBF is an agent from Google DeepMind and Mila. It gets these two hours of play, and it starts with no knowledge of the game.

The two hours are game time, not computer time. We trained BBF on four games, one game at a time, on a laptop GPU. Each game took about 6 hours, because the agent trains its network after each move. On this cabinet, the agent plays these games on the same emulator that it learned on. Use the practice lever to select a point in its practice.

The benchmark

Atari 100k gives an agent 100,000 moves in each of 26 Atari 2600 games. Space Invaders is not one of these 26 games. The BBF paper also gives scores for it, with the same rules. The agent does each move for 4 frames. Thus 100,000 moves are 400,000 frames. At 60 frames a second, this is 1 hour and 51 minutes of play. After the practice, the agent plays 100 games and does not learn. Its score is the average of these 100 games.

The agent gets only the screen and the points. During practice, all points from one move count as 1. A move that gives 100 points and a move that gives 5,000 points have the same value. When the agent loses a life, the current part of the practice stops. Nothing else tells the agent what the game is or what the joystick does.

The agent

BBF (“Bigger, Better, Faster”, Schwarzer et al., 2023) is a value-based agent. Its network looks at the last four frames. For each move, the network calculates how many points will follow that move. The agent does the move with the highest value.

Three parts of BBF make it learn quickly from a small quantity of play:

  • A large network. It is an Impala ResNet at four times the usual width, followed by 2,048 hidden units. After each move, the agent does two training updates. Each update uses 32 moves, selected from all of its play until that time.
  • A second task. The network gets the current frames and the next five moves. From these, it predicts its own output for the next five frames. This task trains the part of the network that reads the screen, also before the agent gets points.
  • Resets. After each 20,000 moves, the last layers get new random weights. The other weights move halfway to random weights. Without resets, a network that fits early data too closely learns less from new data. After each reset, the agent looks 10 moves ahead. Over the next 10,000 moves, this decreases to 3 moves.

What we measured

How it runs here

The games run on the Arcade Learning Environment, compiled to WebAssembly. This is the same emulator that the agent trained on. We compared the two emulators frame by frame. They give the same screens, points and lives. The network runs on TensorFlow.js. Its weights use 8 or 16 bits. On our test frames, it selects the same moves as the original network.

“What it sees” shows the four gray frames of 84 × 84 pixels that go into the network. The newest frame is on the right. Each pixel is the brighter of two screens. “Expected points for each move” shows the points that the network expects after each joystick move. Points that come later have less value. The scale goes from the worst move to the best move, and the agent does the best move.