Why reason about computation?

Reinforcement learning agents do not become more computationally efficient as they improve. Humans do tasks can require less cognitive effort as we become proficient.

If agents could reason about their computation, could they learn to use computation toward improved performance?

We show that agents can be given actions that control when they use compute perform better and are more efficient to hand-crafted agents.

Method: Using actions to reason about compute

The agent's computational processes were changed by adding actions.

Agent's Processes Planning Learning subjective state Observation Action Last action Reward Perception Transition model Reactive policy Value function

Our method selects an option consisting of an Atari action paired with a duration. The agents processes (seen above) are turned on only for option selection. The processes are off during execution (during action repeats). An agent can learn to change how it processes its stream of experience, saving computational resources.

University of Alberta Amii — Alberta Machine Intelligence Institute RLAI — Reinforcement Learning and Artificial Intelligence Lab

Results:

Our method improves in 75% of 46 games, a 233% geometric-mean human-normalised score against DQN's 189%, while averaging 3.6 Hz against DQN's fixed 12 Hz.

Per-game improvement of Compute DQN over DQN and learned decision rates across the Arcade Learning Environment, sharing one game axis, with per-game green dots showing Compute DQN using fewer decisions per second than DQN's fixed rate.
Distribution of compute during play in Pong, Breakout, and Asterix.
QR code linking to the paper on arXiv
arXiv
QR code linking to the code on GitHub
GitHub