ModelRefs / Reinforcement Learning Intro — Tutorial
Reinforcement Learning Intro — Tutorial
Agents, environments, rewards, and the Q-learning algorithm that taught AI to play games. Covers The RL framework: agent, environment, reward.
Overview
Agents, environments, rewards, and the Q-learning algorithm that taught AI to play games
Level: Intermediate. Estimated reading time: 22 minutes.
The RL framework: agent, environment, reward
Reinforcement learning is learning by trial and error. An agent takes actions in an environment, receives rewards, and learns a policy that maximises total future reward.
The core loop: the agent observes the current state s, chooses an action a (according to its policy π), the environment transitions to state s', and returns a reward r. This repeats until a terminal state.
Key terms: - Policy π(s) → a: a function mapping states to actions (what the agent does) - Reward r: a scalar signal from the environment (positive for good outcomes, negative for bad) - Return G: the total discounted reward from time t: G = r + γr' + γ²r'' + ... - Value function V(s): expected return from state s under policy π - Q-function Q(s,a): expected return from taking action a in state s, then following π
Unlike supervised learning, there is no labelled dataset. The agent must discover good actions by exploring.
Q-learning: learning action values
Q-learning learns Q(s, a) — the value of taking action a in state s. The update rule is:
Q(s, a) ← Q(s, a) + α [ r + γ · max_a' Q(s', a') − Q(s, a) ]
Translation: update Q(s, a) toward the target r + γ · max_a' Q(s', a'). The target is the actual reward received plus the discounted best possible future value from the next state.
This is the Bellman equation: the value of an action equals the immediate reward plus discounted future value. Q-learning converges to the optimal Q-function given enough exploration.
Exploration vs exploitation: if the agent always takes the best-known action (exploit), it might never discover better ones. ε-greedy policy: take a random action with probability ε, otherwise take the best-known action. Start with ε=1.0 (full exploration), decay toward 0 (exploit learned policy).
From Q-tables to Deep Q-Networks (DQN)
Q-learning with a table works when the state space is small (e.g., a 4×4 grid). For Atari games with 210×160 pixel screens, the table has more states than atoms in the universe.
Deep Q-Network (DQN, DeepMind 2013): replace the Q-table with a neural network that takes a state as input and outputs Q-values for all actions. Key innovations: - Experience replay: store (s, a, r, s') transitions in a buffer, sample random batches to break correlations - Target network: use a slowly-updating copy of the network to compute targets, stabilising training
DQN achieved superhuman performance on 49 Atari games from raw pixels — the moment RL became a serious research area. Subsequent work: PPO (OpenAI, 2017), AlphaZero (DeepMind, 2017), which beat humans at Chess, Go, and Shogi using self-play.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Reinforcement Learning Intro — Tutorial.