1. The learning environment
+50
0.30
−2
0.90
Ready. Press Run learning or take individual steps.
→ ↑ ↓ ← = action direction
number = Q-value
bright arrow = highest Q-value
● = agent
2. Bellman equation — update the Q-value
Q(s,a) ← Q(s,a) + α [ r + γ max Q(s′,a′) − Q(s,a) ]
OLD Q(s,a)0.00
REWARD r−2.00
MAX NEXT Q0.00
NEW Q(s,a)0.00
Latest update, calculated live
Q(new) = Q(old) + α × [reward + γ × max(next Q) − Q(old)]
Q(new) = 0 + 0.30 × [−2 + 0.90 × 0 − 0] = −0.60
Q(new) = 0 + 0.30 × [−2 + 0.90 × 0 − 0] = −0.60
What is happening?
The agent is in a state s and chooses an action a. It receives a reward, moves to a new state s′, then looks at the best Q-value available from that new state. The Bellman equation uses this information to move the old Q-value toward a better estimate.
The agent is in a state s and chooses an action a. It receives a reward, moves to a new state s′, then looks at the best Q-value available from that new state. The Bellman equation uses this information to move the old Q-value toward a better estimate.
3. Learning history
No updates yet.