Total Eps: The total number of Tic-Tac-Toe games the AI will play to train itself.
Interval: How often (in episodes) the system saves a model checkpoint to the vault.
LR (Learning Rate): The step size the Adam optimizer takes when updating the neural network weights. Too high and it becomes unstable; too low and it learns too slowly.
Gamma (ฮณ): The discount factor. Determines how much the AI cares about long-term vs short-term rewards. A value of 0.95 means it values future moves almost as much as immediate ones.
Decay: The rate at which Epsilon (ฮต) decays. Epsilon controls exploration vs exploitation. A value of 0.0008 means the AI will slowly stop taking random moves and start using its learned network over the course of training.
Opponent: The strategy the opponent uses during training.
Minimax (The Grandmaster): The perfect mathematical oracle of Tic-Tac-Toe. Training against Minimax is akin to supervised learningโthe network is constantly punished by optimal traps and forced to learn perfect defense. Having the right data (expert gameplay) and deeply understanding the problem yields drastically better results than blind reinforcement!
Self-Play: The DQN plays against older copies of itself. While it can eventually learn via bootstrapping, it suffers from the "blind leading the blind" effect early on. It might develop bad habits because its opponent is equally terrible, requiring far more episodes to converge.
Random: The opponent randomly guesses. The DQN will quickly learn to win, but will fail miserably when evaluated against a real expert because it was never exposed to complex traps. It proves that learning from randomly guessing data can never yield the robust intelligence of an expert teacher.
๐ก Training Analysis & Recommendations
๐ง Deep Q-Network Studio Architecture
Welcome to the Deep Q-Network Studio. This environment demystifies Reinforcement Learning by showing exactly how an agent learns Tic-Tac-Toe from scratchโnot by being programmed with rules, but by experiencing rewards and updating its neural network via the Bellman Equation.
The Bellman Update
Q(s, a) = R(s, a) + ฮณ * max Q(s', a')
Unlike Minimax, which exhaustively traverses a static game tree to find the guaranteed optimal path, our DQN uses a multi-layer PyTorch network to approximate the expected future reward Q(s, a) for any given board state. The network's hidden layers dynamically map patterns (like forks and blocks) directly to scalar values.
Overcoming Catastrophic Forgetting
Neural networks suffer from "catastrophic forgetting"โif they only learn from consecutive games, they forget how to block old strategies while learning to beat new ones. Our agent solves this using an Experience Replay Buffer. As it plays, it stores memories as (State, Action, Reward, Next State) and constantly trains on randomized batches from its past!
Why a 9 โ 256 โ 128 โ 9 Architecture?
While a network with ~36,000 parameters is mathematically "overkill" for a game as simple as Tic-Tac-Toe (which has only ~5,478 legal states), overparameterization provides massive benefits in Deep Reinforcement Learning. The excess capacity creates a smoother mathematical landscape for the optimizer, leading to much faster convergence. It also acts as a robust "memory bank" to combat catastrophic forgetting, allowing the network to easily retain old defensive rules while actively learning new offensive traps. Plus, it provides a gorgeous, complex mapping for the Synapse Evolution heatmap!
The Three Phases of Learning
Phase 1: Exploration (High ฮต) - The AI acts almost completely randomly to discover the environment, lose horribly, and fill its memory buffer.
Phase 2: Strategy Transition (Decaying ฮต) - The optimizer adjusts the Synapse Weights. You will see the Loss converge as the network balances exploiting known winning moves with exploring edge cases.
Phase 3: Perfect Exploitation (ฮต = 0) - The AI exclusively selects the move with the highest Q-Value output. It becomes an impenetrable wall.
Tip: Watch the Q-Values (the "Mind Reader" heatmap) overlay on the Live Evaluation Arena to see the network's exact mathematical reasoning in real-time!
๐งช Exhaustive Minimax Evaluation Report
๐ค Why are there so few games?
You might notice the exhaustive evaluation only plays around 140 games. Aren't there 255,168 possible Tic-Tac-Toe games? Yes! But the tree shrinks massively for two reasons:
The DQN is Deterministic: On its turn, the DQN calculates the Q-values and plays its single favorite move. It never explores mistakes, so its branching factor is exactly 1.
Minimax is Optimal: Minimax evaluates all moves but throws away anything that leads to a loss. It only splits the timeline if there are multiple moves mathematically tied for "best" (like opening in any of the 9 squares).
The Ultimate Gauntlet: Those ~140 games represent the exact number of reachable realities when these two specific AIs face off. Minimax tried every possible optimal sequence of traps, forks, and openings it could mathematically generate. If the DQN survives all of them, it means Minimax was never given the opportunity to branch into the thousands of other states that only happen when an opponent messes up!