Date: 31-05-2026

In a hurry? Play the browser demo here: Clash Royale web play demo.

This is a technical article about implementing the PPO self-play RL algorithm on a popular real-time-strategy game: Clash Royale.

Note: This article is written to be accessible to anyone who knows basics of machine learning, so it attempts to detail RL lingo and concepts.

Here’s a quick refresher on what Clash Royal, RL and self-play are if someone doesn’t know or has been rusty on them. Have a glance at them anyways:

• Clash Royale (CR)

  It’s a 1v1 tower defence mobile game first released in 2016 (and basically everyone knows this game from that era)

  This quick video below is a good demo of the game in its early days:

  https://www.youtube.com/watch?v=_hNxfiXmeAE

  Arena: The game has “arenas” which unlock incrementally based on your rating. Each arena unlocks new “cards” to be chosen from.

  Deck and Hand: Among many cards which you unlock, a fixed set of 8 can be chosen before you go into a game and only a hand of 4 cards are available at any point in the game to choose and deploy from.

  Partial information: The opponents don’t know about the other player’s elixirs and cards in their deck or their hand.

• Reinforcement Learning (RL)

  It is a form of machine learning where the model performs actions given an environment and a clear win condition. It’s typically used to learn playing games, popular ones include Chess, Go, Starcraft II, Dota 2 and of course, now CR!

  Every RL algorithm needs a few things:

  1. Environment: here, that’s the CR simulator.

  2. Agent: the one who interacts with the environment. Here there are two agents, one on each side.

  3. Observation or state: the agent must be able to see the environment in some capacity before acting on it. Here, its the whole game state with partial observability because each player doesn’t know the other player’s hand nor their elixir reserve.

  4. Action: an interface for the agents to interact with the environment.
     • Policy $\pi(a|s)$ is a conditional probability distribution used to sample an action given the current state, which can be approximated with a neural network.

  5. Reward: the agent must have some signal about how well it did. The simplest reward function would just be “did I win the game” at the end of the game. Though rewards can be given at each point in the game.

  Here’s a general representation of how the above interact:

  

  There are a lot of RL algorithms like DQN, REINFORCE, A2C, TRPO, PPO, SAC, DreamerV3, etc. each with there own pros and cons.

  We’ll dive a lot deeper into the algorithm and design decisions later.

• Self-play

  As the name suggests, both the agents are some instances of the same underlying policy which play off each other and learn through it.

  All sorts of weird and boring equilibria emerge through such a setup which needs careful mitigation. More on this later.

I built a very simplified clone of this game Pygame renderer. Only training camp cards are implemented as off now.

• But why CR of all games?

  The long term plan for this project (not in this version sorry :( ) is to study continual learning dynamics in RL, something I had been exploring in computer vision over the past few years but was now interested to implement in one of my favourite childhood games!

  Continual Learning (CL): its the ability of an agent to continuously adapt to new environments. This is a very general concept in machine learning, and arguably one of the most important dissection between humans and the current state of AI (even the frontier ones!).

  Here, this looks like this: imagine being in arena 9, you have a set of card all the way from training camp till now. Now when you move to arena 10, you’ll have a new set of cards. A good continually learning model will reuse the knowledge it knows about existing cards to learn playing the new cards quickly (aka forward transfer), while not forgetting how to play the old ones. There’s also a possibility to learn playing older cards better, which is called backward transfer.

  Now this is non-trivial, naively done, its well known that the distribution shift most often completely forgets the past ones unless we regularly force it to “revise” the past ones (among many other strategies). This is especially VERY hard in RL compared to computer vision lets say. But more about that in the next article!

The Simulator

• Why build a simulator from scratch? Why not learn from gameplay videos (TV Royale)?

  PPO is an on-policy algorithm, so it must collect fresh trajectories from the policy currently being trained. A simulator was therefore required to let the agents play games and produce new experience after every policy update.

  Recorded TV Royale matches are not a direct substitute. Learning from them would require an offline reinforcement learning or imitation learning pipeline, along with reliable extraction of actions, rewards, timing and partially hidden game state from video.

  The simulator also exposes exact actions, rewards, terminal states and privileged debugging information, which gameplay footage does not provide.

Scope: this is a performant RL playground inspired by Clash Royale, not a faithful clone. Rules and mechanics are simplified where useful for training speed or experimentation.

Here’s a gameplay video of two agents randomly choosing cards, and deployment position:

Implementation Details

Implemented rules

• Eight Training Camp cards on an 18 by 32 tile arena.
• The agents make four decisions per second.
• Elixir generation doubles at 2:00, then triples at 3:00.
• At 3:00, the simulator enters a custom sudden-death phase. At 5:00, the lowest-health tower determines the tiebreaker.

• Cards and towers

  Crown towers use level 9 statistics, while cards use level 11 statistics.

  Stats picked from the actual game at the time of writing. For unavailable stats, followed CR Wiki.

  • Feel free to skip this dropdown if you’ve ever played this game. Presented below for completeness:

    • Knight:

      

      • Ground troop
      • Melee attack
      • Targets ground troops and buildings

    • Giant:

      

      • Ground troop
      • Melee attack
      • Targets buildings only

    • Mini PEKKA

      

      • Ground troop
      • Melee attack
      • Targets ground troops and buildings

    • Musketeer

      

      • Ground troop
      • Ranged attack
      • Targets ground, air troops and buildings (all)

    • Minions

      

      • Air troop
      • Ranged attack
      • Targets all

    • Archers

      

      • Ground troops (2x)
      • Ranged attack
      • Targets all

    • Fireball

      

      • Spell
      • Area damage to all entities

    • Arrows

      

      • Spell
      • Area damage to all entities

• Game Phase Rules
  • 0:00 to 2:00 : 1 elixir every 3 sec
  • 2:00 to 3:00: 2x elixir
  • 3:00 to 5:00: 3x elixir and custom sudden death, where destroying any remaining tower ends the game immediately
  • 5:00: the player owning the lowest-health tower loses. Equal minimum health produces a draw.

Intentional simplifications

• Card and tower levels are fixed instead of being configurable.
• The card pool is restricted to eight Training Camp cards.
• Deployment uses a static own-half mask, keeping the position distribution consistent throughout a game.
• Movement, collision and targeting physics approximate the official game rather than reproducing it exactly.

Known simulator defects

• Deployable territory does not expand after a princess tower falls.
• Spells incorrectly share the troop deployment mask, so they cannot target the opponent half.
• Bridge and wall pathfinding can still produce edge-case routes that differ from the official game.

Fidelity matrix

You can play against a random bot or trained run 30/run 31 checkpoints in the browser here: Clash Royale web play demo.

The Environment

The simulator is exposed through a standard Gymnasium environment. Gymnasium defines the raw observation and action spaces, while a wrapper flattens and normalizes observations before they enter the policy network.

Each policy sees its own cards, towers, hand, next card and elixir, plus the opponent’s deployed cards and towers. The opponent’s hand, next card and elixir remain hidden.

The simulator runs at 4 FPS resulting in a full overtime game of 300 * 4 = 1200 frames .

TL;DR:
Each entity becomes a normalized 26-feature vector.
Variable deployed-entity sequences are padded to 32 slots per player.
An action is composed of skip | card | position decisions.
The default reward is zero-sum and combines tower damage, tower destruction and the terminal outcome.

Reward function:

The environment returns one reward per player at every step. The default reward is zero-sum: tower damage, tower destruction and game outcome are equal and opposite between players.

• Tower damage reward: each time our troop inflicts damage to any of the opponent’s towers, we give a positive reward proportional to $\Delta\text{tower\_hp}$, and likewise negative reward upon receiving such.

• Tower destruction reward: when we destroy any tower of our opponent’s we get a reward bonus and similarly penalty if our tower is destroyed.

• Winning reward: upon winning, at the last frame, the winner gets a sizeable positive reward. Likewise, equal and opposite when the game is lost.

• Step penalty: each step during the game, inflicting a small negative reward is generally done to encourage ending the game sooner.

Weighted sum of the above components is the effective reward function the agent receives at each step.

The intended priority of the cumulative reward components is:

This balances low-frequency outcome signals against high-frequency shaping signals. The terminal outcome remains the clearest objective, while damage and destruction provide earlier feedback about actions that move toward it.

Reward shaping provides denser feedback because win and loss alone can be too sparse early in training. Too much shaping can encode unwanted inductive biases and lead the policy toward strategies that optimize the proxy rather than winning.

The current default configuration is:

Current implementation summary

The environment returns one reward per player on every step. Tower damage, tower destruction and game outcome are equal and opposite between players. The optional step penalty is applied to both players, but its current default is zero, so the default total reward is zero-sum.

The terminal outcome reward also applies when the 5:00 time limit is reached and the hit-point tiebreaker selects a winner. A true draw receives no terminal bonus.

Step-penalty evidence is inconclusive: run 13 was slightly worse, while run 19 improved some curves but did not materially change the termination-to-truncation ratio. The default therefore remains 0.0.

Performance Optimisation

Profiling showed that rollout collection, not the PPO update, was the main bottleneck. The largest measured improvement came from reducing repeated pathfinding work.

The percentages are each phase’s share of collection, GAE and PPO-update time combined. Even after the optimisation, rollout collection consumed 96.1% of the measured time, so simulator work remained much higher leverage than network-update micro-optimisations.

What changed: the arena now caches its 18 × 32 ground and air occupancy grids and the corresponding free-cell lists. These caches are rebuilt only when a building changes the grid. Troops also reuse existing waypoints, recomputing a path only when the target moves by more than one tile, the occupancy grid changes, the path is exhausted, or a periodic refresh is reached. This removes repeated grid reduction and path searches from most simulator ticks.

Rollout inference is separately batched on the CPU, while PPO minibatch updates run on the GPU. This avoids many small accelerator calls during environment stepping and reserves the GPU for the larger training batches. This device split is useful, but it is not part of the isolated pathfinding speedup reported above.

The Algorithm

The PPO update, GAE, rollout storage, self-play checkpoint matchmaking and model architectures were implemented directly with PyTorch. The project does not use an external RL training framework such as Stable-Baselines3, CleanRL or TorchRL. It still uses Gymnasium for the environment interface, Pygame for simulation and rendering, and Weights & Biases for experiment logging. This section focuses on the directly implemented training and architecture decisions. The basic PPO derivation is explained in the appendix below.

Network

Deep Sets Network

The observation contains a fixed set of towers and cards in hand, but the number and ordering of deployed entities change throughout a match. A Deep Sets style network handles this by encoding every entity with the same function, then pooling the deployed entities into fixed-size summaries. Towers, the hand and the next card retain their explicit slots because their ordering has known meaning.

The default model uses separate actor and critic encoders and trunks. Both receive the same observation, but they learn independent representations for choosing actions and estimating state value.

Observation
	├── game completion fraction: 	(B, 1)
	├── elixir: 					(B, 1)
	├── my deployed entities: 		(B, 32, 26)
	├── opponent deployed entities: (B, 32, 26)
	├── my towers: 					(B, 3, 26)
	├── opponent towers: 			(B, 3, 26)
	├── hand: 						(B, 4, 26)
	└── next card: 					(B, 1, 26)

All entities: (B, 75, 26)
	→ shared-form entity encoder applied independently to every entity
		Linear(26, 64), LayerNorm, Tanh
		Linear(64, 64), LayerNorm, Tanh
		Linear(64, 32), LayerNorm, Tanh
	→ entity embeddings: (B, 75, 32)
	→ 4-head self-attention
	→ residual connection and LayerNorm
	→ contextual entity embeddings: (B, 75, 32)

Trunk input: (B, 418)
	├── game completion fraction: 					1
	├── elixir: 									1
	├── my towers, flattened: 						3 × 32
	├── opponent towers, flattened: 				3 × 32
	├── masked mean of my deployed entities: 		32
	├── masked mean of opponent deployed entities:  32
	├── hand embeddings, flattened: 				4 × 32
	└── next-card embedding: 						32

Actor path
	→ actor trunk: 418 → 209 → 209 → 128
	→ pointer query: (B, 128) → (B, 32)
	→ candidates: [learned skip token, four hand embeddings] = (B, 5, 32)
	→ query-candidate dot products: (B, 5)
		├── candidate 0: skip logit
		└── candidates 1 to 4: card logits
	→ sample the card
	→ concatenate actor trunk output with chosen card embedding: (B, 160)
	→ transposed-convolution position decoder
	→ position logits: (B, 18 × 32)

Critic path
	→ critic trunk: 418 → 209 → 209 → 128
	→ linear value head
	→ state value: (B,)

Fallback variants
	├── linear skip and card heads instead of the pointer decoder
	├── position prediction without the chosen card embedding
	├── linear 576-way position head instead of the CNN decoder
	└── shared actor-critic encoder and trunk

Entity encoding and pooling

Every entity is represented by the same 26 normalized features and passed through the same three-layer MLP. This weight sharing means that the representation of a Knight does not depend on whether it occupies the first or tenth deployed-entity slot.

The deployed entities are padded to 32 slots per player. Zero-padded slots are excluded using a mask, then the remaining embeddings are averaged. Masked mean pooling makes the battlefield summary invariant to entity ordering and keeps its scale relatively stable as troop count changes.

This pooling is also the main information bottleneck. A mean does not directly preserve entity count, the strongest individual activation or multimodal groups of troops. Mean plus max pooling, learned pooling or a spatial feature-map encoder would retain more information, but the current model does not implement them.

Towers, hand cards and the next card are not pooled. Their embeddings are flattened into the trunk so the model can distinguish the left tower from the right tower and each current hand slot. Hand and next-card entities also participate in self-attention before entering the trunk. This gives them access to battlefield context, although it also creates a direct trunk path and an attention-mediated path for the same information.

Entity attention

The attention layer lets every encoded entity use information from every other entity before pooling. For example, a troop embedding can change depending on nearby enemies, towers and cards in hand.

The implementation uses one four-head MultiheadAttention layer, followed by a residual connection and LayerNorm. It is not a complete Transformer block because it has no attention-specific feed-forward sublayer. Zero-padded entities are masked as keys and values so they do not affect valid entities.

Disjoint actor and critic

The default model constructs separate entity encoders, attention layers and trunks for the actor and critic. The actor can therefore learn features useful for comparing actions, while the critic can learn features useful for predicting return. In a shared network, value-loss gradients also modify the representation used by the policy, which can create gradient interference.

The tradeoff is cost: the disjoint configuration nearly doubles representation compute and parameter count. It is a plausible design choice, not an isolated empirical conclusion in this project.

Pointer card decoder

A normal linear card head predicts four logits tied to the four hand positions. The pointer decoder instead derives a query from the current actor state and compares it with five candidates: one learned skip token and the four contextual hand embeddings.

The dot product between the query and each candidate produces one skip logit and four card logits. Card selection is therefore based on the encoded properties of the available cards, rather than only on fixed hand-slot identities. The same mechanism also gives skip a representation that competes directly with playing a card.

Card-conditioned position decoder

Where a card should be deployed depends strongly on which card was selected. The model samples a card, retrieves that card's 32-dimensional hand embedding, concatenates it with the 128-dimensional actor trunk output, then predicts the deployment location from the resulting 160-dimensional vector.

The default position head is a transposed-convolution decoder that expands this vector into an 18 × 32 arena grid. This gives the output a spatial structure that a single 576-way linear layer does not encode. It also has substantially more capacity than the linear alternative, so comparisons between them are not parameter matched.

The decoder still reconstructs the complete spatial grid from entity coordinates embedded in vectors. A spatial input representation would provide a more direct correspondence between battlefield locations and output locations.

Action masking and probability calculation

The environment action contains three values: skip, card and position. Cards costing more than the current elixir are masked before sampling. If all four cards are unaffordable, skip is forced with a large finite logit, and the card logits are reset to finite values to avoid invalid categorical distributions.

When skip is selected, the sampled card and position have no effect on the environment. Their log probabilities and entropy terms are therefore excluded from the PPO objective for that step. This prevents the policy from being trained on arbitrary card and position samples attached to a skip action.

During rollout collection, the network samples actions. During PPO updates, the stored actions are passed back into the same distributions so their new log probabilities, entropy and value estimates can be evaluated.

Normalization, activation and initialization

The entity encoder and trunk use LayerNorm after every linear layer, followed by Tanh. Layer normalization limits changes in activation scale across observations, while Tanh keeps hidden activations bounded. These are stability-motivated choices, but this project has not isolated their effects against ReLU or ELU without changing other architecture settings.

Linear layers use orthogonal initialization. Hidden layers use gain sqrt(2), actor output projections use 0.01, and the critic output uses 1.0. Small actor-output initialization begins with relatively uncommitted action logits, while the larger critic scale avoids constraining initial value predictions to the same narrow range. The CNN position decoder's final layer also uses 0.01.

Skip, card and position logits each have a learned positive temperature. The parameters are initialized to zero and transformed with softplus, so the initial temperature is approximately 0.693, not exactly 1.0. Each action component can then independently adjust how sharp or diffuse its distribution becomes.

Transformer Network

The Transformer replaces Deep Sets pooling with a learned sequence representation. Every tower, deployed entity, hand card and next card becomes a token, allowing repeated attention blocks to model interactions before producing a global state representation.

Entity features
	├── six towers: 				(B, 6, 26)
	├── my deployed entities: 		(B, 32, 26)
	├── opponent deployed entities: (B, 32, 26)
	├── hand: 						(B, 4, 26)
	└── next card: 					(B, 1, 26)
	→ linear entity projection:     26 → 64
	→ entity tokens: 				(B, 75, 64)

Meta features
	├── game completion fraction: (B, 1)
	└── elixir: 				  (B, 1)
	→ concatenate: 				  (B, 2)
	→ linear projection and Tanh: (B, 64)
	→ add to learned CLS token

Token sequence: (B, 76, 64)
	├── meta-enriched CLS token
	├── six tower tokens
	├── 64 deployed-entity tokens
	├── four hand tokens
	└── one next-card token
	→ add segment embeddings
		├── CLS
		├── towers
		├── deployed entities
		└── hand and next card
	→ mask zero-padded deployed-entity tokens
	→ two pre-LayerNorm Transformer encoder blocks
		├── 4-head bidirectional self-attention
		├── feed-forward width: 4 × 64 = 256
		└── zero dropout
	→ contextual token sequence: (B, 76, 64)

Outputs
	├── CLS token: 					 (B, 64)
	│	→ post-Transformer MLP:      64 → 128 → 128
	│	→ global trunk output:       (B, 128)
	└── four contextual hand tokens: (B, 4, 64)

Token construction

Each 26-feature entity is projected directly to a 64-dimensional token. Unlike Deep Sets, the Transformer does not average deployed entities into one vector before reasoning about them. Each entity remains available as an individual token throughout the encoder.

The sequence contains 76 positions: one CLS token, six towers, 64 padded deployed-entity slots, four hand cards and one next card. Zero-padded deployed slots are hidden with a key-padding mask.

Meta-enriched CLS token

Game completion and elixir are projected together from two scalars to 64 features, then added to a learned CLS token. This gives the global token immediate access to match phase and available resources without allocating another sequence position.

After the Transformer, the encoded CLS token acts as the global state summary. A two-layer MLP maps it from 64 to 128 dimensions before the actor or critic head consumes it.

Segment embeddings

The same entity projection is used for every card and tower, so learned segment embeddings identify the role of each token. Four segment types distinguish CLS, towers, deployed entities, and hand plus next-card tokens. Individual entity features still encode properties such as ownership, position and card type.

Transformer encoder

The default encoder contains two full Transformer blocks with four attention heads and a 256-dimensional feed-forward sublayer. Attention is bidirectional: every unmasked token can use information from every other unmasked token.

The blocks use pre-LayerNorm and zero dropout. Pre-LayerNorm improves gradient flow through stacked blocks, while dropout was disabled to avoid injecting noise into policy and value predictions. These are stability-motivated choices rather than isolated conclusions from this project.

Contextual hand tokens

The encoded hand tokens are retained alongside the CLS output. They have attended to the complete game state, so the pointer decoder compares the global actor query against card representations already conditioned on towers, troops, elixir and match phase.

The remaining actor-critic split, pointer card decoder, card-conditioned CNN position decoder, action masking, learned temperatures and initialization are the same as in the Deep Sets network.

Training Algorithm

Training alternates between collecting fresh self-play experience and reusing that rollout for several PPO epochs. Environment workers and rollout inference run on the CPU. The merged rollout is then shuffled into minibatches and used to update the main network on the selected training device. The implementation follows the clipped PPO objective derived in the appendix, with GAE for credit assignment and an Elo-rated checkpoint pool for opponent selection.

Repeat until 10M environment steps
	├── copy the current policy to the CPU rollout network
	├── step 8 environment workers synchronously
	│	├── current policy controls player 2
	│	└── sampled checkpoint controls player 1
	├── store states, actions, old log probabilities, rewards and values
	├── compute GAE separately for every environment trajectory
	├── merge the per-environment buffers
	├── optionally filter or normalize the rollout
	├── run 8 PPO epochs over shuffled minibatches of 2,048 transitions
	└── update diagnostics, Elo ratings and the checkpoint pool

Rollout buffer and GAE

RolloutBuffer stores the observation, sampled action, old policy log probability, reward, value estimate and termination flag for every transition. After collection, it computes generalized advantage estimates (GAE) backwards through each trajectory, then forms critic targets as return = advantage + value. Time-limit truncations remain bootstrapable, while true terminal states stop the recursion.

The discount factor γ = 0.997 determines how much distant reward matters. The GAE factor λ = 0.95 controls the bias-variance tradeoff: lower values rely more on the critic and reduce variance, while values nearer one use longer sampled returns and reduce bias. Advantages are normalized independently inside every minibatch before the policy loss is computed.

Rollout length

The rollout target is calculated as one full game's maximum number of frames multiplied by num_games_in_buffer. With a 1,200-step maximum game and the CLI default of eight games, each PPO update collects 9,600 transitions across all workers.

Larger rollouts produce more varied and less correlated training data, but increase the delay between policy updates and require more memory. Smaller rollouts update more frequently, but each batch has noisier advantage estimates and is easier to overfit. The useful target is therefore measured in complete games, not an arbitrary power-of-two step count.

Parallel rollout collection

ParallelEnvManager starts one independent Gymnasium environment in each worker process. The parent sends one action to every worker, then waits for all results. This synchronous design is simple and keeps each update based on policies from the same point in training, although every step is limited by the slowest worker.

The current default is 8 workers. Active-agent observations are batched for one CPU network call, while opponent policies are evaluated separately. A distinct buffer is maintained for each worker so episode boundaries and GAE recursion are not mixed across environments.

PPO minibatch update

For each stored action, on_batch_update recomputes its log probability under the current policy and forms the probability ratio r = exp(log π_new − log π_old). The actor loss is the negative minimum of the unclipped objective and the objective with r clipped to [0.8, 1.2]. This is the same clipped surrogate objective developed in the PPO appendix.

The critic minimizes mean squared error against the GAE return target. The complete minimized loss is actor loss + 0.5 × critic loss − 0.01 × entropy. The critic coefficient prevents the numerically larger value error from dominating the shared optimization step. The entropy term rewards broad action distributions and slows premature policy collapse.

Optimizer and learning rate

Adam updates all trainable parameters. Its moving estimates of gradient mean and variance give each parameter an adaptive step size, which is useful here because the actor heads, critic and shared representations can produce gradients on different scales. Adam is a stability-motivated default, not an optimizer comparison established by this project.

The CLI default starts at 1.5 × 10⁻⁴ and decays linearly to zero as global_step approaches max_steps = 10,000,000. The same progress fraction can interpolate the entropy coefficient, although both current endpoints are 0.01, so it remains constant.

Optional learning-rate finder

The learning-rate finder is disabled by default. When enabled, it runs once on the first rollout using a temporary network and Adam optimizer. It increases the rate exponentially from 10⁻⁷ to 10⁻¹ over at most 200 minibatches, tracks an exponentially smoothed critic loss, stops after clear divergence, then selects 30% of the rate with the lowest smoothed loss.

Only critic loss is used as the search signal because large trial steps immediately invalidate PPO's policy ratio and make the combined actor objective a poor learning-rate probe. The selected rate becomes the starting point for the normal linear schedule.

Epoch count, clipping and KL control

Each rollout is reused for eight PPO epochs. More epochs extract more optimization from expensive simulator data, but they also move the policy farther from the behavior policy that generated that data. The clip coefficient 0.2 limits how much any sampled action can influence the surrogate objective once its probability ratio moves outside the accepted interval.

Approximate KL divergence measures the aggregate policy shift after each minibatch. Optional KL early stopping ends the epoch loop when an epoch's mean approximate KL exceeds 0.01. It is disabled by default, so all eight epochs run. clip_fraction remains the most direct signal for tuning rollout reuse: very low values mean the data is underused, while high values mean many samples have already reached PPO's trust-region boundary.

Gradient clipping

The default applies one global gradient-norm limit of 0.5 across the complete network. This preserves the relative contribution of every component while preventing a single unstable minibatch from producing an excessive Adam step.

The optional per-head mode separately clips critic, position, skip, deck, pointer and remaining backbone parameters. It prevents a large head, especially the 576-way position decoder, from using the entire norm budget. The tradeoff is that independently clipped groups can have a combined norm greater than 0.5. It is therefore retained as an ablation and disabled by default.

Elo self-play and checkpoint sampling

The current policy and every saved checkpoint begin at Elo 1,200, with scale 400 and update factor 32. After each game, both the active policy and its checkpoint opponent are updated from the result and expected score. Elo is used as a matchmaking estimate inside one evolving training run, not as an absolute measure comparable across unrelated runs.

Here R is Elo rating, E is expected score, and S is the observed score: 1 for a win, 0.5 for a draw and 0 for a loss. The opponent uses the same update with the player labels reversed.

A new checkpoint is admitted only after at least 100 games and a recent mean score of at least 0.55. For each new opponent, the sampler chooses the latest eligible checkpoint with 50% probability. Otherwise it weights checkpoint Elo values toward an expected 50-50 matchup, then samples a checkpoint at the chosen rating. Checkpoints already active in other workers are avoided when alternatives exist, increasing opponent diversity.

Optional rollout preprocessing

Forced-skip removal is disabled by default. When enabled, transitions where the agent cannot afford any card are removed before PPO. This avoids training heavily on actions with no meaningful choice, but it also changes the sampled state distribution and discards value-learning data from low-elixir states.

Observation normalization is also disabled. Its optional implementation maintains exponential moving mean and standard-deviation vectors from complete rollouts and applies them during both collection and updates. Value normalization, also disabled, standardizes return targets over the current rollout. The raw entity features are already normalized by the environment wrapper, while enabling value normalization would require the critic's inference scale to be handled consistently.

Update diagnostics

The PPO update logs the following aggregate measurements. They are diagnostic signals, not objectives that should be optimized independently.

A separate diagnostics logger records Elo, returns, scores, game endings, tower kills, average elixir, skip frequency, card and position usage, and per-head entropy and KL. These distinguish an optimizer problem from policy collapse, degenerate action usage or changes in opponent difficulty.

Default training configuration

This table freezes the defaults used by the current command-line training entry point. Historical runs changed several of these values, so their experiment records should be read as overrides rather than descriptions of the final default system.

Experiments

The experiments had three roles: verify that the training loop could overfit simple opponents, diagnose failed runs, and compare the final network variants in a common evaluation pool.

How to read the training diagnostics

No single curve defines a good run. A healthy run should improve against a fixed opponent, retain multiple viable actions, learn a useful value function and update the policy without repeatedly hitting PPO's clipping limits.

BotNet overfitting tests

Before full self-play, the learner was tested against opponents whose behavior did not change during training. This isolates whether PPO can learn the environment at all.

For these tests, score and win rate are meaningful because opponent strength is fixed. A run should not be called solved from its training curve alone: the final checkpoint should be evaluated in fresh games from both player sides and several seeds.

Pivotal changes

The appendix records many exploratory runs. The following changes were the ones that materially changed the direction of the project.

• Checkpoint opponent pool: early self-play against only the latest policy collapsed into both agents preferring not to play. Sampling older checkpoints introduced strategic diversity and prevented this immediate equilibrium.
• Conditional action probabilities: card and position log probabilities and entropy were excluded on skip actions. This stopped PPO from training irrelevant action components when no card was played.
• Longer credit assignment: increasing γ from 0.99 to 0.997 produced the first major improvement against the scripted opponent.
• Elixir-aware action masking: unaffordable cards were masked and skip was forced when every card was unaffordable. This produced the largest clear behavioral jump, taking the random-bot run beyond 80% training win rate.
• Dynamic checkpoint Elo: saved opponents initially kept their original Elo forever, creating misleading rating growth. Updating both learner and checkpoint ratings made matchmaking and within-run Elo more coherent.
• Entity-based networks: the final system moved from early flat actor-critic models to Deep Sets and Transformer representations that could process a variable battlefield while retaining explicit hand and tower information.

Deep Sets and Transformer training logs

The recorded dashboard below compares the complete 10 million-step run 30 Deep Sets model with the run 31 Transformer. Only the exported dashboard image is available locally, not the underlying history, so exact windowed statistics cannot be recomputed.

Shared pattern: both runs reached a self-play score around 0.6, positive return and high explained variance. Neither curve establishes superior playing strength because the runs trained against different checkpoint pools.

Deep Sets: explained variance rose quickly to roughly 0.95. Entropy declined gradually to about 0.1, while critic loss remained mostly near 0.1 before becoming noisier late in training. Return peaked around the middle of the run and declined toward the end even as within-run Elo continued rising. This divergence means the final checkpoint should be tested directly rather than selected from Elo alone.

Transformer: explained variance learned more slowly but eventually approached 0.9. Its entropy fell much closer to zero, indicating a substantially more deterministic policy. Critic loss stayed much larger and noisier than Deep Sets, approximately 0.2 to 0.4 for much of the run. Score and return remained comparable or slightly higher near the end, but the recorded gameplay was more passive. This can be a stable self-play equilibrium without being the strongest policy against other architectures.

Final contestant-pool results

Training Elo is not used for the final comparison. Freeze one checkpoint from each approach, reset every contestant to Elo 1,200, and let all contestants play in the same pool with equal games and both player-side assignments. The final reported result is the Elo reached by each approach in this shared pool.

Ablations

Each ablation changes one component relative to a fixed baseline. The main result is playing strength: the ablated checkpoint and baseline are placed in the same evaluation pool, initialized at the same Elo and given equal games from both player sides.

Simulator fidelity

Reward

Log HP-delta, tower-destruction, terminal-outcome and step-penalty contributions separately, alongside win rate, termination rate and episode duration.

Network

Architecture comparisons must report parameter count and inference cost. Prefer parameter-matched variants where possible.

Training

Ablation protocol

Use identical environment versions, training steps, reward settings and matched seeds. Change one component at a time and train at least three seeds per variant. Freeze the final checkpoints, then evaluate the ablated variant and baseline in the same Elo pool with equal games and both player-side assignments. Report mean and uncertainty across seeds.

Prioritize the bundled run-25 to run-26 changes, then the pointer decoder, entity attention and Deep Sets versus Transformer comparisons. Training curves alone do not complete a TODO.

Appendix

PPO: Proximal Policy Optimisation

This section attempts to explain arguably the most popular online RL algorithm used out there.

It builds from the basics of RL → REINFORCE → A2C → PPO.

Basic RL Terminology

At every timestep $t$, the agent:

1. Observes state $s_t$ from the environment

2. Picks action $a_t$ using its policy $\pi(a|s_t)$

3. Environment transitions to $s_{t+1}$ and emits reward $r_t$

An entire RL system can be described mathematically with a Markov Decision Process:

• Markov property

  $s_{t+1}$ depends only on $s_t$ and $a_t$, not on history. The state is a sufficient statistic.

$\mathcal{M} = (\mathcal{S},\ \mathcal{A},\ P,\ R,\ \gamma)$

The return $G_t$ is the discounted sum of future rewards from time $t$:

This project is more precisely a partially observable MDP. The simulator state may be Markov, but the agent cannot observe the opponent's hand or elixir, so its current input is not sufficient to recover the full state. The feed-forward policy is therefore reactive: it chooses an action from only the current observation.

Frame stacking, recurrent memory and opponent modeling are natural extensions because they could use recent behavior to infer hidden information. In the terminology below, s_t should be read as the observation available to the policy unless the full simulator state is explicitly mentioned.

REINFORCE Algorithm

This is an algorithm on top of which PPO builds.

The objective function we care about is:

The objective uses the same discounting as the return defined above. More explicitly, J(θ) = E[Σₜ γᵗrₜ]. The displayed Σₜrₜ is only valid for the special case γ = 1.

Where $\tau = (s_0, a_0, r_0, s_1, \ldots)$ is a full episode trajectory.

But $\nabla_\theta p_\theta(\tau)$ is not something we can sample directly.

We can only sample trajectories $\tau$, not their probability gradients.

Here we use the Log Derivative Trick:

Substituting back:

Expand $\log p_\theta(\tau)$:

Using $\nabla_\theta \log p_\theta(\tau) = \sum_t \nabla_\theta \log \pi_\theta(a_t|s_t)$ :

For $t' < t$: reward $r_{t'}$ happened before action $a_t$ was taken. It carries no information with respect to $\pi_\theta(a_t|s_t)$. Hence, all $t' < t$ cross terms vanish.

The Algorithm

Estimate the gradient with a single sampled trajectory, then do gradient ascent:

1. Run episode under $\pi_\theta$, collect $(s_t, a_t, r_t)$

2. Compute returns $G_t = \sum_{k \geq t} \gamma^k r_{t+k}$: future return from each step, not $G_0$

3. Update: $\theta \leftarrow \theta + \alpha \sum_t G_t \nabla_\theta \log \pi_\theta(a_t|s_t)$

Actor Critic Algorithm (A2C)

REINFORCE uses $G_t$ . That’s a full Monte Carlo (MC) return, as the weight on $\log \pi_\theta$. This is unbiased but high variance: every future step adds noise, and a single bad episode can produce a catastrophic gradient step.

The update rule can be written like below:

Where $b(s_t)$ is any function depending only on $s_t$. The above estimator is unbiased since taking expectation on $b(s_t)$ doesn’t affect the estimate since expectation is over the actions taken by the policy.

To fix the variance issue, use $b(s_t) = V_\phi(s_t)$.

Define Advantage as follows:

By minimising $\sum_t A(s_t, a_t)^2$ , the value function: $V_\phi(s_t) \approx V^\pi(s_t)=E[G_t|s=s_t]$

Advantage is a measure of how better or worse than expected did this transition turn out to be.

We could also use TD Error:

The above approach has a lower variance than MC at the cost of some bias from $V_\phi$ being imperfect.

The Algorithm

The actor learns the policy while the critic learns how well the actor is performing by estimating the value function.

Entropy Regularisation

Higher entropy means the actions are more spread out in the action space and vice versa.

When entropy collapses early into training, this is essentially foreclosure to a suboptimal minima. To prevent this we add an entropy bonus as follows:

Although actor-critic diagrams often share one backbone, the implemented default uses disjoint actor and critic networks. This prevents value-loss gradients from reshaping features used to choose actions, and prevents policy updates from destabilizing the value representation. The tradeoff is nearly twice the representation compute and parameter count.

PPO Algorithm

One of the biggest issues with A2C now is its sample inefficiency. Just can’t reuse the transitions because they quickly get stale since we use the underlying log_probs of a policy that changes every gradient ascent.

To mitigate this, PPO uses importance sampling.

While collecting transitions, save the log_probs of the old policy and evaluate the following ratio:

Here, the probability of a complete action follows the action's conditional structure. The joint log probability is the skip log probability plus the card and position log probabilities only when the agent does not skip. On skip steps, card and position choices are irrelevant and contribute zero. Entropy is combined in the same way, preventing irrelevant card and position entropy from dominating skip actions.

Now optimise the following surrogate objective:

1. $\hat{A_t}$ is Generalised Advantage Estimate defined by: $\hat{A}_t^{\text{GAE}} = \sum_{k=0}^{T} (\gamma\lambda)^k\, \delta_{t+k}$ which interpolates between 1-step TD error and full MC error.

2. PPO clips the ratio between a small range around 1. Usually $\epsilon = 0.2$.

3. Out side the clipped range, PPO takes a pessimistic estimate to not over exploit a good action nor be overly punished for a bad one resulting in moderate gradients hence stable training.

The Algorithm

1. Collect rollouts across which include transitions in multiple episodes.

2. Compute GAE backwards on each ordered trajectory, then shuffle transitions into random mini-batches. Shuffling reduces correlation, but SGD and Adam do not require perfectly IID samples. The ordering requirement belongs to GAE, which must be computed before shuffling.

3. Train for multiple epochs on the same rollout before discarding it.

4. The underlying loss function would be as follows:

In the implementation, advantages are normalized to zero mean and unit variance within each mini-batch. Return normalization is optional and uses rollout-level statistics, while observation normalization is a separate optional feature based on running statistics.

The optimized loss uses the clipped policy objective, an unclipped critic mean squared error MSE(return, value), and an entropy bonus. The squared-advantage critic term shown above is only a simplified actor-critic form, not the implemented critic target. Optional KL early stopping can end the remaining PPO epochs when approximate KL exceeds kl_threshold. It is an additional safeguard, not part of the base clipped objective.

Notation and implementation names

How the model and training evolved

Pipeline debugging: runs 1 to 6

• Run 1: used only three cards, Knight, Mini P.E.K.K.A. and Giant. The position head was a Linear layer rather than a CNN.

• Run 2: both agents converged on “no play” as the dominant strategy.
  • Cause: self-play always used the current model as its own opponent.
  • Fix: added an opponent pool and sampled across multiple checkpoints.

• Run 3: entropy collapsed and training stalled.

  

  • The next configuration raised entropy_weight from 0.005 to 0.05.
  • Only 2 to 4 games were collected per buffer. Later configurations increased rollout size toward the 10 to 100 game range, added parallel collection, and raised the minibatch size. These changes were bundled, so recovery cannot be attributed to entropy weight alone.

• Run 4: trained against a random agent as an overfit test.

  

  • Observations:
    • EMA score never exceeded 0.6.
    • Elo, loss and entropy oscillated.

• Run 6:
  • Changes from run 4:
    • Excluded card and position entropy on skip actions.
    • Redesigned reward: tower HP deltas (1.0 to 2.0), tower kill bonus (0.5) and game outcome (5.0).
    • Added per-minibatch advantage normalization.

  

  • Explained variance stayed below 0.5, and the other metrics remained poor.
  • At this point, none of the changes had produced reliable learning.

Fixed-opponent validation: runs 7 to 13

• Run 9: trained against a scripted bot that alternated Knight placements between the left and right lanes.

  

  • Still showed no clear improvement.
  • Working hypothesis:
    • Critic loss was spiky. Reward scale was a suspected cause, but this was not isolated or demonstrated.

• Run 12: changed gae_gamma from 0.99 to 0.997.

  

  • Performance improved substantially.

• Run 13: added a 0.001 step penalty.

  

  • Performed slightly worse, so the step penalty was removed.

• Milestone: the agent reached a high score against the scripted opponent, so training moved to vs-random.
  • Run 13 gameplay at approximately 1.8 million steps.
  • The scripted player is blue and the trained agent is red. The right panel shows diagnostics: green bars are card probabilities, the red overlay shows position logits, the blue square marks the maximum logit, and the yellow square marks the sampled position.

  attachment:39513fb9-e9a5-45d1-8540-f60764e70284:step1780000-episode-0.mp4

Random-opponent scaling: runs 14 to 23

• Run 14: trained against a random opponent.

  

  • Training was stable, but score never exceeded 0.6.

• Run 16: masked unaffordable cards and forced skip when no card was affordable.

  

  • Produced a major improvement, exceeding an 80% training win rate.

• Run 17: run 16 with approximate KL early stopping.
  • Performed worse than run 16.

• Run 18: replaced per-minibatch advantage normalization with running mean and standard deviation.
  • Performance collapsed.

• Run 19: run 16 with a 0.001 step penalty.
  • The termination-to-truncation ratio barely changed, so the penalty was not retained.

• Run 20: run 16 with 100 games per rollout and forced-skip transitions removed, leaving roughly 7,000 samples, similar to run 16.
  • Performed poorly. The larger early rollouts were expensive, but this run bundled rollout size with forced-skip removal, so the cause was not isolated.

• Run 21: run 17 with the maximum PPO epochs raised to 120.
  • Performed poorly.

• Run 22: run 16 with minibatch size reduced to 256.
  • Improved over the run-16 family.

Self-play and the eight-card environment: runs 24 to 31

• Run 24: moved run 22 into full self-play training.

  

  • The near-quadratic Elo growth was a checkpoint-rating bookkeeping bug, not rapid policy improvement.
  • Fix: update checkpoint Elo ratings instead of freezing them when saved.

• Run 25: expanded to eight cards using the previous vs-random configuration.

  

  • Elo oscillated, indicating unstable progress.

• Run 26: changed the architecture bundle to Deep Sets with Tanh, LayerNorm, orthogonal initialization, disjoint actor and critic encoders, a CNN position decoder, and card information appended to the position head. Entity attention and the pointer decoder were disabled.

  

  • All recorded training statistics improved.
  • This configuration reached strong performance against the random opponent in the eight-card environment.

Eight-card full training

• Run 27: moved run 26 into full self-play training.

  

  • Elo increased steadily before plateauing.
  • Adopted as the new baseline.
  • Gameplay after 6.2 million steps:

    attachment:18e86b62-6416-45e6-8c4f-7eea44b2e4e1:step6240000-episode-0.mp4

• Run 28: added the pointer decoder.

  

  • Appeared worse than run 27, but this was not a controlled, multi-seed comparison.

• Run 30: added entity attention.

  

  • Curves appeared stronger than run 27, but no shared frozen head-to-head evaluation was run.
  • Gameplay at 10 million steps:

    attachment:9629386f-8c0b-4366-9a22-45a20cd3cc44:run-30_10M.mp4

• Run 31: replaced Deep Sets with the Transformer model.

  

  • The aggregate training metrics appeared reasonable.
  • Gameplay at 10 million steps:

    attachment:e8869d80-30c4-4e33-8334-f3c0fd2edfc0:run-31_10M.mp4

  • The recorded Transformer gameplay was more passive than Deep Sets: both sides settled into a balanced strategy and left more of the outcome to chance. This observation does not establish lower playing strength without a shared frozen evaluation.
    • Deep Sets may also have benefited from more random action selection, while the Transformer appeared to settle into a stable equilibrium.

Additional changes that can be made

Several architecture improvements, novel training concepts, and engineering adjustments were identified as high-potential directions for further performance improvements:

Deep Sets Enhancements

• Robust Entity Masking: Replace the fragile zero-vector masking heuristic (which assumes active entities can never have all-zero feature vectors) with an explicit boolean mask tensor passed directly from the environment to prevent silent unit dropouts.
• Full Transformer Blocks for Deep Sets: Add a feed-forward network (FFN) block (Linear-GELU-Linear) with residual connections and LayerNorm after the self-attention layer to bring Deep Sets attention to full Transformer expressiveness.
• Configurable Multi-Layer Attention: Stack multiple attention blocks (e.g., 2 layers) controllable via a num_attention_layers parameter. Ensure strict input validation to avoid silent configuration failures.
• Deduplicated Hand Card Representation: Remove hand cards and the next card features from trunk flattening when using attention, keeping the dual path optional under an explicit control flag.
• Combined Mean and Max Pooling: Augment set pooling by concatenating mean-pooled and max-pooled representations to capture both the average battlefield state and extreme outliers (such as the highest-threat unit).

Novel Architectural Concepts

• Spatial Feature Map Encoder: Represent the arena grid as an 18×32 multi-channel image (encoding HP, side, and troop types), and process it with a CNN before fusing with the entity encoder via cross-attention to provide better spatial inductive bias.
• Explicit Opponent Modeling: Add an auxiliary prediction head to predict the opponent's next action (skip, card, position), utilizing the gradient to enrich representation learning.
• Temporal Context & Memory: Add a recurrent unit (LSTM/GRU) or stack frames over the last K observations to break the single-frame reactive limitation and allow the model to learn units' velocity and intent.
• Mixture of Experts (MoE) Trunk: Replace the standard trunk MLP with a Mixture of Experts layer to dynamically route processing based on the game phase (e.g., defense vs. attack).