23. Self-Play Evaluation: Win-Rate Gate and Elo Update

Problem

AlphaGo Zero's training loop doesn't just keep improving one network forever — after each round of self-play training, the newly trained “candidate” network plays an evaluation match against the current “best” network, and only replaces it if it wins convincingly enough. The published gate was a ≥55% win rate over 400 games; below that, the old best network keeps generating the self-play data for the next round.

This problem asks for the evaluation-loop bookkeeping around that gate: given the match results and both networks' running Elo ratings, compute the observed win rate, update the candidate's Elo rating from the match outcome using the standard logistic Elo update, and decide whether the candidate should replace the current best. Treating the whole match as one batched Elo update (rather than one update per individual game) keeps this a single, clean vectorized computation.

Source: src/23_self_play_evaluation_gate.py

def self_play_evaluation_update(
    results: torch.Tensor, candidate_elo: torch.Tensor, best_elo: torch.Tensor,
    k: float = 32.0, win_rate_threshold: float = 0.55,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: ...

>>> results = torch.tensor([1.0, 1.0, 0.5, 1.0, 0.0, 1.0])
>>> candidate_elo, best_elo = torch.tensor(1500.0), torch.tensor(1500.0)
>>> win_rate, new_elo, replace = self_play_evaluation_update(results, candidate_elo, best_elo)
>>> replace.item()
True

Step-by-Step Approach

1. Compute the observed win rate as the mean of the per-game results (1.0 win, 0.5 draw, 0.0 loss).
2. Compute the Elo-implied expected score from the rating gap: 1 / (1 + 10^((best_elo - candidate_elo) / 400)) — how often the candidate “should” win given the ratings alone, before this match's outcome is known.
3. Update the candidate's Elo rating toward the observed outcome: candidate_elo + k * (win_rate - expected_score).
4. Apply the gate as a strict > comparison against the threshold, not >= — a candidate exactly at the threshold should not replace the incumbent.
5. Return all three quantities as separate tensors so the caller can log the win rate, persist the updated Elo, and branch on the replacement decision independently.

The Elo update and the replacement gate are two independent decisions computed from the same win_rate value — the gate only looks at whether win_rate clears a fixed threshold, while the Elo update also factors in how surprising that win rate was relative to the current rating gap, so a strong favorite winning narrowly barely moves its rating even if it clears the gate easily.

Reference solution

import torch


def self_play_evaluation_update(
    results: torch.Tensor, candidate_elo: torch.Tensor, best_elo: torch.Tensor,
    k: float = 32.0, win_rate_threshold: float = 0.55,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
    # observed score over the match: 1.0/0.5/0.0 per game, averaged
    win_rate = results.mean()

    # logistic Elo expectation: how often the candidate "should" win given
    # the rating gap, before this match's actual outcome is known
    expected_score = 1.0 / (1.0 + 10.0 ** ((best_elo - candidate_elo) / 400.0))

    # standard Elo update: move the rating toward the observed outcome,
    # scaled by k -- positive surprise (won more than expected) raises it
    updated_elo = candidate_elo + k * (win_rate - expected_score)

    # AlphaGo Zero's published gate is a strict >, not >=, on the threshold
    should_replace = win_rate > win_rate_threshold

    return win_rate, updated_elo, should_replace

Key Functions & Tricks

• results.mean() — the observed win rate over the evaluation match, from the candidate's perspective
• 10.0 ** ((best_elo - candidate_elo) / 400.0) — the core Elo logistic-curve term; a 400-point rating gap corresponds to a 10:1 expected-score ratio
• win_rate > win_rate_threshold — the AlphaGo Zero-style gate, deliberately a strict inequality so an exact-threshold result does not trigger replacement
• torch.tensor(1500.0) — a 0-d scalar tensor, the natural representation for a single running Elo rating that still participates in tensor arithmetic

How to Recognize This Pattern