26. Soft (Polyak) Target-Network Update

Problem

DQN's target network is frozen and copied over in hard, infrequent jumps (every N steps, replace it wholesale). Continuous-control methods like DDPG found that hard copies were too disruptive for their setting and instead update the target network a little every single step, via an exponential moving average of the online network's weights: target_param ← tau * online_param + (1 - tau) * target_param, with tau typically very small (e.g. 0.005), so the target drifts smoothly toward the online network instead of jumping.

This is the same “Polyak averaging” mechanism reused by nearly every actor-critic method with target networks since (TD3, SAC, and DDPG's own critic and actor target networks). It has to be applied per-parameter, since a real network's parameters come as a list of tensors of different shapes (weight matrices, bias vectors, ...), not one flat tensor.

Source: src/26_polyak_target_update.py

def polyak_update(
    online_params: list[torch.Tensor], target_params: list[torch.Tensor], tau: float,
) -> list[torch.Tensor]: ...

>>> online = [torch.ones(2, 2), torch.zeros(2)]
>>> target = [torch.zeros(2, 2), torch.ones(2)]
>>> polyak_update(online, target, tau=0.1)[0]
tensor([[0.1000, 0.1000],
        [0.1000, 0.1000]])

Step-by-Step Approach

1. Wrap the whole update in torch.no_grad() — a target-network update is bookkeeping between two sets of weights, never part of a training graph, and should never carry gradient.
2. Zip the two parameter lists together so each online tensor is paired with its corresponding target tensor by position.
3. Apply the exponential-moving-average formula elementwise to each pair: tau * p_online + (1 - tau) * p_target.
4. Collect the results into a new list, preserving the same order and shapes as the input lists.
5. Sanity-check the two edge cases: tau=1 should reduce to an exact hard copy of the online parameters, and tau=0 should leave the target parameters completely unchanged.

The update itself is a one-line elementwise formula; the only real engineering concern is applying it independently to every parameter tensor in the list, since a real network's weight matrices and bias vectors have different shapes and can't be stacked into one big tensor first.

Reference solution

import torch


def polyak_update(
    online_params: list[torch.Tensor], target_params: list[torch.Tensor], tau: float,
) -> list[torch.Tensor]:
    # target-network updates are never part of a training graph, so this
    # should never carry gradient regardless of what the caller passed in
    with torch.no_grad():
        return [
            tau * p_online + (1.0 - tau) * p_target
            for p_online, p_target in zip(online_params, target_params)
        ]

Key Functions & Tricks

• torch.no_grad() — documents and enforces that this bookkeeping update never contributes to autograd, matching how a real target network is maintained outside the training graph
• zip(online_params, target_params) — pairs each online parameter tensor with its corresponding target tensor by position, without assuming they can be concatenated into one tensor
• tau * p_online + (1.0 - tau) * p_target — the exponential-moving-average update itself, applied identically regardless of each parameter tensor's shape
• list comprehension over zip — keeps the whole update vectorized-per-tensor rather than looping over individual scalar weights

How to Recognize This Pattern