3. Team-Draft Interleaving of Two Ranked Lists

Problem

Fin's research compares multiple reranker candidates — their own trained cross-encoder against off-the-shelf options like Cohere Rerank v3.5. A standard offline technique for comparing two rankers head-to-head, without running a full separate online experiment for each, is team-draft interleaving: blend the two ranked lists into one combined list by alternating picks between them, and show that single blended list to evaluators or use click data on it to infer which ranker's picks tend to win.

The purpose is to cancel out position bias. If you evaluated the two rankers' lists side by side, whichever list happens to put a good result higher just because of list structure (not actual quality) has an unfair advantage. Interleaving removes that bias by fusing both rankers' opinions into one ordering, so credit for a doc's position can be attributed back to whichever team picked it.

Naive concatenation (list A followed by list B) or simple round-robin zipping doesn't work here, because both lists can rank the same document, and a document should only ever appear once in the blended result — whichever team reaches it first gets credit for the pick, and the other team's copy is skipped.

Source: src/3_team_draft_interleaving.py

def interleave(list_a: list[str], list_b: list[str]) -> list[str]:
    ...

Examples:
>>> interleave(["p1", "p2", "p3"], ["p2", "p4", "p1"])
["p1", "p2", "p3", "p4"]

>>> interleave(["x", "y"], ["y", "x"])
["x", "y"]

Step-by-Step Approach

1. Recognize this as a two-pointer merge, but driven by strict turn-taking rather than by comparing values — team A always moves first, then the turn alternates A, B, A, B, ... regardless of what happens on each turn.
2. Track a pointer into each list (i for A, j for B) and a seen set of every doc_id already placed in the result, so a document already picked by one team is never picked again by the other.
3. On a team's turn, first advance that team's pointer past any doc_ids already in seen — its next real candidate might be several slots ahead if the other team already grabbed everything before it.
4. If a candidate remains after skipping, append it to the result and add it to seen; if the team's list is exhausted, it simply makes no pick this turn.
5. Hand the turn to the other team unconditionally — a skipped/empty turn still counts as a turn, it doesn't repeat or give the other team two picks in a row.
6. Stop once both pointers have run past the end of their lists; handle the edge cases where one or both input lists are empty, and where the lists fully overlap (the result then just dedupes down to the shared items in A's pick order).

The key insight is that "seen" and "turn" are tracked independently: skipping an already-picked doc never costs a team its turn, but failing to find any candidate on a turn still passes the turn along — those two rules together are what make the algorithm fair and deterministic.

Reference solution

def interleave(list_a: list[str], list_b: list[str]) -> list[str]:
    # alternating two-pointer merge with a seen-set, O(n + m) time, O(n + m) space
    result = []
    seen = set()
    i = j = 0
    turn = "A"
    while i < len(list_a) or j < len(list_b):
        if turn == "A":
            # advance past any doc_ids B already claimed
            while i < len(list_a) and list_a[i] in seen:
                i += 1
            if i < len(list_a):
                result.append(list_a[i])
                seen.add(list_a[i])
                i += 1
            # turn passes to B even if A had no candidate left
            turn = "B"
        else:
            # advance past any doc_ids A already claimed
            while j < len(list_b) and list_b[j] in seen:
                j += 1
            if j < len(list_b):
                result.append(list_b[j])
                seen.add(list_b[j])
                j += 1
            # turn passes to A even if B had no candidate left
            turn = "A"
    return result

Key Functions & Tricks

• doc_id in seen — set membership gives an O(1) "already picked" check per skip
• while i < len(list_a) and list_a[i] in seen: i += 1 — inner skip-loop fast-forwards a pointer past stale picks before considering a new one
• turn = "A" / "B" — a plain string flag alternates control between the two teams each iteration
• while i < len(list_a) or j < len(list_b) — the outer loop's OR condition keeps going until both lists are exhausted, not just one
• Turn always advances, pick doesn't have to happen — separating "whose turn" from "was there a candidate" is what keeps the algorithm fair when one list runs out early
• seen.add(list_a[i]) — marking a pick seen immediately prevents the other team from re-picking it on its very next turn

How to Recognize This Pattern