38. Threshold Maximizing F1 Score

Problem

Fin gates its final answer on a certainty/confidence score, deciding whether to answer directly or escalate to a human (see the companion problem on minimum threshold for a precision target). A threshold set too high avoids wrong answers but escalates too many queries a human didn't need to see; a threshold set too low answers more queries directly but lets in more mistakes. Precision and recall move in opposite directions as the threshold slides, so neither one alone tells you where the best operating point is.

F1 score — the harmonic mean of precision and recall — collapses that tradeoff into a single scalar to optimize: it's high only when both precision and recall are reasonably high, and punishes lopsided outcomes (near-perfect precision with terrible recall, or vice versa) far more than a simple average would. Given historical (certainty_score, was_correct) pairs, find the certainty threshold that maximizes F1.

Trying every distinct score present as a candidate threshold is sufficient — nothing about the predicted-positive set changes for any threshold value strictly between two consecutive distinct scores, so scanning a continuous range would just repeat the same precision/recall computation over and over for no new information.

Source: src/38_threshold_maximizing_f1.py

def best_f1_threshold(scored_outcomes: list[tuple[float, bool]]) -> tuple[float, float]:
    ...

Examples:
>>> best_f1_threshold([(0.9, True), (0.8, True), (0.7, False), (0.6, True), (0.5, False)])
(0.6, 0.8571428571428571)

Step-by-Step Approach

1. Compute total_positive — the total count of actually-correct outcomes across the whole dataset — once upfront, since it never changes as the threshold sweeps down.
2. Sort all outcomes descending by certainty score once, so the sweep can move from the strictest threshold down to the most inclusive one, extending the "predicted positive" set monotonically instead of recomputing it from scratch per candidate.
3. Walk the sorted list grouping every run of tied scores together — a single threshold value can't split entries that share the same score, so they must enter the "predicted positive" set as one unit.
4. Maintain running tp and fp counters as each group is absorbed; precision and recall fall out directly from those two counters (plus the fixed total_positive) at every group boundary.
5. Compute F1 as the harmonic mean of precision and recall at each boundary, guarding every division against a zero denominator.
6. Compare against the best-so-far using the tie-break rule: a strictly higher F1 always replaces it; an equal F1 replaces it only if the new threshold is smaller (more inclusive, higher recall).
7. Continue through every group in the sorted list; the final (best_threshold, best_f1) pair is the answer once the whole list has been consumed.

The key insight: this sweep is walking the precision-recall curve left to right in a single pass, maintaining running counts instead of recomputing precision and recall from scratch at each candidate threshold — O(n log n) total instead of the O(n²) that direct recomputation per threshold would cost.

Reference solution

def best_f1_threshold(scored_outcomes: list[tuple[float, bool]]) -> tuple[float, float]:
    # sort once descending, single pass grouping tied scores and maintaining running tp/fp, O(n log n) time, O(n) space
    if not scored_outcomes:
        raise ValueError("empty input")
    total_positive = sum(1 for _, label in scored_outcomes if label)
    ordered = sorted(scored_outcomes, key=lambda x: -x[0])
    tp = fp = 0
    best_f1 = -1.0
    best_threshold = None
    i, n = 0, len(ordered)
    while i < n:
        j = i
        # group every entry tied at this score together, since a threshold can't split them
        while j < n and ordered[j][0] == ordered[i][0]:
            if ordered[j][1]:
                tp += 1
            else:
                fp += 1
            j += 1
        precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        recall = tp / total_positive if total_positive > 0 else 0.0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
        # strictly better F1 wins outright; equal F1 prefers the smaller (more inclusive) threshold
        if f1 > best_f1 or (f1 == best_f1 and (best_threshold is None or ordered[i][0] < best_threshold)):
            best_f1 = f1
            best_threshold = ordered[i][0]
        i = j
    return (best_threshold, best_f1)

Key Functions & Tricks

• sorted(scored_outcomes, key=lambda x: -x[0]) — descending sort via negated key, enabling a single monotonic sweep.
• total_positive = sum(1 for _, label in scored_outcomes if label) — computed once upfront since it's constant across every candidate threshold.
• Inner while j < n and ordered[j][0] == ordered[i][0] — groups tied scores so a threshold can't split them mid-group.
• tp / (tp + fp) if (tp + fp) > 0 else 0.0 — precision guard for before anything has been predicted positive.
• tp / total_positive if total_positive > 0 else 0.0 — recall guard for when there are zero ground-truth positives.
• ... if (precision + recall) > 0 else 0.0 — avoids a ZeroDivisionError in the harmonic mean when both precision and recall are 0.
• f1 == best_f1 and ordered[i][0] < best_threshold — tie-break rule that keeps the smallest threshold among equal-F1 candidates.

How to Recognize This Pattern