Football Prediction Calibration: How to Judge Probability Forecasts

What is prediction calibration?

A model says a team has a 70% chance to win. The team loses. Was the prediction bad? Not necessarily: a 30% outcome should happen sometimes. The more revealing question is what happens across hundreds of similarly confident forecasts. Calibration measures whether predicted probabilities line up with observed frequencies over many events. If a model assigns a 70% chance to a group of comparable outcomes, a well-calibrated model should see those outcomes occur about 70% of the time over a sufficiently large, representative sample. A 70% prediction can still be wrong on any individual match.

Calibration is not the same as knowing the result in advance. Football is uncertain, and even strong forecasts cannot remove that uncertainty. The goal is to produce probabilities that are useful and statistically defensible.

A simple football example

Imagine a model gives a home-win probability between 60% and 70% for 100 matches. If the average forecast in that group is 65%, roughly 65 home wins would be consistent with good calibration, allowing for sampling variation. The number is illustrative, not a claim about SmartMatchData's historical results.

With only ten matches in a group, one additional win changes the observed rate by ten percentage points. That is too much weight to place on a single result. That is why calibration charts need sample counts and why tiny bins should not be interpreted as precise evidence of model quality.

Calibration versus prediction accuracy

Accuracy asks how often a selected outcome was correct. For 1X2 football forecasts, that might mean choosing the most probable of home win, draw or away win and checking whether it occurred. Accuracy does not tell us how good the probability estimates were. A model assigning 51% to the correct outcome and another assigning 95% can receive the same accuracy credit, even though their confidence is very different.

Calibration instead examines the match between stated confidence and actual frequency. A model can have respectable accuracy but be overconfident or underconfident. Both dimensions matter when judging forecasts.

What is log loss?

Log loss is a proper scoring rule that penalizes assigning low probability to outcomes that actually happen. For a single binary outcome, it is calculated as −[y log(p) + (1−y) log(1−p)], where y is 1 if the event occurs and 0 otherwise. For 1X2, the multiclass version uses the negative logarithm of the probability assigned to the actual home-win, draw or away-win outcome.

Lower average log loss is better. Extremely confident mistakes receive large penalties. Scores should be compared on the same outcomes and evaluation periods, and the treatment of missing or invalid predictions should be transparent. A low log loss on a small or selectively chosen sample does not establish future reliability.

Brier score: a second way to assess probabilities

The **Brier score** measures the squared difference between a forecast probability and what actually happened. For a binary event, the calculation is **(p − y)²**, where p is the predicted probability and y is 1 when the event occurs and 0 when it does not. The score is averaged over settled predictions, and lower values are better.

Suppose a model gives a home team a 70% chance to win. If it wins, the binary Brier score for that forecast is **(0.70 − 1)² = 0.09**. If it does not win, the score is **(0.70 − 0)² = 0.49**. A confident miss is penalized more heavily than a correct confident prediction.

For three-way **1X2** forecasts, a multiclass Brier score can sum the squared differences for home win, draw and away win. Some reports divide that sum by the number of classes, while others do not. The convention must be stated before comparing published scores. A binary BTTS Brier score should not be compared directly with a differently scaled multiclass 1X2 score.

The Brier score and log loss both evaluate probability quality, but they penalize errors differently. Log loss is especially harsh on predictions that assign extremely small probabilities to outcomes that occur. The Brier score uses squared errors and is often intuitive to explain. Neither should be judged without an appropriate baseline and sample size.

Compare the model with a baseline, not just itself

An accuracy figure of 55% may sound impressive until the evaluation set is examined. If one outcome dominates the sample, a simplistic strategy that always selects that outcome might achieve a similar headline accuracy. The question is whether the model improves on a sensible benchmark.

Possible baselines include historical outcome frequencies from an appropriate training period or a simple probability model built from past data. A baseline must be defined without using results that were unavailable when the forecast was made. Otherwise the comparison benefits from information leakage.

Consider a hypothetical set of 100 binary forecasts. A model with an average Brier score of **0.20** outperforms a baseline scoring **0.24** on the same fixtures, under the same scoring convention. The absolute improvement is **0.04**. This is an illustrative calculation, not a SmartMatchData performance claim. It is not enough by itself to establish statistical significance or superiority on future fixtures.

Always compare identical fixtures, prediction horizons and settled outcomes. Missing forecasts, postponed games and changes in competition coverage can alter the apparent performance if handled inconsistently.

Accuracy, discrimination and calibration are different tests

Three useful questions should be kept separate. **Accuracy** asks whether the most likely outcome was selected correctly. **Calibration** asks whether events assigned a particular probability occur at roughly that frequency. **Discrimination** asks whether the model assigns higher probabilities to events that happen than to events that do not.

A model can be calibrated but uninformative. For example, assigning a constant 50% probability to a balanced binary event can be reasonably calibrated across a large sample, yet it cannot distinguish easier cases from harder ones. Conversely, a model can rank matches effectively but express its confidence too aggressively.

A complete evaluation therefore considers calibration plots, a proper scoring rule such as log loss or Brier score, relevant baselines and the number of settled predictions. No single number describes every dimension of forecast quality.

Walk-forward testing and changing model versions

Football data changes over time. A model evaluated on fixtures it was trained on may appear better than one tested on genuinely unseen future games. A more credible approach trains or updates a model using past information, locks predictions before kickoff, and evaluates the subsequent fixtures without revising those forecasts.

In **walk-forward evaluation**, the training window moves through time. Each test period occurs after the data used to fit the model. This better reflects operational forecasting than randomly mixing older and newer matches, although implementation details and data leakage still matter.

If the model changes, retain a version identifier with each locked prediction. Reporting a single performance number across different versions can hide whether an apparent improvement came from a new model, a different league mix or a change in the evaluation sample.

How to read a calibration report responsibly

Start with the **number of settled forecasts**, the competition coverage and the evaluation dates. Check whether predictions were locked before kickoff and whether cancelled or missing matches were excluded consistently. Then inspect the observed frequencies within probability ranges, paying attention to how many observations each range contains.

Next, compare log loss or Brier score with an explicitly defined baseline on the same fixtures. Look for differences between 1X2, BTTS and Over 2.5 rather than assuming one market's score transfers to another. Finally, inspect whether recent performance differs from longer-term performance without overreacting to a short streak.

The small sample size guide explains why a handful of settled predictions cannot establish model quality. The football prediction guide provides context for how match probabilities are generated.

Reliability diagrams and probability bins

A reliability diagram groups forecasts into probability ranges and compares each group's average forecast with the observed event frequency. Points near the diagonal indicate better calibration for that sample. Points below or above it may suggest overconfidence or underconfidence, depending on the axis convention.

Bins with few observations can fluctuate widely. Different bin widths can change how a diagram looks. Include sample counts, use consistent definitions and avoid reading too much into minor departures from the diagonal.

Why locked pre-match forecasts matter

A fair forward evaluation uses predictions recorded before kickoff and leaves those values unchanged after the result becomes known. Otherwise, retrospective adjustments can make a model appear stronger than it was in real use. SmartMatchData publishes locked pre-kickoff probabilities and tracks settled outcomes separately.

Models and competitions should be evaluated consistently. When model versions change, combining their forecasts without explanation can hide differences in performance. The Model Methodology explains SmartMatchData's modelling approach, while Model Performance provides public evaluation information where sufficient settled results are available.

Comparing 1X2, BTTS and Over 2.5

These are different prediction tasks. 1X2 has three mutually exclusive outcomes; BTTS Yes and Over 2.5 are binary events. Their base rates and scoring-rule baselines differ, so raw scores should not automatically be ranked across markets. Evaluate each market using its own outcome definition and a consistent sample.

For context, read BTTS Probability Explained and the Expected Goals guide. A strong model assessment looks beyond a single headline accuracy number.

What a good-looking score can hide

Sample size, fixture selection, league coverage, postponed games and missing data affect reported metrics. Calibration can change over time when team strengths, data providers or modelling assumptions change. Performance on past matches does not guarantee performance on future matches.

Use prediction probabilities as uncertain estimates, not guaranteed results or promises of financial return. Transparent evaluation should report both successful and unsuccessful settled forecasts rather than selecting only favorable examples.