Predicting tennis matches with machine learning

A while ago I wanted to know whether you could beat the laziest possible prediction: the higher-ranked player wins. The short answer is yes. The slightly longer answer is "yes, by about six percentage points, and it will cost you a few weekends". This project is the long answer.

The goal was simple to state. Give the system two players and a tournament context, get back a win probability. Stating it was the easy part.

The data

Everything comes from tennis_atp, Jeff Sackmann's public repository of ATP results (Note that for now they have been removed for some reason, I leave the link in case they come back). The archive goes back to 1968, but the pipeline only loads from 1990 onward, because pre-1990 rows are missing ranking points and most of the match statistics, and a feature that is absent for a third of your data is not a feature, it's a liability.

That leaves 112,056 matches. Each row is one match: winner, loser, tournament, surface, round, and for the more recent ones the detailed stuff (aces, double faults, first-serve percentage, points won behind the second serve).

Reading three decades of CSVs on every run is a few seconds of pure waste, so there's a Parquet cache in front of it. First run costs you a coffee, every run after that is instant.

Then the cleaning pass, which is mostly a list of things that are technically matches but tell you nothing about who is better:

• retirements and walkovers, because a player pulling out with a torn abdominal is not evidence about level
• impossible durations, under 20 minutes or over 6 hours
• rankings above 2000, which in practice means the row is dirty

About 96% of the data survives, which leaves 107,921 usable matches.

The format problem

The raw files are asymmetric. Every row has winner_* and loser_* columns, which means the target is sitting in the schema itself. Train on that and the model learns, immediately and with great confidence, that the player in the winner column tends to win. Accuracy: 100%. Usefulness: zero.

The fix is to rewrite every match as a symmetric record with player_a and player_b, assigning the suffix by coin flip. Half the matches get mirrored, so the loser becomes player_a and the winner becomes player_b. The target is 1 when player_a wins.

Now the dataset is antisymmetric by construction. Column position carries no signal, so the model has no choice but to learn the actual differences between the two players.

The pipeline

Why XGBoost, in a repo called random-forest

The folder is named random-forest and the model is XGBoost. I swapped the estimator early on and never renamed the directory, which I like to think of as archaeology rather than negligence.

Both are tree ensembles, but they get there differently. Random Forest grows trees in parallel on bootstrap samples and averages the votes. XGBoost grows them sequentially, each one fitting the residual errors of everything before it. For this problem the boosting version wins on three counts that actually matter:

• Explicit regularisation. L1 and L2 are separate knobs. With 40 correlated features, several of which are different ways of measuring the same thing, that control is the difference between a model and a memorisation device.
• Native missing-value handling. It learns which side of a split to send a NaN instead of making me pick an imputation strategy and then defend it.
• Finer interactions. Sequential boosting carves up feature combinations at a granularity that averaged trees don't reach.

I will not pretend speed was a factor. At 108k rows both algorithms finish before you can get up for water.

The cost of boosting is that it overfits enthusiastically if you let it. Hence the final config: reg_alpha=5.0, reg_lambda=8.0, max_depth=5, max_leaves=32. Shallow, heavily penalised, slightly boring. That's the point.

Feature engineering, where the whole thing is actually decided

Sixty candidate features go in, SHAP selection keeps 40. If the model is any good, most of the credit lives here rather than in the estimator.

ELO, or "how good is this person really"

ELO is the chess rating system: beat someone stronger and you gain a lot, lose to someone weaker and you give a lot back. Three variants ended up in the model.

Global ELO across all surfaces. Per-surface ELO, separate for clay, hard and grass, because compressing Nadal on clay and Federer on grass into one number is an act of violence against both. Peak ELO, the historical maximum, which is a decent proxy for "how good was this player when healthy".

The K factor is not the textbook constant 32. It decays with experience and scales with tournament level:

K = 250 / (matches_played + 5)^0.4  ×  level_multiplier  ×  margin_multiplier

New players move fast because we know nothing about them; veterans with 400 matches on record move slowly because we already do. The level multiplier runs from 1.5 at a Grand Slam down to 0.8 at a Challenger, so beating Djokovic at Wimbledon is worth roughly twice as much as beating him at a 250. Which feels right, and more importantly it measures better.

elo_diff is the most predictive single feature in the model by a wide margin. Nothing else is close.

Rolling win rate

Win rate over the last 3, 5, 10 and 20 matches. Recent form matters. Less than the commentary would have you believe, but it matters.

The implementation detail that actually bites: this has to be strictly look-ahead free. For each match you use only results strictly prior to it, in date order. Everyone knows this. Everyone also writes a .shift() on an insufficiently sorted frame at least once and spends an afternoon delighted by the resulting metrics.

Head-to-head

How many times the two have met, and the win rate between them. Surprisingly sharp on specific pairs. Some players simply own an opponent in a way that neither ranking nor ELO predicts, and the H2H column is the only place that shows up.

Fatigue and momentum

• Matches in the last 7 days, a blunt workload proxy
• Consecutive wins and losses, for short-term streaks
• Trend in recent form, the slope over the last 5 matches, so improving and declining look different

One casualty here: I had a days_since_last_match feature and cut it, because tourney_date is the tournament's start date, not the match date. Every match in a two-week Slam shares one timestamp, which makes the feature confidently wrong. Better absent than misleading.

Tournament context

Surface as a one-hot. Tournament prestige on a scale from Grand Slam (4) down to Challenger (1). Round encoded from first round (1) to final (7), which captures difficulty but also preparation, since nobody approaches a Wednesday R64 the way they approach a final.

Best-of format is in there too, and it earns its place. Five sets suppress variance, and suppressed variance favours the better player. Fluke wins need short matches.

Interaction terms

Seven explicit products, for dynamics that no single column exposes:

• elo_surface_diff × sign(rank_diff), for when surface ELO and global ranking disagree, which is exactly when the upset happens
• age_diff × tournament_prestige, since young players do better in smaller draws and older ones hold up in the big ones
• form × prestige, because form travels further at a Slam
• height_diff × surface_speed, since height pays on fast grass and much less on slow clay

Recency weighting

Sample weights decay exponentially with a 7-year half-life. A 2004 match still contributes, at a quarter of the weight of a 2025 one. Tennis in 2004 was a different sport with different racquets and different court speeds, and the model should know that without me throwing the data away.

Temporal cross-validation, the part everyone gets wrong

This is where sports ML projects go to die.

Run standard k-fold and your validation set contains 2015 matches scored by a model that has already read 2019. The leakage is subtle, it never throws an error, and it inflates every metric you care about. Then you run it on live matches and the whole thing quietly falls apart.

The fix is an expanding-window temporal split. Train on the past, validate on the immediate future, move the boundary forward, repeat. The fold boundaries are picked by date quantiles rather than round years, so each validation window carries a comparable number of matches:

Fold 1 trains on 36k matches, fold 5 on 91k. The 2023-2024 holdout (5,839 matches) is never touched during tuning, feature selection or anything else. It is the only number in this article I fully trust.

The cost is variance. Early folds have thinner history behind their ELO and rolling features, so per-fold accuracy swings by about 1.7 points. Honest metrics are noisier metrics. That trade is always worth making.

Hyperparameter tuning with Optuna

Optuna runs Bayesian optimisation with a TPE sampler. Instead of grinding through a grid or sampling blindly, it builds a probabilistic model of the search space and keeps drawing from the promising regions. A MedianPruner kills trials that are clearly going nowhere partway through, which is where most of the wall-clock savings actually come from.

The objective is ROC-AUC on the current trial's validation fold. 25 trials with a 30-minute ceiling, which with pruning covers roughly the same ground as 50 unpruned ones. The full pipeline, loading through artifacts, runs in about six minutes on a laptop CPU.

Results

Cross-validation, averaged over 5 temporal folds:

• Accuracy: 68.9% ± 1.8%
• ROC-AUC: 75.8% ± 2.0%
• Log loss: 0.581

Holdout, 2023-2024, never seen during any part of training:

• Accuracy: 65.5%
• ROC-AUC: 72.2%
• Log loss: 0.613

That's a 3.6 point AUC drop from CV to holdout. Not nothing, but not alarming either. Part of it is structural: a model trained through 2022 has never heard of the players who broke through in 2023, and it has to rate Alcaraz's clay game from a handful of matches.

Segmented, where it gets interesting

By surface, hard 65.4%, grass 65.7%, clay 65.8%. Which is to say: no difference at all. I fully expected clay to be harder, on the theory that longer rallies mean more chances for the underdog. The data disagrees, and per-surface ELO seems to be doing its job well enough that the surface stops mattering as a source of error.

By tournament level, Grand Slams 71.0%, Challengers 66.9%, Masters 1000 64.5%, ATP 250/500 63.8%. The Slam number is the best-of-5 effect again: three sets to win is a long time to be lucky. The Challenger result is less flattering and more mundane, since the level gaps down there are enormous and picking the favourite is easy.

By round, early rounds 66.2%, middle 64.1%, semi-finals and finals 63.2%. Exactly the shape you'd expect. The deeper the draw, the more evenly matched the survivors, and a final between two players inside 50 ELO points is a coin flip wearing a headband.

Top 10 by SHAP importance

1. elo_diff (0.41), global ELO difference
2. elo_surface_diff (0.20), surface-specific ELO difference
3. rank_diff (0.12), ATP ranking difference
4. venue_experience_diff (0.09), how familiar each player is with the venue
5. age_prestige_interaction (0.09), age × tournament prestige
6. peak_elo_diff (0.09), peak ELO difference
7. surface_elo_rank_interaction (0.07), surface ELO × rank sign
8. rank_points_diff (0.06), ranking points difference
9. form_elo_diff (0.06), form-weighted ELO
10. win_rate_diff_last_20 (0.06), medium-term form

ELO and ranking dominating is not news. The one that raised an eyebrow was venue_experience_diff at fourth. Having played a tournament before, independent of how well, is apparently worth real signal. Familiarity with the courts, the balls, the altitude, the walk from the locker room. Tennis players are creatures of habit and the model noticed before I did.

Calibration

The model outputs a probability, not a verdict, and a probability is only useful if it means something. If it says 70%, that player should win about 70% of the time, otherwise you can't compare it to a bookmaker's line or size a bet or do anything except admire it.

Post-training, a CalibratedClassifierCV with the sigmoid method (Platt scaling) is fitted on the validation set, over a frozen estimator so the base model isn't refit. The calibrated version only replaces the raw one if it actually improves validation log loss, which sounds paranoid until the first time isotonic regression cheerfully overfits a validation fold and you ship a worse model.

Measured Expected Calibration Error on the holdout: 0.037. Not perfect. Good enough that the probabilities are worth reading as probabilities.

What I learned

The ceiling is low and that's the sport's fault. Tennis has enormous intrinsic variance. A set turns on three loose points. The model cannot see a niggling wrist, a bad night's sleep, a personal crisis, or a court that plays completely differently at 2pm and at 7pm under lights. Around 65% on genuinely unseen data is, in that context, a real result.

Custom ELO earns its complexity. Per-surface, experience-adaptive K, tournament-weighted. Each piece is a small gain over textbook ELO, and small gains stack. elo_surface_diff sitting at number two in the importance table is the whole argument.

Temporal validation is non-negotiable. My first attempt with standard k-fold hit 72% accuracy and I was briefly delighted. Temporal CV took it to 68.9%, and the holdout to 65.5%. That's not the model getting worse, that's the number getting honest.

More data beats cleaner data, at least here. I tried cutting everything before 2005, reasoning that pre-2005 tennis is a different sport. It closed the CV-to-holdout gap from 0.034 to 0.024, which looked like a win for about four minutes, until I noticed holdout AUC hadn't moved (0.7223 to 0.7214). The gap closed because CV came down to meet the holdout, not because the holdout came up. I had thrown away 40% of the data to make one diagnostic number prettier. Recency weighting was already handling the era problem properly. Reverted.

Interaction features are more subtle than "do they help". Two of the seven land in the SHAP top ten, which looks like a resounding success, but the AUC gain from adding all seven was about 0.5%. Both things are true: they're genuinely informative, and they're mostly informative about things gradient boosting was already picking up on its own. Importance and marginal value are different questions and it took me longer than I'd like to admit to internalise that.

Deleting features is underrated. The per-surface win rates were 65 to 71% missing and imputed to 0.0, which quietly told the model that "no data" and "perfectly even" are the same thing. They were also redundant with surface ELO. Cutting them changed holdout performance by nothing at all and made the model easier to reason about.

Where this goes next

• Bookmaker odds as a feature. Odds encode everything I can't see: injury whispers, practice-court reports, insider confidence. Adding them would probably help, but the more interesting question is how much they help, since that's a direct measurement of how much of the market's edge is information I simply don't have access to.
• Match statistics as first-class inputs. Aces, first-serve percentage and break points saved only exist for a fraction of historical matches, so they're currently reduced to a single serve-dominance feature. A model trained only on fully-instrumented matches would have less data and much richer data. Worth testing.
• A per-surface model. A clay specialist model might beat the generalist on clay. It would also have a third of the data, which is exactly the trade the segmented results suggest isn't worth making, but I'd like to be proven wrong.
• Player embeddings. Instead of hand-built aggregates, a learned vector per player. Enough matches per player exist to make this plausible, and it might capture stylistic matchups that no difference-of-scalars feature can.

The code is on GitHub. The trained model ships with the repo, so you can run predictions without retraining, though retraining is only about six minutes if you want to watch it happen.