football-llm-scaling
Is a fine-tuned model forecasting football results, or remembering them?
LLM evaluation · solo research · in progress
The question
When a fine-tuned language model beats prompting at predicting football results, is it forecasting, or remembering results it already saw? Memory is a known risk for language-model forecasts of past events (Gao et al., 2025 (opens in new tab); Gorla & Puduppully, 2026 (opens in new tab)). This study tests one fine-tuned model with three controls on the same matches.
How I’m testing it
Llama 3.1 8B, fine-tuned with QLoRA on 192 World Cup matches from 2010 to 2018, against 5-shot prompting of the same model. Tested on the 64 matches of the 2022 World Cup, which are in the model’s training data, and the 104 matches of the 2026 World Cup, played after it ends.
What’s next
Testing whether this holds across several model families, two pairs of leagues and a 70B model, and finding where each model’s knowledge really ends compared with the date its makers state.
Early results
- It writes the exact final score of about 62% of named 2022 matches, and leads prompting by about 20 points.
- Trained on plausible fake scores, it recalls 10% of them and the lead is gone.
- On the 2026 matches, after its training data ends, there is no clear lead.
- With team names hidden, recall falls to 5% and the lead shrinks by 28.6 points.
Early results, in detail
The lead is memory
On named 2022 matches the fine-tuned model writes the exact final score of about 62% of them (three training seeds; a constant guess gets 11%). When the score decides the result, it leads prompting by about 20 points (ten seeds, p = 0.003).
Fake scores remove it
Training on the same 192 matches with plausible fake scores, and the same stated results, drops exact recall from about 62% to 10%. The lead becomes -5.7 points, not significant. Mistral 7B and Qwen2.5 14B also recall the real scores after fine-tuning, and fake scores remove that recall; their lead over prompting was small to begin with.
Unseen matches remove it
On the 104 matches of 2026 the lead is +2.2 points (90% interval -7.1 to +11.2): no clear lead.
Hidden names remove it
On the same 2022 matches with team names hidden, exact recall falls to 5% and the lead shrinks by 28.6 points (p = 0.0002).
The grading rule
A rule that lets the score override the stated result turned the model’s remembered scores into hits.
A simple baseline
A team-rating model (Elo) picks the winner as often as prompting does: 54.7% against 56.3% in 2022, and 64.4% against 61.5% in 2026.