Learn / Skill or luck? Sample size, p-values and overfitting

Skill or luck? Sample size, p-values and overfitting

Flip a fair coin 100 times and you’ll rarely get exactly 50 heads. Betting records work the same way. Here’s how to tell a real edge from a lucky streak, and why most winning systems turn out to be the lucky kind.

How much records swing by chance

Imagine a strategy with no edge at all: each bet is a true 50/50. Over 100 bets, its win rate will land within about ±5 percentage points of 50% most of the time, so anything from 45% to 55% is ordinary noise. Over 1,000 bets the typical swing shrinks to about ±1.6 points.

That's the core problem. A 56% record over 100 bets looks like a great system, but it's well within what blind luck produces. The same 56% over 1,000 bets would be far more convincing.

What the luck check tells you

Snap Theory's luck check is a one-sided statistical test (a p-value) on profit per bet. It answers: if this strategy really had no edge, how often would it show at least this much profit?

Above 20%: very common for no-edge strategies. Treat the result as noise.
5–20%: interesting but weak. Needs more evidence.
Below 5%: luck is a less likely explanation, but read the next section first.
Below 1%: a strong signal, if the strategy wasn't found by searching through many others.

The many-tries trap

Test 100 strategies that have no edge, and about 5 of them will pass a 5% luck check by chance alone. Test 10,000 and about 500 will. That's why a strategy you found by trying lots of filter combinations deserves much more suspicion than one you thought of first and tested once.

This is also why an impressive backtest posted online proves little: you don't see the hundreds of versions that didn't work.

Overfitting

Overfitting means tailoring a rule so closely to past data that it captures that data's random quirks rather than anything real. Warning signs:

Many filters, each narrowing the sample further ("road underdogs, weeks 10–14, on grass, after a loss, in a divisional game").
Thresholds that look oddly specific (a spread of exactly +3.5 to +6, but not +3 or +6.5).
No reason you could have explained before you saw the result.
Great results in a few seasons, flat or negative in the rest.

The best defense: out-of-sample testing

Split history in two. Find or tune the rule using the earlier seasons only, then check it on the later seasons without changing anything. If the edge was real, it should still show up, usually smaller. If it vanishes, it was noise.

The strategy finder does exactly this. It ranks thousands of rules on "discovery" seasons, then shows how each did on later "check" seasons it never saw. Typically about half of the top discovery results fail the check, which is a useful reminder of how much noise there is.

A practical checklist

Hundreds of bets, not dozens.
A luck check under 5%, ideally well under.
A reason that makes sense on its own.
Profitable across most seasons, not just a few.
Still profitable on seasons it wasn't built on.
Even then, expect the edge to shrink as markets adjust.

Try it: run the strategy finder and switch Show to "All" to see how many top discovery results faded in the check seasons.

Snap Theory is a research tool, not betting advice. Past results don't guarantee future results. 21+.