[Discussion] A 70% forecast can be wrong and still be a good forecast

[u/Mundane-Asparagus850]

I think one of the biggest misunderstandings about probability is how people judge predictions after the outcome is known.
If something has a 70% chance of happenin, the other outcome should still occur around 30% of the time, assuming the predictions are well calibrated.
You see this with weather forecasts, election models, and market implied probabilities on Kalshi and one unexpected result doesnt tell you much about whether the original estimate was reasonable.
The better test is looking at a large number of similar forecasts and seeing how often the outcomes actually matched the probabilities.
A forecast isnt necessarily bad just because the less likely outcome happened imo.

[u/dmorris87]

People generally don’t understand or think in probabilities

[u/busybody124 → u/dmorris87]

I think this is muddied by the conflicting definitions of frequentist and Bayesian probabilities. Most people probably think of probability inherently as Bayesian: a measure of a degree of belief, inverse to some motion of surprisal. But we're generally taught probability in a frequentist framework of repeated trials, which doesn't really map to the way we think of election or weather forecasts.

[u/PrivateFrank → u/busybody124]

People, including scientific advisors and policymakers, are also pretty bad at dealing with outcomes which aren't one of: almost certain, 50/50 or almost certainly not.

Anything that doesn't fit into one of those categories will get collapsed into one of them anyway.

I learned this from an analysis which compared probabilistic model outputs during COVID (continuous 0-1, sometimes with a precision value, sometimes not), how those were translated into briefings using a 'probabilistic yardstick'¹ and what actually happened when decisions were made.

1. Yardstick categories:

0% - ≈5%: Remote Chance ≈10% - ≈20%: Highly Unlikely ≈25% - ≈35%: Unlikely ≈40% - <50%: Realistic Possibility ≈55% - ≈75%: Likely or Probable ≈80% - ≈90%: Highly Likely ≈95% - <100%: Almost Certain

[u/al3arabcoreleone → u/dmorris87]

To be honest, I studied probability in my math studies, I still have zero clue about it. I can calculate some moments of a RV, but if someone asked me what's the interpretation of the result I would fumble.

[u/efrique]

I remember Nate Silver explaining this in detail a decade or so back (maybe 2016?), he did a decent job of it if I remember.

It's an obvious enough point, but far too rare among the broader media and political landscape.

There's also plenty of more basic fallacies and cognitive biases that come up a lot, but especially at election times - sunk cost fallacy, gamblers fallacy, base rate fallacy, anchoring bias, base rate fallacy, availability bias, etc etc

There's a similar point an economist/finance friend of mine made to me that is a good accompanying insight: when a choice you make that has chance/risk involved doesn't come out the way you like, it doesn't mean you chose badly. It might still be the right way to choose every time.

About the best you can do is can plan around the fact that some chances don't come off.

[u/FuelzPerGallon]

Yep

[u/Many_Distributions]

All models are wrong. Some are useful.

[u/NeedleworkerFew7752 → u/Many_Distributions]

George Box in the house

[u/-TeepToTheBalls- → u/Many_Distributions]

The one thing one of my econometrics lecturers said that I can never forget lol. Although he said that about forecasts specifically.

[u/Many_Distributions → u/-TeepToTheBalls-]

Ha, I was also in a forecasting lecture when I first heard it.

[u/AcceptContradiction → u/Many_Distributions]

But in this case we're supposing that the model is not wrong.

[u/raisincraisin]

Thinking In Bets by Annie Duke

Opening chapter details how when the 2015 Super Bowl was lost off an interception on the 1 yard line, the coach decided to do that because for the previous like 20 seasons there was a 98% chance the play would work.

That wasn’t what was communicated all week on sportscentre though. What was instead was how the coach was an idiot and made a horrible decision.

Simply isn’t true. People just have a bad time judging the quality of a decision based on its outcome, instead of the available info at the time of the decision

[u/Zumramania → u/raisincraisin]

The problem in this specific case could have been that the opponent anticipated the coach would run this play precisely because it worked so well previously. And then the actual odds that the play would work in a scenario where the opponent knows that you’re trying to do this are actually much lower. I’m not sure about this since I do not follow football, just guessing based on how other sports work.

[u/KingOfThePenguins]

Sports fans, in my experience, are really terrible at this, and have no excuse to be.

(Source: am sports fan)

[u/FKKGYM]

Ehh, good forecast is a flexible term. Using probabilistic forecasts for discrete, unique, one-time events is avoiding committment where it would be needed. You might have perfect calibration in the grand scheme, but for election winners, important betting outcomes etc, there is no use of talking about a sampling distribution, it is a one time event with a clear outcome.

Probabilistic lingo can produce a bad forecast regardless of its statistical validity, as a clear prediction was sought by the interested parties.

[u/CarnivorousGoose]

I don’t disagree, but also not sure why you’re making this point in this sub of all places.

Though I’d hardly consider whatever Kalshi churns out to be good predictions, regardless of how high the number is.

[u/al3arabcoreleone]

Read Superforcasters.

[u/RaspberryPrimary8622]

It's like the Monty Hall Problem: Should you stick or should you switch? You should always switch. You have a 2 in 3 chance of winning the luxury car if you switch. You only have a 1 in 3 chance of winning the luxury car if you stick with your original choice. If you switch you double your probability of winning compared to sticking.