I bought this in paperback, and I suspected at the time it would annoy me but I wasn’t entirely certain what was wrong with it. I attempted to read it a couple times; I finally have read it, and can now articulate my issues with it.
I just want to be clear up front here. Criticizing pundits for making predictions that turn out to be absolutely false in a short time frame is a _great hobby_ and I am _here_ for it. More of this please.
Also, I fucking loathe Nate Silver’s operation (which at least started out doing the above) and I also don’t much care for this book (again, which mostly started out doing the above). (A little digression: one of the things I dislike about their descriptions of super forecasters / advice to people who want to emulate them comes down to do you listen to people you agree with more or people you disagree with. As if that is a simple way to divide up the world. Who are these people hanging out with?)
On the one hand, Silver and Tetlock have very similar issues: they are attempting to do some quantitative assessment of predictions with a view to Better Predictions. As more or less a direct, predictable result of that, they push predictors in the direction of shorter term, clearly defined predictions. They _also_ both tend to move in the direction of closer to when the answer will be Known. Their approaches _do_ improve prediction accuracy! So there’s that. Woot. *confetti*
If you are trying to nail down what is going to happen within, say, the next 2 weeks to 3 months, this is a _really_ worthwhile thing to get good at, and I have no particular reason to believe that the Good Judgment Project’s approach to things is bad. (I don’t care about this time frame.)
On the other hand, they have very similar issues: the process of converting a prediction to something that can be quantified puts some limitations on which predictions can be assessed and which cannot. I applaud their time limits, honestly. Letting people say, but inflation is about to happen, for … ever … is a stupid thing. Putting a stop to that by saying, okay fine, but by this metric over the next quarter is an admirable strategy that will expose people acting in bad faith. Altho expecting that to in turn change anything is a degree of optimism I, personally, do not suffer from.
I will note that just relentlessly mocking perma-inflationistas, permabears, etc. is probably more effective.
Overall, their efforts to create quantifiable predictions, and the strategy of converting the question into more answerable ones, however, is another way to move people away from what they actually want to something they legit do not want or at least do not particularly care about. So, kind of a negative there. If I ask for a cookie, and I get a graham cracker with chocolate frosting on it, and I go, that’s not really a cookie, and you say, Peg Bracken calls it a cookie, I might roll my eyes, eat it and say yeah, sure, it’s tasty, but honestly, we don’t think it’s a cookie.
_Peg Bracken_ didn’t think it was a cookie. When you name something an “Afterthought Cookie”, and embed it in the _I Hate to Cook Cookbook_, I don’t think you honestly believe it is a cookie. You’re here to make people shut up and quit asking for things — you’re not here to give them what they want.
At times, Tetlock and Gardner glance off the question of the questions, but they never systematically engage with it, and that is where most of the problem lies. They come closest when talking about Taleb, but mostly so they can weaken Taleb’s side of the debate (fair — not gonna get in the way of that, honestly).
They also frequently take an overtly abrasive approach to their audience. It feels very, oh, the audience thinks strawman but here is data, so that the actual reader will feel clever for not believing strawman. It’s annoying. It’s a common tactic, but that doesn’t make me like it. This is at its worst in the chapter about the German armed forces. They start by talking about Moltke, in the 19th century. They leap silently over the Great War and the Kaiser, and land squarely in the interwar period. They talk up the Wehrmacht, and then suggest that if the reader is uncomfortable with this, well, let’s just go with a quote:
“Understanding what worked in the Wehrmacht requires engaging in the toughest of all forms of perspective taking: acknowledging that something we despite possess impressive qualities. Forecasters who can’t cope with the dissonance risk making the most serious possible forecasting error in a conflict: underestimating your opponent.”
We called The History Channel the Nazi Channel. I remember board gamer culture including an entire cohort of people who adored Axis and Allies above all other games. Lauding the German War Machine and saying, sure, they’re evil, but they were sooooo good at the war staff is an absolute fucking cliche. It’s being used here to club the reader. I don’t know why they are doing this, but it’s weird. Are there people reading this book who think that one cannot / should not learn from their enemies? Who the hell would that even be? That said, the prowess of the Germans / Nazis has been waaayyyyy oversold. The difference was starkest at the beginning of the war, and was largely an artifact of they intended war, and everyone else was _not_ intending war. Guess who prepared more. Surprise! The ones who prepared more were more prepared. *confetti* You win a prize. Here is the prize. <— that period is your prize.
There are also really oddball moments, like when they talk about the Bay of Pigs and then the Cuban Missile Crisis. I don’t know anyone who defends the Bay of Pigs thing — and they don’t either — but they’re still defending JFK and his administration during the Cuban Missile Crisis. Granted, this was published in 2015. It has not aged well.
I wish more people took seriously what Gorbachev did, and how. Tetlock and Gardner use him as a bit of a punchline in a couple of different places — to mock all the people who utterly failed to predict that he would attain power and what he would do with it and who then turned around and claimed they had predicted the collapse of the USSR all along, and then also because Gorby cashed out at one point. The background reality, of course, is that Gorbachev intentionally misrepresented himself very successfully to the entire Soviet system, got into a position of power, and then used that position to accomplish goals he and Raisa had set for themselves decades earlier. I’m no believer in Great Men, but I _do_ believe that some positions give people quite a lot of power, and that people who attain those positions with previously under-appreciated goals can then use that power in unexpected ways that permanently change the course of history. The failure of Tetlock and Gardner to engage fully with Gorbachev’s personal story is of a piece with their failure to really engage with Shinzo Abe’s character. Individuals in powerful positions can have significant impacts on the broader world, but they are still individuals. When Flack says he shouldn’t be at Davos, on some level, Flack is saying that his own lack of awareness of stuff that one is assumed to be aware of by dint of being at Davos could make putting Flack at Davos a real risk to the larger world. I’m not at all convinced that Tetlock and Gardner heard that ; they heard “humility”. I’m like, whatever dude.
Anyway.
The GJP is still around, so if you have a lot of time on your hands and want to play, let me know what you think. If you would like to participate in the GJP _and you know me other than through reading my posts on this blog_, and that $99 fee is a stopper for you, get in touch with me. I’m tentatively willing to fund participation, to get the opinion of people whose opinion I value. If I don’t know you, well, I don’t know whether I value your opinion, and I’m not going to try to answer that question now.
TL;DR: Should I read Super Forecasting? No.
Executive summary: like a lot of efforts to “science-ify” things that are difficult to “science-ify”, they change making predictions so much that it is not at all clear to me that I have much interest left once they are done with it. I’m sure not interested in their metrics or time frames.
Where I’m coming from: I’ve spent my entire adult life trying to predict the future, and having very interesting, highly rewarding failures. My default time horizon is 30 years. I was unable to find anything in this book to help me with that lifelong project, altho I was able to use it to clarify how I think about things and thus hopefully I will be better able to articulate what I do, how I do it, and why I think it works.
ETA the next day: After a good night’s sleep, I realize that I missed something very obviously weird and wrong about this book. The forecasting questions are more or less all questions that “normal people” have limited understanding of even the nouns (country names, technical criteria about the economy, etc.). The “super forecasters” had an excess of computer programmers / scientists _at least in the examples given in the book_. The methods involved first coming up to speed on what the question meant and then getting enough background to come up with a baseline. Finally, the authors “extremized” the resulting prediction unless the group was very homogenous, on the principle that the participants had varying information / processes and some of that was applicable / relevant / successful in the resulting prediction and some was not and exaggerating the prediction would benefit a successful forecast less than it would harm an average forecast. That last bit is a massaging of the data that I would kinda would like to see a bunch of very wise data scientists weigh in on, so that I can hear the level of contempt / how grudging the acceptance is, to know whether it is just clever, or if it is actively malign.
Anyway. The two takeaways I get from this whole thing are that the intelligence community and corporate forecasters in general are (or at least were, at time of writing) a little too homogenous, and that they do not have the most up-to-date / best-in-class ways of interpreting data. Again — not surprising. If we are taking a bunch of elites who are or might otherwise be lawyers and making them analyze data, we’re going to get lower quality results than if we took a bunch of basically numerate people, trained them in state-of-the-art data analysis, and then turned both groups loose on the same data and questions about that data. Fixing this problem is likely to be tricky, because the communities in question already exist and will tend to self-replicate and there are some unaddressed questions here about who do we trust with sensitive information. I think having access to the secret stuff is probably super useful, useful enough to _almost_ compensate for the homogenity and the low quality of data interpretation.
![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}