Hey there, I'm Alix Dunn, and welcome back to Computer Says Maybe, where I talk to the people pushing back on tech power and fighting for something better. It is officially October, which means the US midterms and other global elections are right around the corner. So this month, we are looking at politics and the role that AI is playing in these elections, really through one question: who gets to shape the future?

We're gonna look at how powerful people use predictions about technology and the future as political tools to make certain things feel inevitable and in turn narrow the realm of the possible. Then we're gonna get into what happens when people stop believing those predictions or stop trusting the institutions making them.

How can that mistrust become leverage? And how can local resistance become sustained political power with a new vision for what the future should look like? Today, I'm joined by Carissa Velez, author of Prophecy: Prediction, Power, and the Fight for the Future, From Ancient Oracles to AI. Carissa makes a compelling argument that predictions made by people in power aren't forecasts.

They are orders, self-fulfilling prophecies. So when we hear big tech talking about the inevitability of AI, we have to ask ourselves, are they predicting what's actually going to happen, or are they just telling us what they want to happen and using their power to make it so? Throughout the month, we'll be in conversation with people tackling these questions on the podcast and in the Maybe Collective.

The Collective is a community we recently launched where you can meet co-conspirators, join events with guests from the show, and dig deeper into the conversations we're having every week. We would love to have you join us, so we'll drop a link in the show notes. Now let's get into my conversation with Carissa.

I'm Carissa Veliz. I'm an associate professor at the Institute for Ethics and AI at the University of Oxford, and I'm the author of Privacy is Power, which is a book about the machinery of surveillance, and Prophecy, which is a book about how the machinery of surveillance feeds the machinery of prediction, and it's about how prediction has been used and abused for all of history.

Prediction is a command disguised as description. I'm gonna say that one more time for listeners 'cause I feel like it's such a dense but also interesting sentence. So prediction is a command disguised as a description. Can you explain a little bit what you mean by that? Because we tend to read predictions as scientific enterprises or as statements that have to do with facts, I thought it was very important to emphasize the history of prediction, which is quite questionable.

And one of the things that make history useful is that when you're not lured in by the distractions of the present time ab- and about feeling fear or admiration or, or just falling into different tricks or trends, and then when you see it with a perspective of history, you see things more clearly. And one of the things that becomes salient when you look at the history of prediction is that there's a lot of power involved.

And that led me to take out my philosopher's toolkit and try to analyze prediction from a different point of view, from a more further away point of view. And one of the tools in the philosopher's toolkit is, of course, analyzing language. And when you analyze predictions, any kind of prediction, including very simple ones, like if I tell you, "Alix, it's gonna rain in a couple of hours."

You might think that is a statement of fact, like I'm describing the state of the world in the future, and that might be the case in a way if it rains. And if I'm basing my statement on an app that every time it says there's a 60% of chance of raining, it rains 60% of those ch- of those times, then even when it's validated, it's actually not a description.

At best, it's an educated guess. And, you know, if there's reason, you know, if I can see the clouds and my app says it's gonna rain and it smells the way it smells when, when it's gonna rain, it might be a very good educated guess and, and it might be very useful. But even then, it is not a description about the world because descriptions about the world are facts, and facts belong to the past and the present, never to the future.

So that's the first thing. A prediction can never be a fact, even in the most scientific of context. But furthermore, most of the time we predict with a certain function in mind. So we don't predict just out of curiosity. We're looking for something. We want something. So there's a lot of what Nietzsche called will to power weaved into predictions.

And when I tell you, "Alex, it's gonna rain in a couple of hours," maybe I'm telling you we shouldn't go out for a walk, or maybe I'm telling you we should take an umbrella. And so even with very simple predictions, usually there is an implicit command or an implicit suggestion. Now, when we look at predictions in the public sphere and see how they are actually used by public figures, we realize that those characteristics are magnified to the extreme.

So when a tech executive says that we're gonna use AI for this and that task next year, it's not only that it's not a fact, is that it's not an educated guess. It's a, it's very far from that. It's actually marketing or a kind of power grab. By making us believe in this future, th- they hope to trigger the mechanisms whereby predictions can become self-fulfilling prophecies.

And that is if they succeed in making you feel like you're missing out on something if you don't adapt to that future that they often describe as inevitable, then you will work for them. And in a way, when you believe uncritically the predictions of prophets, what you do is you obey in advance. It's not an explicit order, but it might just as well be.

Because when, you know, if the most popular tech executive of the day says, "We're gonna use AI for this," and you go out and you buy the AI and use it exactly for that, then th- they've won. They've done exactly what the prediction was meant to do. I feel like this idea of self-fulfilling prophecy is kind of an understood, like, daily expression.

What I found really interesting in your book, though, is applying it systematically to several examples in the context of technology. So do you mind, if they come to mind, other examples where someone predicts something in the realm of a technology or encourages someone to do something based on a prediction within the context of a technology, and then that actually ends up bringing about the very prediction?

Are there examples that you can share in that? 'Cause I think it's a really powerful idea. My favorite example is that of giving out loans or jobs or certain opportunities, because we are so Reliant on assumptions that when we use an algorithm to determine, to filter candidates, we tend to assume that those algorithms are tracking something real in the world, that if the algorithm says that you're probably a gonna be a bad employee, that it's tracking some characteristic in you that will make you a bad employee.

Mm-hmm. But when you look at actual examples and when you look at the data that is being used and how correlations works, you realize that many times they're not tracking anything of substance. They're just tracking correlations that can be very spurious correlations. So in the case of Amazon tried an algorithm a few years ago, and they realized that the algorithm was selecting men, and that if you had anything to do with the word woman in your application, say you had played for the women's soccer team, that would count against you.

And of course, that, that is outrageous. And the interesting thing about self-fulfilling prophecies, and in particular in the case of algorithms, because they're already quite opaque, is that they don't create error signals. So they don't create any kind of crumbs that will lead you to realize there has been a mistake.

Because they identify certain people as not being employable, they don't... Those, those people never get a job because everybody's using more or less the same algorithm. And then the algorithm can claim, or the company that produces the algorithm can claim, that their algorithm is 99.9% accurate. Hmm. But unless we have randomized control trials, we can never tell whether the outcome brought about was because there was some tracking of some characteristic or because the algorithm brought about that outcome in itself.

Do you wanna describe a little bit about this relationship between correlation and causation and this kind of fantasy that a lot of tech people have that that's gonna somehow be able to allow us to kind of see what is obviously impossible to see? If you have anything to do with the corporate world, you will hear this time and again in meetings about, "Well, the data don't lie," you know?

The, the data is all you need. Um, writing that chapter, too, was incredibly hard and incredibly fun and interesting because it's so in- so important to take a fresh look at numbers. Numbers get used so often as a kind of exclusionary door. It's like somebody cites a number, and the implication is that the conversation ends there.

And the people who cite numbers often count on other people not questioning them, not feeling confident enough that they can understand it, or not feeling confident enough to say, "I don't understand that, and therefore I'm gonna ask questions, because I should be able to understand it." And so numbers get used as this, as this other kind of power play, um, that is also an expression of the power play that the predictions are in themselves.

And yeah, when you look at the history of the relationship between correlation and causation, it's fascinating. So in the Middle Ages and before that, in, in, in ancient times, there was this general belief that the ultimate explanation for something and what it meant to understand something is to discover the ultimate cause.

In theological terms- People would think about it as God. But in non-theological terms, you would think about it as where the why question stops. So if you ask, you know, what caused this, and what caused this, and what caused this, like, what, what is the ultimate cause? And the further you could go, the more you understood the phenomenon in question.

With correlation, at the beginning of our understanding of correlation, we thought it was the second best. When you cannot achieve an understanding of the causes, then you rely on correlation. And famously, some of the, uh, scientists like Rutherford said that if your experiment relied on correlation, then it was a bad experiment, and you should go back and redo it.

And then some of the weakest sciences, and by weak I mean the less trusted sciences with less of a pedigree or less of social standing, started pushing for statistics. And statistics have been roughly developed by bureaucrats and gamblers. To simplify. Not by scientists, which is interesting. And then, but, like, sciences like psychology, and within psychology, parapsychology, started relying more on pushing for statistics.

And then eventually, yeah, a few generations of social scientists, including very f- famously Quetelet and Galton, transformed what used to be thought as s- a second best, a kind of not a deep truth about reality, into an explanation in itself. So before they came along, a correlation could only be interpreted as symptomatic, as it, as it, it's pointing towards a cause that we don't know, but it might, it might be there.

And these social scientists relied so much on statistics and showed some of the magic of statistics, like, for example, the normal curve, to, to such an extent that they gained the social status to justify statistics as an explanation in itself. But what we lose with that is an understanding of causality, which arguably is something that makes human beings quite smart when we're smart.

We also have a lot of capability for stupidity. But, like, when we're smart, it's usually because we have some causal understanding of the world. One of the phenomena that, that I think I'm not sure we appreciate enough is that the more we use statistics and the more we turn everything into a statistical analysis, the more spurious correlations we're going to find, just- Mm-hmm

as a matter of statistics. Mm-hmm. Probability, yeah. Yeah. One of the examples I give in the book, and that I get from Gerd Gigerenzer, is the incredible correlation between chocolate consumption and Nobel Prize winners. And so it turns out that Switzerland has a very high chocolate consumption and very high ratio of Nobel Prize winners, and countries like Brazil less so.

And it shows you how if you only rely on correlation, it would lead you to do some ridiculous things, like just increase the co- chocolate consumption in your country, hoping that will produce Nobel Prize winners, and good luck with that. Yeah, it was a really good one. Can you say a little bit more about the extent to which you think trying to guess, even educated guesses, is, like, just pointless because of the world we're in?

Yes. So on the one hand, I don't wanna give the impression that I am against any kind of prediction, um, because sometimes an educated guess is quite useful. But what I'm against is this unreflective use of prediction. And in particular, whether you're a business person or in policy making or just a responsible citizen or just a, like a human being who wants to navigate the world as competently as possible, it's very important to realize the things you can predict and the things you can't predict.

And so can we predict what the next season is going to be? Yes. And why we can predict it is not because we have hundreds of years of data that show that after the summer comes the autumn. It's not because of that. It's because we have causal understanding. Or not only because of that. So phenomena that tend to be cyclical, that tend to follow a normal curve, are the ones that are most predictable.

And the key to successful prediction is to know the limits of prediction. So even when you make a prediction, to have a sense of what could make the prediction not go the way you think it's going to go. So one example would be the pandemic. Pandemics are notoriously unpredictable because they vary a lot, and they can vary a lot, for example, in how contagious they are.

And depending on that number, it can be exponential. So anything that is exponential that defies the normal curve is unpredictable because a sufficiently extreme outlier will negate the normal curve. So whenever we make predictions about, say, supply chain or whatever it might be, you have to bear in mind that prediction will only apply if there is nothing like a pandemic that will disrupt everything and, and make everything different.

And so, yes, the world is fundamentally unpredictable. What's interesting is also that the events that are most unpredictable are the ones that are most disruptive. And so it's precisely the kinds of events that we would like to predict most that are the hardest or impossible to predict. And that suggests that we should change our mentality.

And instead of trying to predict those, which we know we can't predict, like how, how can we fall into the illusion, you know, throughout centuries? We should have a mentality of preparedness. So we don't know when the next pandemic is gonna be. We don't know what it's gonna look like, whether it's gonna be airbo- borne or some other kind of virus, but we know there's gonna be a next pandemic.

It might be in six months, it might be in a year, it might be in 100 years. Mm-hmm. But we can do things to prepare for when that happens I love this idea that prediction, distraction, preparation, and that example of the, I think it was geese, maybe just flock of birds in air travel. Can you share that example that someone shared with you?

This, this example comes from Margaret Heffernan, and she makes the excellent point that part of what makes airplanes very safe is that they are designed with a mentality of preparation, not of prediction. So every time you take off on an airplane, there isn't a team of engineers on the ground trying to predict whether a goose is going to impact your engine.

That would be extremely expensive and very dangerous. No, you build the airplane to withstand that kind of impact because you know that sooner or later it's gonna happen, and that way you don't have to predict. Are there other examples that you think are helpful for helping people understand that kind of zero sum tension between predictive capability and preparation?

I guess anything that can be catastrophic, either for your individual life or for our collective life, falls into that kind of setting in which you, you should actually prepare and try to minimize your, your losses in, in, in the event of a catastrophic happening, and that's why we have insurance. One of the messages of the book is how the world is full of con artists.

That is not where I thought you were going, but true. True. Um, there have been moments in history in which a whole industry is dominated by con artists. So- I couldn't possibly think of what you're referring to currently. One example is the financial industry, right? For decades and decades and decades and decades, people thought that you could hire a financial advisor, and they would show you how they could beat the market.

And how would they show you? By having many funds they invest in. You know, half of them do better than the market, half of them do worse. They delete the data on the half that didn't do well. They keep the data on the half that, that did well, and they show you, you know, what a, what a great, um, track record they have.

And for decades, this was a whole industry. It worked. And then came Vanguard and others and said like, "Well, actually you can have ETFs." And people like Warren Buffett have, have warned of the limits of this mentality of beating the market, but it worked for a long time, and arguably it still works. And in the same way, there are other industries and other institutions and other- And m- maybe in all institutions there is a ratio of con artists.

And prediction, it is a field which lends itself very well to con artists because prediction reaps benefits from people's fear, and that is a very dangerous thing. When you go to an expert and ask them about the future, you're usually, you usually put yourself in a position of vulnerability. What you're saying is, "I'm afraid.

I don't know what to do, and I want you to tell me what to do because I think you can see the future." And you can imagine how not everybody's a great person. Not everybody's a scientist who just wants to avoid people being harmed by a pandemic, and some people will capitalize on this k- vacuum of power that is available to them when we fixate on prediction.

So can we talk about tech bros? Like, they've been obviously making lots of kind of wild proclamations, but they say it in this very, like, mathy, engineer-coded kind of ways. How much do you think of these AI bros as con artists? I s- I, I don't wanna be overly harsh, and I don't wanna be overly generous.

Yeah, fair. And that's hard because- A con artist knows they're a con artist. So it's hard to assess people who it's not clear what they actually believe. You know, when these people talk about, there have been some bizarre claims of like, "I want Claude to be happy." Do you believe this? This is incredibly concerning.

Do you not believe this? This is incredibly concerning in a different way. When I, well, actually, when I, when I was thinking about con artists, I was lis- thinking about lesser con artists- Yeah. ... that profit from predictions, from this kind of discourse, from selling to governments the ability to predict this or that through consultancies and things like that.

With tech executives, it's harder to tell because I think that what they are doing, and I think they know they're doing it, is marketing. And so from one perspective, many of these tech executives, like Zuckerberg and Amodei and others, ha- have manifestos about the future, and some of them have published them quite recently, and Altman as well.

And so I think that if I were to sue them and we would meet in a court of law, I think a plausible response from them is, "Look, I'm a business person, and I was doing marketing, and everybody knows I have this company, and I have this product, and if people are clueless, people are clueless. I wasn't presenting this as scientific fact.

This was a piece of marketing." And that's partly why I wrote the book because o- one of the greatest values of writing is figuring out what you think. And so it's not only that you write the book, it's that the book writes you as well. It changes the way you perceive the world. For example, this idea about m- predictions a- as speech acts, which is the ph- the philosophical- Mm-hmm

jargon for which we were talking about them being commands in disguise. I didn't start with that idea when I started writing the book. I discovered that idea along the way. What prophecy did for me is that I can never look at a prediction in the same way again, and predictions that might have just blended into the background of facts in the past now jump at me like red flags.

And I hope that the book can do that for other people, so that when a tech executive says anything about the future, it jumps at you as marketing. Yeah. And there's a paper by Thomas Nagel called Concealment and Exposure, and it's about privacy. But in this paper, he has this brilliant section about how with- withholding certain information, certain private information from people, is not lying, just like being, wearing clothes is not hiding the fact that you don't have clothes underneath.

Yeah. And he gives this analogy of clueless tourists who think that the friendliness of the Americans is actual friendship, or that a British person telling you that your idea is very interesting is actually interesting. Trying to make us a little bit less clueless as tourists in, in, in this environment.

And I've been thinking about, okay, so, you know, these tech executives are businesspeople, they are doing marketing, they are selling a product. If they weren't businesspeople, if they were academics and let's say honest academics with integrity, which is, you know, not to be taken for granted- Yeah ... how would they phrase what they're saying, and how should I phrase it to other people?

And here's what I've come up with. I think that an honest version of what they're doing is saying, "Look, here is a vision of the future. This is where I think we should go because I think this is a good future." Mm-hmm. "This is how I think we get there. Do you want to join me and help me build this future?"

I think that's the honest version. There's also a really great quote that I think summarizes some of this stuff in the book that's near this, which is, "If you're predicting the future of humanity, then you're not doing science. Whatever you think you're doing is closer to reading animals' entrails, and most likely you're playing power games."

Which I feel like- When, when somebody listens to our conversation and thinks, "What do you mean the world is not predictable? Sure there are things that are predictable." Part of the nuance is not only things that are cyclical, like, like seasons, but also if you exercise enough power over a person, sure, you can predict what they do.

Yeah. So if you want to be absolutely certain where somebody's gonna be the next day, jail them. The, the price of predicting human behavior is modifying it and ultimately, and in the extreme, determining it. So I really, I would love to dig in a little bit more, 'cause you bring Hannah Arendt in a lot on how authoritarians- relate to predicting and that basically they need to be able to make predictions, but then they also need to guarantee that those predictions come true, and then basically their kind of power gets oriented towards making these things true.

And I think we can see that constantly. I thought that was really interesting. I hadn't actually read some of that work from Arendt and found it really useful and interesting. Do you wanna talk a little bit about this relationship between trying to build autocratic power and predicting things? Yeah. So both Hannah Arendt and Karl Popper were fascinated by prediction and issued a lot of warnings regarding prediction in politics because they had lived through the Second World War.

And I feel like those lessons have been forgotten, partly because, you know, Hannah Arendt is so... Her work is so broad, and when you read The Origins of Totalitarianism, there is so much in there that the comments on prediction can easily be lost. But I think if prediction is, is a more important kind of, um, key to understanding totalitarianism than even Arendt recognized, just judging by how m- how s- how much space she dedicated to it.

But her main argument is super dark and based on her experience and citing examples, and she claimed that the Nazis often phrased their views about the future in terms of inevitable prophecies, um, that play the role of justifying what they did, shielding them from accountability because it's not that they were murderers, they were just fulfilling a prophecy that was gonna be fulfilled anyway.

And she claims that they went to the extreme of being willing to hurt their own people just to make their prophecies come true, to validate their judgment. It's very interesting to see those comments in light of prophecies in other totalitarian systems like the Soviet Union Which was famous for its attempt to plan the economy.

And what is it to plan if not to predict and carry out that kind of projection into the future? So I think that it's the perfect time to reread Hannah Arendt. Yeah, exactly. So she makes the case that debating, quote, "About the truth or falsity of a totalin- totalitarian dictator's prediction is as weird as arguing with a potential murderer about whether his future vi- victim is dead or alive.

The only appropriate response is therefore to rescue the person whose death is predicted" is, like, such a- So powerful and so good. So I cited her word for word in my TED Talk. That's how much I like that quote. I put it in connection with a prediction from Larry Ellison, who is the CEO of Oracle. He makes the prediction that in, that we will live in a modern surveillance state in which citizens will be on their best behavior because they know they're being watched all the time.

And of course, he stands to earn a lot of money from this. But to argue with it is exactly like arguing w- with a murderer about whether the victim is dead or alive, and the only appropriate response is to rescue democracy from these kinds of dystopian views about the future that these people are trying to create, to bring into existence.

Yeah, you also pull her back in to explain that basically why are these hyper-powerful people seemingly willing to destroy the world, and she says that, "Only destruction can satisfy the veracity of someone who can't accumulate more power through wealth," which is like, this is so dark. But she's so right.

I mean, speak about, I feel a bit strange in a conversation about how prediction is oftentimes inaccurate. She's extremely prescient. I know, yeah. Because she understands human psychology. And I was reminded in this passage of the similarities there are between tyrants and tyrannical personalities and domestic abusers.

So why does a domestic abuser murder his spouse? Well, because that's a way to possess her forever, and then nobody else can have her. And so they are willing to not have her either by murdering them through that kind of unquenchable thirst for power. Yeah. Okay, we're going in a totally different direction.

I found some of the philosophy frameworks that you provide in the book really helpful, and especially around utilitarianism when you get into effective altruism, and you start... It's like a different kind of pathology. So can you describe what a utility monster is in utilitarian philosophy? Yes. So very briefly, utilitarian philosophy was born out of people like Jeremy Bentham and John Stuart Mill in the UK, and it's the ethical theory that what is ethically correct is to maximize utility.

And utility often gets cashed out as pleasure or wellbeing, and so sometimes it's described as, um, do- doing the most good for the greater number. This philosophy has been so successful in its marketing and implementation that it's essentially dominated large parts of public policy for a long time And even looking at literature, Charles Dickens was already quite annoyed at that.

And many of his novels react against this kind of mentality. But it's been very successful. And part of why it's been successful is because it gives a little number for utility. And so as soon as you can measure things, you tend to feel like you have a grasp on them, even though it might be an illusion.

And the utility monster is this problem for utilitarianism because a utility monster is somebody who reaps great benefits from something. So let's say you and I both l- like chocolate, but whereas you get one unit of pleasure from chocolate, I get 1,000 units of pleasure from chocolate. And so because the purpose of utilitarianism is to maximize these numbers, then you should give all your chocolate to me because I'm just gonna enjoy it so much more.

Sorry. Yeah. For the greater good, I guess. For the greater good. Yeah. Of course. Yeah. And this kind of rationalization has been used to defend very questionable courses of action. Like for example, donating more money to somebody in a rich country because they stand a better chance of changing things than somebody who's worse off.

And you can see how this could go very wrong. Yeah. And of course, in a way, AI is like a utility monster because if you describe a machine that will solve every problem that you have ever thought about- Yeah ... for the rest of history, then you're going to try to bet everything on it because it will- Yeah

solve any problem. Climate change, oh, don't worry. Oh, AI will solve it. War, oh, don't worry. AI will solve it, a- and anything in between. Yeah. And so we pour more and more resources into this utility monster, betting that it's going to promise everything that the tech executives are predicting, when, if you look at the history of prediction, that's probably not, not the way it's gonna pan out.

I mean, I know that there is no such thing as a free market, but it is really interesting watching the kind of self-fulfilling prophecy of the AI marketplace and the amount of money that has now been invested, air quotes, because it's like investment kind of belies a business model that is reasonable, and this more feels like just giant bets on this future coming about, and it puts us all in peril because it's essentially, like, our whole economy is chained to this thing.

Do you wanna say a little bit about effective altruism as, like, a, a way of rationalizing basically anything as a kind of, uh, a framework, and then potentially helping people understand why that's maybe not the locus of good decision-making for our sort of social organizing? Effective altruism is a movement that was born out of utilitarianism, born out of Oxford, I'm afraid to say.

And I think originally it was a group of very young guys who- I think we're probably well-intentioned. You're in your 20s, you wanna change the world, you wanna save the world. You think you're smart, and you are smart in many ways. You're clever. And you think, "Oh, yeah, I'm gonna solve pr- poverty," or whatever it might be.

And, and, you know, effective altruists started being interested in poverty. And the main idea is one that is incredibly intuitive, and especially for a 20-year-old who's an idealist. Um, and the main idea is that you should be an altruist, that you should try to benefit the world, to make the world a better place, and that you should do so effectively.

What's wrong with that? What's not to like? If you can donate 10 pounds to an organization that will save more people than org- another organization that with 10 pounds le- does less, then you should donate to the organization that does more with 10 pounds. That makes sense. Sounds so rational. Yeah. It sounds great, except that when you start getting into the details and the implications, it start getting really dark.

Because one of the implications of aggregating utility, aggregating wellbeing or saving lives or whatever it might be, is that you're willing to sacrifice others. And for example, if you're gonna save 10 lives, then it's okay to kill one because you're gonna save 10 lives. And so the utilitarian has no limits, no, no red lines, no human rights.

N- no red lines. As long as, you know, the good outweighs the bad, the bad isn't bad. It's good. And so one of the criticisms of a philosopher like Bernard Williams is that utilitarianism not only allows for very wrong things like killing, but it can encourage it in some scenarios. And instead of Portraying the more intuitive and I think factually correct view that there are things that you sometimes have to do that are wrong, and there's sometimes life puts you in a situation in which you have to do the least wrong thing.

It portrays it as a no-brainer, as just, that's just the right thing to do. And the reason why I say that you can't ever trust a utilitarian is because you never know what the calculation is gonna be. Yeah. Like, you might be the price to pay for saving the 10 lives. Along with, with utilitarianism not having any red lines, one, one of those examples is that the utilitarian is perfectly happy to lie.

That, you know, they will tell you, "Oh, you're, we're going to this very nice bar," to make you happy while they're leading you to, to a- That's my dad. Yeah. Yeah. I got it. Exactly. And so this, this idea of going into the jungle comes from the biography of Derek Parfit, who was a utilitarian, and he was a very important figure for effective altruists.

And he spent all his professional career at All Souls in Oxford, and his biographer interviews some of his colleagues, and one of his colleagues at All Souls says, "Would I go into the jungle with Derek? No." "Because he would think that his work is so important and so impactful, that he would throw me to the jaguar so that the jaguar eats me and not him."

So good. 'Cause it also, like, it has this air that you can engineer outcomes, which I feel like connects to this, these AI bros and just the people that assert that they know the future, is that they have this very condescending or self-anointing engineering instinct when it comes to society that's just really gross.

It's like, what gives you the right to think about how to engineer everything and everyone and have these huge effects on people's lives so casually as though that's your role in the world? And it feels very connected, even for people that aren't effective altruists, but the people that are working within AI and have the hubris, that's the word I was looking for, to like assert that not only do they know the future, but they're shaping the future, and that you should let them, and that you should give them money to do it.

And it's just this whole godlike complex. Yeah. Bernard Williams also said that utilitarianism was, was particularly appropriate or fitting for a mentality of colonialism, which it is. And another utilitarian called Henry Si- Sidgwick wrote in this book that it was okay for a small utilitarian elite to make decisions for the rest of the world and lie about it, because if they told the truth, people might be upset and it might break down the project.

And it's this mentality of it's okay to do whatever you want to do because you're justifying it for the greater good, and it doesn't matter whether all these people don't want it, you know better. And one of the quirky things about the book is that it covers a lot of topics, and they might seem disconnected, but I...

They're actually not. Yeah. Because if you don't have faith in prediction, then effective altruism falls apart. And one of the characteristics of current effective altruists' discourse is they're not trying to predict what's gonna happen in six months or 10 years, which by the way, they have been incredibly unsuccessful at doing.

Like, dramatically unsuccessful. Like, really bad at that, yeah. Really bad. So for, m- I think more than a decade, they, for example, pushed hard for Using deworming as one of the most effective ways to help people- Yeah ... in disadvantaged countries. And it turns out it's not effective at all. And furthermore- Yeah

they impose deworming on kids without the consent of their parents, which created a lot of animosity and distrust, and kind of secondary effects that were not positive at all. So it's not that they're trying to predict within 10 years, which they've already shown to be incredibly bad at. They're predicting what the world will look like in a thousand years, and relying on assumptions like we will be to, able to upload our brains, which what does that even mean?

But anyway, and back to where we started in the conversation, we said that utilitarianism was willing to sacrifice the few for the greater good. And one of the ideologies that, that is gaining prominence within effective altruism is called longtermism. And the idea is that we should look at the world in the long term and not be too short-termist, which sounds like a great idea.

Again, many times in politics, politicians are famous for focusing on the election and not on, on the greater kind of project of saving democracy, say. But what they mean is, okay, so w- they're gonna predict how many people are going to live in the future, and their prediction is that it is in the trillions, so they floated different numbers, but like 80 trillion or whatever.

Sure. Once you reach a certain threshold- Sure ... like it's who, who cares? Whatever. Like, it's, yeah. And then they compare that with the billions that live today. And now clearly trillions trump billions, and so they are willing to sacrifice the wellbeing of the billions that are not hypothetical people, that are alive and breathing today, for hypothetical future people.

And this is all faith on prediction that has shown time and again to be unsuccessful. So it's really quite remarkable. Effective altruism is relevant in the context of AI, not only because it's a philosophy that greatly depends on prediction, but because many tech executives have embraced effective altruism and justified some of what they've done- By, by the principles of effective altruism.

And because effective altruism has used longtermism to justify investing a lot of money, for example, in AI safety. They claim that reducing the possibility of existential risk from AI, so AI destroying the world, by even, like, a billionth of a percentage, is more effective when you take into account the trillions of people that will live in the future than, say, helping people today.

And so that is why it's so important and so relevant to think more carefully about this philosophy. And one key thing is because they tend to make projections so far into the future, they use a trick, which is to use the infinite. And once you use the infinite, you can justify everything. Because just like we were saying, like, if AI is gonna be infinitely good for everything, then it's worth any kind of investment, then what- whatever you do, no matter how bad, is gonna be justified by the infinite.

Because in the infinite, you can't make the world better or worse. It's just whatever you do, whatever act you do, it's just a blip in the infinite. You can't add a good act to infinite amount of good, and you can't add something bad to infinite amount of bad. And so by making these large predictions and using these large numbers, what they are effectively doing is justifying anything through the mathematical trick of using infinity.

Yeah, it's like a zero gravity moral environment where, like- Exactly ... nothing really. What role do you see democracy playing in helping us navigate the fact that we don't know what's coming? So I empathize with the desire to make the world more predictable. It's nerve-wracking to acknowledge that your life could change any second, for better or worse.

So I understand the anxiety. But that impulse to make the world more predictable can be very dangerous. A writer called Helmut Rosa makes a point that when we try to control the world, it's like the, the fabric of reality fights back, and ironically - Mm ... we create more monstrous forms of unpredictability.

So we can create the illusion of predictability in the short term, but in the long term, we might create these monstrous, unpredictable events, like for example, nuclear going wrong, like we've seen a few times in history. And when you think about it more carefully, would you really like a completely predictable life?

And I think it's quite obvious that you would not. It would mean that you wouldn't get to meet someone who you didn't expect, who might change your life for the better, who might teach you new things, that you would never get surprised in any positive way, that you would know exactly where you're gonna be not only tomorrow, but a year from now and 10 years from now.

That probably means you're in a police state. Um- That's my nightmare. I don't wanna- Yeah ... that happening. Like, I Exactly. Part of what makes democracy is precisely that we don't know who's gonna win the election. Because if we did know, then it wouldn't be democracy at all. Yeah. And that's scary, but it's also exciting because it invites us to be more of agents in, in, in our world.

Part of what annoys me so much about the predictive mentality is that it turns us into passive subjects. The mood of prediction is the future is written, and all I can do is try to get a glimpse of it and adapt. Whereas the mentality of embracing uncertainty is, no, the future is not written. It's partly up to me to write it, and I have to get off the couch and stop doom scrolling and actually build the future that I want to live in.

Because if not, somebody else will build it for me, and it won't be my world, it will be theirs.

Thanks so much to Carissa for that conversation. One idea that's really stuck with me from reading the book is not just how predictions can make a particular vision for the future feel inevitable, but how they can then be used to justify decisions in the present. So we're seeing that play out with AI now as theoretical promises about what the technology might make possible are spurring wildly speculative investments and competition over local resources.

Data centers have become one of the hottest issues on this year's ballots, and for many people, they're an entry point into these questions about AI, power, and who gets to shape the future. So next week, I am talking with researcher and writer Ketan Joshi about how data centers are amassing so much power, both literally and politically, on a global scale.

He will talk us through who's building all these data centers and how, who benefits, and who's left paying for it in energy bills, climate costs, and higher prices. Again, if this conversation got you thinking and you're looking for a space to share your thoughts, do join us in the Maybe Collective, where we'll be unpacking the series together all month.

As always, you can catch up on last month's episodes wherever you get your podcasts or watch full episodes on our YouTube channel. So thank you to our podcast team, Kushal Dev, Marion Wellington, Van Newman, and Zoe Trout for all their work making this podcast happen every week, and we will see you next time.

‍