Monstrous rationality
AI isn't dangerous because it will abandon reason. It's dangerous because reason isn't enough.
I studied political philosophy at university.
I fell into what’s probably a pretty common trap for a first year student. I read about Jeremy Bentham’s theory of Utilitarianism.
The greatest good for the greatest number.
Clean and simple and rational. It doesn’t support privilege, sentiment, private interests. It’s a simple calculation that asks what produces the most good for the most people. What could be fairer?
My tutor at Durham, the ever provocative Dr Dawson, asked me a question.
Doesn’t that potentially justify slavery?
His point was not that every utilitarian is a deliberately pro-slavery.
But if “good” is just math, then people are just variables. And abstract. So the benefit of enough people can counterbalance the suffering...degradation even...of a minority.
That’s elegant. It’s also horrific.
And we’re making systems that are getting very good at the elegant part.
If every human being on the planet could be guaranteed a good life, at the expense of one person being tortured forever, is that a reasonable bargain?
And if it is, when does it stop being reasonable? How many “good” lives are enough. A hundred? A million? Everyone?
That is not to say that reason has no place in ethics. It’s vital.
If it isn’t there, then our ethics are just instinctual. Private feelings. Tribal or national loyalty. Superstition. Power. So reason exists to help us look beyond just ourselves. Without reason we wouldn’t have the principles of consistency, fairness, and restraint.
We need it.
But not by itself.
Pure rationality has no human element.
Which is a worry about how we build AI. AI isn’t irrational. It’s not evil. It doesn’t hate us. It doesn’t want anything.
It calculates and recognizes and optimizes. It tries to reach a clean result.
And it has no body. It can’t be wounded. It doesn’t have emotion. No memory of shame. It won’t - as most people would - instinctively recoil at someone suffering.
AI never has a moment where, even if the numbers work, it would choose to say no. People do.
And it doesn’t need to hate anyone to produce inhuman outcomes. It only needs to optimize “good” when it can’t feel “bad”.
It’s not just a machine problem. We can see that lack of empathy in people, and how it applies to human policy when the people in charge narrow the frame to a binary view of efficiency.
Take foreign aid.
In a very simplistic view, the dismantling of USAID can look rational. The US saved tens of billions of dollars, and that money stayed in the US. That expenditure didn’t benefit US citizens directly, so it was a waste.
Those cuts and dismantling are estimated to cost 14 million additional deaths across the world by 2030. A third of those will be children under the age of five.
US foreign aid is a tiny share of federal spending. The total (including USAID) is about 1% of the total outlay.
Who accepts that bargain? Or, to bring it down to an individual level...how many people would you let die to save 1% of your annual budget?
The horror here is that the calculation is rational if you remove human reality. Those children are somewhere else. You don’t need to see the suffering. The savings are immediate. Those deaths - just statistics.
It’s even more disturbing in a human being. They have to refuse the empathy. But it can scale more rapidly in a machine. It can’t feel that empathy in the first place.
This is where Hannah Arendt’s phrase, “the banality of evil,” applies.
She’s talking about a moral vacancy. It’s not just that ordinary people can do terrible things. It’s that terrible things can be processed through ordinary systems. Compliance, language, bureaucracy. So the evil becomes banal, because nobody believes that they are choosing it. It’s just doing the work, even more impersonal than “just following orders.”
AI can - and if we’re not careful, will - intensify that.
An AI makes a recommendation. A dashboard says it should be accepted. A manager takes a look at that metric - it’s optimized. They approve it. A person is harmed, but it disappears into the workflow.
That’s harm as procedure. As an entirely reasonable output of a system that’s defined narrow terms and doesn’t feel the weight of what’s being done.
You don’t need to be Catholic, or religious at all, to find value in the Vatican’s recent warnings about artificial intelligence. The Pope isn’t talking about doctrine. He’s talking about dignity.
Dignity is something that doesn’t fit into the math.
I’m a person.
I’ve no doubt that I can be - and probably am - measured in various ways. My utility, productivity, and efficiency by my company. My citizenship by my nation. My purchasing power and risk score by stores and credit agencies.
But if you believe in dignity then you believe that a human can’t be reduced to those things.
It doesn’t remove tragedy. But it might at least view it as a tragedy. An understanding that scarce resource distribution creates pain, even if it can’t avoided.
A purely rational system might always justify one more acceptable loss.
The human question is different.
What can we not do? What must we not do?
AI can’t answer that for us. We struggle to answer it as a culture ourselves. I’m shocked by how much evil so many people will justify, again and again. Those same people would probably look at me as an impractical bleeding heart.
AI can help us reason. It can’t tell us when reason has to stop.
We talk about human in the loop (HITL) at work. AI systems that have human review, oversight, approval.
This is the same writ large. Human judgment, human dignity, human refusal.
This is for society as a whole.
That can only be us.
Further reading:
Renic, N. & Schwarz, E. Inhuman-in-the-loop: AI-targeting and the Erosion of Moral Restraint. OpinioJuris, Dec 2023.
Stonebridge, Prof L. Hannah Arendt’s lessons for our times: the banality of evil, totalitarianism and statelessness. The British Academy, Aug 2024.
Utilitarianism - Ethics Unwrapped. UT McCombs School of Business.