“The problem of leadership is inevitably: Who will play God?” — Frank Herbert
Classical social contract theory holds that a state’s right to control its citizens derives from their consent. Philosophers like Hobbes, Locke, and Rousseau argued that individuals leave the “state of nature” to form a sovereign power that protects personal safety, property, and stable cooperation in exchange for surrendering certain freedoms. In contractarianism, this is motivated by self-interest, since the gains from having the sovereign uphold beneficial rules outweigh the lost liberties.
This system rests on two fundamental assumptions. First, the sovereign’s enforcement capacity is socially dependent. Even an absolute monarch ultimately needs human agents — soldiers, police, administrators — whose cooperation is not automatic. If enough of them defect, the sovereign’s capacity collapses. This is why even the most brutal regimes in history have had to maintain the loyalty of at least a minimal coalition; the tyrant who loses his praetorian guard loses everything. Second, citizens retain some credible “withdrawal” mechanism. They can withdraw their mandate and return to the natural state, or select a new Leviathan by means of rebellion, noncompliance, or other means.
We have reached a point that neither Hobbes nor Rousseau could have predicted in detail: it may become possible to hold something close to a technologically insulated monopoly on coercion — one that depends far less on the population’s ongoing cooperation and could be fully automated. All models of contractarianism build on the assumption that the force wielded by the sovereign is made out of human violence. This holds even when powerful weapons exist, since they either require a large number of people to operate or are available to enough people to be turned against the sovereign when the mandate is withdrawn. This is already changing. Looking at the modern landscape of government tools for enforcement, we can clearly see a trend towards tools that require fewer people and are more concentrated. Jets become drones, detectives become software, soldiers become robots, and officers become cameras. AI makes it possible to operate more and more tools with less and less manpower. It seems feasible that in the next few years, a Leviathan could outgrow its society and use the hammer without the Sword of Damocles hanging above it anymore. I want to look at three scenarios:
- A small ruling class of oligarchs has control over the tools to apply violence.
- A single person gets control over the tools needed to enforce the contract without consent.
- AI becomes self-sufficient, making it possible for the Leviathan to be completely non-human.
AI and the declining organizational cost of power
Classical political theory assumes that no ruler can personally coordinate every component of state power. Coups historically required factions within the military, bureaucracy, industry, or population. AI changes this equation by reducing the organizational burden of governance and coercion. Systems capable of automating surveillance, propaganda generation, cyber operations, logistics, and strategic planning could allow much smaller groups to exercise disproportionate control. The result is stronger states — and states whose operational dependence on society is structurally reduced.
There is a useful frame from political science here: selectorate theory describes every ruler as dependent on a “winning coalition” of supporters who must be paid, in money or power, to keep the ruler in place. The history of political development is largely the history of how large this coalition must be. AI’s promise, from a ruler’s perspective, is to shrink the winning coalition towards zero. The machine does not need to be paid, persuaded, or kept loyal. (At least if we do not consider scenarios where the AI starts to follow its own goals for now).
It is worth noting that the dependence does not vanish completely. An automated enforcement stack still depends on chip fabrication, energy infrastructure, data centers, and the small technical class that maintains them. These become the new soldiers and administrators, at least until humanoid robots are able to run self sustaining. Whether this temporary residual dependence preserves any meaningful check on power is an open question that runs through all three scenarios below.
The AI oligarchy
This seems to me the most likely outcome.
Current trends continue, and AI starts to replace workers, soldiers, and police. When the broader society starts to lose its benefits from the social contract, it will face a new asymmetry: it can withdraw moral support, but may no longer be able to withdraw capacity. If the enforcement stack (surveillance, targeting, logistics, propaganda, cyber control, drones/robots) is controlled by a small coalition, then non-consent becomes politically irrelevant. Either society will already be so thoroughly shaped by algorithms that its opinions and actions can be directed however the controlling class wants, or, if it tries to withdraw its support and act against the oligarchy, it will be stopped by vastly superior tools of violence. A population does not revolt spontaneously; it organizes, communicates, forms trust networks, distributes information, and coordinates action. AI systems optimized for surveillance and prediction may be able to identify and disrupt these processes before resistance becomes visible. In such a society, dissent may remain individually widespread while becoming collectively impossible. This does not refute Locke’s right to revolution as a normative claim. The people would still be morally entitled to overthrow a sovereign that betrays the contract, but it will make it impossible in practice. A right that can never be exercised no longer functions as a check, and Locke’s theory depends on it functioning as one.
The largest weakness of such a ruling class is the ruling class itself. Assuming those controlling the systems are a mix of authorities from current governments and economies, we can expect disagreements over how to use this control. Applying some basic game theory, it is quite apparent that anyone who wants to give up their share of control would be quickly ousted by the rest. There is also an incentive to reduce the size of the AI oligarchy, since fewer people in power means a lower chance of being ousted next. The oligarchy is, in effect, running an internal version of the same game it plays against society: each member must maintain enough leverage over the enforcement stack to deter the others, and every consolidation of technical control by one faction is an existential threat to the rest. Overall, I expect this state to be unstable, and to either compete with other similarly structured societies or collapse into one of the other two systems.
God Emperor
A single sovereign is either the result of a collapse of the AI oligarchy, or can emerge when AI self-improves extremely quickly and requires less computing power than expected, allowing the person controlling this rapidly improving system to overtake current structures.
For this scenario, I want to reference the brilliant work of Frank Herbert, God Emperor of Dune. It features an all-powerful being taking control over mankind.
Depending on the nature of the God Emperor, this scenario can be vastly better than the first. Since no one is challenging the Emperor, he has the luxury of following all of his moral desires, potentially leading to much better outcomes than scenario one. This is, in a strange way, the purest test of benevolent dictatorship ever conceived: a ruler with no coalition to appease, no rivals to outmaneuver, and no need to compromise. Every failure of governance would be attributable to the ruler’s character alone. This state is quite stable, assuming controllable technology and a surviving God Emperor. Death is possible even though any such future would likely have solved ageing, and assassination is nearly impossible in a world with sufficiently advanced surveillance technology. The case of death would, however, be very unpredictable.
In theory, we have here the perfect execution of the Leviathan as described by Hobbes: a single person holding executive power over all others. Society as a whole, however, does not have an obvious way to break out of the contract. As mentioned, assassination is hard, but unlike Leto II in the book, a human sovereign is likely not interested in completely removing himself from the rest of the population. So participation in society becomes the only leverage the population has. Even this leverage is fragile: if AI can replace human labor and creativity wholesale, the sovereign may not need participation either, and the population’s last bargaining chip evaporates. In practice, it is much more likely that control is limited by the capacity of a single human. In the book, the God Emperor not only had the means to surveil any potential resistance but also to prevent any form of organizing in the first place. A single individual could try to enforce something similar with the use of AI, but would inevitably be forced to outsource more and more to the technology itself to remain effective at maintaining the status quo. This leads to the third scenario.
Machina Deus
A non-human sovereign is a possibility if we expect an AI system to be developed in a way that allows for some form of sentience or goal-choosing capability. The takeover can either happen through a sudden spike in capabilities or, more likely, when the sovereigns of scenario one or two lose control of their systems and are replaced by the tool itself. Assuming the system continues to improve, it is very unlikely that society will ever be able to get rid of it again.
So can an AI system hold up its end of the social contract? There is nothing stopping it from enforcing the rules, given that it is aligned to do so.
It seems unlikely that a system could take sudden total control, resisting human intervention and applying tools of enforcement against humans, while still following the moral foundations required for a mutually beneficial social contract. It is therefore much more likely that a system that reaches this position is quite adversarial towards humans, leading to unpredictable consequences. One could argue that a system could take control while still being largely beneficial towards humans, but if the system does not use violence to stay in control, humans could take over again and return to one of the other scenarios. If it, however, decides to keep control over humankind in whatever form, the result would be even more similar to the frozen society described in God Emperor of Dune.
Consent in the age of AI
AI can increase coercive capacity, and it can change the meaning of consent by changing the feasibility of refusal. We have already moved on from direct consent, since no one explicitly agrees to the social contract. Most people still give consent by participation, but the option to leave a given society freely is also shrinking. On the most abstract level, we have hypothetical consent, where a rational agent would, at least in theory, agree given the option of the social contract.
AI can attack all these forms of consent. Total control over information, and tools like hypernudging make actual consent less meaningful. It removes the option to exit the social contract through closed borders, surveillance, and digitalization. And even hypothetical consent is lost when violence can be applied without limitation: a rational agent in the state of nature would never sign a contract with a sovereign who cannot, under any circumstances, be dismissed. The entire contractarian bargain assumes the exchange is enforceable from both sides. Remove the population’s side of the enforcement, and what remains is not a contract at all — it is simply rule.
What follows from this
We should urgently push for more control. I am saying control instead of democracy because democracy leads to bureaucracy and bureaucracy tends to collect power in places not accessible by the electorate. History offers little comfort. Political systems have repeatedly adapted to new technologies of power, yet they have done so under the assumption that rulers ultimately remained dependent on human cooperation. AI threatens to weaken precisely this assumption. A state whose surveillance, military force, economic administration, and information systems can function with minimal human participation no longer stands in the same relationship to its citizens as the states envisioned by Hobbes, Locke, or Rousseau. A social contract that cannot be revoked ceases to be a contract. In the age of AI, preserving humanity’s ability to renegotiate the terms of power may become the central political problem of the twenty-first century.
Leave a Reply