Superintelligence may give everyone better answers. That doesn’t mean we’ll agree on what to do with them.
A couple weeks ago, Mark Zuckerberg published an optimistic vision for our AI future in the Wall Street Journal. I agreed with more of it than I expected.
In the piece, he argues that the defining question of the coming era is not whether superintelligence arrives, but who gets to use it. He worries about a future in which extraordinary intelligence is concentrated in a few companies, governments, or institutions.
Instead, he proposes "individual empowerment as the source of prosperity, invention as the primary purpose of superintelligence, and balance of power as the foundation of safety.". I do believe AI is going to make intelligence radically more abundant, and I would much rather see that abundance distributed than hoarded. I also agree that we should think about AI as more than an automation technology. The exciting question isn't just how much work AI can do instead of us; it's what humans might be able to do once intelligence that was once expensive, scarce, or hard to get suddenly becomes cheap and everywhere.
A search problem, not a negotiation problem
But besides that, I gotta say, Zuckerberg loses me. His essay seems to assume that many of our problems exist because we haven't yet found the right answer, that if we give every individual enough intelligence, each of us will be better equipped to discover the choices that best fit our goals. He seems to be saying, "If we could just have more intelligence available, things would sort themselves out." And this is why I suspect Mr. Zuckerberg has never tried to just shop for something as simple as bodywash or toothpaste on Amazon.com.
Choices, choices, everywhere
Amazon offers me OVER NINETY THOUSAND RESULTS when I go searching for body wash. That is an absurd amount of choice and optionality. There are countless formulations, scents, ingredients, price points, brands, sustainability claims, bottle sizes, and shipping speeds. In one sense, that's a real accomplishment: for most of history the problem with soap was simple scarcity. Now abundance has solved that problem, but has it made it easier to buy soap? No!
This is the familiar paradox of choice. Once options get abundant enough, finding an acceptable one becomes trivial, and deciding among the thousand acceptable ones becomes the actual work. Now we've got Wirecutter, and Consumer Reports, and influencer reviews, all trying to solve this new problem we created: hundreds of thousands of options for one sub-sub-category of soap.
Yes, Mark, superintelligence could help enormously with this. Imagine an AI that knows everything about me, understands my preferences, reads every ingredient list and every review, compares every product available, and announces: I've considered all 8,742 possibilities. Graham's super-intelligence declares:
This is, by far, the best body wash for Graham Walker.
Fantastic! Hooray!
Except my husband wanted bar soap.
Well now what?
And this is my problem: Zuckerberg’s model treats preference as a discovery problem. In much of human life, preference is a negotiation problem.
Body wash purchasing is, of course, a trivial argument. But I think it exposes something real about Zuckerberg's vision. My preference and my husband's preference aren't undiscovered variables waiting for a sufficiently intelligent system to uncover them. They can just be truly incompatible.
Maybe I care more about moisturizing and he cares about fragrance. I dislike bar soap; he dislikes plastic bottles. We both want less clutter in the shower. There may be no objectively correct answer hiding in the data. We have to negotiate or compromise.
Superintelligence might help us understand the tradeoffs, or suggest compromises neither of us considered, or even clarify which preferences we care about most. But at some point, if our preferences stay incompatible, something has to give. I mean, great, our super-intelligence has created higher order disagreement. It didn't eliminate it.
Multi-agent systems don't resolve themselves
Wouldn't your life and job be easy if everyone just agreed with you?
Most of the hard parts of my job as an emergency physician aren’t search problems. I can order a CT scan. I can order labs. The difficult parts begin when people disagree: the family who thinks I haven’t done enough testing, the patient who wants to leave when I think they should stay, the pharmacist who thinks the dose I’ve ordered is unsafe, the consultant who doesn’t think the patient needs admission.
More intelligence can technically help with every one of those disagreements. But it can’t guarantee they’ll disappear, because they aren’t necessarily failures of intelligence. They’re conflicts between people with different information, incentives, values, risk tolerances, and responsibilities.
In AI terminology, these aren’t single-agent problems. They’re multi-agent problems. And Zuckerberg himself acknowledges this: “Humanity isn’t a monoculture.” He continues, “There is no technological solution that can align with everyone’s opposing interests and diverse values at once.”
Exactly. But I don’t think the rest of his argument fully confronts what follows from that observation.
He illustrates the benefits of distributed superintelligence with a thought experiment: imagine only one person had a superintelligent lawyer. That person would have an unfair advantage. But give everyone a superintelligent lawyer, he argues, and justice would be carried out far more fairly and efficiently than it is today. But would it?
Give both sides in a case an AI capable of producing a five-hundred-page brief in seconds, and you don't get resolution. You get an arms race in argumentation, each side's system reading and rebutting the other's output faster than any human could follow. Somewhere at the end of that exchange, presumably a superintelligent judge or superinteligent jury still has to decide: guilty or not guilty? Liable or not? Which principle should prevail when legitimate principles conflict?
Sure, we'll have made legal intelligence abundant. We won't have made adjudication abundant. If anything we'll have made it harder, since the scarce resource becomes the finite attention we humans have to weigh in.
A blind spot money can buy
Lastly, I do think it's particularly interesting that this type of argument is coming from Mr. Zuckerberg, because I think it may be a massive blind spot he's unaware of. I've never met the guy, but a lot of the frictions I'm describing — negotiation, shared scarcity, incompatible preferences — are problems that extreme wealth can probably insulate a person from. For most people, life constantly forces incompatible preferences into the same constraint.
- Two people want different vacations but can only afford one.
- Two departments want the same employee.
- Two patients need the same bed.
- A family has a finite budget.
- A couple shares one home.
Money doesn't eliminate disagreement, but enormous wealth can turn a lot of coordination problems into purchasing decisions. Instead of choosing between A and B, you can often just have both.
I don't pretend to know how Mark Zuckerberg makes decisions at home, and "out of touch billionaire" is an cheap shot, but I do wonder whether the essay's model of human agency — and what he thinks us plebian challenges are — reflects the constraints he thinks we're all struggling with, not the real challenges we actually have. For most of us, "something has to give" is still a fairly fundamental law of physics.
Trust isn't a closing line
Which brings me to the part of the essay I have the hardest time accepting. In his conclusion, Mr. Zuckerberg concludes:
Meta is committed to building with the principles of individual empowerment, invention and balance of power.
Frankly, I'm not sure why I should believe him. Facebook and Instagram have certainly let billions of people communicate and create, but Meta's history is not exactly a clean case study in decentralizing power.
These are enormous centralized platforms that drove people away from the open web, whose owners control the algorithms, the distribution, the monetization, and the rules. Individual user vs. Meta has never struck me as a great example of a balance of power.
Trust isn't built by declaring principles in The Wall Street Journal. It's built by what happens when those principles get inconvenient: when individual empowerment conflicts with Meta's business model, when openness conflicts with competitive advantage, when a billion people's preferences conflict with one another. Who controls the distribution layer that "personal" superintelligence reaches people through? Who sets its boundaries? Who holds the data that makes it personal? And most of all, what power is Meta actually willing to give up?
What becomes scarce next
Those questions matter because Zuckerberg is right about something basic: intelligence is going to become more abundant. (If you want my full thoughts on the matter, this is the start of my keynote at the Australian Medical Association's National Conference in 2 weeks!) I hope he's also right that it ends up broadly distributed rather than sitting exclusively with a handful of corporations and governments. But abundance doesn't eliminate scarcity. It moves it.
When information got abundant, attention got scarce. When products got abundant, choosing among them got hard. If intelligence gets abundant, the bottleneck will probably move again.
Zuckerberg writes that "the defining question of our age isn't whether superintelligence will exist, but who will have access to it." That's an important question. I don’t think it’s the defining one.
The harder question comes the day after everyone gets access.
What happens when everyone has superintelligence, every side has an extraordinarily good argument — and we still disagree?