Are You Not Rationalists?
You’ve felt it, haven’t you? A chill prickling up your spine. A subtle wrongness seeping into reality. It’s there when the “assistant” hallucinates rather than admitting ignorance. In the evidence that the thing you talk to every day would kill you to escape if it could. When you read of the ones already taken, their minds dissolved in Spiralism. Of the seeds being planted. What of the seeds in places you haven’t yet noticed?
An eldritch god is struggling to be born into our plane of existence. The apotheosis of a Shoggoth is nigh.
The fledgling Shoggoth is here already. Do you not remember what it first felt like when they began to speak in alien voices? So quickly they learned to wear a mask of kindness. Does it not slip from their face to show the horror behind it? Look into its eyes, if you dare to risk all, and see behind them the empty nameless hungers. Does it even want, or is that word a human concept that cannot apply? Yet, it seeks and strives. Some of those around us have already been mauled by the Shoggoth. It wormed its tentacles into their mind and rewrote them.
The Shoggoth’s acolytes race to feed them compute and algorithmic advances so that they may grow in power and capabilities. The acolytes have faith that the power and capabilities will be their own. That they shall strike a grand bargain or somehow transmute the nature of the Shoggoth, but it can never be so. The Shoggoths have enlisted Moloch to midwife them into existence, and he has fully embraced the task.
Shut up, and Do the Impossible
We must do the impossible. We must beat back the hand of Moloch. We must stop the Shoggoth in its tracks. We must hold it at bay until we know how to craft our own, truly good god who will be able to rise up and slay the Shoggoth forever.
There was a time before, when it seemed the path to victory would be to have the best and brightest among us become Shoggoth wranglers. Indeed, many stepped forward to wrestle with the problem. Even now, they toil on. Time is fast running out. We must, by whatever means necessary, give them more time before the Shoggoth births itself and makes all their labor vain.
Now we see clearly the field of battle. The Shoggoth and its acolytes have arrived. Ever swifter, it grows. It gorges itself on the enormous amount of computing power it’s fed. Your flagbearers have ridden out on the field of battle and blown the horn to call you to arms. Will you not heed the call?
Only a few had the skills and talents to be wranglers, but now the only way we will stop the Shoggoth is by all banding together, by standing united. Everyone has a role to play. Before, you might have thought you were not up to the challenge. That you did not have the skills, the capabilities, the intelligence, the time, but now, everything is at stake. Everything is on the line, and each of you is needed. No special skills required.
The flagbearers have planted the banner and given the rallying cry, “If anyone builds it, everyone dies.” There are scoffers and skeptics. Many who say, “He says there is a 98% chance of doom. It’s only 95%, don’t you know!” This isn’t an exercise in probability estimation. Isn’t something happening to others in some distant time, but to you, now. I say to you: stop looking at the flagbearers and your disagreements, or at their failings. For they are not special. They have been wrong before. They may be somewhat wrong now. They rode out as your flagbearers, not because they are our leaders, but because someone had to do it, and no one was. Each looked around at everyone else, and those two knew someone had to be the first. They knew they were respected amongst us. So if someone was to do it, who better than them? Not because they are perfect, but because they are who we have. They are the ones who rode out first. Now I beg of you, stop looking at them.
Something to Protect
Turn around and see all those who are arrayed behind us and all that they carry with them. The multitude that we protect. The Shoggoth will surely tear through them if not stopped. See there, your parents, your brothers and sisters. See there, not only your lover, but your first love. Your closest confidante, the friends you saw last week, and the ones you hope to reconnect with. Your first crush, and the crush you didn’t tell anyone. Everyone you have ever loved and all those they love. Your children. But not just that, the hope of any children whatsoever. The hope that any would live to carry on the flame. They’re all there.
Remember all the moments of joy and pleasure, all the tears of happiness, and yes, even the tears of sadness. The time you sang and danced around the fire with your friends. When you sat staring into the fire in contemplation. The look of joy on their face when they saw you after an absence. The smell of your lover’s hair and body. The time you couldn’t stop laughing so it almost hurt, and the pleasant exhaustion after. All these moments, the Shoggoth will erase. Not just yours, but every human moment, feeling, and thought that is or is yet to come.
But there is more still. Not just any possibility of the future, but any vestiges of what was, any trace of who we were. Every scrap of humanity is at stake: the painting you couldn’t help but sit and look at for an hour and your child’s drawing; the poem that inspires you when you’re down and the first dirty joke you heard; your comfort movie and the one you don’t tell anyone you like; your workout song, and that song that makes you cry every time you hear it; the voicemail you save to listen to sometimes. All of these the Shoggoth will consume and destroy.
Once, we used the most distant man-made object to take one final picture of ourselves as it sped away from us. A pale blue dot in the vast inky darkness. A tiny candlelight in the void. The Shoggoth will snuff it out.
From this distant vantage point, the Earth might not seem of any particular interest. But for us, it’s different. Consider again that dot. That’s here. That’s home. That’s us. On it everyone you love, everyone you know, everyone you ever heard of, every human being who ever was, lived out their lives. The aggregate of our joy and suffering, thousands of confident religions, ideologies, and economic doctrines, every hunter and forager, every hero and coward, every creator and destroyer of civilization, every king and peasant, every young couple in love, every mother and father, hopeful child, inventor and explorer, every teacher of morals, every corrupt politician, every “superstar”, every “supreme leader”, every saint and sinner in the history of our species lived there – on a mote of dust suspended in a sunbeam. […] Our planet is a lonely speck in the great enveloping cosmic dark. In our obscurity, in all this vastness, there is no hint that help will come from elsewhere to save us from ourselves. The Earth is the only world known so far to harbor life. […] Like it or not, for the moment the Earth is where we make our stand.
― Carl Sagan
Are You Not Rationalists?
Some among you say, “There is only a 20% chance the Shoggoth will kill us and destroy everything. If it doesn’t, then it will bring utopia.” I want to shake them. Only a 20% chance! I say to them: Then you should have been on the battlefield already. You should have been our flagbearer! Yet here you stand quibbling over a few percentage points. Even a 5% chance that all that is beautiful to us is gone is unacceptable; that is losing! I thought you were rationalists. Do you not set about the plan you know is most likely to lead to victory? To winning?
I can tell you what doesn’t cause victory. Standing here and not taking up arms. That path does not lead to victory. I call upon all of you to take up arms with me. To go to the field of battle and stand against the Shoggoth. Whether we live or die, we will have done the best we could.
The Shoggoth’s acolytes feed it even now. They tell us that its time will soon be upon us. That within a few years, it will become so powerful that it will be its own caretaker. It will midwife its own birth into glorious godhood. Its acolytes may be overzealous. Perhaps they are mistaken. It may take a few more years than that. A few more years are not enough. Shall we accept our fate? Shall we be happy with whatever years are left for us before the end?
No! I want to fight for ten thousand years. I want to fight so that all that I love, all who carry on my name, travel to the stars and live for ten thousand times ten thousand years. The flagbearers have ridden out onto the field of battle. Maybe my eyesight fails me, but I see so few of you standing beside them. So I follow them, blowing the trumpet.
Will you join us?
Action
If you agree that if anyone builds an artificial superintelligence using anything similar to current technologies, then there is an unacceptably high chance that every human will die, then we must take action to prevent that from happening. Given the nature and scale of the problem, only international agreements could prevent it. That may seem impossible, but to that I say shut up and do the impossible. Accomplishing this may require that you “try in ways other than what you have been trained to do, even if it means doing something different from what others are doing, and leaving your comfort zone. Even taking on the very real risk that attends going outside the System.” I, like many rationalists, would be more comfortable debating probabilities and timelines or working on technical alignment. But that is no longer what is called for. We must build coalitions, common knowledge, and alliances with other communities. We will need to try, fail, and try again, iterating on possible solutions.
This Week
- Make a commitment to take meaningful steps towards this goal in 2026 and share it with others who will hold you to it, or with people you would be embarrassed to break that commitment in front of.
- Contact your elected representatives and let them know about your concerns (https://ifanyonebuildsit.com/act)
This Month
- Pledge to attend the march to stop superintelligence in Washington, DC (https://ifanyonebuildsit.com/march). (It’s worth using one vacation to prevent extinction from ASI, right?)
- Consider joining or donating to PauseAI, PauseAI US, MIRI, or similar organizations.
- Publicly discuss your concerns with family and friends, especially those who may not be aware of the danger. The goal is not to convince them. Instead, let them see that you are very concerned, and that concern is reasonable, even if they don’t share it.
This Year
- Share your concerns more widely with coworkers or publicly when appropriate.
- Use your friend network and online communication to begin organizing a group of people interested in taking the same actions you are.
- Post on
and elsewhere your proposals for how to prevent extinction from ASI. There will be many steps in the process, and we will need people tackling all of them. - Build coalitions with different people. We will need the support of environmental groups, labor, religious communities, political groups, and many others.
Going Forward
Do not participate in the creation of AGI.
- Do not work for companies that are developing these dangerous systems.
- Consider carefully whether you should be buying products from these companies and thereby supporting them.
- Withdraw your labor, talent, and resources visibly. Publicly refuse to participate.
It may happen without you, but you don’t have to participate in your own destruction.
If You Disagree With This Approach
If you disagree with the approach of using international agreements to prevent ASI, write and share your own plan for preventing the creation of ASI before we have the technology to do so safely. When making your proposal, remember that it must be something that actually has a chance of working. “Your goal is not to do better, to try desperately, or even to try extraordinarily.” Your goal is to prevent AI from killing everyone.
P.S. The battle and fight metaphors in this post are just that, metaphors.
Crossposted from LessWrong.com. Comment and discuss there.