I recently had the pleasure and frustration of reading Life 3.0: Being Human in the Age of Artificial Intelligence by Max Tegmark.¹ I think I first heard about it from the recent “Future of Intelligence” episode of Sam Harris’ podcast Waking Up.² Robert Hanson also wrote a review worth reading on his blog OvercomingBias.com.³ From hearing about it there, I inferred I was part of the target audience. However, upon reading it, I found that I’m probably not. Indeed, I doubt that most of the rationality community is either. I previously read Bostrom’s seminal book Superintelligence and found it rather dense, dry and a bit of a slog even for someone interested in the topic.⁴ Tegmark instead wrote for a general audience and, to me, the book falls in the pop-science genre. It has many reader-friendly features. When appropriate, it includes helpful charts and diagrams, though these are sometimes cartoonish. Each chapter concludes with a bullet point summary. Perhaps most importantly, the book is written in an approachable, and engaging style. I found myself enjoying reading the book and curious where he would go next. At the same time, I found myself frustrated with the presentation of AI safety topics. Tegmark understands the issues and states each of them eventually. However, this often gets lost among the other topics and fictionalized accounts. I’m concerned the average reader may not come away with an adequate understanding.

The prologue gives a brief fictional account of the creation of a superintelligence, starting from an idealistic, secret team in an unnamed fictional tech company, all the way to global domination. By the end, the AI and the company control essentially a world government that has brought prosperity and happiness to most people and all but eliminated conflict and war. Fiction can make things that might have seemed implausible feel like tangible reality. By opening this way, Tegmark both engages the reader and creates a viscerally real sense of the possibility of superintelligence. In it, he illustrates a very plausible scenario for AI takeover and the creators’ concern for AI safety. However, I worry that it appears too easy to contain and control a superintelligent AI. Being compelling and coming as it does at the beginning of the book, it will be very memorable. While consciously people will remember that the message of the book was the dangers of AI, they may reason from the most salient example: the fictional evidence of an AI that was easily contained and brought enormous benefits.

In the first two chapters, Tegmark engagingly introduces the problem of AI alignment and the state of the discussion. He does this with plenty of charts and diagrams. The framing of discussing the various positions on the problem and of listing and addressing the most common myths works well. This section also has the first good illustrations of the problem, including the example that “humans control tigers not because we’re stronger, but because we’re smarter. This means that if we cede our position as smartest on our planet, it’s possible we might also cede our control.”⁵

Next, the book tackles the subject of near-term AI advances and issues. I found this section to be one of the most interesting and useful in the book, perhaps because I haven’t read much on these topics before. I would have liked to see more detail about some areas. Nevertheless, the survey was helpful. Tegmark first covers the state of the art in AI. From there he moves on to issues of software bugs and security. I was glad to see this put front and center. It doesn’t get adequate concern in our society. He then considers AI’s impact on law, weapons, and employment. Specifically, the question of whether large portions of the population might become not only unemployed but unemployable. I particularly enjoyed the end of the chapter about how to give people a sense of purpose if there is no work. That often gets glossed over in discussions of technological unemployment.

At more than one hundred pages in, Life 3.0 finally comes to the issue of the dangers of human-level AGI and superintelligence. This section is far too short. There isn’t enough space to adequately cover the issues and make clear how grave the threat is. He points out that an AI will be motivated to circumvent our control even if its intentions are good using a fun illustration involving a world of 5-year olds imprisoning you. The difficulties of containing or boxing a superintelligent AI are pointed out. Here, he returns to the fictional story from the prologue to describe several ways the AI might escape. While this illustrates possible escape paths, I worry that it will fail to convey the message. Coming in the middle of the book, separated from the initial story, it is less likely to be remembered. Also, specific scenarios often lead people to think they can address them with some easy fixes and the problem is solved. Tegmark tries to make clear that there are many other potential means of escape and that we, being less intelligent than the AI, may not even be able to imagine how it might escape. However, those are brief abstract arguments compared to the compelling fictional scenarios.

From issues of the dangers of superintelligence, the book moves on to future scenarios. It describes twelve different scenarios, giving each two to five pages of explanation. They run the gamut from very good to very bad. Some come off as if they aren’t genuine possible futures, but rather dreamed up by someone who is ideologically motivated in the first ten minutes of thinking about the problem. Even worse, the list is missing some essential ones. In two of these, AI truly tries to create human happiness, not by some person’s ideology-driven philosophy, but by doing what might genuinely make humans happy. One such scenario is where the AI creates for each human a separate virtual world optimized for their happiness. Perhaps there is still a way to interact between the worlds, to visit your friend’s world, but people spend the majority of their time in their own. The second is what I would describe as a true utopia. Where the AI tries to create the kind of world that would make humans happy. Eliminating suffering, but also creating situations where the humans are challenged to learn and grow. The closest he comes to that is his “benevolent dictator” scenario, but that seems more like a caricature of what someone who values “diversity” in humans might imagine. Perhaps the biggest oversight is ignoring transhumanism as both a likely occurrence and moral good. His “libertarian utopia” does include cyborgs and augmented humans, but there is no discussion of elimination of all disease, radical life extension, dramatically enhancing human happiness, enhancing human cognition, and ultimately giving humans godlike powers. This chapter is meant to be thought-provoking and raise in the reader’s mind the question of what kind of future they would want. It concludes by directing people to AgeOfAi.org where there is a survey covering when they think superintelligent AI might arrive, and which of the twelve scenarios they would prefer.

The sixth chapter felt out of place to me. Ostensibly about how we might acquire resource for the very long term future, i.e. 10 000 years, it reads like a tour of cool ideas from modern science about what might someday be possible. It describes ideas including Dyson spheres, nuclear fusion, evaporating and spinning black holes, quasars, exotic matter states, quantum computers and the limits of computation, nuclear rockets, light sails, the Kardashev scale, von Neumann probes, and wormholes. Then tackles the end of the universe and what might happen if we are not the only intelligent species in the universe. This chapter seemed to be there mostly for the gee-whiz factor, and I think the book would have better off without it.

At this point, Tegmark finally tackles the subject of aligning an AI’s goals with our own. This chapter should have come immediately after the one on the dangers of AI and the difficulty of keeping it boxed. It does, however, do a good job of laying the foundation of what goals are and why it makes sense to talk about machines and AIs having goals. Then of explaining some of the challenges in goal alignment. I was pleased to see his emphasis on the fact that “the real risk with AGI isn’t malice but competence” (italics in original).⁶ That is to say, that an AI that was excellent at optimizing something slightly different from what we wanted would be extremely dangerous even though it wasn’t intentionally programmed to harm us.

The final chapter of the book takes a strange turn into consciousness. Tegmark defines consciousness as having subjective experience. What philosophers would call qualia. As with the chapter on acquiring resources, this one seemed to be a tour of fascinating ideas. This time about the philosophy, neuroscience, and ethics of consciousness. The relevant point is that the AI’s moral worth may be determined by whether it is conscious. This part could have been omitted from the book, so the choice to use it as the official conclusion was strange.

After the concluding chapter, there is an epilogue. However, its length and presentation make it feel like a chapter. In my mind, this became the conclusion of the book. The epilogue describes the founding of the Future of Life Institute by Tegmark and AI conferences organized by them. It ends with an encouragement to develop positive visions of the future. One can understand why Tegmark wanted to include this material in the book somewhere, but the shift of topic and tone at the end was jarring. It further weakened the already weak ending created by discussing consciousness.

Altogether, Life 3.0 was engaging and readable. The chapter on the near term impacts of AI was the most interesting to me, in part because I haven’t seen this topic discussed as much. I also came away with some illustrations that will be useful for explaining the issues in AI safety to others. It seems Tegmark is hoping to create awareness of AI safety among the general populace by encouraging them to imagine their own potential futures. It may do that. However, an uninformed thinker’s ideas on the subject are more than simply unhelpful. Rather, they are often detrimental to launching a proper response. In trying to reach a broad audience, Tegmark has failed to convey the gravity and nuance of the problem. The many tangentially related topics covered obscure the vital message of AI Safety. I believe Tegmark’s contributions to AI safety through the Future of Life Institute are more effective than this book.

A comment on my first post, “Ferocious Truth,” raised an interesting topic. The commenter, Simon, described two ways of handling what might be termed a folk ontology when it doesn’t properly correspond to reality. I’ve coined the term folk ontology to refer to concepts and categories held by ordinary people with regard to an idea, particularly in reference to when they have not reflected on the nature of the idea. The issue of folk ontologies that don’t correspond to reality often comes up with ideas philosophy deals with such as free will, personal identity, and morality. However, it also arises with prescientific concepts about things as diverse as warmth, magic, astrology, and the sun rising in the east. Many folk ontologies seem to have their roots in evolved human biases and perceptions. How else can we explain beliefs about magic and personal identity that are widely shared across cultures and even amongst isolated tribes?

The first response Simon described is to rescue the concept by remapping it onto something that does exist. For example, a prescientific conception of warmth might view it as a magical intrinsic property of some objects that can be temporarily transferred to others. Thus fire and living humans are intrinsically warm, whereas a hot cast-iron skillet has had warmth transferred to it from the fire it was set on. Science has revealed this idea to be incorrect. It also offers us an explanation of what we perceive as warmth that we can map our concept of warmth to. Namely, the transference of energy between two bodies due to their temperature difference by means such as conduction, convection, or radiation. This example is developed at some length in the article “Rescuing the utility function” on arbital.com.¹ In this case, the mapping is so precise that we hardly notice the shift. In other cases, the mapping does not apply as well, and we find ourselves stumbling over the difference between our expectations and reality.

The second response is to reject the concept altogether. To rule it out of bounds and taboo its use. Here, it is recognized that the idea does not correspond to anything in reality and there is nothing in reality even close to it. Many people and cultures have believed in magic and astrology, but science has revealed that they don’t exist. There is nothing remotely like magic in the world. Rather everything people have taken as magic is fraud, altered states of consciousness, magical thinking, confirmation bias, wishful thinking, or other tricks of the mind. To try to map magic onto some existing reality would invite people to continue to hold on to false beliefs rather than acknowledging their falsehood. In philosophy, rejection of an idea is often labeled as nihilism with regard to that idea.

In addition to the first two responses elucidated by Simon, there is a third available response to incorrect folk ontologies. This is to restrict the use of the idea to the circumstances or ways in which it can be reasonably applied while recognizing that it is fundamentally unreal. A prescientific view might hold that the sun is a giant disk of white-hot fire that rises above the flat earth in the east and sets in the west. Science tells us that the sun is the closest star to the earth about which we orbit. The apparent rising of the sun in the east is instead the rotation of the earth bringing the sun into view. Yet, the idea is not remapped to something else. Nor is it rejected in favor of speaking only of the earth rotating the sun into view. Instead, the sun is frequently described as rising in the east in the limited contexts for which this is well defined. However, as soon as the discussion moves outside those contexts, as when discussing celestial mechanics, the error of describing the sun as rising in the east is acknowledged and substituted with descriptions of the earth’s rotation and orbit around the sun.

These three responses to incorrect folk ontologies seem to cover the range of reasonable responses. From the examples I’ve given so far, it can be seen that none of the responses is the correct response for all folk ontologies. For some ideas, more than one response can seem to have merit. It is not always clear which response is best. Indeed, some of the fundamental debates in philosophy and between people are really debates over which of these responses is proper. Simon’s example of personal identity provides a good illustration of this. Other concepts from philosophy shed light on the detailed workings of each response.

It is commonly held that each individual has a self. That they were the same person in the past as they are now and will be the same person in the future. This simple view is problematic. What if one were to awaken tomorrow with partial or complete amnesia? Would they then be the same person? What of people in comas? Indeed, am I even the same person moment to moment as my mind changes and adapts, and the atoms in my body are replaced with ones from food and air? Thought experiments even more radically call into question the idea of personal identity. Imagine making an atomically precise copy of someone, is the copy the same person? Or, consider someone stepping into a Star Trek style teleporter and being beamed to a distant place, is the teleported person the same person? Both of these involve instances where there is a radical discontinuity of identity and yet the physical reality, the atomic structure, is identical. This indicates that the folk concept of identity somehow includes a magical property that isn’t captured by the state of the material world. Given that the folk ontology of personal identity is flawed and does not correspond to reality, how should we respond?

Rescuing

There have been many attempts to rescue the concept of personal identity. Some people try to map personal identity to physical continuity. So that as long as there is continuity of the physical body, the person is considered to be the same. This runs afoul of difficulties in determining the continued identity of physical objects as illustrated by the Ship of Theseus problem. It also seems to fail in scenarios such as mind uploading where the person might feel that there was continuity of themselves despite the lack of bodily continuity. More fundamentally, a solid understanding of quantum mechanics rules out the idea that identity could somehow be contained in the particular atoms that make up a body or indeed that the idea of the continuity of the atoms in a body from moment to moment is even coherent.²

Alternatively, people have attempted to reduce the idea of personal identity to some form of psychological continuity. This may take the form of focusing on things such as memories or personality. However, this reduction is also very problematic. Imagine that someone will have all their memories erased and then be tortured. Under the memory model of identity, they should not object to this any more than they object to the torture of a stranger. Yet most people’s intuition is that the future person being tortured will, in some sense, be them and so they will object more strongly. An explanation of identity that focuses on personality would have to admit that a significant change in personality could change someone’s identity. Such changes in personality can be caused by starting or stopping a medication, experiencing a brain injury or having a brain tumor removed. Yet, most people would not think someone’s identity had fundamentally changed in these situations.

Both of these attempts to rescue personal identity are ultimately unsatisfying. Still, some philosophers argue for each of these views. Perhaps instead of attempting to rescue personal identity, it should be rejected. While rescuing may not be the correct approach for personal identity, it may be the correct approach for other ideas. One idea that can be rescued to a reasonable degree is free will.

Free will is often defined in folk ontologies to be the ability to make choices that are not determined by past events but are instead fully willed by the individual. This view persists today in some forms of libertarian free will. However, the scientific evidence indicates that humans are fully deterministic systems. Even theories that admit the possibility of quantum randomness influencing decision-making don’t seem to be compatible with this view of free will. In this case, one’s choices aren’t freely made, they are determined randomly by a process outside one’s control. Accepting that determinism is incompatible with free will leads one to the reject it. The philosophical position that rejects free will as non-existent is called hard determinism. However, I believe a better response is to remap the idea of free will. Redefine free will as the ability to make choices that are independent of outside influence even if they may be fully determined by an individual’s past experiences and present state of mind. This is a compatibilist view of free will. This view enables us to continue to use the concept of free will in situations where it seems appropriate. We can still convey useful information by describing some actions as freely willed and others at not. While at the same time not being misled into positing some non-physical source of free will or believing that people’s actions aren’t fully the result of the complete set of circumstances that brought them to the point of decision.

Rejecting

Instead of attempting to rescue personal identity, one can reject it. In this view, the self is an illusion arising as an evolutionary artifact. No personal identity is destroyed at death. People should discard the idea and language of personal identity and the self. This view has the virtue of being consistent and also corresponding to reality. However, it is ultimately very unsatisfying as people experience daily the identification with their past and future selves. Our values and preferences are bound up in the idea of personal identity. Why should I even plan for tomorrow if I don’t believe I will be there to receive the benefits?

In my opinion, an idea that can be fruitfully rejected is that of objective morality. The human intuition is that morality is a fact of reality. However, it can be seen that events and actions don’t carry around magical tags labeling them as right or wrong. It is only our minds that label actions as such. Morality does not exist in the territory only in each individual’s map of the territory. This leads some people to try to rescue morality, leading to various forms of moral relativism. However, a more productive response is to reject morality and hold a moral nihilist view. This view sees morality as a mixture of erroneous statements about the world and statements about the speaker’s values and feelings. Many moral nihilists will adopt the third response of restricting and recognizing by accepting moral language and recognizing that it doesn’t properly refer to anything. However, to me a complete rejection of all moral language and ideas is a better response. Morality is wired into the human brain in such a way that even committed moral nihilists can find themselves falling into thinking in moral categories. By completely rejecting all moral language and ideas one can better avoid the pitfall of moral thinking.

Restricting & Recognizing

Having considered rescuing and rejecting personal identity and found both lacking, I prefer to restrict its use and recognize its limitations. Identity is a useful concept we hold and apply in our everyday lives. It can’t be reduced to a simple formulation; rather it is inherently complex. The sum of many factors. Still, most people agree what identity refers to in the common cases. It is nebulous but patterned, meaning that it can’t be entirely pinned down, but is still a distinct, definite concept. However, we must restrict its use to the mundane situations of life that our brains evolved. Outside of those situations, it leads us astray because it doesn’t correspond to the territory. When considering thought experiments like teleporters, automatically precise clones, or mind uploading, we must recognize that our concept of identity should be discarded and is not helpful to thinking about the problem anymore.

Though I take the stance that we should reject the use of moral language, restricting and recognizing is probably a more common approach amongst moral nihilists. Last year, I found myself recognizing the limitations of moral language when a friend challenged me to answer the torture vs. dust specks dilemma. This asks: would you prefer that one person be horribly tortured for fifty years without hope or rest, or that inconceivably large number of people get dust specks in their eyes? Here “get dust specks in their eyes” is functioning as the least bad thing that could happen to a person. Most people will accept the question and then begin to try to compare the moral weight of the torture versus so many people having this very tiny bad thing happen to them. Which they prefer often hinges on questions of how to aggregate the harm caused by the dust specks and whether there is some threshold below which harm should not be considered. Instead, I recognized this as a situation where we had moved past the proper applicability of moral language. Instead, I fell back to the reality which is that we each have certain moral intuitions or preferences which we develop based on evolved moral reasoning and cultural ideas. These do not feel like preferences because our brains present them differently and they can be in conflict which what we think of as our preferences. However, they function as a set of preferences coming from a separate module in our brain. Those preferences are complex and irrational. Given this, it isn’t appropriate to try to reason through the torture vs. dust speck dilemma. Instead, I answered that I would prefer dust specks because when I imagined the two outcomes, that is what my moral intuitions said were better. Hence, that is the outcome I would feel better about after the fact. Morality is not objective; there is no right answer. All I can try to do is minimize my own negative feelings caused by my moral intuitions. My friend called me a “dirty dust specker”, but had to acknowledge the soundness of my reasoning.

Choosing a Response

How is one to choose the correct response? Different folk ontologies call for a different response. None of the three responses is always applicable. I’ve given examples where I think each response is best. Rescuing is only viable when there is a sufficiently similar entity in the territory to map the concept too. This is why magic can’t be rescued. Without that, rescuing can lead to confusion. Words exist not just as categories for one’s own thinking, but for communication. All parties to a conversation must be on board with how a folk ontology has been rescued. When this is not the case, confusion, misunderstandings, and even deception can occur. Rescuing a concept can also leave a void of terminology to refer to the original folk ontology, making discussion of it difficult. This means that the correct response may be context dependent. A rescued idea may need to be tabooed when speaking with people who don’t share the same view or when discussing the folk ontology itself. Rejecting is a far less fraught response. The primary downside is increased difficulty in communication caused by a loss of terminology to describe a phenomenon that people have previously found useful. When rejection is a good response, this drawback is more than offset by the clarity of thought brought about by avoiding the erroneous concept. To me, morality is a good example of this. Avoiding describing things as right or wrong can be cumbersome. However, it brings a great deal of clarity as it forces one to focus on optimizing goals. This highlights the differing goals people have and prevents moral intuitions from clouding judgment. Restricting and recognizing requires a nuanced understanding of when a concept is applicable and practice noticing when it is not. It can also lead others to assume the folk ontology is believed.

Even with the above trade-offs in mind, it can be difficult to select the best response. Indeed, the response may be context dependent. Varying based on who one is interacting with and what is being discussed. Furthermore, these three responses aren’t fully distinct. There is a spectrum between them. Feel free to craft a response that is an amalgam of these as appropriate. For example, it might make sense to apply a particular response to one aspect of a folk ontology and a different response to another aspect.

all the stories are shattered.
the meaning they ensconced is poured out,
    a drink offering on the altar of truth,
    a propitiation for ignorance.
how shall we who have no hands build new vessels?
vessels to carry us across the sea.
rather, we shall drown.
gasping for life, we swallow lies.
how shall we live unless we become fishes?

Recently I read Freezing People is (Not) Easy: My Adventures in Cryonics by Bob Nelson.¹ It recounts his time as president of the Cryonics Society of California (CSC) during which he cryopreserved and then attempted to maintain the cryopreservation of a number of early cryonics patients. I’ve been thinking about and investigating cryonics of late, so I went looking for any and all related material I could get my hands on. Since this book was the only one I could source from my local library, it ended up being the first one I read. I was looking to gain insight into why cryonics has not become more popular and to learn some of the challenges faced when attempting to run a cryonics organization. What I found instead was a sad tale of typical human failings. It should be noted that the author has been a very controversial figure and some might claim that the veracity of this book is questionable. He has been called a fraud and a con man. If his memoir is to be believed, he was simply seeking to help people fulfill their desires to be cryopreserved, but foolishly failed to adequately plan, prepare, or ensure monetary provision for the task.

It’s often said that one should start a story at the beginning. The question is: what is the beginning of the story one is telling? The first chapter describes Mr. Nelson’s difficult childhood and very early marriage under challenging circumstances. I found this chapter boring and extraneous. Even after finishing the whole book, I’m not sure what the purpose of including this material was. Perhaps it was to show why he cared about others so much that he fell into his later mistakes? If so, it did a poor job of that besides one anecdote of him helping a severely injured teenager when he was himself a teenage runaway. More likely it was in the hope that the reader might take pity on him, viewing him as a tragic hero.

Nelson first learned of cryonics in 1965² when an acquaintance pointed out a newspaper article discussing the recently published book The Prospect of Immortality by Robert Ettinger.³ He was immediately consumed by the idea of cryonics: that a person’s death could be postponed until technology advanced to the point where they could be healed. Searching local bookstores for a copy, he finally located a store that had them. Upon arrival, he found they hadn’t even removed the books from the shipping box because they had received them so recently. Reading through Mr. Ettinger’s book his excitement grew. Nelson loved astronomy and space travel, and was immediately thrilled by the potential of using cryonics technology to place astronauts in suspended animation for long-term spaceflight. He envisioned a new race, in the vein of the space race, to be the first country to revive an individual from cryopreservation. As soon as he became convinced of the scientific validity of the possibility of cryonics by consulting with several friends and the recognition of natural examples of suspended animation, he was a true convert. The strength of his faith in the advancement of technology to achieve near-miraculous results is evident.

I find it fascinating that a small number of individuals take immediately to the idea of cryonics upon learning of it while the vast majority dismiss it or are repulsed. What is it that sets these people apart? Nelson himself wonders about this in the book but offers no insights. Understanding this would seem to be a crucial first step to anyone seeking to encourage others to sign up for cryonics. This information is even more important for a cryonics organization whose success depends on it.

Nelson’s excitement led him to attend the first meeting of the Life Extension Society in California on May 13, 1966⁴. He found the attendees’ scientific credentials and financial resources underwhelming. So, it should not have been so surprising to him when, at the second meeting, he was elected president despite being a TV repairman. He then established the nonprofit Cryonics Society of California (CSC) whose initial plan was to promote research into cryonics. A scientific advisory council was organized, and the CSC was to provide funding. The council informed them that if the CSC cryopreserved someone then the council would walk away and have nothing to do with them. The book is unclear on this point, but it appears the council, being composed of researchers in existing fields like cryobiology, thought that more direct involvement of the CSC with cryopreservation would hurt their reputations. This restriction was not concerning to Nelson as he felt more research was needed and didn’t plan to cryopreserve anyone. He was giddy from the success of forming the council and receiving the validation of scientists. Indeed, his excitement at this stage seems incredible. He states that “in December 1966, cryonics was a bright golden vista with limitless possibilities on the horizon.”⁵

I would have liked to hear more about the CSC’s structure and plans. Later we learn that they had membership dues, yet it is never stated how much the dues were or approximately how many members they had. The funding for the scientific advisory council was to come from CSC funds, and the researchers would apply for external grants as well. Details of how Nelson expected this to develop would have been helpful.

Early in 1967, Nelson was unexpectedly approached by the son of Dr. James Bedford, a psychology professor at the University of California, with a request to cryopreserve his father upon legal death. Despite having agreed with the advisory council not to cryopreserve anyone and being entirely unprepared, they decided to proceed with the cryopreservation of Dr. Bedford. The CSC scrambled to acquire the necessary equipment. Bedford’s son stated that there was “three hundred thousand dollars in a foundation for cryobiological research, and [he was] director of that foundation.”⁶ However, Bedford died before they had collected any money and almost before they had the necessary chemicals. They performed the cryopreservation regardless and promptly turned his body over to his son. The CSC never received a dime for their efforts.

Much later Nelson learned that the money from the estate had been spent on legal fees as Bedford’s wife and son defended his will and cryopreservation from lawsuits brought by other relatives. Bedford is widely considered to be the first genuine cryonics patient because his body was preserved immediately after his death and cryoprotection measures were taken. Over the years, Bedford’s son diligently maintained his father’s cryopreservation. Long-term support from relatives like this was rare, and Bedford is the only patient prior to 1974 that remains cryopreserved.⁷ His body is now in the care and custody of the Alcor Life Extension Foundation. While the cryopreservation of Dr. Bedford generated copious publicity for the CSC and cryonics, it also immediately led to the scientific advisory council withdrawing, just as they’d warned.

This began a series of further cryopreservations by Nelson and his associates. Three people were cryopreserved and their bodies held at a mortuary while arrangements were made to keep them at a cemetery. (The patients were legally regarded as dead and as such their bodies were required to be stored at a mortuary or cemetery.) In each case, they collected no money up front, and the CSC received only $10 300 in donations after the procedures. By treating all fees as donations and handling the transfer of the body under the Anatomical Gift Act, the CSC avoided creating any contractual obligation to care for the patients and hoped to shelter themselves from any legal problems. However, this did not prevent patients’ relatives from later suing.

In all these cases, the plan was that relatives would pay for care after cryopreservation. This approach was predictably an egregious mistake, yet the reason for adopting it is never stated. These patients needed potentially indefinite care that relatives might lack the resolve or means to provide. What if relatives passed away before the patient was revived? I wish there had been some discussion of why this approach was chosen and considered viable. Nevertheless, it appears to have been the assumed practice at the time as all three organizations that performed cryopreservations followed it.⁷ In taking on these patients it seems Nelson was acting out of what he felt was a duty to care for these people. Yet, as any hospital will attest, one can’t give away unlimited free care. Doing so means an end to one’s ability to care for anyone.

In addition to the three patients CSC cryopreserved, they took over care of a fourth, who had been frozen by Cryo-Care Equipment Corporation. At the time, this business was one of only three organizations performing cryonics, but it made no attempt at cryoprotection, advertising its services for cosmetic purposes rather than eventual revival.⁷ Nelson was given charge of this patient because he offered the family a lower rate for ongoing care. He made this offer partially because he wanted the cryonics capsule the body was in as he had not been able to acquire a usable one himself. Despite promising to pay $150 per month for storage, the family never paid any money and instead explicitly left his fate to the CSC.

Nelson’s “predicted boom in cryonics patients never materialized.”⁸ Without more paying patients, his failure to collect adequate fees or establish a revenue stream to pay for the patients’ ongoing care left the CSC in dire financial straights. He found himself paying out of pocket for the ongoing patient care. In 1970, Nelson was unable to continue the care of these patients and was left no other option than to allow them to thaw.⁹ According to Nelson, this was a very emotional moment for him, and he went out into the desert to decide if it was the right course of action. This is how he lost all four patients in his care at the time.

Despite this failure to care for the patients in his charge, Nelson took on the cryopreservation of two additional patients at a price of $10 000 for cryopreservation and a suggested donation of $100–$300 per month for storage costs. The $10 000 donation did not cover the costs of a cryonics capsule and was not collected from most patients. As he had done previously, he also took charge of another patient who was cryopreserved by a separate organization, the Cryonics Society of New York (CSNY). Nelson repeated all his mistakes. He failed to collect money in advance and had no better plan for ongoing care when, again, relatives failed to pay. He continued to pay for patient care out of pocket and using the CSC membership dues. Eventually, when he was out of town for a week, having left the patients under the watch of the cemetery’s groundskeeper, disaster struck. While Nelson was away, there was a failure of the vacuum pump that was maintaining the insulation of the poorly constructed capsule. The groundskeeper, speaking in heavily-accented, broken English, was unable to communicate the problem to the liquid nitrogen supplier, who had contracted to maintain the temperature of the capsule. Nelson returned to find that the bodies had completely thawed.

After some time spent in despair, Nelson refroze them. He did this despite knowing that they were almost certainly no longer viable for revival since he estimated they may have been without liquid nitrogen for five days. To his surprise, when he contacted the families they expressed the belief that there was nothing to be done but to keep them frozen and hope for the best. He had clearly explained to them when they initially signed up the consequences of a cryopreservation failure. Still, the families’ reactions are understandable given their emotional involvement. On the other hand, Nelson’s decision is inexplicable, especially since some of the families were not paying for ongoing care.

While all this was going on, Nelson also assisted a man named Nicholas DeBlasio in setting up a storage location on the grounds of a cemetery in New York for his wife Ann who had been cryopreserved by the CSNY. Mr. DeBlasio was unhappy with his dealings with the CSNY and wanted to handle everything himself. He did in fact successfully maintain his wife’s cryopreservation for some years, and even took charge of another patient, storing them in the same capsule. However, infrequent checking of and issues with the capsule eventually led to a partial thaw and the subsequent decision to terminate the patients. This case highlights that long-term cryopreservation necessitates a dedicated professional team.

Nelson’s wife had divorced him. Exhausted and disillusioned by his efforts and failures to maintain the patients, he eventually decided to leave the field of cryonics. Without funding to continue patient care he contacted the families of the patients and with their consent removed the patients from cryopreservation. However, Nelson’s story was not over, as several family members sued him and the cemetery he had worked with to store the bodies. Even though Nelson was broke, they sought damages from the cemetery’s liability insurance and needed him as part of their case to improve their chances of winning. At the time, there was also a lot of bad publicity as news organizations picked up the story of the thawing and trial. The, at times farcical, trial is described at some length. It did make for a good climax to the story but was ultimately not very enlightening.

The book concludes by describing Nelson’s life since that time and how he came to the decision to write his story. As with the first chapter describing his life before his involvement with cryonics, this felt extraneous to the story. It was made somewhat more palatable by having come to see him as a pitiable figure so that it was nice to hear he was able to move on with his life.

The image sometimes painted of Nelson is that of a fraud and swindler. That was how the attorneys of the families suing him attempted to portray him. In this book, he comes off as a sincere man trying to do his best to help others but ultimately failing. By not acting as a professional he repeatedly set himself up to fail. No one can provide a service without taking adequate payment. Indefinitely maintaining care of cryonics patients is a tremendous undertaking and not to be taken on without significant planning. Organizations with long-term financial obligations such as cemeteries and insurances companies have been known to fail. With cryonics one hopes that no room for failure would be left open. Despite this, Nelson and all the cryonics organizations operating at the time foolishly relied on family members for ongoing patient care funding. To me, this is simply another example of human irrationality. Modern cryonics organizations are, by comparison, much better managed. Still, it is not clear to me if they have adequately planned for the rigors of perpetual care.

Overall, the book was an enjoyable read for someone interested in cryonics. I think it serves as valuable documentation of the history of cryonics. However, it would have served that purpose much better had it detailed more the financial, organizational, and legal structure of the CSC; and explained the thinking behind having relatives fund patient care and not demanding payment up front for cryopreservation and the cryonics capsule. While I think it is good this book was written, I do worry that it will hurt the cryonics movement by dredging up the early failures. Readers may not make the distinction between the CSC and contemporary cryonics organizations.

[For a quick review, jump to the summary or chart of signs.]

Group discussions can easily become unwieldy. Certain individuals have a tendency to dominate the conversation, limiting others’ ability to contribute. When the goal is collaborative truth seeking, this is counter productive. Those who are excluded from the discussion may be exactly the ones with the necessary insights to advance the dialogue toward the truth. To deal with this and other issues, a moderator is sometimes appointed. However, traditional moderation may introduce too much friction and is geared toward potentially combative dialogue. A lighter-weight system intended for more collaborative discussions is helpful. To address this, a system of hand signals called automoderation was created. Automoderation has been used in the rationality community in Columbus, Ohio, for almost two years now with good success.

The rationality community should ostensibly be seeking the truth in any discussion. One might imagine that rationalists, cognizant of the pitfalls of group discussion and genuinely seeking truth, would manage their discussions better than others. However, human nature still dominates and there are often individuals involved who have underdeveloped social skills. For example, they may struggle to signal that they wish to speak or fail to notice the social signals that they are inappropriately dominating the conversation. Additionally, a focus on thinking through the subject and developing genuine insight often draws cognitive resources away from preparing a quick response as might be needed to jump into the middle of a conversation. It can be helpful, in that situation, to have a method of signaling one’s desire to speak, and be given adequate time by the group to formulate one’s thoughts.

In larger and more formal settings a moderator can be appointed. They become responsible for determining who may speak and for how long. They facilitate discussion, ensuring that all members with standing are given a chance to speak and no one inappropriately dominates the conversation. A moderator can also facilitate metadiscussion about procedure, topics, and participant needs while ensuring the debate isn’t derailed by them. However, in smaller or less formal settings a traditional moderator may not be called for and there may be no one willing to fill the role. Yet, the smaller or less formal setting does not always obviate the need for some degree of moderation.

The rationality community in Columbus, Ohio, found itself in the position of needing a system of moderation for their discussions, in particular for a rationality dojo. A little over two years ago Max Harms along with another member of the community created a system of hand signals supporting moderation in smaller, less formal settings. This system was inspired by the Occupy movement hand signals. When all participants know the hand signals, a moderator may not even be needed. A moderator is still useful, but often does little besides clarifying the system and consequently introduces very little friction. This system of hand signals is called automoderation. It has been used successfully in groups as small as 3 to 4 people and as large as 15 to 20.

Automoderation embodies a wait rather than an interrupt culture. Interrupt culture may be fine for a causal fun conversation, but collaborative truth seeking is aided by a wait culture. Traditional moderation systems such as Robert’s Rules of Order are intended to handle parliamentary proceedings and combative discussion. This system is intended for use in a more cooperative group discussion.

Automoderation Hand Signals

The automoderation system requires participants to sit in a circle. The flow of who speaks proceeds clockwise based on hand signals used to indicate desire for and purpose of speaking. There are five hand signals each having a different priority. Higher priority signals take precedence. The primary signal is simply raising the hand indicating the desire to expand on the current discussion topic. People new to automoderation are encouraged to use expand if they are in doubt as to the correct signal to use. Each signal indicates a different intent of the speaker. In addition, some signals modify the flow of the discussion. In order from lowest to highest priority, they are a raised fist indicating change topic, the pinkie finger extended indicating a probing question, a raised hand indicating a desire to expand on the topic, the pointer finger extended indicating a clarifying question, and two hands forming a triangle with the thumbs and forefingers indicating meta point. The hand signals are illustrated in the chart below in order from lowest to highest priority. A high-resolution version of this chart suitable for printing as a poster can also be downloaded.

The meta signal is used to raise points regarding the nature or situation of the discussion. Meta signals are relatively rare but often important. They include such things as: keeping the discourse focused on the topic at hand; clarifying the rules of automoderation; requesting changes to the environment or situation such as getting a drink, taking a break, or going to the bathroom; requesting information about the discussion, for example, how long it is planned to run; addressing concerns of tone and undesirable behavior; and dealing with strong emotional reactions.

A clarifying question asks a speaker to clarify something they said. This should be confined to instances of genuine ambiguity or confusion about what someone meant. After the question has been asked, the person is given a chance to briefly respond to the question but should stay focused on answering the question and avoid making additional statements. Typically, a clarifying question is raised while someone is speaking and the question is asked of them. Technically though, a clarifying question could be asked of anyone, not just the last speaker. This can happen when the confusion is not noticed until after another speaker has started.

Expand signals a desire to continue the conversation and expand on what has been said. This includes sharing ideas, opinions, experiences, and facts that are relevant. It is also used to respond to other speakers’ points and raise further questions or areas to be discussed. The majority of the signals used in a discussion are typically expand. Do not waste time making statements indicating agreement with other speakers; instead, use nonverbal hand signals described later to indicate agreement or disagreement.

A probing question is any question directed at a particular participant that is not a clarifying question. Probing questions can be used to explore a point in more depth. As with clarifying questions, the person the question is directed to is given a chance to respond. A probing question could be asked of anyone but is normally directed at the last speaker. Often, it makes more sense to transform a probing question into a statement of confusion or curiosity that can be directed to the whole group using the expand signal instead. This avoids the lower priority of the probing question signal and the potential for creating too much back and forth. It also gives other participants who may be better able to address the question the chance to weigh in.

Change topic signals that a participant wishes to change the topic. This can be because the current topic seems to have been discussed fully or there is simply a more interesting topic that they would like to discuss instead. Sometimes it is clear to multiple people that a topic is drawing to a close and one signal for a change topic will elicit multiple change topic signals in response. Generally, after a change in topic is proposed, a group decision is made, often using the approve and disapprove hand signals described later. One more pass around the circle may be allowed on the current topic so people can make closing remarks. If multiple people are requesting a change of topic, it is often worth seeing if there are multiple suggestions for the next topic and allowing the group to decide amongst them. There are situations when it may be appropriate to use meta to signal a desire to change topic instead. This might be the case if the current topic is too emotional for a participant to continue with or if, for reasons of time or agenda, it is important to move on to another topic even though the current topic is not complete. A change topic signal does not have to include a suggestion for the new topic.

Variations

The automoderation system should be taken as a set of guidelines rather than rules. Each community and group should adapt it to their needs. Someday it may become more codified and it might make sense to be more strict. Automoderation is still new and certainly has room for improvements to be discovered through experimentation. Additionally, there are situations it doesn’t cover. Those are instances where a moderator may need to step in and make a judgment call.

As originally created, the flow of who speaks always continued clockwise from whoever the last speaker was, regardless of which hand signal led to their speaking. If multiple people were signaling, the one with the highest priority was selected. Ties between signals of the same priority were broken by selecting the first person clockwise from the last speaker. This has the virtue of being very simple and clear. However, it means that people can be skipped over. For example, someone wishing to expand might be skipped because a person after them is using the meta signal. The skipped person would then not be able to speak until the flow came back around the circle. The same problem can arise when a question is asked since the flow continues from the respondent.

Simple Rules

1. When someone is done speaking, call on people who are signaling a desire to speak.
2. If two or more people are signaling, call on the one with the highest priority signal; break ties by going in a circle, clockwise from the last speaker.
3. If someone asks a question (probing or clarifying), the person they ask should respond; flow continues from the question answerer.

In practice, the Columbus rationality community has not followed the simple rules but rather continues flow from a point that allows the skipped participants to speak next. This works well but is more complicated to keep track of. In this variation, flow may continue either from the last speaker who was not responding to a question or from a previous speaker depending on the relative priority of the signals involved. When multiple people are signaling, the one with the highest priority is selected. Ties are broken by selecting the first person clockwise from the appropriate participant. When the last speaker was expanding or requesting a change of topic, flow continues from their left. When the last speaker was responding to a probing question, flow continues from the left of the questioner. When first going up to the priority of a clarifying question or meta point from a lower priority signal, note the last speaker or if the last speaker was responding to a probing question, note the questioner. Flow continues from the noted person when returning to lower a priority. When the last speaker was making a meta point, flow continues to their left if there are more meta signals; otherwise, flow continues from the noted speaker. When the last speaker was responding to a clarifying question, flow continues from the questioner’s left if there are more clarifying questions or meta points; otherwise, flow continues from the noted speaker. If your group feels comfortable with this approach, it is the one currently recommended by the Columbus rationality community.

Standard Rules

1. When someone is done speaking, call on people who are signaling a desire to speak.
2. If two or more people are signaling, call on the one with the highest priority signal; break ties by going in a circle, clockwise from the appropriate participant.
3. When the last speaker was expanding or changing the topic, flow continues from their left. If the next speaker is asking a clarifying question or raising a meta point, note the current speaker. This is the point flow will continue from when returning to a lower priority.
4. When the last speaker was responding to a probing question, flow continues from the questioner’s left. If the next speaker is asking a clarifying question or raising a meta point, note the questioner. This is the point flow will continue from when returning to a lower priority.
5. When the last speaker was making a meta point, flow continues from their left if the next speaker is also making a meta point; otherwise, flow continues from the previously noted participant.
6. When the last speaker was responding to a clarifying question, flow continues from the questioner’s left if the next speaker is asking a clarifying question or making a meta point; otherwise, flow continues from the previously noted participant.
7. If someone asks a question (probing or clarifying), the person they ask should respond.

Other variations on the system have been suggested but not tried to determine whether they are better. Some community members have suggested eliminating the probing question signal as it is rarely used. Its priority below expand discourages use. It can lead to back and forth discussion, and expand is typically a better signal to use instead. Other community members have suggested making the change topic signal not be a request to speak. Like the signals described below, it would be used only to communicate nonverbally to the group. If one wished to talk about the change of topic, they could raise a hand at the same time. This would go well with the removal of probing question as it would eliminate all signals with priority below expand.

In practice in Columbus, the change topic and probing question signals are sometimes treated as having the same priority as expand. This simplifies things slightly and doesn’t seem to cause problems. However, it does open up the possibility that someone could force discussion on a change of topic before other people are ready.

Other Hand Signals

In addition to the automoderation hand signals, the community has found it helpful to have a few other hand signals to facilitate communication without adding any auditory distraction to the discussion. The first few of these are so useful that they could be considered for official inclusion in the automoderation system. The rest of the signals should probably be taken as possible suggestions if the situation arises or a community finds the need for them.

A thumbs up is used to signal approval or agreement and a thumbs down to signal disapproval or disagreement. These can be used for informal voting when a change of topic or meta change is proposed. They may be used when someone makes a particularly good or enlightening point. It is important that they be interpreted and used to signal about the content of a speakers message and never as a judgment of the speaker themselves.

Two more signals are in occasional use. The first is the OK hand sign, which is used to indicate that you are listening and interested in what someone has to say. The second is to point the flat hand, palm down toward the speaker and wiggle the finger tips. This sign was imported by a few community members from CFAR and means “I feel you” or “This resonates with me”. That is subtly different from the thumbs up which implies something more like agreement. These signals have rarely been used in practice and may not be worth the complexity of having them.

It may be helpful to add a hand signal to keep track of where flow should start from when continuing after a meta or clarifying question. The noted speaker would make this sign so that others don’t have to mentally keep track of this. Flow would then continue from their left. Two suggestions for this signal have been made. The first is the “shaka sign” with the thumb and pinkie extended and the other fingers curled closed. The second suggestion is pointing to the left.

Examples

To help clarify the rules, here are some examples following the standard rules described above. Imagine Amy, Ben, Cora, and Dan wish to hold a discussion. They sit in a circle in that order. Here are how a number of scenarios would play out. For clarity, each scenario starts with Amy speaking.

While Amy is speaking, Cora raises her hand to expand. When Amy is done speaking, Cora speaks next because she is the only one with a hand raised. Flow continues from Cora when she is done.

While Amy is speaking, Ben and Dan raise their hands to expand and Cora signals a meta point. When Amy is done speaking, there are signals of priority higher than expand, so it is noted that flow will continue from Amy in the future. Cora speaks next because her signal is higher priority. When Cora is done, flow continues from Amy, so Ben will speak next.

While Amy is speaking, Ben signals a probing question and Dan wishes to expand. When Amy is done speaking, Dan speaks next because expand is higher priority than a probing question. When Dan is done speaking, since no other people wish to expand, Ben may ask his probing question. He asks it of Amy, who then responds. Flow then continues from Dan because flow continues from the questioner rather than the respondent.

Advantages

On the whole, the automoderation system has proved very successful and continues to be used in the Columbus, Ohio, rationality community. It is mainly used at the regularly scheduled meetings that are open to rationalists only; at these, everyone present knows the hand signals or can be expected to learn them as part of their initiation into the community. The advantages of using automoderation include:

• It ensures that everyone is given an opportunity to speak. This is particularly valuable for those members who are more introverted, take longer to formulate a response, or have underdeveloped social skills. Women have also reported that this helps them participate in the discussion as they may have been socialized to wait longer before speaking or to be less aggressive in entering a conversation. When women are trying to participate in a male-dominated discussion, the automoderation system highlights the shift to a wait culture for all involved.
• Having the system as a community norm enables members to initiate it when they see value in doing so. It is not uncommon, if there is a discussion going on in which someone feels they are not able to interject, for them to raise a hand. This serves as a reminder to the others present to make sure everyone is included in the discussion and often immediately triggers the use of automoderation.
• A group of people who know the system often need no moderator at all as it is clear to all participants who the next speaker is.
• The hand signals provide an easy nonverbal channel of communication to moderate the discussion that does not interfere in any way with the current speaker.
• Many raised hands and other signals can indicate to a speaker that others wish to speak and it may be a good idea to bring their current remarks to a close to allow others to speak. This is made more palatable by the knowledge they will have another chance to speak when it comes around the circle again.
• The requirement to select the appropriate hand signal can force one to clarify in their own mind the purpose of speaking and better plan their remarks.
• The hand signals can be useful in situations where automoderation is not being used. In particular, the meta symbol can be used to good effect outside automoderation to indicate a desire to interrupt a conversation with a meta item. Often this is something the conversation members will feel constitutes a reasonable interruption.
• By promoting wait culture, automoderation can help to train members of the community away from speaking as much as possible or as quickly as possible simply for the, often unconscious, purpose of gaining social status.

Disadvantages

While automoderation is a useful social tool, it is not without its drawbacks. It is important to be aware of these to help mitigate them.

• There are times it would be better to allow a back and forth discussion between a few participants. Automoderation does not support this. This is a situation where a moderator can step in if needed.
• Discussion points can pile up, so that by the time one has an opportunity to speak, there are now many things to respond to. It can be hard to keep track of all the threads that are now being discussed in parallel.
• It is easy to lose track of what one was planning to say while waiting for others to speak.
• The pile up of points can encourage people to move on rather than continuing to explore a point that it might be valuable to delve into more.
• The topic can naturally drift as participants expand on each other’s statements.
• If a participant is long winded or doesn’t make positive contributions to the discussion, it can be difficult to limit their disruption of the discussion.
• Overuse of automoderation removes opportunity to practice valuable social discussion skills. Often, a conversation with fellow rationalists would otherwise be an ideal situation for this practice since they are likely to be more forgiving and supportive of one’s efforts to grow.

Applicability

There are situations it may not be appropriate to use automoderation. It is good to be aware of these so that one can judge when it would be more appropriate to simply have a discussion or to use another moderation system.

• Automoderation breaks down in large groups. While it has been used with some success in groups of 15 to 20 that was only because not many participants actually wished to speak. Had all of them wished to speak, it likely would not have gone well.
• It should not be the default for casual conversations amongst groups of friends.
• Don’t use it for small groups of 3 to 4 unless the need becomes apparent or it is a more formal meeting.
• Automoderation is probably best suited for groups of 5 to 10 people.
• Automoderation is best suited for groups where all members are cognizant of the degree to which they can genuinely contribute to the discussion.
• Don’t use automoderation when not enough participants are familiar with the system.

Etiquette

Some etiquette can help the automoderation system run more smoothly and ensure all participants feel included.

• If someone has had their hand up for a while either because they have been skipped or for a change of topic or probing question, it is polite to avoid raising one’s hand so that they may speak.
• If someone is wishing to speak, it is polite for the speaker to try and finish their point rather than going on at length, especially if they have a clarifying question.
• Don’t waste time using expand to state agreement; instead, use the nonverbal hand signal of a thumbs up.

Summary

The hand signals in priority order, low to high, are:

• Change Topic (Fist) – request to move on to another topic.
• Probing Question (Pinkie Extended) – ask a question and receive a response.
• Expand (Raised Hand) – make further statements about the topic.
• Clarifying Question (Pointer Finger Extended) – ask a question to clear up confusion or ambiguity and receive a response.
• Meta (Triangle) – raise an issue about the nature or context of the discussion.

Hand signals that are not a request to speak:

• Approve/Agree (Thumbs Up)
• Disapprove/Disagree (Thumbs Down)

Simple Variant

1. When someone is done speaking, call on people who are signaling a desire to speak.
2. If two or more people are signaling, call on the one with the highest priority signal; break ties by going in a circle, clockwise from the last speaker.
3. If someone asks a question (probing or clarifying), the person they ask should respond; then flow continues from the question answerer.

Standard Variant

1. When someone is done speaking, call on people who are signaling a desire to speak.
2. If two or more people are signaling, call on the one with the highest priority signal; break ties by going in a circle, clockwise from the appropriate participant.
3. When the last speaker was expanding or changing the topic, flow continues from their left. If the next speaker is asking a clarifying question or raising a meta point, note the current speaker. This is the point flow will continue from when returning to a lower priority.
4. When the last speaker was responding to a probing question, flow continues from the questioner’s left. If the next speaker is asking a clarifying question or raising a meta point, note the questioner. This is the point flow will continue from when returning to a lower priority.
5. When the last speaker was making a meta point, flow continues from their left if the next speaker is also making a meta point; otherwise, flow continues from the previously noted participant.
6. When the last speaker was responding to a clarifying question, flow continues from the questioner’s left if the next speaker is asking a clarifying question or making a meta point; otherwise, flow continues from the previously noted participant.
7. If someone asks a question (probing or clarifying), the person they ask should respond.

Experimental Variant

This experimental variant is based on feedback from members in the Columbus rationality community and has not been tried yet. It is like the standard variant with the following changes. Remove the probing question signal. The change topic signal is no longer a request to speak but is simply a nonverbal signal like approve or disapprove. Use the flow marker signal of pointing to the person to one’s left when going to a meta or clarifying question to indicate where flow should continue from.

Conclusion

Automoderation has been a very useful addition to the social toolbox of the Columbus rationality community. It has helped many discussions go smoothly and people to contribute and feel more comfortable participating in the dialogue. This post was written in the hope that other rationalist communities will find it useful and spread and improve the system. Take whatever parts seem useful and apply them as seems best.

Reactions

• Automoderation by Brienne Yudkowsky
• On Automoderation by The Zvi
• Thoughts on Automoderation by Proof Of Logic

EDIT 2017-04-21: Added reactions, clarified chart order.

Humans frequently fail to make their beliefs match what is true. There are many areas of life where this is typically not the case. When asked what a ball thrown in the air will do or whether the keys will be where they were left, almost all people will express beliefs that match reality. However, in many other areas of life they hold beliefs that bear no relation to what is true. This is most common in the areas of religion, morality, and philosophy. It is not limited to those though, but appears often where feedback on the veracity of belief is absent or indirect. Examples include astrology, crystal healing, and conspiracy theories. False beliefs are very widespread. According to a 2016 Gallup Poll, 79% of American’s believe in God.¹ Another survey shows that 72% of American’s believe in an afterlife.² Indeed, even a small percentage of “atheists” claim to believe in an afterlife according to a pew research poll.³ Large percentages of Americans believe in various conspiracy theories. A 2012 survey showed that 10% of Americans believe astrology is “very scientific” and 32% thought it was “sort of scientific”.⁴ Those are only a few of the more egregious examples of false beliefs. Everyone holds many minor beliefs that don’t comport with reality.

What are we to make of the fact that so many people hold false beliefs when most profess to be competent to determine the truth? Perhaps a better question to ask is: what reason is there to think that humans will believe true things? Humans are evolved animals, and the process of natural selection does not inherently select for individuals that hold true beliefs. Evolution selects for individuals that propagate their genes. In many instances, that is actually a reasonably good approximation for selecting for true beliefs. An individual holding proper beliefs on which animals are its predators and what is edible, will be more likely to survive and pass on its genes. Thus, there is reason to believe that in some areas human beliefs should tend to be true. However, in other areas there may have been no selection pressure or indeed have been selection for a tendency to false belief. It has been speculated that widespread belief in God is a result of over active agency detectors in the human brain. That is, a bias toward seeing external events as the result of the actions of another individual or animal rather than as mere coincidence. A human progenitor might be more likely to survive if they ascribe the rustle of some nearby grass to a predator rather than the wind. Even if doing so means they run for safety when it is not necessary, it ensures they will run truly when needed. Consistent with the hypothesis that we evolved to often hold false beliefs, cognitive psychology has identified many biases common to humans.

We humans have many reasons for holding the beliefs we do besides that the beliefs are the best explanation of the evidence. Holding a belief may be beneficial for fitting in with a social group. One may believe that they “ought” to hold a certain belief. For example, that it is a right or proper belief to hold, or that virtuous people hold that belief. Beliefs may have been learned from an authority and not questioned. Often, people do not have the spare cognitive resources to spend on questioning a belief, or feel that getting to the truth may be beyond their capabilities and so not worth the attempt. Frequently beliefs aren’t selected for truth, but for expediency or benefit. Cognitive science finds that humans are prone to engage in motivated reasoning. Where the desired belief has already been selected for some other purpose and the reasoning mind is engaged to provide justification to others or to oneself. Often motivated reasoning is used because facing the actual truth has lots of negative emotions attached to it. Ultimately, too many simply fear the truth. They find truth to be ferocious.

It is not so much the mundane beliefs and facts of everyday life that people find ferocious, but the answers to the great questions of morality, meaning, religion, and philosophy. These are what they find too ferocious to face. Yet, they already live in the world as it actually is. Psychologist Eugene Gendlin wrote about this in his book Focusing.

What is true is already so. Owning up to it doesn’t make it worse. Not being open about it doesn’t make it go away. And because it’s true, it is what is there to be interacted with. Anything untrue isn’t there to be lived. People can stand what is true, for they are already enduring it.

This quote is referred to as the “Litany of Gendlin” in the rationality community.⁵ This community grew on the website Less Wrong started in 2009 using material written by Eliezer Yudkowsky for the blog Overcoming Bias. In this community, rationality is considered to have two aspects: epistemic rationality and instrumental rationality. Epistemic rationality is having accurate beliefs about the world. Instrumental rationality is taking the best actions to achieve your goals, whatever those may be. In pursuit of being more rational, they focus a great deal on learning about and counteracting their own cognitive biases. When thinking about whether their beliefs are accurate, they like to use the metaphor of the map and the territory.

This metaphor originated with amateur philosopher Alfred Korzybski who remarked that “the map is not the territory.”⁶ The actual reality is compared to a territory and one’s beliefs to a map of the territory. This serves as an important reminder that our beliefs don’t always match reality, just as a map may not always be an accurate reflection of the territory it is meant to depict. Embedded in this metaphor is the view of truth philosophers term the correspondence theory of truth. That what makes something true or false is whether it properly corresponds to reality.

Map Problems

Using the map and territory metaphor, we can more easily see that there are a wide variety of ways our beliefs can fail to match reality. Put another way, it is not simply that we hold false beliefs, but that there are many ways our beliefs can fail to correspond to reality.

Gaps

One of the least problematic ways a map can fail to match the territory is to simply have gaps or blank spaces. Modern maps generally don’t have this. However, historical maps often show distant lands fade off into blank spaces or the unexplored interiors of regions are blank. Here, the map makes it clear that there is something there, but does not indicate what it is. We experience this in our beliefs when we consider something and find that we don’t know the answer. We are knowingly ignorant of a topic. This is generally less dangerous than other kinds of map/territory errors because at least we know that we don’t know.

Omissions

A more serious issue is an omission from a map. As when a map fails to indicate the presence of a hazard or barrier. Reading the map, it seems there are no gaps; that nothing is missing. There is no indication of the omission. In our beliefs this is experienced as unknown ignorance. We believe we know about a given subject, but, unbeknownst to us, our beliefs have important omissions. This can be problematic because we make decisions believing we are adequately informed when we are not. We do not know to seek out more information to help us make a better decision, and so may make the wrong one.

Mismatch

Before modern cartographic methods it was more common for a map to show the shape of a coastline or other feature incorrectly. The map indicates the presence of a feature, but describes that feature incorrectly. This can be almost as insidious as an omission. Again, we do not recognize the error in our map, in our beliefs. Yet, our beliefs are subtle distortions of reality, leading us astray.

Mythical Lands

Perhaps the most insidious of errors is a map that shows features that don’t exist in the territory, as when ancient maps showed mythical lands. We may, like bearers of maps of lost pirate treasure, set off on hopeless quests. Beliefs that bear no correspondence to anything in the world are more likely to cause active pursuit of incorrect goals. The other map/territory errors tend to manifest in failure to act. This error, tends to manifest in incorrect action. Great harm can be caused when people act on this, often attempting to force others to conform to their false beliefs about the world.

Correcting Maps

Given the many ways our maps can fail to match the territory and the potentially high price of the resulting mistakes, it behooves us to attempt to correct our maps. Indeed, that is the very project of epistemic rationality. Yet has the rationality movement actually made its members more rational? Consider that in a 2016 survey of Less Wrong users, only 48 of 1 660 or 2.9% of respondents answering the question said that they were “signed up or just finishing up paperwork” for cryonics.⁷ This despite the fact that Eliezer Yudkowsky has argued in the strongest terms possible that it is the rational thing to do, saying “I want you to live.”⁸ While this is certainly a much higher portion than the essentially 0% of Americans who are signed up for cryonics based on published membership numbers,⁹ it is still a tiny percentage when considering that cryonics is the most direct action one can take to increase the probability of living past one’s natural lifespan. If “rationality is systematized winning”¹⁰ then it would seem that involvement in the rationality community hasn’t been able to increase rationality very much. It has been objected to this characterization that the problem may not be a failure of epistemic rationality, but rather of instrumental rationality. That the Less Wrong site does not focus on this aspect of rationality. This objection is consistent with the fact that 515 or 31% of respondents to the question answered that they “would like to sign up,” but haven’t for various reasons. Beyond that, when asked “Do you think cryonics, as currently practiced by Alcor/Cryonics Institute will work?”, 71% of respondents answered yes or maybe.⁷ I will concede that Less Wrong does not focus on training instrumental rationality and there is a disconnect between beliefs and actions. However, the distinction between instrumental and epistemic rationality is not so clear cut.

If a reliable and trustworthy source said that for the entire day, a major company or government was giving out $100 000 checks to everyone who showed up at a nearby location, what would be the rational course of action? It might be argued that it is more likely the source is mistaken or lying, but assume for the sake of the argument that one does not believe this to be the case. Any course of action not involving going down and collecting the $100 000 would likely not be rational. To do so would be an indication that one didn’t actually believe they would receive the money upon going to claim it. Likewise, when offered a chance to live beyond our natural lifespan, responses that fail to result in actually taking that opportunity must call into question whether the opportunity is believed to be real. Yet, there is a large gap between those claiming to believe and those acting as we would expect based on that belief. This is the divide between professed and actual belief that too often occurs. So, we must find ways to correct not only our professed beliefs, but our actual beliefs.

Very few individuals are able to routinely seek and discover the truth. It is my experience that those who do place a great value on knowing the truth. They hold curiosity as a virtue. Curiosity admits its ignorance and seeks to replace it with knowledge. But curiosity is not sufficient unless it is paired with diligent study. Otherwise, it is simply idle curiosity. Diligent study comes when we wholeheartedly follow after something. Personally, I place a high value on knowing the truth and learning new things. I wish this value was more widely shared. And so I say to all: let us be curious disciples of ferocious truth.