I think a formative moment for any rationalist– our “Uncle Ben shot by the mugger” moment, if you will– is the moment you go “holy shit, everyone in the world is fucking insane.”
Your dad buys lottery tickets. Your best friend treats her colds with homeopathic remedies. Your sister thinks she can beat the stock market. The value of a life is something like twenty million dollars and you can save one for $3,500. The most common cause of condom failure is not wearing one. Ninety percent of people believe in God. You have– multiple times– started watching a movie and it turned out to be terrible and you kept watching it because “we’re already half an hour in” and at no point did it occur to you that this doesn’t make any goddamn sense.
I don’t think it’s an accident that a lot of rationalists are mentally ill. Those of us who are mentally ill learn early and painfully that your brain is constantly lying to you for no reason. I don’t think our brains lie to us more than neurotypicals’ brains do; but they lie more dramatically, about things society is not set up to accommodate, and so the lesson is drilled in.
Now, there are basically two ways you can respond to this.
First, you can say “holy shit, everyone in the world is fucking insane. Therefore, if I adopt the radical new policy of not being fucking insane, I can pick up these giant piles of utility everyone is leaving on the ground, and then I win.”
And this leads to: Polyamory. Modafinil use. Effective altruism. Pretty much any CFAR technique. Attempting to solve medicine single-handedly. Cryonics. SENS. MIRI. Half the things that make Eliezer haters go “Jesus Christ, what is he up to now.”
This is the strategy of discovering a hot new stock tip, investing all your money, winning big, and retiring to Maui.
Second, you can say “holy shit, everyone in the world is fucking insane. However, none of them seem to realize that they’re insane. By extension, I am probably insane. I should take careful steps to minimize the damage I do.”
And this leads to: The emphasis on applying meta-level moral rules that you would wish your enemies abode by. The Principle of Charity. The cults sequence. Trusting experts. The virtue of scholarship. Chesterton’s Fence. The outside view.
This is the strategy of discovering a hot new stock tip, realizing that most stock tips are bogus, and not going bankrupt.
I want to emphasize that these are not mutually exclusive. In fact, they’re a dialectic (…okay, look, this hammer I found is really neat and I want to find some place to use it). Trying to minimize the damage from your insanity is, in fact, a strategy for picking up some of that utility on the ground and then winning.
Nevertheless, there are definitely some people who are more on the becoming-sane side of things, and other people who are more on the insanity-harm-reduction side of things. Eliezer is way over on the “I can become sane!” side; Scott seems to be somewhere in the middle; I’m a grouchy asshole who wanders around waving my cane and saying “you kids are not going to outperform the people with actual degrees“; Topher is somehow even grouchier than I am, and opposes the Principle of Charity on the grounds that people are too insane to be able to ever do it. And I feel like a lot of conflicts in the rationalist community boil down to this conversation:
Becoming Sane Side: “Hey! Guys! I found out how to take over the world using only the power of my mind and a toothpick.”
Harm Reduction Side: “You can’t do that. Nobody’s done that before.”
Becoming Sane Side: “Of course they didn’t, they were completely irrational.”
Harm Reduction Side: “But they thought they were rational, too.”
Becoming Sane Side: “The difference is that I’m right.”
Harm Reduction Side: “They thought that, too!”
Becoming Sane Side: “So, what, I’m not supposed to try to outperform the people who literally spend their money on lottery tickets?”
Harm Reduction Side: “No, you can try to outperform them by not trying to take over the world.”
Both Sides, Simultaneously: “YOU ARE INSANE.”
But, you know, Eliezer wrote Ethical Injunctions, and Topher is a modafinil-using polyamorous effective altruist who’s signed up for cryonics. Whichever one we favor, we all have both impulses in our soul. And I hope that next time we notice a conversation happening along those lines we can at least come to an understanding about what we’re arguing about.