Tags

, ,

In a perfect world, if someone had a psychological problem, I’d be able to tell them all about the rich peer-reviewed psychological literature that has to do with specifically their problem, all of which has effect sizes of a billion and definitely and certainly applies to everyone.

Unfortunately, we live in this world.

For the past decade or so, the social sciences (particularly psychology) and medicine have been going through the replication crisis. Basically, when you repeat social scientific and medical experiments, many of the effects that existed the first time you did the experiment disappear. Some projects suggest that fewer than half of psychological studies show a reproducible effect.

Why is this a problem? Imagine if, when we repeated the experiment of dropping a feather and a bowling ball in a vacuum chamber, sometimes they fell at the same rate, but sometimes the feather fell faster, sometimes the bowling ball fell faster, and sometimes the bowling ball fell up and hit the experimenter in the face. It would be pretty clear that we don’t know anything about how gravity works, and all the physics that built on the idea that feathers and bowling balls fall at the same rate in a vacuum chamber was totally wrong.

It would also bring up the question of how nobody noticed that sometimes bowling balls fall up before. The answer for psychology is that many psychologists use questionable research practices: one survey suggests that more than half of psychologists have. Many psychologists check whether the data shows an effect; if it is, they publish, and if it isn’t, they recruit fifty more participants to see if it will show an effect. Other psychologists will write a paper that includes multiple studies and only report studies that showed an effect, even though they conducted other studies that showed no effect. Still other psychologists will decide whether or not to leave out outliers depending on which way shows an effect as existing. There are lots of other questionable research practices, many of which require more statistics to explain, but I hope that gives a sense of what people are doing.

The other problem is that humans behave much more inconsistently than bowling balls do. You’d expect that if you drop a bowling ball in a vacuum chamber, it’ll be exactly the same as every other bowling ball in every other vacuum chamber around the globe. You only need to drop one bowling ball in a vacuum chamber to find out how bowling balls in vacuum chambers work everywhere. But if you do a study with a hundred and fifty people, you might have happened to find a hundred and fifty really weird people. Even if the effect is real within your sample, it might not apply to any other group of a hundred and fifty people, because people are weird.

It is hard to overstate the implications of the replication crisis. I would go so far as to say that you should not read quantitative psychological research, because it will make you actively stupider and have more inaccurate beliefs. You should view quantitative social scientific research more generally with grave suspicion. (Qualitative social science research wasn’t ever really supposed to replicate in the first place; ethnographies are still likely to provide an accurate insight into what it’s like to be a person very different from you, although you might be well-advised to skip the chapter about the generalizable implications.)

Now, you might think we’re in a complete state of epistemic helplessness, unable to know what facts about people are true and what facts are false. But that’s not true.

People are actually very good at figuring out whether psychological results are true or false. I’d cite the research on this but I just finished explaining why that’s bad, so instead I’ll direct you to 80,000 Hours’s psychology replication quiz. If you don’t get better than half, I will write you a drabble on the subject of your choice.

Consider priming, one of the results that has most completely and utterly failed to replicate. It turns out that doing a bunch of anagrams of words related to old people does not actually make you walk slower, imagining doing something morally wrong does not make you want to buy cleaning products, and reading the word ‘stupid’ does not make you assume people you’re talking to are less intelligent. Ask yourself: have I ever in my life experienced reading a bunch of words related to old people and then walking more slowly? Have you ever been like “sorry I’m so slow walking, I was helping out at the nursing home this morning”? No?

Most people spend much of their time thinking about other people. The human brain is in many ways optimizing for understanding other people. (Compare the abilities of your laptop computer and a five-year-old child with regards to comforting you when you’re sad, then compare their abilities with regards to quantum physics.) We are, ourselves, people. We have natural expertise and tons of information about the subject. We can outperform science.

A lot of people think of knowledge as coming from science, and science as coming from people in white labcoats doing complicated things with numbers and publishing peer-reviewed journal articles. But science is just a particular kind of empiricism. Empiricism means going out into the world, looking at things, coming up with an idea about how they might work, trying it out, and seeing if you’re right. You do empiricism all the time– when you try a new recipe, when you see if a good night’s sleep will make you feel better, when you check whether this movie reviewer has any clue what she’s talking about.

And you can do empiricism about your own psychological problems.

There are a bunch of meta-analyses that suggest that SSRIs, in general, are just barely better than placebo. There are also a lot of people who take an antidepressant, go off their antidepressant, get depressed, get back on their antidepressant, stop being depressed, go off their antidepressant, get depressed, get back on their antidepressant, stop being depressed, go off their antidepressant, etc. If you are in that group of people, what should you expect to happen if you go off your antidepressant?

You should expect to be depressed.

In a certain sense, this is totally ignoring science– you’re looking at that peer-reviewed meta-analysis with a bunch of complicated statistics and a sample size larger than one and going “…nah.” In another sense, this is doing science of the sort that is most closely connected to the question. The thing you care about is what will happen if you go off your antidepressant, you have empirically investigated it, and the answer is that you will be depressed.

At this point I should take a moment to discuss the placebo effect. Unfortunately, many people use the phrase “placebo effect” in a sloppy way. In a broad sense, the placebo effect refers to the ways a disease or condition changes if you take a sugar pill instead of a pill with an active ingredient, or talk to a kind intelligent person with no therapeutic training instead of a therapist, or whatever. In a narrow sense, the placebo effect refers to a particular source of these effects: the mind-body effect where if you do something at all to help the condition then you feel better. (More precisely, the former is called the “placebo response” and the latter is called the “placebo effect.”)

Actually, there are lots of causes of the placebo response. For example, a lot of conditions like depression fluctuate: you feel worse for a time, then you feel better, then you feel worse, then you feel better again. People tend to seek out treatment when they feel really bad and stop treatment when they feel better, so it looks like the treatment works, but it’s actually just the natural course of the disease. Similarly, some diseases are time-limited: whether or not you drink orange juice, your cold is going to take about a week to get better. Another example is that some people feel bad that people have put all this effort into giving them medicine and then they don’t feel better, so they rate themselves as feeling better even though they really don’t.

Ideally, you would do a placebo-controlled study on yourself. The process is fairly simple for medications you expect to work within one day, and I think more people should consider doing it. However, there are lots of situations where placebo-controlled studies don’t work at all: it is very difficult to do, say, placebo self-therapy or placebo exercise, and many medications such as SSRIs have to be taken for weeks to work properly.

A more promising strategy is to think about whether your experiences are consistent with the response being a placebo response. For example, if you’ve been depressed for years, you can rule out “it was just the sort of thing that would get better in a week anyway” as an explanation. If you typically miss your antidepressant because of a snafu with the pharmacy, that suggests your post-antidepressant depression is a real effect; if you typically miss your antidepressant because you’re too sad to take it, that suggests it may be a placebo response. If you’ve tried six antidepressants and only the seventh worked (despite them all being similar-looking little pills with similar side effects), it’s unlikely to be a placebo response, since there’s no reason to believe pill #7 would have a placebo response the others didn’t.

How do you choose things to experiment with? I offer the following guidelines:

  • Good safety profile; few to no side effects. SSRIs are widely used and if they caused sudden horrible death we would probably know. It is difficult to imagine pomodoros or Facebook blockers having health effects one way or the other. Exercise actually makes you live longer. Conversely, taking random research chemicals off Longecity has all kinds of possible horrible side effects, and antipsychotics are routinely rated as “the only thing worse than psychosis” or “actually, no, I think I would rather be psychotic”.
  • Easy to figure out if it works. If an intervention promises to help you get more work done or be less anxious, it is easy to figure out whether your to-do list has fewer items on it or you’re able to talk to strangers instead of hiding in the corner out of fright. If an intervention promises to help you reach enlightenment, this is harder to determine.
  • Easy to do. Installing Facebook-blocking software or using a pomodoro tracker is very easy; anyone can do it in about one minute. Exercise, dietary changes, meditating for an hour a day, and so on are very difficult.
  • Quick to test. It takes one afternoon to check whether pomodoros work for you: if you didn’t get more work done that afternoon, it probably didn’t. It can take months or years of work to find out whether meditation works for you. The former is a better thing to experiment with.
  • Explanation makes sense. Ask the person why it works. If the explanation is something like “when you do something you’re scared of and it isn’t a disaster you’re likely to be less scared,” then the thing is more likely to really work. If the explanation involves unscrambling anagrams about old people making you walk slower, then maybe don’t try that one.
  • Good reviews. If it worked for someone else, maybe it’s worth trying. You can ask your friends, but it’s also worth mining historical figures for ideas (maybe Marcus Aurelius has some good tips) or, sigh, reading the psychological literature.
Advertisements