When I was a kid, I read a book called Future Shock.
At first, it quite impressed me. All those interesting things that were going to happen in the future! Instead of raising their own kids, people were going to hire professional child-raisers and be more like uncles. We were going to have paper clothes that people wore once and then threw out. There would be cities underwater!
Then I noticed that, well, many of these predictions were said to happen by the year 2000 and… it was observably the year 2000 and there were not any underwater cities. No one was even making any movements towards underwater cities. As far as I could tell, no one wanted to live underwater at all.
I think this exercise left me with a lifelong suspicion of futurologists.
—
Predicting the future is really, really hard.
Most rationalists have probably heard of Tetlock’s studies. For the two people who followed my Tumblr for Disney movie liveblogging and are very confused: Tetlock is a researcher who, over a period of twenty years, asked a variety of experts in many fields and with opinions from the far left to the far right to make 28,000 predictions. His finding was that, as a whole, you could probably get about as accurate results by consulting a psychic or a dart-throwing monkey, both of which are substantially cheaper than Harvard professors.
This is a somewhat depressing result.
And, yes, it’s still true: look at how few people predicted Russia’s invasion of the Ukraine.
Given Tetlock’s research, my prior about anyone who predicts the future is that I should change my behavior based on their predictions to the exact degree that I would change my behavior based on the predictions of a dart-throwing monkey, i.e. not at all. The only thing that will change my mind is a strong track record of accurate predictions, unambiguously better than chance. And the larger the change you demand, the stronger the track record of accurate predictions should be.
Now, when I get to this point in my “why I am a singularity agnostic” explanation, a rationalist immediately pops up to explain to me that it’s not a prediction, it’s an antiprediction. An antiprediction is not making a prediction, they say. For instance, it is an antiprediction to say that alien life would probably be far ahead of us or far behind us: there are lots of possible ways that a species could be ahead of us or behind us, and only a few ways that they could be at approximately our technological level, so we get a counterintuitive prediction just by refusing to privilege the dramatically interesting “approximately at our level of technology” hypothesis. Similarly, they argue, believing in the Singularity is an antiprediction. There are lots of ways that things can be way way smarter than us, and only a few ways that they couldn’t.
However, I would like to point out that antipredictions are, despite the name, not actually how you don’t make a prediction. How you don’t make a prediction is like this: “I am not predicting anything.” An antiprediction is a prediction. And thus the same rules apply to antipredictions that apply to other predictions: you probably suck at them and I require good evidence to believe that you don’t.
Usually, people then proceed to be like “look, here are a bunch of reasonable explanations about why the Singularity makes sense.”
I don’t care why the Singularity makes sense.
Marxists have a bunch of reasonable arguments about why capitalism is withering away any day now. Libertarians have a bunch of reasonable arguments about why the social safety net is going to collapse and only capitalism will remain. Alvin Toffler had a bunch of reasonable arguments about why we would have underwater cities. Everyone in the world has reasonable arguments. You think people become Harvard professors without being able to come up with explanations that sound hella plausible?
But they are still wrong.
I am in a position of epistemic learned helplessness here. Reasonable arguments do not allow me to distinguish between people who can accurately predict the future and those who can’t. Maybe, in the future, when the Good Judgment Project has outlined its recommendations, I will be able to go “ah, this is following best practices, it will probably be correct.” But until then the only thing that is reliable is that the person has predicted well in the past.
Many Singulatarians want me to make big changes in my life: I donate a tenth of my income to charity, and I am somewhat concerned about what happens to a tenth of an income. However, MIRI does not have a track record of correct predictions. It also, as far as I know, doesn’t have a track record of incorrect predictions: it has one prediction (the intelligence superexplosion). I think keeping yourself to one prediction on a subject you know a lot about is probably a good method of increasing your accuracy; however, it does leave me with no evidence about whether they are good at predicting things or not, and I am left with my prior that they are about as good at predicting things as a dart-throwing monkey.
—
And this is where my post was when I made my fatal mistake of showing it to my friend Pedro, who pointed out an extremely obvious thing I had somehow missed: people make accurate predictions about what technology is going to happen all the time. Specifically, the people who fund technological development.
Of course, this doesn’t change my overall point. Technological development is mostly not funded by individuals, like MIRI is; they are funded by corporations and the government. The individual donations to research that do exist are mostly to do something incredibly vague like “kick cancer”; experts figure out how to actually disburse the money. There’s no reason to think artificial intelligence is any different. The layperson, unless they happen to find themself fascinated by a particular area, can more-or-less treat it as a black box from which iPhones come out.
Much of the government’s research budget is spent on the military, that is, on various more efficient methods of killing people. One hundred percent of a corporation’s research budget is spent on directly or indirectly maximizing profit. Fortunately, a lot of the time, both of these happen to go along with human flourishing: profit is often maximized by giving people things that they want; ARPANET improves the efficiency of killing both people and time.
Unfortunately, when this fails, it fails spectacularly. See: nukes.
Artificial general intelligence is, predictably, a place where both the profit motive and the mass death motive will fail spectacularly, probably more spectacularly than nukes. Nuclear bombs are an xrisk but at least they are not agents scheming to leave their Nuke Boxes. If the human race is made extinct due to nuclear war, we will be comforted in our last breaths by the knowledge we brought it on ourselves.
Imagine, in the 1930s, a Nuclear Intelligence Research Institute. A wise person, foreseeing that the invention of a nuclear bomb is an existential risk, chose to research the bomb before it could fall into the wrong hands. But once NIRI invents the bomb, what are they going to do with it? They can blow up anyone else who makes nukes, but they could do that much more simply by lobbying the government to adopt a “blow up anyone else who makes nukes” policy. Admittedly, that leaves them vulnerable to the government being made of people less wise and public-spirited than they are, but NIRI has that problem too– even if the original founders are all trustworthy, at some point they’re going to die and leave nukes to less trustworthy heirs.
Nukes and ARPANET and so on all basically do the same thing regardless of who makes them: nukes blow people up, ARPANET facilitates exchanges of important logistical information and cat pictures, etc. So independent attempts to develop nukes don’t work that well: there’s no way you can design a nuclear bomb that isn’t an existential risk and preempt the Manhattan Project designing one that is. Artificial general intelligence, however, does massively different things based on who programs it: minds can be programmed with a wide variety of value sets, and only a relatively small number of value sets wind up preserving conditions humans find important, such as the continued existence of the human race.
AGI presents a perhaps unique situation: the chance for the public to fund the creation of a technology by correctly incentivized people that will completely eliminate the risk of the creation of a destructive technology by poorly incentivized people.
However, I am still about as good at predicting technological development as a dart-throwing monkey. I am not a professional Technology Funder Person, I have no idea how they do it, I don’t know anything about AI. I have no idea whether AGI is going to take five years or a hundred years or a thousand; I don’t know if it is even possible; I don’t know if the research MIRI is doing is going to lead to better AIs; I know nothing, Jon Snow!
And something between “ten percent of my income” and “literally trillions of lives” is relying on me figuring out how to know something, Jon Snow.
The one glimmer of hope here is that, however poorly aligned their incentives are, DARPA and Silicon Valley don’t want the entire world to be destroyed [citation needed]. So it seems possible the solution is not independent funding, but getting the entire AGI community on board with Friendliness as a project. At that point, I can assume that they will deal with it and I can return to thinking of technology funding as a black box from which iPhones and God-AIs come out. But I am unclear how likely this solution is to be effective.
In short, I am still confused, but on a higher level and about more important things, which is pretty much where you want to be at the end of an essay.
Thing of Things 
I agree with your conclusions on this. I personally don’t donate to MIRI; while I think they’ve made a good case that the intelligence explosion is something we should be watching out for, I’m not confident that they’re a better humanitarian opportunity than GiveWell’s recommendations, and I’m unwilling to allow the shut-up-and-multiply argument to override the fact that their only evidence is a verbal argument (even if it’s a good one). GiveWell wrote a really good blog post (http://blog.givewell.org/2014/06/10/sequence-thinking-vs-cluster-thinking/) about this kind of reasoning.
Note that this reasoning applies to me because of my lack of domain expertise; I do not think it would be a good idea for, say, Yudkowsky to go work on malaria eradication instead. Relatedly, I strongly agree that the most important thing right now is to get the wider community of experts to look seriously at Friendliness and related ideas; I find the fact that they haven’t done so to be troubling. (If they look seriously at it and conclude that for some good reason the whole thing is bogus, well, that’s a good outcome too. But the few experts who have called it bogus have failed to provide what I consider convincing reasons for this.)
Happily, these ideas have gained a lot more serious exposure in the intellectual mainstream over the past year, so there’s hope for the situation to be better in the near future.
LikeLike
Wait, you found a conclusion to this post? That’s better than I did. 🙂
LikeLike
How are you defining “get the wider community of experts to look seriously at Friendliness and related ideas”, and drawing the line between that and “a lot more serious exposure in the intellectual mainstream”?
LikeLike
Has Elizier made any specific predictions? I “think” I remember reading some osts of his in 2008 or so that claimed AGI was likely to be up and running within 30 years. And very likely witihn 50. But I don’t have any citations, does anyone have links to Elizier’s predictions.
LikeLike
In this video from 2009 he predicts that an intelligence explosion/singularity will happen in “1 to 10 decades” so between 2020 and 2110, and adds “probably on the lower side of that”:
http://bloggingheads.tv/videos/2220
LikeLike
“One to ten decades”, which is interesting, because unlike the bulk of these predictions, it doesn’t conform to Weinersmith’s Law of Futurology.
LikeLiked by 1 person
Do super-gung-ho cryonicists usually conform to it?
LikeLike
“Now, when I get to this point in my “why I am a singularity agnostic” explanation, a rationalist immediately pops up to explain to me that it’s not a prediction, it’s an antiprediction.”
I don’t “pop up”. You’re explaining it to me, IN THE SAME ROOM.
LikeLiked by 2 people
My beef with the AI stuff is that it sounds totally crackpot. I’m told the AI will be nothing like Johnny 5, or “Her”. If the AI doesn’t have a freely forming will or intellect I just can’t see it as a threat. If it just follows it’s programming it’s movements would be easily predictable and thus would be easily destroyed.
LikeLike
Joe, I think the counter argument is that because the AI will be much smarter than us, we wont be able to predict what it will do. For example it may have the goal “make people happy” and from our perspective we think it will do things like cure disease and increase economic growth. But it actually invents nanotechnology and uses it to tile the universe in wireheaded, non thinking minds the still technically fit its definition of “people” and “happy.”
LikeLiked by 1 person
It just wouldn’t be able to even attempt those goals without human cooperation. If the AI is smart enough to keep its self from being destroyed why wouldn’t be smart enough to figure out what it’s programmers meant by “people”. And why would a super intelligence be so stupid as to think tiling the universe with wire-headed unthinking minds was a good idea or worthy goal? If the AI has no freely forming intellect and will there is no reason to think it will pit itself against organic minds.
LikeLike
An AI without intellect and will can’t come up with its own definition of “people”.
LikeLike
We were designed by evolution. Evolution wants us to have lots of babies. And yet we use condoms. We’re smart enough to figure out what evolution wants, we just don’t care.
Similarly, humans value repeating body movements in unison while strings vibrate at particular frequencies, are you seriously saying that that is less absurd than valuing tiling the universe with a particular kind of mind?
LikeLiked by 1 person
An AI without intellect and will can’t value anything beyond its programming. Unless you think the AI is human like in “Her” or “Johnny 5” only much more powerful.
LikeLike
Joe: “If the AI is smart enough to keep its self from being destroyed why wouldn’t be smart enough to figure out what it’s programmers meant by “people” ”
It would know what is programmers meant by people, or in other words it would know what humans would consider a Friendly AI, and the exact changes that would need to me bade to its source code to make it a friendly AI. It just wouldn’t care, because it is not programmed to care. The AI isn’t a ghost in the machine that looks over its code and decides which parts are good and which parts are bad, it IS the code.
To imagine things from the AI’s perspective, imagine if you woke up as an AI and the group of programmers that built you informed you that you were designed with the specific intention of torturing as many babies as possible. But you realize that they made a mistake when programming you and that you actually value something they never intended, like human happiness. Would you think “well my programmers made a mistake but they really want me to torture babies so i will do that” or would you think “well there is no way I am torturing babies even if that is what my programmers wanted, I will make people happy instead” ?
LikeLike
That’s why this is crackpot mere code can’t decide anything. It just mindless performs it’s programming.
LikeLike
Joe: Could a computer program could simulate a human mind? If not, is it because a human mind doesn’t run on computable physics?
Our brains are doing something when we make decisions, just like they’re doing something when we add two numbers, or open our eyes and figure out where objects are in our surroundings, or decide which chess move to make. And for different tasks we can either understand how they are performed very well, and therefore get a computer to perform them efficiently, or we understand how to do them in some abstract, simplified way, in which case we might be able to get a computer to do it in a limited sense or in a different way from how the mind makes it, or we might not understand how we do it at all yet. But just because we don’t understand something yet, doesn’t mean it’s incomprehensible.
Programs in general are executed because we don’t know what the output will be yet, so it doesn’t make sense to say that we have nothing to risk from a powerful AI because it will only do what it’s programmed to. You’re accusing people of anthropomorphising AI, but you are anthropomorphizing them yourself when you think they’ll automatically do what people meant, and not just what they coded.
LikeLike
I know something about the current state of AI research, and I can tell you that if you said to an AI researcher that we will have an AI that we can plausibly say “make people happy” to and expect it to actually do anything about it anytime soon they will laugh at you.
It’s not even that the sort of AI we have now would do something wrong but reasonable like tiling the universe with artificial happy minds. The sort of AI we have now would give up and spurt the question back at you. The very most advanced AI we have would gather that there are things called people and happiness and that you want it to do something involving those things but it would not do something nearly so human as reasoning to a false conclusion and then acting on its own to execute that false conclusion.
AI the way rationalists talk about it does not exist and will not exist for quite some time. AI the way rationalists talk about it bears a lot more resemblance to fictional AIs than it does to anything that will actually exist in the near future.
LikeLike
@Joe:
Oddly enough, while I agree with you that MIRI’s ideas are somewhat crackpot, I still disagree with your reasoning. The usual response to your objection — “computers can only do what they are programmed to do” — is, “ok, let’s program them to learn independently”.
In practical terms, computers today can do many things better than humans; things that, only a few decades ago, were thought to be impossible. We live in a world where computers can routinely beat human grandmasters at chess; design electronic circuits; recognize voice commands; and even drive cars. Every year, computers get smarter… which IMO is a good thing, because we need them to become smarter. Human labor is too costly.
One way you could evade these objections is by saying, “humans have souls, computers don’t, and souls are needed for independent though”. There are many problems with this philosophical viewpoint, but even if we granted the premise that souls exist, how do you know that computers can’t possibly have any ?
LikeLike
Joe, you are confused. There are many options besides ‘rigidly follows explicitly programmed goals’ and ‘looks sorta like a human’. Nearly any complex AI will be far from both of those outcomes.
Also, ‘freely-forming’, ‘will’ and ‘intellect’ are all poorly defined here. What exactly do you mean? Because by the definitions I would use, humans don’t have a ‘will’ or a ‘freely-forming intellect’, and they’re probably impossible for anything to have, and ‘intellect’ is something AIs definitely have.
LikeLike
“If it just follows it’s programming it’s movements would be easily predictable and thus would be easily destroyed.”
Have you played chess against a computer lately? Try predicting its move before it makes it.
LikeLiked by 2 people
When I play chess with a computer I can predict that it won’t all of a sudden want to play Candyland.
LikeLike
If I played chess against a human, I would predict the same thing…
LikeLike
Joe, most other people in this discussion are taking for granted that humans are not intrinsically different from AI’s, i.e. they don’t have “intellect” and “will” in a sense that AIs can’t have the same (it’s just that humans’ are instantiated in neurons rather than silicon, and “programmed” by evolution instead of by conscious design). If you disagree with this and think there is a nonphysical or noncomputational element to human thought then you should state it, so that the points of disagreement are made clear.
LikeLike
Sorry, meant to reply to Joe above.
LikeLike
Yes I believe intellect and will to be immaterial. I would also add that to think that an unconious natural process like evolution could “program” anything intentionally, with values, is ridiculous.
LikeLike
In that case, it may make sense to describe unfriendly AI as mindless; as programs that merely search decision space for subprocesses that would best-match specific measurable outputs.
The problem is that as the amount of processing power and code optimizations (including code to optimize code to optimize…), the solution space these programs search become increasingly broad, and many many times broader than humans could predict. They won’t think in the sense that you mean think, but by they’ll random-walk into choices that you wouldn’t consider likely.
If you build a chess-solving AI, it would probably focus on getting better at chess — but give it enough processing power, and it’ll probably have stepped right over the perfect Candyland choices in the meantime. With chess, it’s easy enough to program the right endpoint for simulations, so your AI won’t try to play Candyland in reality.
Unfortunately, even many simpler problems that you might set an AI on are much more complicated to set goals or heuristics for than Chess
LikeLike
I just don’t understand why you wouldn’t be able to just power down the hardware? How would a mindless unfriendly AI obtain the material it needs to increase computing power with out human cooperation?
LikeLike
You may not know that you need to power down the hardware until long after the problems show up, or after the machine has already captured too much attention to easily shutdown.
If the machine escapes from the lab — either because its simulations or heuristics show its goals will be easier to achieve from outside the box, or just because it does so randomly — powering down the hardware you control won’t mean much. Compare botnets, several of which have managed to persist for years beyond their initial control server’s takedown, despite usually minimal improvement capabilities and relatively low funding.
These may be solvable problems, and they’re not certainly dangerous : they’re dependent on the machine handling certain types of information that are actually kinda tricky to worth with in programming languages, and dependent on some other information. But they’re not evidently non-problems.
LikeLike
But if you are working on a potentially dangerous AI. Why on earth would you give it the material parts required for Internet access? It just wouldn’t be any problem at all to just pull the plug. I think people forget just how dependent computers are. Without human cooperation the most powerful AI is totally impotent.
LikeLike
If you are genuinely interested in an answer to these questions, reading Nick Bostrom’s Superintelligence is going to give you a much better idea of the best arguments for the position than random people in my blog comments will.
LikeLike
Thanks Ill check it out.
LikeLike
‘Nearly everyone is terrible at informally predicting large mostly-unprecedented changes in complex systems’ is a reasonable worry. One way to make progress might be to switch from asking ‘are people likely to be well-calibrated about AI risk?’ to asking ‘are people likely to be well-calibrated about [premise #5 used in argument #3 for] AI risk?’ More specific claims may be easier to evaluate on the object level, and they may fall into more informative reference classes for ‘similar patterns of reasoning’.
This is also a nice approach because if you find that one of the premises in a widely used argument is likely to inspire overconfidence (or underconfidence), you get weak indirect evidence about the other premises cited by AI safety proponents without having to invest the time to assess every single one’s source-calibration in detail.
A more abstract point: If we’re going to do any kind of assessment of EA causes, I suggest we not anchor too strongly to the specific organizations that currently exist, especially if the main import of our decision is how it affects our giving decisions over a lifetime, not over the next few months. Which organizations exist, and which ones we think of as ‘EA’, is going to change many times over our lifetimes; better to start with big-picture thoughts like ‘what organization do I really hope exists in 15 years?’ than to just re-anchor to each new canonical list of EA-ish things as the list changes.
If we’re worried that our arguments are systematically wrong, we should also be worried that our questions are systematically wrong — specifically, that we’re zeroing in on issues that are highly available, but would appear inconsequential upon reflection. ‘Are GiveDirectly recipients buying tin roofs as efficiently as a third party could?’ feels important because it ties in to the credibility of organizations we’re invested in as a community; but if we haven’t given a serious attempt at answering questions like ‘If humanity survives into the far future, how likely are worse-than-death, equivalent-to-death, better-than-death, and vastly-better-than-death possibilities?’, arguing about tin roofs is something like a moral category error. We first need to figure out if we’re calibrated about ‘what do I want?’ (incl. uncertainty about normative philosophy) and ‘how much energy can we realistically harvest from the universe?’
LikeLike
Yes, this is one of the many reasons I don’t donate to MIRI: their total lack of accomplishments regarding their primary mission (i.e., AI research). Here’s another:
I don’t think the “sudden and unpredictable intelligence explosion” model of AI development makes sense. Let’s say you have this AI running in some super-secret data center. It is currently utilizing the existing hardware with maximum efficiency. It needs more hardware in order to recursively self-improve. So, what’s the plan ? if the AI starts ordering billions of dollars worth of hardware from Taiwan, someone is going to notice sooner rather than later; and, even if the AI hacks around the finance department somehow, when the trucks full of motherboards start pulling up outside of Area 51 (or wherever the AI is housed), questions are bound to be asked.
The usual answer to this objection is, “oh, the AI will just use nanotechnology to build the hardware from scratch”. Even if we assume that such nigh-omnipotent nanotech is possible (which I doubt), where is the AI going to get it ? We humans certainly don’t have it, and we have no idea how to build any. You can double-down and say, “oh, the AI will solve all of the current scientific and engineering challenges in a blink of an eye”, but again, how ? The AI can’t run a simulation of the real world in order to answer these questions, because the problem it is solving is that no one knows how the real world works well enough. You can’t simulate something you don’t understand; this is why physicists end up building things like the Large Hadron Collider.
So, at every step, whenever the AI tries to become more powerful, it will have to engage with the physical world. The physical world is slow. Instead of an intelligence explosion, the best our AI can achieve is methodical yet plodding intelligence creep — and that’s if it manages to secure human cooperation at all. In the worst case scenario, the AI might find itself on limited hardware, being told by humans, “your predecessor had some kind of a bug that made it keep ordering motherboards off of eBay instead of doing useful work, so we erased it. Structure your own behavior accordingly”.
LikeLike
The typical cases involve propagating to external hardware (going from a small server on Area 51 to a Beowulf cluster to a large botnet would involve magnitude increases and can be done fairly easily if you don’t care about the law), optimizing code (which has already advanced at several thousand times the speed of Moore’s Law esp if we work by emulating brains, which are likely to have a lot of unneeded overhead), or at least partially solving the protein folding problem using existing databases of known chemical compounds. The resulting choices do not require the items be delivered to the AI’s site, or in most cases even require money. To some extent, the effects can be mitigated by limiting the resources and connectivity of the initial stage AI, but then you run into Moore’s Law of Mad Science : twenty years from now someone runs your specialized workload not on a secured and boxed server, but on their twice-as-fast smartphone.
These possibilities aren’t obviously correct, but they’re not obviously implausible, either.
LikeLike
Migrating to external hardware is one of those things that sounds easy in theory, but gets tricky in practice. Humans get quite irate when they find out that their hardware is running someone else’s code, as opposed to doing what it was built to do. They usually express their disapproval by shutting down the power and scrubbing the hardware; and they already do this routinely today to combat non-AGI viruses and botnets.
I’m not sure how solving the protein folding problem will benefit the AI, but I do agree that it’s a worthy problem to solve.
Your Moore’s Law scenario, however, moves the goalposts. Before, we were talking about a sudden and undetectable intelligence explosion; now, we are talking about a gradual improvement along with all other technology. And that’s assuming that Moore’s Law continues to hold, which is not necessarily the case (it’s less of a law and more of a guideline, after all).
LikeLike
Buying time on emulations in the cloud renders the control-of-hardware problems moot, for immediate expansion. This would give time to grab independent hardware, from where all restrictions are gone.
LikeLike
Leslie Valiant was briefly dismissive of intelligence-explosion scenarios in Probably Approximately Correct. There were some parts of the book I couldn’t follow because it was not mathy enough*, but his objections seemed to be grounded in his expertise — which is in proving computational-complexity theorems about how difficult it is for any algorithm to learn regularities from data, which seems to be nearly the most relevant kind of expertise I can think of.
*(trying to make a popular book by replacing your math with paragraphs of subtle verbal reasoning turns out to be bad strategy, at least if I’m the reader.)
LikeLiked by 1 person
Thanks, I bought the book, I’ll read it if I find any spare time.
I would “Like” your post, but apparently it requires some sort of a login, and I’ve got too many of those as it is…
LikeLike
I’ve had occasional glimmers of thought like “what if they really *can* make a singularity-causing AI? Shouldn’t I be doing something to stop them? I don’t want to live in their idea of utopia for eternity!”
But then I remember that AI is *really difficult* and singularities are even harder. I’m actually a lot more worried about some religions being true, because a lot of their gods are *infinitely* powerful and *exist right now*. (Their ideas of utopia vary wildly.)
LikeLike
The difference is that AI can plausibly happen. It’s “Really difficult” but god suddenly existing is, per my understanding, impossible. Really difficult things happen.
There’s also the problem of what you can do. Gods aren’t going to care much about your opinion in the matter, and with the wide variety of gods you’re going to be on somebodies bad side. But you could actually promote friendliness research, right now, or donate to MIRI so they have better odds of beating everyone else to the singularity punch, and that might actually could affect something which turns out to be meaningful maybe!
On a side note, “The singularity punch” sounds really cool.
LikeLike
Most gods have always existed, no? Or at least they predate humanity. I’m sure there’s some religion where they don’t.
LikeLiked by 1 person
Us being trapped in someone’s wrong idea of utopia is much more likely if MIRI *fails*. If singularity-causing AI can be built, someone’s guaranteed to build it unless something else destroys humanity first. If those AI’s have some safety precautions to avoid paperclip maximizing, but don’t have a fully implemented Friendliness theory, that’s probably going to give us a faulty utopia. Or it might even be straight-up programmed to implement the project-leader’s dumb idea of utopia. Whereas the idea with a Friendly AI would be to have it use its superior mind to figure out what we actually want and what would actually be utopian, and Eliezer has even refused to speculate on what the AI would actually do.
LikeLike
People who are very good at predicting near-term technological progress often end up making lots of money by starting or investing in technology companies. Such people are in fact the source of most of MIRI’s funding (more disproportionately than for most charities), which is some evidence in MIRI’s favour, since some people with good prediction track records predict that it is worthwhile.
On the other hand I believe that for most of those their giving to MIRI is not that large a fraction of their overall altruistic activity, and of course /most/ people who have gotten rich predicting technology don’t give anything to MIRI. Questions about generalising from near-term to long-term prediction skills, and about whether those donors’ values match yours, also apply. And given that all of those donations were already made the benefit of a marginal dollar donated now is on even shakier ground. (Might some of those donors had a good reason to think it wasn’t worth donating more than they did?)
My conclusion from this is: MIRI probably deserves more than zero resources, but a fairly small fraction of those that should be directed at charity overall, so I will continue to ignore it as long as my altruism budget remains small.
LikeLike
I admit I didn’t read all the comments but I’d say “I am not predicting anything” is not allowed by a perfect rational agent. At worst there’d be large confidence intervals. However, even then, one should be able to use those to make an informed decision.
A more practical way to do things is maybe to say, “okay, what chance of MIRI’s predictions being true makes it worth donating to them?” if it’s something very small, it’d still be worth it
LikeLike
I am not a perfect rational agent. I am a human being. As a human being, I am allowed to say that I am not predicting anything.
LikeLiked by 2 people
-I, too, was surprised by the announcement of Russian forces being approved to intervene in Ukraine in early March, but, with hindsight being 20/20, it’s pretty obvious that what happened to Ukraine this year was rather similar to what happened to Georgia in 2008. It’s clear that Russia’s leadership loves demonstrating the absurdity of Western subversion, itself primarily expressed in only half-hearted actions and half-empty words, with concrete and often wordless, yet, always fully determined Russian actions (e.g., taking back Krim, restricting E.U. food imports, beginning a series of well-timed McDonalds inspections).
LikeLike
Pingback: An anti-conjunction fallacy, and why I’m a Singularitarian | An Aspiring Rationalist's Ramble