Hi. Your big point (the amount of steering we can hope to do is more like swimming in a river than altering its course) is a good one which I pretty much take for granted, so you won't get any argument from me there. Instead, I'll nitpick about ems.

Much as I like the general idea ("Humans 3.0" in my essay) I have yet to see a convincing description of the technology. A couple of things jumped out at me while reading your essay:

"Ems would often spin off copies to do short term tasks and then end when those tasks are done."

How? A straight copy would be indistinguishable from the original. So you'd have two identical individuals, both thinking that the other one should just do its assignment and then have the good taste to commit suicide. If you propose to modify the copy to behave as expected, you lose the proposed advantage of ems over other approaches to AI, not needing to understand their inner workings in detail. Or is your proposal to create the copy in a torture chamber, suitably equipped to compel obedience and then kill upon task completion?

"Ems would split by speed into status castes, with faster ems being higher status."

Why would any em willingly run slower than technically possible? Would you willingly take a drug which slows you down? You mention gender imbalances as a possible cause, but if copying is so trivial, the gender in low supply could just duplicate itself.

Another speed-related problem: are your ems running on general purpose computers? The implied trivial ease of copying suggests as much. Have you tried to estimate the computational requirements for emulating even a single human brain in real time? Which technology are you suggesting will be used to run trillions of them, as you say, within a century?

When I inventoried computing technologies currently on the horizon, the only realistic candidate I found was neuromorphic hardware. It is far from obvious that it could be used for the kind of general purpose computing needed to run virtual worlds, and if it can't, ems will have a big speed problem.

    Hello Robin

    I agree that we have a limited vision of the river ahead and a limited steering ability. Your markets give us some vision. They are a form of crowdsourcing. I try other forms: I try to crowdsource help in steering. Check out my essay. I should add markets. The one thing I question is your faith in uploads. I will assign them some small subjective probability, but I really doubt that we can determine what seems necessary for a high resolution version: the state of trillions of synapses and their interconnections. I doubt that remote scanning will ever have adequate resolution. Invading the brain with billions of nanobots might determine the state, but would seem to have problems determining interconnections. I doubt that we could slice the brain thinly enough to scan it without trauma (but I am no expert.) We already have low resolution emulations: actors playing folks who were prolific writers. I have seen these versions of Franklin, Twain, and Lincoln. When robots are smart enough they could also play the part of a historical figure. But I do not think that is what you mean. What am I missing that makes you think that uploads are likely? You say that the technology for scanning is likely to be ready within a century. On what do you base that?

      • [deleted]

      • Post #4

      The first uploads would be created via destructive scans, which we already have a lot of experience doing. We don't have sufficient resolution scans yet, in part because we aren't sure what chemical resolution is needed.

      It is trivial when making a copy to mark one as the copy and inform it of that fact. I would have agreed ahead of time that in this situation it would do the agreed task and then end. If you wouldn't agree to this sort of thing, you wouldn't be a good candidate for this world. Others can be found would will agree.

      There is a cost to run faster, which prevents everyone running as fast as possible. The computers are likely to be specialized for the purpose of running ems. It is quite possible to make such machine and still make it easy to copy their internal state to another such machine.

      All right, so you would select suicidal humans as clan templates. That's reassuring. :)

      Yes, there is a cost to run faster: energy. But your whole premise is that ems are made viable by their ability to run faster than organic humans. The faster they are, the more work they get done per unit time, the more energy they can afford, the more profitable they become.

      One interesting consequence of having them run on specialized hardware is that they could be very fast compared to us, maybe too fast for their own comfort. We find virtual worlds attractive because they are easily modifiable, thanks to the versatility of general purpose computers. But general purpose computers may not be able to keep pace with ems running on specialized hardware... and maybe that's what would keep them from wanting to run at full speed. It would make their world too dull.

      Ems can beat humans even if they aren't faster; they need only be cheaper. If needed supporting computers couldn't keep up with some speed, they you wouldn't run at that speed. You'd run slower, and cheaper.

      Robert,

      Thank you for your original, entertaining and thought-provoking essay. I really like your metaphor of the future being like a wide, fast, murky river at night. I've been thinking a lot about your opening statement, that we should work hard to clearly see the future we will get if we do NOTHING to steer it (because most likely humanity will do little to steer it).

      If what you're saying is true, it should also apply to the past: for instance, people in 1900 should have thought very hard to clearly see the year 2000 they would get if humanity did nothing in particular to steer the future. The interesting question to ask ourselves, then, is this :

      In retrospect, how inevitable was the general course taken by the history of the world during the 20th century?

      This suggests a fascinating (but difficult) thought experiment: imagine that we rewind the clock to 1900, and consider the range of the most plausible possible 20th centuries that humanity could have had. If we eliminate "wild cards" like a major asteroid strike or an extraterrestrial invasion, do we always get to a year 2000 that looks reasonably like the one we got --- 6 billion people, capitalism "victorious", reasonably cheap personal computers, the Internet growing rapidly, cell phones, cars running on fossil fuel everywhere, almost no use of nuclear power in warfare but also in electricity generation (except in a few countries), a timid space program with only a few humans in low Earth orbit, huge inequalities between countries, but rising wealth overall, falling death rates and birth rates, etc. --- or was there a fairly wide range of plausible 20th centuries that could have happened instead, if humanity (or some of its most dominant actors) had steered the future differently?

      Studying this alternate-historical thought experiment could help in finding out the degree of "steerability" of the stretch of river that we have ahead of us! I think it could make for an interesting side-project for the Institute for the Future of Humanity, or for other groups working in futures studies.

      In my essay, "To Steer Well We Need to See Clearly: the Need for a Worldwide Futurocentric Education Initiative", I propose that we should work hard to put together a "futurocentric curriculum" aimed at schools but also at lifelong education, in order to raise the level of public interest and knowledge about the topics that are the most relevant to the future of humanity. I think that the question of the degree of "steerability" of the future that you raise, and the alternate-past thought experiments that it suggests, would be quite interesting for the history component of the futurocentric curriculum!

      Marc

        Answers to the question "How different could today have been from what it is" vary with the kinds of changes one allows in the past, relative to what actually happened. For the purpose of my essay I'm focused on changes caused by people trying to influence the distant future, changes that are in the direction they anticipated in order to obtain the effects that they desired. This is a very restrictive standard, a standard that few changes can meet. If you allow many more kinds of changes in your hypothetical, you can get a lot more possible change from what we have now.

        It may well be that the world we have now depends sensitively on many random factors, so that had those gone differently the world we see would have been different. Even so, it might be that almost no one can foresee the effects of their actions with enough clarity to useful act in order to obtain changes they desire.

        Okay, page 1 is all about arguing that attempts to steer the future are futile. We just have to surrender to whatever technology, that autonomous force, is bringing us.

        Page 2 introduces the idea that actually, we can do a reasonable job of foreseeing what technology, like it or not, is bringing us.

        Then we get to the fun part: Ems. They're like Fisher-Price little people, only smarter and less physical. There's going to be this whole economy and society of ems, or uploads in conventional parlance. Why? Because otherwise AI is going to fail, but uploading will succeed.

        The essay, and apparently the book, is full of confident assertions about ems, what they will be like, how they will fit into our world. And I mean confident. There's not the slightest doubt - Ems are going to do all the work, and they're going to have a class hierarchy, money, retirement....

        Call me skeptical. I don't see how you get to whole-brain scanning at the level of detail necessary to make an upload work, even assuming you have the hardware to run it on. If you did have that level of technology, I don't see why you wouldn't have an understanding of the brain's circuits and learning mechanisms and more generally, how to make AI of human and superhuman capability that is tailored to what you want it to do, rather than being inherently egoist and potentially dangerous. Claims of expertise in AI research from more than a decade ago don't impress me that much, I'm afraid. Anyone who says there's not been much progress since then isn't paying attention.

        I may agree that we're heading into the rapids, but that suggests maybe we'd best paddle to the shore, or look for a rock to wash up on or a branch to grab, i.e. STOP. At least until we can figure out what we do and don't want to do. I don't think the smart move would be to start planning for the age of the elves, I mean, ems.

          I don't think I offered any explicit modifiers to connote confidence, and certainly not at the "not the slightest doubt" level. I do in fact have great doubts. Nevertheless it seems worth trying to puzzle out a best estimate, even in the face of great uncertainty.

          Perhaps you aren't aware of the level of detail already provided by current brain scanning tech. The detail is impressive. Of the three techs required to main ems work, scanning seems the tech most likely to be ready first. But our impressive scanning tech does not translate into an understanding of how the brain works; we are still a very long way away on that.

          I, and most AI researchers, disagree that everything has changed in the last decade, and most stuff from before then is irrelevant.

          You are talking about maps of gross pathways or small sections of microscopic detail. If you are going to directly "upload" a fully entrained brain and have it work you need whole-brain data at sub-cellular, perhaps even molecular levels of detail. The whole thing all at once, or it will be damaged, and a large fraction of it at least, or it won't work at all. We're nowhere near that. Also, these maps of axons/dendrites don't reveal the synapse strengths or the cell-level learning mechanisms; we really don't know what is going on below all that spiking activity. If we had that kind of data, we'd probably know enough about how it works that we could design other systems based on the same mechanisms.

          I didn't say everything has changed in the last decade but a lot has in terms of the performance level of systems in use and under development today. This has been driven not only by theoretical advances but to a large extent by hardware and data sets. We're still nowhere near brain levels of complexity but if we had the hardware that could run uploads it could as well run brain-derived and hybrid algorithms that would probably be more useful, safer, and ethical than the idea of creating human-like minds to use as slaves or indentured servants - or worse, to replace humanity wholesale.

          Brains have ceased activity and then restarted fine, so we don't need dynamic state info. So we could freeze a brain and slowly read off the needed info at our leisure. We could destructively scan a brain by slicing a layer off and then scanning the next layer; this is how it is done now. While we are not that far away from being able to do this at sub-cellular levels, many experts doubt that this level of detail is require; most of the info found there may well be redundant with info in the structure and connections of the cells.

          Destructive scanning is almost certainly necessary, which of course means death of the human being. I don't know what experts believe you need only connection data; nearly everyone would agree you need synapse type and strength and it may well turn out that there is important processing going on inside the cell, i.e. the neuron may be a fairly complex cellular automaton. I expect this is essential to learning as well as emotional and likely even cognitive states, i.e. the axonal output is not a strict function of dendritic input but also depends on internal states which respond to chemical modulators and history. Plus of course there are some huge number of different neuron types, different synapse types, and your scan has to identify each one.

          Perhaps if you have only gross connection data that will be enough to pull out something that sort of works, like a person recovering from alcohol coma or some huge pervasive brain trauma, and then your em, while not a faithful upload, can retrain and adapt. But if we know enough to facilitate such self-repair we probably know enough to engineer AI.

          Really, the weakest part of your argument (apart from the elf economics) is the notion that we will be able to make uploading work before we can make AI work. Both are very much dependent on having the hardware and for uploads you also need the scans. The assumption that we can make uploads work without understanding how the brain works in detail seems especially unrealistic. But also, neuromorphic circuits are evidently the way to go for brainlike computing, and they are not going to run scan data, because they will be neuromorphic, not neuron simulations.

          We will continue to study the brain and use what we learn for medicine and to gain insight which will be used to engineer hybrid neuromorphic and algorithmic AI systems, which hopefully we will use as powerful tools rather than something to replace human beings.

          Dear Professor Hanson,

          Let me get this straight. According to some credentialed scientists, it has taken thousands of years of evolution for nature to produce the human brain. There appears to be about 7 billion human brains presently active on earth. Each of these human brains had to commence in a baby that was then born to a woman. You are saying that that was a waste of time for women. Here you are, you can fabricate a brain that is far superior to a natural born human one. The mighty Hanson brain only operates to Hanson's concepts of superior scientific knowledge. Hanson knows better than God what a brain should do.

          Joe Fisher

          No, Joe, you have got it wrong. Most artificial intelligence researchers want to create intelligence artificially, from scratch. Robin Hanson wants to read the soul, God's work, from actual human brains. If he can do it, God's work is not wasted. Indeed some speculate that God is using us to create heaven. The environment inside a computer would look a lot like what heaven is supposed be. The souls living there would live forever, or at least as long as computers could be kept running, perhaps until the heat death of the universe in 101000 years. "When we've been there ten thousand years; Bright shining as the sun; We've no less days to sing God's praise; Than when we'd first begun."

            I really enjoyed your essay, Robin. I agree with your overall point that we have a limited ability to influence the future, so our interventions will have to be well thought out. I also agree that seeing the future--to an extent--is not impossible.

            But as a fellow social scientist I'm skeptical about our ability to see the future in as much detail as you seem to. Your vision of an em future is plausible, but still seems to me to assume too many things that we can't know. I don't see how, for example, we can possibly what ems--who will in a way be quite alien to us--will feel over hundreds of years about members of their clan. It likewise strikes me as extremely unlikely that ems will stay recognizably the same for long while information technology is developing rapidly. And although the rise of ems would be an enormously important development, I think there are other important developments that could radically alter the trajectory of the future.

            You're right that some of the predictions people have made in the past were pretty good. But when enough people make predictions some are bound to be right even if they're just throwing darts. So I don't think we should be too confident in our predictions when we're making plans for the future. We certainly should prepare and plan for something like the scenario you outline. But I also think it is just one of many possible futures we need consider.

            Best,

            Robert de Neufville

              I think we are all over the place about speed because we are making unspoken assumptions about performance over cost. Here are a couple of real life examples. If an em runs on hardware with the same kind of profile as the HD 4870 (first graph), the economically optimal choice will be to run as fast as possible; if it's more like the GeForce GTX 260 (second graph) it will prefer to work at the lower "sweet spot" rate.

              As long as we don't know what the hardware is like, we can imagine all sorts of profiles, including weird, monotonically decreasing ones. How much any of that will matter depends on how steep the graph is, i.e. on marginal cost. Larger slopes will cause more herding at the optimal rate. If I had to bet, I would bet on that kind of tight range, with ems down-clocking only if idle and up-clocking (if at all possible) only to cope with temporary spikes in workload too small to warrant delegation. But, who knows what the real profile will be like.

              I dont't have a strong opinion (yet...) on the feasibility of scanning, but I think we should at least be aware of what the guy says who currently seems closest to actually doing it: "The key insights that allowed us to bring the HBP [Human Brain Project] to fruition were the realisation that it is just not possible to map the brain experimentally [...]". :/

              James,

              You honestly think that 7 billion women have given birth to 7 billion inferior brains? Then why did nature choose to create brains in this fashion? Why did nature create Hanson's brain that allowed Hanson to develop a superior brain to the one he was born with? Why did nature not grant you that same gift? The existence of God has never been proven in any court of law despite the fact that there is a standing reward of $1 million for the first parson that can provide such proof. Why do you need a fabricated intelligence to help you to prove that your God exists?

              Joe Fisher

              Robert, I agree we shouldn't be overconfident, that many past predictions were bad, that my scenario is just one of many to consider, and that other big changes could also make big differences. But even so, it still seems we should try our best to think through each scenario that we consider, using all the best standard results from all relevant fields.

              I'd be happy to share my 94K word book draft with you, and would seriously consider any specific critiques you might have. But there isn't much one can say in response to "we can't possibly know" or "surely things won't stay recognizably the same." Those seem to me to be conversation-enders, and I want to continue the conversation. We can't be trying our best if we just quit merely because we realize that eventually we must reach limits to our abilities to foresee. Instead, we should only quit when the complexity of the task exceeds our patience to work through the many relevant details.