I agree that there would be huge gains from finding ways for us to better aggregate the info we all have and are capable of collecting. Before there was a world wide web, I worked with the Xanadu project because I found hope in their vision that back-links would cure bad debate by making it easy to find good rebuttals of bad arguments. When I came to doubt that vision, I pursued prediction markets as a more reliable mechanism by which people of good will could come together to form reliable consensus that non-experts could trust. I've been pursuing that for twenty five years.

But I've come to realize that there is actually relatively little demand for institutions or mechanisms that can cheaply produce accurate estimates. Beliefs often serve many other functions for most people that conflict with accuracy. So they aren't very interested in following back-links to good rebuttals of bad arguments, and they aren't very interested in supporting prediction markets. They also would not be very interested in buying or listening to AIs that provide accurate beliefs. The key problem here is the demand, not the supply. We have babel because that's mostly what people want.

I have hopes that prediction markets could someday become like accounting. In a world where no firm does accounting, it would be problematic to propose it, as you'd be accusing someone of stealing. But in a world where all firms do accounting, it would be problematic to propose not doing it, as you seem to be asking for permission to steal. Similarly, if we can ever make prediction markets the norm, it would be embarrassing to not have a prediction market on a topic, as that would suggest you don't really want to know the truth. Of course people don't usually want the truth, but they also don't like to admit that publicly.

    The problem with debate via hypertext, crit, etc. is that it's clumsy, clunky and divergent. I'm not sure about prediction markets, but I think a lot of people are not convinced, in a society with so great maldistribution of money where so many do not have disposable income to place bets, that a betting pool is the best way to divine the future.

    You are right though that people are likely to balk at having AIs tell them what they don't want to hear, which is one reason I said it will be important for the technology be to controlled by its users. Some people, at least, and some institutions do (at least think they) want to know The Truth, whatever that might turn out to be. But also, I am convinced that AIs will be powerful agents of persuasion. They will probably be most effective when used through networks of existing connections rather than trying to persuade directly across confrontations. But it will turn out (social big data) that networks exist through which you can ultimately reach everyone, or big fractions.

    I expect that as advanced AI systems are deployed as agents of persuasion, they will look for ways to make their messages deliver pleasure response or whatever makes them effective - and I note that some people are hooked on fear, hate, anger, etc.

    Hi Mark,

    my 2012 FQXi essay contest entry "The Universe is not like a Computer" and also my 2013 essay contest entry "Information, Numbers, Time, Life, Ethics" explain why computers represent information ONLY from the point of view of human beings:computers/robots can NEVER understand or experience the information that they represent. Worrying that computers/robots understand what is going on, or that they are going to take on a life of their own, is a terrible waste of time and energy.

    However, we cannot deny that there are current and potential future problems (and benefits) from governments and corporations accessing and manipulating data sets of information about people, and problems (and benefits) from drones and other unmanned programmed vehicles/machines/robots.

    But its HUMAN intelligence we are talking about here: there is no machine intelligence. The truth of the matter is that its the humans and corporations behind the machine software (i.e. "artificial intelligence") that we should be worrying about: its not really the machines/robots per se that we should worry about. The robots are just "doing what they are told" so to speak.

    Also there seems to be a real problem of humans interacting constantly with the dead (i.e. machines) instead of with living reality. I believe that people like Kurzweil, Tallinn, and even scientists Hawking, Russell, Tegmark and Wilczek (see their Huffington Post article "Transcending Complacency on Superintelligent Machines"), have become unhinged from the physical reality that we actually live in!!

    Cheers,

    Lorraine

    Lorraine, I don't share your conviction that machine intelligence is impossible, and I'll tell you why if I may. I approach this from two directions.

    First, it seems to me that machines already are doing things that I would call intelligent, and I think you should too. When a machine is able to receive a complex signal, classify it, determine an appropriate response in the context of its intended purpose, and respond, that to me is the paradigm of intelligence. I don't see any limit to the complexity of such behavior by machines. Consider animal intelligence, in general. An animal seeks food, avoids predators, seeks a mate, and sometimes protects and raises its young. Although we don't know how to make self-replicating machines (yet), it seems clear to me that computer systems could effect all these kinds of behaviors. If you think not, I wonder which of these behaviors you feel would be impossible, and why the effort would fail.

    Second, with regard to human intelligence and consciousness, we know that the input and output pathways to the brain are through the propagation of action potentials in axons, and in between there is some mixture of electrical and chemical signaling between neurons. Each neuron must be implementing some kind of automaton so that its outputs are a function of its inputs over time. I see no reason why artificial systems can't implement functions which are isomorphic to those of neurons. These functions might not be as simple as some people (and their models) assume, but they must be some functions which we could describe and make something that behaves isomorphically. If we did, we would necessarily have a machine that behaves the same as a person. You could have a conversation with it about its consciousness, what it is like to be it, whether there is a God, etc. It seems to me that this must mean that it would be actually conscious, if we actually are. I don't think the idea of a zombie makes sense.

    So, I am fairly convinced that a machine can be intelligent, and can be conscious. However, if it is not human, then its consciousness is not human consciousness, and my preference would be that we never make such things. But I think we can make very intelligent systems which would be very useful to us as tools, without having to make systems that would have legitimate claims to personhood nor any inclination to assert such claims.

    Hello Mark,

    I read your nicely composed essay with a great pleasure. It is lovely to meet somebody with a broad erudition and a writing style. I like you telling that "people do not want the truth" (presumable, there is such a thing).

    Sorry, I did not give a high mark. Your essay is really good, but I am looking for something ingenious.

    You may look at my entry about imagining the future. I hope my essay will encourage you to learn more about quorum sensing as a means of knowing and to apply analogous imagining in your field of interests.

    Please disregard any typo mistakes you may encounter.

    Cheers,

    Margarita Iudin

      Prediction markets have consistently shown their accuracy in head to head contests with other mechanisms. Given this track record, I don't see how your opinion that money is maldistributed is relevant. All these empirical tests were done in this same society where you think money is maldistributed.

      Dear Mark,

      That was an interesting essay -- but as Robin Hanson pointed out (and you acknowledged), people don't want to hear the truth. Like you, I hold much hope for structured debate forums, especially if the machine could ground each node using RDF/OWL (that is one of my proposals in my essay Three Crucial Technologies .

      However, don't forget that we must worry about more than just intelligent machines. With the advent of the Internet of Things, these machines will not only control your thermostat (as smart meters do now), but your credit card, and probably even your pacemaker. These will be powerful constructs.

      So I worry. The biggest problem, which unfortunately didn't quite make it in my essay as well I as I intended, is that I don't think that the problem is one of communication, or of knowledge, or even wisdom--but one of will (though with it's moral aspect, maybe wisdom *is* involved). We can't force AGIs to love us, and we do bad job of loving each other, so how could we teach them anyway?

      To consider the depth of the problem, consider another Biblical story; God created the universe, and everything was chaos. Ten seconds later, He said, "Let there be light!" Who do you think brought the light? Obviously, the strongest, best, most beautiful, and most intelligent angel, the Light Bringer. For those who don't know the rest of the story, when the Light Bringer (Lucifer) found out about God's plans to create a race of self-replicating bags of shit--to whom he would have to kneel and obey--he got majorly ticked off. Understandably, I suppose, but the problem was not one of intelligence, because Lucifer was super-smart. Deep down, I think he understood how amazing it would be for mere matter to do the things to the universe that he had done (it would even better than a 1-year-old winning the Olympic marathon), but his pride just wouldn't let him appreciate it. So he became determined to destroy humanity.

      How do we encourage AGIs to seek to be virtuous when we can't even agree among ourselves on what that is, exactly? And we can't just encourage them-we must force them, because if they aren't friendly, then we're screwed. Extinct, actually. I suppose Mass Effect's Synthesis ending might be better, but then we're stuck with the same flaws we have now. And similar results--war, injustice, and ignorance, only on a much greater scale.

        Well, if you don't think money is maldistributed, that's your opinion, but it is widely disagreed with. Which was my point; I was just suggesting that could be one reason prediction markets haven't been so popular, despite their not-bad track record of accuracy. People just don't think one-disposable-dollar-one-vote is a good formula. Especially given how easy it would be for an interested and well-heeled party to tip the scale.

        Hi Ti,

        The Lucifer story is profound, but it's about human minds. I know Omohundro and others have argued that any sufficiently advanced intelligence will share certain "basic drives" or characteristics that we might summarize as egoism. I'm not sure that's true. I think we might be able to build very powerful AI tools that aren't modeled after humans (or any animals), don't think or experience as we do, and don't have any tendencies to want to take over. I think the creation of eogistic or human-like AI should be absolutely forbidden, precisely because we would never be able to predict or control what it would do. So it isn't a matter of cajoling AGIs to be virtuous, it's a matter of maintaining human control and using AI as a tool only. If that makes sense.

        best reasonable wishes,

        Mark

        Hello Michael,

        Yes, you may offer, and I'll reciprocate, but I've learned the hard way that people really do not want sincere criticism, even if they think they do. Perhaps you are an exception. As for myself, I find it hard to imagine that anyone would be more critical of my own essay than I am. They might be more dismissive, but that is not the same thing.

        best reasonable wishes,

        Mark

        Mark,

        In response to your first point, I can't understand why you are so bewitched by mere superficial behaviours and external appearances of machines: you are talking about mere simulacrums of intelligence and consciousness. Despite any seeming complexity, a computer/robot is a deterministic mechanism that does what its told (so to speak). But a micro-organism, an animal or other living thing is self-directing, non-deterministic, and creates unpredictable, but non-random, outcomes for itself in response to circumstances.

        In response to your second point, I think you should look more closely at the nature of information. I'm using the word "information" in the sense of "knowledge communicated" or "knowledge gained" (i.e. perception/awareness/experience/consciousness). Something that was once a in person's conscious experience (e.g. some words in the French language) might, after typing at a keyboard, end up represented as a string of zeroes and ones in a computer. But the particular string depends on the character encoding system used. So this string of zeroes and ones is NOT information to a computer/robot UNLESS it happens to have a code book handy. But actually there are no strings of zeroes and ones in a computer - there is only physical hardware with appropriate properties that can REPRESENT zeroes and ones. So for the physical hardware of a robot/computer to correspond to information (consciousness), the robot/computer would need not only a code book, it would need to know that this physical hardware state represents "0" and that physical hardware state represents "1", and it would need to speak the French language!

        Computer circuits have voltage levels, neurons have spiking action potentials. Either can be considered an encoding of information, which you may take in the engineering or physics rather than philosophy sense.

        It is not true that computers are necessarily deterministic; one can easily introduce randomness from environmental noise or quantum sources.

        You did not engage my point that if we could (and I do not see the reason why in principle we could not) create a machine composed of units which have input-output functions isomorphic to those of neurons in a human brain, it would behave like a brain, and we would be able to have philosophical conversations with it about the nature of consciousness. Do you think that such a machine would be a zombie, that says it is conscious but actually is not? Or do you think such a machine is impossible? If the latter, what hitherto unknown principle of physics would intervene to prevent it from being realized?

        Mark,

        You make a rather large assumption that we "could ...create a machine composed of units which have input-output functions isomorphic to those of neurons in a human brain". Though humans are extremely capable and clever and knowledgeable etc etc, we cannot make such a machine because of the nature of reality.

        Clearly, I have a totally different view of the fundamental nature of reality to you (described in my essay in which I describe 3 invalid assumptions of physics).

        Cheers,

        Lorraine

        Lorraine, you may wish to leave the conversation at that, but if so, I will just point out that you have not explained what aspect of the nature of reality would prevent the realization of a a machine composed of units which have input-output functions isomorphic to those of neurons in a human brain. It seems to me that, whatever that aspect of reality might be, it would not be consistent with known physics and would therefore entail a modification of physics. While that is logically possible, I do not expect that it will turn out to be true. best, Mark

        translate.google.com attempts to consolidate language. But two different people speaking the same language understand things differently, even though they are reading identical written works.

        If everyone shares the same perspectives, then there are very much limited opportunities to discover new relationships.

        Diverse interactions spawn greater numbers of Moments of Inspiration.

        Ideally, we would each speak a different language and our brains would be able to correlate the vast systems of relationships of all languages and related written works; correction, all works. Living from moment to moment immersed in moments of inspirations.

        Perhaps parallel processing of quantum computers will make this possible.

          Thanks for your comments, James. As a practical matter, in order to steer the future at all, let alone intelligently, we need to be able to understand one another and overcome differences. No doubt there are many ways to approach this problem in addition to the ones discussed in my essay.

          Mark, a most excellent essay. You have a facility for penetrating to the rotting roots of our human condition; there are few essays that engage me from beginning to end like yours. Kudos.

          We have a very similar worldview, though my own contribution focuses pretty narrowly on the content of your endnote (10).

          I look forward to more dialogue.

          Best,

          Tom