Hi Lawrence,

I'd like to make the point that "Information Theory" is NOT about information - its about CODED information. Do you agree?

Computers only deal with code (coded information), and when people communicate via speech or writing they are also using code. Code is meaningless until it is decoded, and to do this you need a code book or you need to understand the code. I contend in my essay that when all the layers of coded information are decoded, what is left is "pure" information i.e.experience.

Lorraine,

You hit the nail on the head:

"The difficulty is in trying to interpret what the models mean. My interpretation is that the nature of reality is such that not all of fundamental reality is amenable to precise mathematical representation"

But the main problem is not determinism versus non-determinism. The problem is, that physicists have misinterpreted the meaning of the mathematics, due to some very specific misconceptions about the nature of information. This was the subject of my own essay. My major point is this: in addition to the equations per se, one needs to known other things, like initial conditions. It is the vast information content of the initial conditions, not the tiny information content of the equations, that determines almost everything about how the physicists themselves behave. By totally ignoring this fact, they have totally misunderstood how physicists interact with their own observations, and consequently, misinterpreted the observations. The math is not the problem. The problem is the slapped-on interpretations.

Rob McEachern

nmann,

Boy, we are all in deep trouble...

"Any information measure that depends so strongly on the prior knowledge of the recipient is too subjective to be of use and should be discarded."

I guess we will all have to discard this forum, our computers, cell phones, smart phones, big-screen TVs, GPS navigators...

And here I thought that all these devices, that depend so strongly on the prior knowledge of the recipient, for almost every aspect of their functioning, were really pretty cool and even, dare I say, useful. I stand corrected.

On a more serious note: If I was sitting on a park bench, staring at a paper with your credit card number written on it, in incomprehensible Chinese, and then an English-speaking native translated the number for me, do you really believe I will not increase my information content, and perhaps even my bank account?

I agree that the paper was not information for me. But the translated speech sure was.

Could you explain what you mean by "if information in any given context is something that can only be diminished..."? The whole point of communication is to increase information, and this is accomplished very successfully in many contexts.

One other thought occurred to me, about a possible point of confusion:

"while the newspaper's content was objectively information"

It was not. It was merely data, not information. The whole concept of information is not about what is in the message, but about how much you are able to recover, error free, from the message. In other words, it is measured at the output, not the input, of the information recovery process.

Rob McEachern

  • [deleted]

  • Post #53

Pure information theory deals with any string of bits, whether coded or not. Coding comes if you impose some Golay or elliptic curve encryption or error correction. If one sends a signal through a noisy channel an encryption system is used to filter the signal at the other end of the channel. A black hole is a sort of quantum bit channel. A major goal is to figure out the encryption algorithm, or equivalently algebra, which permits information to be communicated through a black hole. By this is means if you send a certain amount of information into a black hole then, how is that information or signal ciphered in the Hawking radiation?

Cheers LC

  • [deleted]

  • Post #54

Robert,

Please forgive me for starting a new internal thread here, but I really hate the layout of these topic forums. Re: Kahre ... and you are going to despise this absolutely, I just know it (from Hans C. von Baeyer's blurb or review)(quote):

In The Mathematical Theory of Information Jan Kahre presents a bold new approach to classical information theory. With profound erudition, refreshing iconoclasm, and an appealing sense of humor he leaps over half a century's received wisdom and starts afresh. From common sense observations such as "information is ABOUT something", "transmission tends to degrade information," and "noise can be beneficial", he constructs a rigorous framework that is general enough to encompass a great variety of conventional results in different fields, yet discriminating enough to prune the accumulation of dead wood. His most lasting accomplishment is the demonstration that Claude Shannon's measure of information, which has dominated the subject since 1948, is by no means the only candidate, and not always the most appropriate one. This remarkable book is sure to stretch the horizon and inspire the imagination of every physicist, computer scientist, mathematician, engineer, and philosopher interested in the theory and practice of information science.

(end quote)

In all fairness it's no fruitier than most blurbs. Anyway, here's Kahre himself, getting more or less technical (and I too tend to wince at "This would make the Second Law of Thermodynamics a special case of the Law of Diminishing Information"):

Toward an Axiomatic Information Theory

He says "Towards" but then his native languages and Finnish and Swedish.

    nmann,

    "Z gets all its information about X from Y." Not too many entities get all the knowledge they possess from a single source. Even if there is only a single input message source, the a priori knowledge built into the receiver, and being exploited to recover info from that source, probably did not come from that source.

    "Shannon's famous entropy measure is not a general information measure, but a consequence of the statistical properties of long strings of symbols." This statement represents a profound misunderstanding. As I mentioned in a post under my essay, Shannon's Capacity measure is simply equal to the number of bits required to "digitize" a band-limited signal. His equality statement simply says that the number of recovered bits of information cannot exceed the number of bits required to digitize a band-limited signal, in the first place.

    The entropy measure is probably the best known, but least significant aspect of Shannon's work. A much more significant aspect was the non-constructive proof he gave, which demonstrated that some coding scheme must exist, which would enable messages to be received error free, right up to the maximum limit. The proof gave no clues how to do it, but it inspired two generations of communications engineers to search for the answers. After fifty years of research, and for the first time ever, that research enabled dial-up telephone modems to operate, virtually error free, at rates over 90% of the maximum limit. When I started graduate school, forty years ago, dial-up modems operated at 110 bits per second. It was a BIG deal, a year or so later, when the Physics dept. bought the new, latest and greatest modems, that operated at 300 bits per second. By the beginning of the 21st century, discoveries in coding theory and signal processing algorithms had pushed it up over 30,000 bits bit second, aided by the recent development of integrated circuit technology, that was powerful enough to run the algorithms.

    Rob,

    looking back over the posts, I think I can't agree with what you've said about the nature of information e.g. in your last 3 posts:

    You say "The problem is, there is no such thing as "uncoded information" in experience. Experience is all about coding, such as sensory coding."

    But the physical coded information we use in everyday life (speech, writing, math equations,computer code) always represents something else. I think experience is uncoded information in that it doesn't represent something else.

    You say "I think that the "subjective information derived from information relationships" IS the code."

    But what does this code represent? Does subjective information represent something else? I don't think so. I think subjective information (experience) is the end of the line - it doesn't represent anything else except maybe itself.Are you saying that experience is equivalent to information, and that this information represents itself?

    Re "My major point is this: in addition to the equations per se, one needs to known other things, like initial conditions. It is the vast information content of the initial conditions, not the tiny information content of the equations, that determines almost everything about how the physicists themselves behave.":

    I would think that a lot of fundamental information (including quantity/number) derives from a sort of network of information category relationships that we humans attempt to symbolically represent by law of nature equations. The trouble is that we may never be able to precisely mathematically represent the source of some of this fundamental information - so from our point of view we can't understand it, and perhaps that is why the mathematical equations don't correspond to our normal view of reality. But given that from our point of view, the equations attempt to represent actual information (number, category, relationship) that is occurring at the foundations of reality - I don't know what you mean by saying that the equations have a "tiny information content".

    Lawrence,

    Re: "A black hole is a sort of quantum bit channel. A major goal is to figure out the encryption algorithm, or equivalently algebra, which permits information to be communicated through a black hole. By this is means if you send a certain amount of information into a black hole then, how is that information or signal ciphered in the Hawking radiation?"

    Sorry, but I wouldn't be too sure that what you say has any relationship to reality - it's very speculative stuff!

    I can only repeat that a string of bits is NOT information -it can only represent information. A string of bits has no information value until every layer of code it represents is decoded, at which stage it is "pure" information i.e. experience.

    Lorraine,

    You asked : "But what does this code represent?"

    It represents how to behave towards future, similar experiences. We learn from experience, but there is no point in accumulating learning for its own sake. If you never actually "use" it, then it is a pointless waste of time, from a "survival of the fittest" point-of-view. What makes this so interesting to me, is that the system never has to learn "why" anything is useful. It merely has to treat it as though it is - and then it will be. In this sense, it becomes a self-fulfilling prophecy.

    • [deleted]

    • Post #59

    Robert,

    The fact that a real X generally doesn't get all its information from any single Y is one of those obvious things a critical reader processes automatically. It's akin to the way thermodynamic entropy is sometimes introduced by discussing two hypothetical molecules, X and Y or A and B with different temperatures, and what occurs when you put them in isolated juxtaposition. Writ large you have what used to be called the heat-death of the Universe.

    Here's the first chapter of the magnum opus with considerably more detail, including the original mention of the Chinese newspaper. Kahre stubbornly insists on calling its content Information.

    1. About Information

    Lorraine,

    "I feel that the content of 'pure' information is necessarily categories and relationships, and that new information categories and even numbers can be built from existing categories and relationships. For example mass is a category of information that we humans can represent with a symbol; and our symbols ' - x /' etc. represent the only types of relationships that categories of information can have. Laws of nature are our representation of information category relationships."

    Okay ... well over a decade ago the physicist David Mermin originated what's called the Ithaca Interpretation of Quantum Mechanics, named after Ithaca, NY where Cornell U is located. In a paper from the end of the last century called "What is Quantum Mechanics Trying to Tell Us?" he presents the concept of "correlations without correlata" (which Piet Hut, a FXQi member, popularized in a piece titled "There Are No Things"). I can supply links. "Correlations have physical reality; that which they correlate does not," Mermin says. Anyway, do you see this as at all analogous to "pure information"?

    • [deleted]

    • Post #60

    Got my X and Y juxtaposed.

    If Kahre wishes to allow the unintelligible text on the newspaper (input, rather than output), to be "information" then he must also allow for the fact, that unbeknownst to him, I secretly genetically engineered the newspaper fibers. To decode my message, he must recover every DNA molecule, from every cell, from every fiber, string them all end-to-end and read my preamble, which then informs him he must acquire every newspaper on earth and do the same in order to read my complete message. Preventing such absurdities is why information is defined on the output of the recovery process, not the input.

    Rob,

    I have read your essay, but I think that at the foundations of reality there can only be fairly simple things like information relationships. You seem to be saying that, effectively, there is a block of code at the foundations of reality that consists of a whole set of logical rules. I can't see that at the level of fundamental particles there has been any time for a "survival of the fittest" set of rules to evolve. Also, where are these rules, in what form do they exist (Platonic realm?), how are these rules enforced, and in what way are they different to our attempted representations of "laws of nature"?

    Of course things have changed by the time you get to complex molecules. I surmise that by the time you get to DNA molecules, a simple information experience has given way to a more complex information encounter in which the physical RNA represents information to itself, and that in relationship to the RNA the physical DNA and its physical environment represents further information. We can describe and represent what is happening from our point of view, but this is not the same as what is happening information-wise from the RNA molecule's point of view.

    • [deleted]

    • Post #63

    Hi Lorraine,

    I hope that I'm making some progress in understanding this problem that you pose.

    I ran into some theorems by Tarski and Godel about the limitations of self-referential languages. They are interesting, but I'm not sure how these may or may not apply to what you are trying here. In any case, these theorems indirectly lead me to the thought that the only representation of the Universe that is not self-referential is the Universe itself. The problem I am seeing is that in order to have such a representation would be for us to create a Universe. This is pretty much where my thought process implodes and my ears begin to emit a little smoke. Have your thoughts about this essay ever traveled down this path? If so, did you make it further than I did?

    - Shawn

      • [deleted]

      • Post #64

      Sorry,

      "The problem I am seeing is that in order to have such a representation would be for us to create a Universe."

      should have read

      "The problem I am seeing is that in order to have such a representation, we would have to create a Universe."

      Lorraine,

      I never said anything at all about "the foundations of reality." It is you, not I, that speaks of such things. I merely spoke about an observer's perception of that reality. And like it or not, all of our perceptions of reality are indeed being filtered through, as you put it, "a block of code", residing in our brains. And, like it or not, that is our reality.

      nmann, re the online chapter 1 of The Mathematical Theory of Information by Jan Kahre:

      I have read it, and as expected, it's about coded information not information. Noticeable is the way it mixes up information and coded information as though there were no essential difference. Look at these representations:

      (1.5.1) "rel(B@A)" Here A represents a coded message sent and B represents a coded message received.

      (1.2.1) "inf(B@A)" Here B represents a (coded) newspaper article about an event, and A represents the event. (The newspaper article is coded because words are a code)

      But actually A does not represent an objective event. So-called event A is actually the reporter's subjective experience of event A, and B is the coded representation of that subjective experience.

      "Information" theories fail to honestly identify what is going on - they fail to properly identify and highlight the reality of subjective experience. Subjective experience is actually part of these theories, but instead of saying that A represents the reporter's subjective experience they claim that A represents an event.

      As part of my essay, I tried to point out the connection with subjective experience.

      • [deleted]

      • Post #67

      I am basing my definition of information on the Khinchin-Shannon measure

      S = -k sum_n P(n) log(P(n))

      For P(n) the probability of the occurrence of the n-th bit. Entropy is the basic measure of information. If there is a process which changes the distribution of probabilities and thus this entropy measure, then information has been lost or erased. If you have a bit stream that enters a channel and at the output side the entropy measure is the same you conclude your information has been conserved. The largest probability occurs when you have the equipartition of probabilities so P(n) = P for all n. and the minimal entropy for P(m) = 1 for a unique m and P(n) = 0 for all n =! M.

      It might be of course you don't understand how it is conserved. Suppose you have bit stream, or a quantum bit stream, with probabilities P(n). You have a code for this bit stream; for example the bit steam is an image and the code is a gif or jpeg algorithm. You then run this into a channel to communicate it to the output. The output is corrupted in some way, the image algorithm gives no picture or only part of it. This means there has been a scrambling of your information with other information. If all that other stuff is given by some density of probabilities ρ for random bits, the total entropy is

      S' = -k sum_n P(n) log(P(n)) - k sum_aρ_a log(ρ_a)

      = S S(channel),

      where you have a problem with mixing your signal with the channel noise. What you want is some error correction system which unscrambles this so you can trace out the channel noise. This can be accomplished by various means of computing Hamming distances or the application of a Steiner system.

      So this I think connects to the different definitions of information used here. The distinction is between information with bit probabilities on a channel

      The entropy of a black hole is given by the Bekenstein formula S = kA/4L_p^2. Here L_p = sqrt{Għ/c^3} is the Planck length, and L_p^2 is a unit of Planck area. A is the area of a black hole event horizon. The entropy is equal to the number of Planck units of area that comprises the event horizon S = Nk/4. This is given by the total density matrix of the black hole, where ρ is the quantum density matrix ρ_{op} = sum_n|ψ_n)(ψ_n|. A trace over the density matrix in the Khinchin-Shannon formula determines entropy. If you threw a bunch of quantum states into a black hole entropy of the black hole is

      S = k sum_nρ_a log(ρ_n) S_{BH}

      The black hole entropy increases. You are not able to disentangle entropy of your signal from the black hole by performing the proper partial trace necessary. However, if you kept an accounting of all states in the black hole and the joint entropy of the states you put into the black hole, which are negative, then you could in principle extract the information you put into the black hole. How joint entropy can be negative is a consequence of quantum entanglement, and by putting a quantum bit stream into a black hole is to entangle it with the black hole.

      A detailed understanding of this requires the use of error correction codes. The most general one is the Leech lattice Λ_{24}, which is constructed from a triplet of E_8 heterotic groups. string theory has a heterotic sector with E_8 ~ SO(32). The so called sporadic groups, such as the Leech lattice, are a system of automorphisms and normalizers which define something called the Fischer Griess (or monster) group.

      At any rate the idea of the universe as a computer is really just a mathematical gadget. It is not as if the universe is identical to the Matrix computer in the movies of that title.

      Cheers LC

        Hi Shawn,

        Self-referential languages, while an interesting topic, don't really apply to what I'm trying to say here. I have just posted to nmann on the subject of The Mathematical Theory of Information by Jan Kahre, and that post might be relevant to what you are trying to follow up.