Georgiana,
Your stated:
"The bit of knowledge is what the observer selects to know, not what the object is."
Not quite. The central issue is not concerned with what the observer wants to know. It is concerned with what the observer is able to know. You might want to know how to decode a secret message. But you might not be able to do it. You might want to "look inside" some physical object, like an atom, but you might not be able to do it.
In Information Theory, a "bit of information" means something very specific, and something rather different than what it means in common parlance. And it has some very, very, very counterintuitive consequences. It is concerned with the information content of "messages".
Think about this message, that I am trying to communicate to you. Think about it as if it arrived in the mail, written on a piece of paper. The attributes of the paper are irrelevant to the attributes of the message. The alphabetic symbols I am using in an attempt to communicate my message are quite independent of the paper's location and momentum etc. The latter are irrelevant to the message content.
The basic question that Information Theory seeks to answer is:
How can the sender of a message encode the message, employing the least number of bits, and still ensure that the message can be received, by the intended recipient, free from all errors?
This may seem to be only remotely related to making observations of naturally occurring objects. But in fact, the exact opposite has been discovered to be true. Why?
Because it can be shown that one's ability to extract information from observations is highly dependent upon what the two entities (sender and receiver) know about each other. This is why, in my essay, I kept stressing the importance of the information content of the "initial conditions" over the information content of the equations. It is not just the case that the former's content is often much larger than the latter's, but that it also play's a much greater role in ones's ability to recover any information from the observation at all. Knowing Maxwell's Equations is not going to get you very far when it comes to trying to "observe" messages being received by a modern smartphone, or from the DNA in your genes.