You raise an interesting point about the finitude of information. In an idealist philosophy my reality is determined only by the finite amount of information in my mind. The external world is an ensemble of possible worlds consistent with that information. I dont allow that there is one external physical reality for the state of the whole universe around me because that would require an infinite amount of information, but each possible state of the universe would require a large or infinite amount of information to describe it. What this provides is an illusion of an external reality described by more information than I really possess in my mind.
I think that symmetry is important to how this works. Suppose that all objects in the universe were distinguishable so they can be labelled in some way. If the universe is infinite and I know that the object with label X is in the room with me then I have an infinite amount of information about it. This can't be right. The paradox is resolved because elementary particles are indistinguishable. All I can really know is that there is an object in the room made up of elementary particles in some configuration. This requires only a finite amount of information. The indistinguishability of particles is a permutation symmetry.
You are right that I have identified information entropy with action, even if I did not say it explicitly. It is an interesting conclusion, and not one that I would have expected.