sorry for this long reply. But I was really intrigued by your essay, which inspired me to a lot of thoughts.
In physics I used to be a bit of a Platonist interested only in theoretical physics, which is reflecting the true forms and not much interested in experimental physics that are concerned only with the shadows of ideal forms. In your essay you made the theory of verification - which I compare with experimental physics - really interesting! Your essay is nice to read and interesting from the beginning to the end.
A third of all mathematical publication contains errors? That is worse than medical publications!
In physics realism that states the truth of ontological propositions should be independent of the means of its verifications (measurements). This of course is questioned by quantum mechanics. And the search for a type of realistic interpretation of physics, which does not relay to the human observer is still going on. In mathematics even more than in physics, we use to have a realistic attitude. Do you defend in your essay an non-realist position, where the truth of mathematical proposition depends on the ability of verification of the verifier?
I am not up to date in AI research and I found your exposition very interesting. Were you able to give to 'comprehensibility' a precise mathematical meaning?
When I was reading your essay my son asked me what I was reading. I sad it is about whether a computer program could be able to check if other programs and himself is working correctly. I instantaneously asked: why don't you write a second program. the two programs could check themselves. So if there were another barber ... In your essay you say that there are minds that can never be understood by a human mind as it has the greater number of states. But could two minds that have the same amount of states comprehend each other?
In the context of physics I always wondered, whether a measurement instrument must have greater complexity than the objects that is measured. For instance for measuring a square one needs at least a field over 4 points in order the be able to distinguish, if a square has been rotated. Also this would lead to an infinite regress.
In your physics section you seem to imply that the probabilistic nature of mathematical verification implies the probabilistic nature of mathematics and
hence the probabilistic nature of physics (=QM) in the MU. Is that so?
I always wondered, if the fact that a system cannot completely know itself, and an external measurer is needed to completely specify the system , could be the cause of the probabilistic nature of QM. If an object has n degrees of freedom, its measurer must have at least n degrees of freedom. Let's say m. So the total system must have n*m degrees of freedom, which is greater than m. Hence there are undetermined elements within the system. Hence probability.
Well in relativistic classical physics only relative distances and velocities are measurable within the system. While the absolute values are measurable only from outside - relative to the location of that system. There is also an infinite regress here, but I think this is completely consistent and classical and no 'problem' arises with that kind of infinite regress.
Last but not least I want to advertise my essay, that I still need to write and that will have a title like: "Changing axioms - the unpredictable evolution of physical laws". There I propose that the definability of basic concepts (quantities) that make up the physical theory (laws) depends on the possibility to separate the system from its environment. For instance relative distances are only defined as long as the system is separable (translation invariant) from its environment. This in my opinion not only solves the objectivity problem raised by Wigner friend type of experiments, since it protects the system from outside interventions and its symmetry and unitarity as condition to have well defined concepts within the system. But the conditioning of the definability of concepts by its environment (which is contingent) means that the basic concepts that make up the theory may change by changing environment and so does the theory itself. I think that gives an interesting view also on AI which is able to adapt itself and change its program according to the environment.