Hi Philip,
It is interesting that we have partly similar opinions on the links between math and physics, that is, a mathematical Platonism carrying all possibilities, and the idea that the physical universe comes as a particular case of deep mathematical theories. Here are my remarks:
"It has been observed for years that the nature of physical laws appears fine-tuned for the convenience of life (...) Almost every natural occurring element of the periodic table plays some essential role in the making of multicellular life form."
Sorry, while a case for fine-tuning can be made indeed, I don't see it well expressed in this specific way. If I saw well, only few of the physical constants really matter in all crucial processes of the evolution of stars and nucleosynthesis, and they already need to be tuned to fulfill the vital requirements. This does not let many available degrees of freedom to also fine-tune the details of chemical composition, which is actually not so fine-tuned but more guided by its own rigid necessities (mathematically necessary list of nuclear orbitals).
To make water, hydrogen came first and did not need to be produced; oxygen is much needed but its abundance is no mystery, as 8 protons is a magic number for nucleus stability, not much sensitive to physical constants.
After oxygen, the next 3 most abundant elements in the Earth's crust are silicon, aluminum and iron. Silicon is only 2テ--10^-5 of mass in the human body, and "very few organisms have a use for it" (wikipedia). Aluminum "has no known function in biology". Iron is useful but in very small proportion only. Phosphorus is needed as 1% of mass of human body but it is only 0.1% of continental crust and 6テ--10^-8 the mass of sea water (as it better stays in rocks).
"Some physicists have speculated that there is an eternal process of inflation with vacua decaying to different solutions so that our own universe is just one bubble inside a larger arena. "
Hum, if I understand well it requires 2 inflation periods, one before and one after decay ; inflation before decay is natural, since, by definition, the void had higher energy, but then the problem is that for different possible lower levels the inflation needs to still go on for some time and then stop in a synchronized manner (but what would happen otherwise ?). Hard stuff.
"I think it is more parsimonious to accept that all solutions exist in some higher sense, whether inside or outside our universe (...) I take it as self-evident that logical possibilities exist even if only in some metaphorical sense that we don't understand. It is just a way of saying that some things are possible"
I agree that, according to the nature of mathematics, all logical possibilities exist. However I see this as very clearly, formally defined and not mysterious at all, only subject to the well-understood limit of undecidability. As you should know, existence is a mathematical concept, expressed by a specific symbol, that only needs an axiomatic theory to describe the shape of a universe where it is interpreted. The typically suitable framework to state the existence of all mathematical possibilities, is set theory, which admits itself many possible variants to specify the details (due to the undecidability of existence of many kinds of infinite systems we cannot construct).
"What then would happen if we treat the whole of mathematics as a statistical physics system or as a path integral over the moduli space of all possible theories ? Would some universal behavior emerge that could describe the meta-laws of physics?"
A big problem to define an integral over a range of "all possibilities", is that it requires some kind of measure to compare their weights. Such a measure usually requires, at least, a kind of fixed size of a local part of body whose variations are considered at a time. However no such comparison is possible between infinite systems that cannot be precisely described by a common specific theoretical framework.
I think that before flying to such highly speculative, ill-defined generalizations (that I would consider not really mathematical anymore, since I see mathematics as the science of definiteness), you need to consider the obvious first step and particular case of application of such ideas of "admitting the coexistence of all possibilities" that combines the advantages of being very well-known and well-defined, a natural direct consequence of the known laws of physics, with a well-defined measure of the weight of the many possibilities that is very directly and massively verified by observations, and for which, at the same time, this idea of coexistence of all possibilities has the amazing advantage of being still highly controversial. You see what I mean ? Answer below.
You wrote "I was going to write about what might happen if there were only mathematicians and no physicists. How many ideas from physics would they invent without any input from the real world. You can imagine that they even have no direct contact with the physical world. They could just be brains in a vat left to ponder on logical problems. It may even be possible one day to see this happen using artificial intelligence. To be more specific we might program an AI system using Sparse Acataleptic Bayesian Inference algorithms to solve integer diophantine equations.(...) Diophantine equations are very rich in terms of the kind of mathematical tools are required to solve even simple cases."
This is quite interesting as I have a similar project of rebuilding mathematics from scratch, starting with purely logical concepts before reaching physics. I use my own intelligence instead of an AI system, and of course I do know much physics at the start but I care to put things into an optimal logical order so as to make every step appear sufficiently motivated by purely logical concerns without any feeling of arbitrariness nor external (physical) source of motivation or inspiration.
Though the properties, and proofs of properties, of integer diophantine equations may potentially involve lots of mathematics indeed, I'm afraid they would be a quite inefficient way of rebuilding maths from scratch. In fact, I guess the power of development of high mathematics is much less a matter of what problem is supposed to be tackled, than a matter of what kind of intelligence or algorithm is tackling it. Indeed, this AI needs to know from the start what is a proof and what is a definition that can be used to make shorter proofs... finally you need to give it a huge a priori knowledge of mathematics beyond diophantine equations, before it starts searching. But the worst point in my opinion is that, even considering deep mathematics as an intrinsic necessary reality to be discovered, I guess the act of discovering them may require a conscious mind (not AI) to be efficiently done. The concepts of elegance and universality in mathematics may be themselves diversely interpreted and not always in mathematically well-defined manners. Insofar as they would be mathematically definite, I would compare discovering deep maths to a problem of breaking a cryptographic key, for which the solution may indeed be mathematically unique and well-verifiable, but purely mathematical systems could not efficiently discover the solution themselves if it is not revealed from the outside.
Of course I was referring above to the many-worlds interpretation of quantum physics. The most famous difficulty with this interpretation is how to make sense of probabilities. You wrote "The Mathematical Universe Hypotheses tells us that all logical possibilities are equal". Consider the simple example of a polarized photon that is measured in another direction with an arbitrary angle. Both possible observed states are well-defined elementary states, not any variably large numbers of possible states, and have no intrinsic difference of quality having anything to do with the entropy of the measurement apparatus or whatever. Still the theory says one is more likely than the other, with a probability that can take any value depending on the angle of measurement. How to make sense of their difference of likeliness if they are equally real ? See my longer analysis of the paradoxes in this interpretation. Finally I invite you to read my essay where I defend another interpretation.