Dear Karen,
Your essay is great! You characterized perfectly the criteria that determine when we can stop digging. There is nothing to add. The essay is also excellently written, with clear explanations.
What I can say more about this may be only a matter of personal taste. I want to make some points that even if QFT and GR may not survive as they presently are in the final theory, some parts of both will, and I want to try and identify which.
QFT1. Many problems of QFT are, as you mention, because we don't have a good mathematical formulation. Maybe such formulation exists, without necessarily requiring input from GR. But even in this case, the problem with the UV limit seems to be that the only way we know how to calculate is perturbative. And maybe this can't be made mathematically rigorous even in principle (by mathematician's standard of rigor, which is the correct one). So we will need not only the true, rigorous QFT, but also the way to do the calculations in a rigorous way, which is independent of the fundamentality, it is just a translation for humans of the predictions of the theory.
QFT2. The way to obtain a QFT is tributary to history of physics. We start with a classical theory, and quantize it. This is a cooking recipe, there is no reason why the true QFT wouldn't be completely independent by the classical theories (except for the condition that they have to emerge in the classical limit). I will give an example. Schrödinger's equation is obtained by quantizing a classical theory. But Dirac's equation is not the quantization of a classical theory. Historically, it appeared because Dirac wanted to make the Pauli-Schrödinger equation relativistic, but it is only because of his genius that he arrived at something completely new. Of course, we know that his equation was not able to explain what Schrödinger's already did, so it had to be fixed by putting it in the same Procrustean bed. For example, to give it a Hamiltonian formulation, which is not friendly with Lorentz invariance, but allowed to obtain the Pauli-Schrödinger QM in nonrelativistic limit. And although Lorentz invariance is restored in the path integral formulation, that historical gene is still there and I think it obfuscates the true lesson. Another historical atavism is the so called second quantization, which is just cooking new food by an old recipe. Because this is the best we know.
QFT3. The QM measurement and emergence of classical problems. These two are really weird. You have such a good theory to describe particles, atoms, and their interactions, and they simply destroy this. I think these problems show that there's something essential we don't understand about the quantum. The theory is not complete, but I don't mean in the sense of needing some hidden variables, but we simply don't have an ontology and its dynamics. And I think all these attempts to find it, called "interpretations", are tailor-made to solve the measurement problem, ignoring much of the bigger picture, for example the lessons from GR, which is always seen as the one to be sacrificed. I think this also blocks the development.
GR1. Singularities are usually considered to make the major case against GR. I think the situation is not as bad as it is presented. Here are some possible answers. i) The singularity theorems rely on three conditions. The energy condition may be broken when QFT is taken into account. At least this happens for some approaches to GR. ii) Another way is that in the Einstein equation the Einstein tensor should be replaced with something else, or equivalently, the GR Lagrangian should be changed. There are various modifications of GR like this, including conformal gravity. They give similar predictions in the regimes where GR was tested, and some of them avoid singularities or give a possible answer to dark energy and dark matter. Note that such changes still keep the lessons of GR, like matter being related to spacetime curvature, the diffeomorphism invariance, and the principle of equivalence. iii) It is possible that the GR equations can be replaced by others which give the same geometry outside the singularities, but stay finite at the singularities. Something like a change of variables. This requires extending semi-Riemannian geometry to work for some relevant cases of singular metrics, and such an extension is known, and gives good results for the usual singularities. This is just standard GR, but puts the equations in a form free of infinities at singularities.
GR2. Dark matter. There are results suggesting that this doesn't require changing GR, being due to unknown forms of matter. But there are also solutions that suggest that modified gravity may solve this, see GR1 ii).
GR3. Do we need to quantize spacetime? In fact, this doesn't mean to discretize it, it means that we may need a generalization of Einstein's equation in which the matter side is quantum, leading to some quantum geometry like superpostion of different geometries or even topologies. But if QFT3 is solved in a way which is based on some ontological fields which have well defined stress-energy tensor, spacetime could remain "classical". While the most common opinion is that this is not possible, it may be. I saw a criticism you raised in a comment about this myth of the Plank scale, and I fully agree.
GR4. We need to make gravity into a gauge theory and quantize it like the other gauge fields. This is debatable. In GR, gravity is inertia on curved spacetime. It admits formulations as a gauge theory, and maybe the gauge curvature of the SM forces can be related to the spacetime curvature, but at this time is premature to say that GR has to be made like those quantum gauge theories.
I think that there is still much for us to understand about QFT and GR. But frankly, GR is much more mathematically mature and better understood than QFT, and it is strange that most approaches take QFT for granted and are eager to throw GR away before learning its lessons. I think the entire fundamental physics needs to be redone from scratch, identifying all assumptions, in particular those tied to the history, and with good mathematics.
Thanks again for your excellent essay!
Best wishes,
Cristi Stoica, Indra's net
