Very interesting essay! I agree that it's important to interrogate the nature of certainty in mathematics, given that many of us feel that it is somehow the last bastion of 'absolute' certainty. The idea that math (or perhaps a branch of math) should embrace uncertainty reminds me of Horgan's famous "Death of Proof" article in Scientific American, wherein he discusses alternative standards of proof or methods of proof. There are computer-generated proofs, probabilistic proofs, zero-knowledge proofs, and so on. Then, of course, there is the fact that a lot of published mathematics doesn't 100% spell out the details of a proof, e.g. things that are in principle 'easily checked' are sketched or glossed over. Finally, there are lots of statements mathematicians believe are true, and even use to prove other results (e.g. the generalized Riemann hypothesis is often used in number theory), but that have not been shown rigorously to be true.
The (possibly apocryphal) story about Hippasus I may not have heard before. Very funny.
You're not alone in wondering how it is that abstract ideas like numbers and addition map so perfectly onto the real world. I've wondered about it for a long time. Did it have to be this way? Because of the 'meta' nature of the question, it's very hard to even begin to come up with a sensible answer. It just seems to be a brute fact about the world.
I did not totally follow the section 'deriving' probability theory and quantum amplitude manipulation rules, but maybe I just need to go back over it or check the relevant papers.
I was pleased to learn about Tarski's theorem from your essay, a striking result in mathematical logic that seems to have been largely overshadowed by Godel's incompleteness theorems. Very interesting point about relationship between Tarki's theorem and probability theory (probability theory cannot specify what's true, so truth must come from some other source).
In closing, I'll say that mathematicians seem to be more comfortable these days with using heuristics and alternative means (e.g. numerical work) to help judge whether or not some claim is true. So to some extent, this kind of approach seems accepted. But it does not seem to me that these standards of proof will ever fully replace a 'rigorous' proof, whatever the issues may be with foundations and defining what 'rigorous' means.