Peter,
Interesting essay and marvelous graphics! (I know others have remarked about this, but it is certainly worthy of one more acknowledgement.)
My view is that simple physical ideas do sometimes lead to horribly complicated mathematics, just because the less you assume, the more you have to explain. On the other hand, if you assume a lot of special structure at the beginning, the mathematics does not necessarily become so bad. Since I think that the physical ideas should be simple and convincing (concepts like cause and effect), I'm willing to accept that the math may have to be hard in order to get the job done.
For example, the usual models of spacetime involve manifolds, which assume a great deal of special structure, but are also mathematically convenient. If you stop taking the manifold for granted, you might be in for some difficult mathematics, but perhaps you can have more confidence that the physical ideas themselves are based on solid ground.
Anyway, I enjoyed reading it! Take care,
Ben Dribus