Dear Lutz,
Thank you for your encouraging words in our address! Your suggestion to "add some examples where math has led physics astray" could give a new extremely interesting essay. At the moment, I'd like to mention just one important case in this respect. Perhaps, you know this story, but still I wish to mention it here for the sake of your wonderful question.
Copernicus was convinced that the planet orbits must be nothing but circles, as the most perfect, most symmetric among figures, corresponding to the symmetry of the Sun's attraction. The idea was beautiful, reasonable,--and still wrong. As a result, Copernicus was forced to introduce his own epicycles, and his heliocentric system was not as beautiful as he expected. Most likely that was why he held over with the publication. It required a genius of Kepler to solve this problem and to prove the heliocentric idea is correct. This was one of the most dramatic moments in the history of science, I am sure. Seeing the failure of Copernicus, Kepler still believed in the beautiful mathematics underlying the world. As well as Copernicus, he was Pythagorean/Platonic, but his field of search of the mathematical beauty in the sky was wider, and he was heavenly rewarded!
Many thanks and all the best,
Alexey Burov.