Dears Anshu and Tejinder,
I fully agree that a connection between physics and mathematics, if to be explained, must be rooted in cognitive science.
Your essay makes very important remarks, not often seen, notably that the mathematics involved in physics are relatively simple (and many current theoretical physics explorations seem to be just picking randomly in the toolbox of established mathematics!).
I would not completely adhere to your claim that ``primordial perceptions such as object, size, shape, pattern and change'' are ``hard-wired'' in humans. I see what you mean, and I agree. But I would not build a whole theory with the present aims, upon these precise primordial terms, as if I could fix them and forget about them. The main reason is that it is extremely problematic to fix a bottom layer once for all, in the faith that it will work universally. We have to live with the tension between the need for fixed basic elements --formal-- to be able to reason with certainty, and the permanently renewed experimental fact that, whenever we dig further into reality, whatever the modality, we find always new structures, without ever finding a bottom layer. Thus different situation require different formalisations. What seems a bottom layer is better viewed as a horizon, an intrinsic limitation of our particular mode of investigation. I am being elusive here, because it would take too long to make the fully the case, so I would warmly recommend Gilles Cohen-Tannoudji's Universal constants in physics (Mcgraw Hill, 1992), for his illuminating interpretations of universal constants as such horizons to physics (to knowledge), and not some absolute, universal constants of nature.
I have approached the case of perception more abstractly. I would have been curious to read your comments on how I have addressed this precise point. I took a starting point very close to your remark that ``physicists ignore or `forget' the brain, treating it as a perfect passive agent''. In addition to having unscientific aspects, this stance --which has been very fruitful, though-- completely neglects that knowledge (included physical) is relative to cognitive subjects. This relativity can be expressed very precisely, in the terms of the cognitive subject being a frame of reference. Since physics has often advanced by discovering new relativities, it should not durably eschew this one. A wider scientific framework must include the cognitive subject as a constitutive part, the key actor of the building of knowledge. And perception should occupy a central position in the framework. Most philosophical traditions have made perception a pivotal phenomenon in the edification of knowledge; by banishing the cognitive subject, physics has forgotten much of the ancient wisdom. Again, this banishing, has had fruitful consequences, but also, inherent limitations. The so-called von Neumann-Wigner interpretation of quantum mechanics is, in my view, an all too clear case of such limitation: when you reach the limits of what your theoretical framework can do, you suddenly call the banished and hold-in-contempt subject to the rescue, to help you collapse the wave function: what you have no way to do from inside the theory. Thus, suddenly, you appeal to perception.
Regards,