Dear Dr. Rajan,
Your essay is beautifully written and a pleasure to read. You ask all the good questions about quantum mechanics - whether randomness is fundamental, the origin of the Born probability rule, the possibility of there being randomness in time, and the deep foundational issues that plague quantum gravity research.
I wanted to mention here that there has been much progress on these questions, in a manner which moved away from `shut up and calculate'. It started with the Ghirardi-Weber-Rimini-Pearle theory of spontaneous localisation in 1986. They provided a falsifiable and dynamical explanation of the quantum measurement problem, and of the absence of macroscopic superpositions. The theory is currently being tested in a few labs in Europe.
Subsequently, Stephen Adler sought to derive quantum theory from a deeper underlying formalism - his theory of trace dynamics. The deeper theory is a deterministic matrix dynamics from where quantum theory, Born rule, randomness, and spontaneous localisation are emergent phenomena.
Recently, I have shown how to include gravity in Adler's framework, using the mathematics of non-commutative geometry. This has lead to the new theory of Spontaneous Quantum Gravity, where indeed God plays dice with time, but only in an emergent sense. Underlying quantum indeterminism is determinism at the Planck scale.
I discuss these developments in my essay in this contest: The pollen and the electron. Since many of the deep questions you raise are addessed and answered in my work, I hope you will find it interesting.
My best wishes to you in this contest,
Tejinder.