Jim,
One way to think of physicalism is a dichotomy of energy and information. Energy manifests information, or it would collapse into a black hole, while information defines energy. How would a wave, for instance, be understood, other than as frequency and amplitude? So particles/quanta are energy with various parameters of information.
Consider that as biological organisms, we evolved the central nervous system to process information and the digestive, respiratory and circulation systems to process energy, to propel that information gathering.
Then consider that as energy is "conserved," it is always and only present. So the natural change of this energy means the information is constantly changing. This creates time. As energy goes from one configuration to the next, it goes past to future, while these forms, the information, goes future to past.
Consider, as an example, that conscious goes past to future, as particular thoughts go future to past.