Continue on with this subject..
Violation of Bell's Inequalities appear to have put us in the position of either rejecting the idea of hidden variables or accepting the notion of hidden variables as they apply to non-locality.
Some theorists who do not like the experimental results will say, "I will concede the existence of non-locality as long as I get to retain the ability to rely on hidden variables." Others will simply reject the experimental results entirely and argue the fine points of the experimental setups. They will concentrate on finding ways to show how the results are invalid. Again, going back to my points earlier, this sounds more like an attempt to hold onto a particular worldview rather than taking an objective look at the evidence and trying to reach the most logical conclusion that represents the most plausible interpretation. If the inequalities had not been violated and the results of experiment had told them what they wanted to hear, would they be pressing the issue of validity? I would tend to believe not.
The experimental results have been replicated many times. If someone were to demonstrate and empirically prove that hidden variables indeed exist, I would have no choice but to accept their existence. That evidence, however, does not exist and nobody has put forth any consistent theory that would account for or explain such hidden variables. This does not mean I am right. It simply means that I have formed my opinion in as objective a way possible, irrespective of my own prejudices and metaphysical bias. It doesn't really matter to me if it turns out that hidden variables do indeed exist anymore than it would matter to me if it turned out that we discovered that the Universe really was the back of a Tortoise floating in a gigantic pool of water. I simply see no reason to currently accept either proposition. They appear to be conjectures based on criteria that appeals to an individuals metaphysical desires and inclinations about how the world should operate.
In the absence of evidence for hidden variables, I can only objectively infer that classical reductionism likely died a very slow death as we discovered more and more about the workings of nature on scales that are far removed from our senses. Many have refused to attend the funeral and hold out hope that it simply went AWOL and will one day reappear or manage to resurrect itself. The theoretical community appears to be going through the progressive states of Kubler-Ross -- Denial, Anger, Bargaining, Depression, and Acceptance. Currently, many seem to be somewhere between the Bargaining-Depression state.
I believe that any resolution of the conceptual and ontological issues we face are likely going to result not from a reformulation of Quantum Mechanics or a discovery of hidden variables, but will involve a major paradigm shift that reorganizes our intuitive and causal notions of how nature is structured on a phenomenological level, not only microscopically but macroscopically as well.
To give you an idea of what I mean, we should consider some examples from History. In the history of our quest to understand the structure and properties of the phenomenon that exist in the natural world, there have been two fundamental reasons for shifts in thought that lead us to reformulate current principles. Such shifts either result from observations that force this change upon, as is the case with QM, or they result from a radical ideological shift in perspective. Both will lead to a reformulation of fundamental notions or concepts about the world but the latter proves more powerful as it forces one to reformulate ideas on a global scale. It forces a new approach to an old problem using entirely novel ontological conceptions about the nature of the Universe itself, not just a specific phenomenon associated with an arbitrary scale of influence.
I would present two examples here that explain each type of methodological shifts .
Contrary to popular opinion and many textbooks, Copernicus did not discover that the Earth orbited the Sun. Copernicus presented a Hypothesis--one that was neither logically or empirically necessary, given the information available at the time. The Ptolemaic system could account for the observations of the behavior of heavenly bodies with equal precision. Unlike our venture into the Quantum world, the Heliocentric hypothesis was not one which was formed as a result of some new observation that could not be accounted for within the confines of the accepted theory of the time. Copernicus did not come out and say to the world, "Hey guys, we've got this all wrong. I was just observing the sky last night and discovered that the Earth is actually circling the Sun. We've got this whole thing wrong !"
The Heliocentric hypothesis was one that had fermented in the cellar for centuries. You can find the hypothesis present in some schools of thought all the way back to the Greeks. Copernicus framed the heliocentric hypothesis on a much more refined and accurate scale, using the precise data available at the time. He was able to do something the Greeks could not--he could match empirical data to a hypothesis and give it some life. Still, I must emphasize that the Heliocentric hypothesis was not necessary to be true in order to account for the data -- the Ptolemaic model did so with equal precision. Since displacing the Earth from it's primal and privileged position was abhorrent to the common sensibilities of the time, why, then, did it gradually gain acceptance? It offered simplicity and elegance that the Ptolemaic system could not. It accounted for the astronomical observations in a much more straightforward and rational way. In short, it made more intuitive sense; whereas the Ptolemaic system, with its bizarre system of epicycles and deferent, did not. One could more easily explain and account for observed phenomenon.
The period that followed represents a time when the ideology of the world was turned totally upside down as a result of the inferences we derived from the theory. It marks the beginning of the scientific revolution and allowed us to proceed in ways previously undreamed of. We have never looked back since. Obviously, the idea was slow to be accepted. As with the Theory of Evolution today, it infers things about our existence that many people may not want to hear. If our position in the Universe is not special then perhaps we are not as special as we thought. In the case of Copernicus, the idea started to take shape that perhaps the world does not revolve around us, either literally or figuratively. IMHO, the Copernican Revolution represents the most radical shift in naturalistic and rational thought in the History of the Western world.
Compare this Paradigm shift to the changes in reasoning that occur when nature forces upon us, by means of direct observation, certain tenets or principles which cannot be avoided but do conflict with the current paradigm.
A misconception in the history of physics as it is portrayed in most texts is that Planck was a willing accomplice in the development of Quantum Theory. Planck's motivation for introducing the notion of quantization was entirely methodological and not thematic. He was simply using quantization as a neat mathematical trick. His goal was to create a mathematical model which could be used to reproduce the spectral observations gleaned from the study of blackbody radiation.
Another misconception is that he was motivated to do so due to the Ultraviolet Catastrophe. This is not the case -- in any way, shape, or form. Planck held great reservations surrounding the methodological application of Boltzmann's statistic to observed phenomenon. He never trusted such an approach as a internally consistent method of relating thermodynamic properties of macroscopic systems to micro-phenomenon. He saw the application of M-B statistics as rather arbitrary and contrived. He set out to create his own methodology that would not rely on assumptions about the mathematical forms of the distributions themselves. In other words, he was asking, 'What mathematical form of the distribution equations would allow one to successful model the spectral phenomenon?' His goal was to model the behavior of the system, not start with assumptions about the statistical behavior of an ideal ensemble.
Using Classic Electrodynamics, Planck tackled the issue by imagining that all of the component structures contained in a body that gave rise to the EM waves could be imagined as charges on the end of springs, vibrating back and forth like a simple harmonic oscillator. He worked out an equation to model the energy distribution among the components that would give the exact results measured in experiment. He gradually found out that he could never apply a continuous distribution of energy among the oscillators and model the empirical results. The only way he could make the mathematics work is if he assigned each oscillator only discreet values of energy which were proportional to an arbitrary constant. We know the rest of the story.
The gist here is that Planck thought of quantization only as a mathematical trick that would allow him to model the phenomenon. His goal was not to explain the nature of the underlying entities which made up the system. He, and others, refused to even entertain the idea that the quantization of energy states had anything to do with the entities themselves. When he published his results, he presented the equations as a mathematical tool to model the observed phenomenon. He was not making a statement about the nature of the underlying entities themselves.
It wasn't until Einstein took up the study of the Photoelectric effect and direct experimental proof of quantization of light was shown to exist that he finally came to grips with the idea that quantization indeed occurs on a fundamental level. He admits he only grudgingly accepted the notion. It is for this reason, btw, that many consider Einstein the founder of Quantum Theory -- or at least co-founder. It is also ironic that both men eventually came to reject the theory which arose from their work. Neither would ever accept Quantum Theory as a complete and accurate representation of nature. Two of the most brilliant men in the history of physics let their prejudices concerning the ontological nature of reality interfere with the acceptance of a theory.
The point here is not to bloviate but to offer a different perspective on the subject of interpretations. There are two different ways in which we come about altering our perspectives.
Shifts in understanding may come from areas of research that are largely ignored and still in their infancy. In our current case, I believe that such areas of study include chaos theory, complexity, and emergence. These are realms where processes take on a more fundamental role than discreet substance and represent different ways of tackling an issue. I believe that work in these areas could possibly allow us to make sense of things that we may be missing in transitions as we go from one level of structure to the next.
We have made a lot of ontological assumptions about the nature of physical reality and how systems are constructed. Many of them are hard to hold onto in the face of current observations of the world beyond our senses. I think we need to ask the question, what is fundamental to nature? The assumption has always been that discreet substance(whatever that means on the subatomic level) is the most fundamental property of nature that is responsible for all observed systems. The notion that substance may not be fundamental may seem absurd and nonsensical but on close inspection it is not as absurd as one may think -- can it be more absurd than believing that a photon possesses certain ontological properties that allow it to simultaneously be both a wave and a particle?