Another issue that is i.m.o. far more interesting than the Chinese Room Argument is the following. If you have an intelligent machine, you can arrange it to be run in a fully determinstic environment. Eg., just simulate its brain inside a virtual environment.
Now, you can then argue that the physical states it evolves through can be mapped one to one to those of a simple system that runs trough the same number of states, like a clock. But how can a clock have the same consciousness as a (digital) brain?
One possible answer is that the digital brain is really running a nontrivial program. If you had fed it counterfactual input, it would have had given you the corresponding counterfactual output. In contrast, the clock can only run in one particular way. But then, one can object to this answer by asking how on Earth counterfactuals that, by definition, refer to things that did not happen, can be relevant at all?
My own take on this is that we should think of the consciousness generated by a brain as an effective Hamiltonian that describes its time evolution, instead of a physical state. In principle, a real brain is described by some microscopic Hamiltonian that describes everything that goes on in the brain. What we call consciousness should then be obtained (in principle, of course) by integrating out all the microscopic details.
If you touch an hot object then the microscopic Hamiltonian will describe exactly all the electric activity in your nerves leading to your arm being pulled back. The effective Hamiltonian describes the same process but now in terms of coarse grained variables (e.g. your fingers, your arm, but not individual nerve cells).
Now, not all of these coarse grained variables can refer to the "trivial" macroscopically observable quantities of the system. E.g. if we apply this procedure to balls that can collide with each other by integrating out the atomic degrees of freedom, in order to obtain a description in terms of only the center of masses of the balls and total momenta, we would be forced to include a few extra variables such as internal energy or temperature, entropy etc. otherwise you cannot get a closed macroscopic description.
So, you could well imagine that the effective Hamiltonian will describe the system in terms of arms, legs and other body parts, but also new variables will enter and they could correspond to the subjective things we can feel such as pain etc.
This then addresses the issue of how counterfactuals can matter. If what we really are is an Hamiltonian, the effective version which, loosely speaking, describes only the things we are conscious of, then many different counterfactual states at the micro level will be mapped to the exact same macro state. The way all these states evolve under the miscoscopic Hamiltonian completely fixes the effective Hamiltonian, even how it operates for counterfactual macrostates.
To illustrate that last point, suppose we look at watermolecules as the can be found in a cup of water at some temperature. The macro description only describes the thermodynamic phase the water is in (ice, water , steam). By studying the water molecules inside the cup of water, you can in principle reconstruct the full Hamiltonian of the interacting water molecules and that then fixes the properties of ice and steam also.
One consequence of this picture is that conscious observers cannot be thought of as living in the a single parallel universe in the Many Worlds Interpreation of QM. A precise microstate does not define an Hamiltonian. An observer having a definite conscious experience is an Hamiltonian of the form:
[math]H = \sum_{r,s}h_{r,s}|r\rangle\langle s|[/math]
where the microstates r and s that are summed over are consistent with the conscious experience. Restricting r and s to be one particular state would not yield a proper Hamiltonian.
