Hi Eugene,
First and foremost I appreciate the feedback. I clearly put a lot of myself into the work and it's immensely rewarding to hear that it resonates with people. It would seem that we agree on the 'spirit' of many aspects of the question but I'm uncertain if we agree in the details of the letter. On our point of contention, I am of the opinion that it is possible to confer self-awareness into a dynamic system, and I would point to our existence and the record of our historical evolution as proof of this possibility. I didn't exactly spell this out in the essay, but my opinion is that the computational complexity gain conferred by self-awareness is, in a sense, the 'teleological' explanation for self-awareness. In evolutionary terms, however, it's important to note that this merely gives the rationale for why a feature was kept or 'selected for,' but it tells us nothing about how the feature emerged originally. This is akin to saying that photosynthesis emerged "so that organisms could use sunlight." This statement may be true in a sense but it tells you nothing about how to build a leaf or perfrom repeated inelastic absorption of sunlight. It does, however, help us in identifying bottlenecks which constrain the net energy/information flow through a system until enough potential stress accumulates that new mechanisms emerge which increase the net flux by discovering new ways of channeling energy and information. Photoynthesis is sort of my paradigm example of what happens when a dynamical bottleneck is overcome by discovery of a novel mechanism for energy and information flux, and the discovery of self-awareness by humans may have also been a similar, bottleneck-opening discovery which allowed us to process more information and degrade more free energy because of the commputaional complexity increase it allowed. Unlike photosynthesis, I don't have a mechanistic explanation to complete the teleological explanation, but I presume that the details of the evolution of our own species contain sufficient information to explain this mechanism eventually. Given that we are self-aware, some aspect of the selective constraints on our evolution must lead to self-awareness. The challenge is to identify those constraints and implement them in a machine learning environment. We may be a long way from accomplishing this feat, but our existence alone is enough to convince me it must be possible.
Best,
Joe