"I'm getting a sense that people are having a hard time figuring out 'what's the great idea?',"
Jack: Exactly
" which since probably very few of you have ever heard of semiotics, let alone semiotics, may not be surprising. I'll start with the assumption that most people's background is in physics, so you will be familiar with quantum mechanics and the question of 'what does it all mean?', with problematic issues such as the fact that we seem to be able to talk only in terms of averages rather in terms of individual events, what is really happening when an observation is made, and the paradox of Schrödinger's cat, and all that. You will also know that 'hidden variable' theories exist that claim to resolve such issues but are really rather a fudge. It would be nice if we had something better, more intuitive."
Jack: Bohm's picture is not a fudge. Indeed, that's what Michael Towler's Cambridge Lectures are all about. So what you have just claimed is very misleading far from the truth. Also you seem to not understand Yakir Aharonov's weak measurements and how they relate to the Von Neumann strong measurements. In fact, we now see individual events in a new kind of statistical sense very different from the old eigenvalue idea you are citing above.
"In this connection, what has happened to me is that in recent years I have been exposed a number of ideas that look like something better, the interesting thing being that they seem to be able to fit together nicely: it is a bit like the old tale where people see different parts of an elephant and think they are seeing completely different things but they all fit with the idea that they are seeing different parts of a single thing, the elephant.
The commonality is basically the idea that this obscure region that, according to orthodoxy, we really can't do anything about is one characterised by a kind of life, and by complexity, and it is the complexity that makes 'business as usual' impossible in dealing with it. But if this is the case, then instead of just giving up we should do the best that we can. One approach is that of 'complexity biology', basically that of treating biology from the perspective of complex systems. This is the approach of my colleague Alex Hankey, who has entered an essay into this competition, but his 2015 paper goes into much more detail and I hope he will upload it to an archive, as I have suggested, so everyone can read it without having to pay the journal to do so (as is allowed under certain circumstances).
Alex and I have compared notes on this, but I have developed more a different side, based on so-called biosemiotics. I came by this through being invited to talk at a conference on semiotics, and more recently I became aware of its application to biology, biosemiotics. This is the study of the role that meaning plays in biology, and it has some very neat ideas. As an introduction to this, you might want to look at the slides for a lecture I gave at a recent conference on Fundamental Physics, starting perhaps at slide 6, and then Hoffmeyer's paper on semiotic scaffolding. These ideas address subtleties in regard to what makes life possible.
Question: why should these esoteric ideas matter as far as physics goes? The answer I think is this: let's suppose that people are right to say that this mystery realm is essentially biological. In that case we need to use biological tools to make sense of what is happening there, and not just blindly hope that the methods currently in use in fundamental physics will in the end do the job. I have addressed the question of how this new direction can proceed theoretically above so will not repeat them, just look at my response to Andrew Beckwith for details. This approach, combined with that addressed in Hankey's essay, may not in the end lead anywhere, but I believe it will, and it is certainly well worth seeing where it can lead.
So to summarise: biology involves a different kind of ordering to regular physical systems -- just consider how different what happens in biological systems is from the case of physical systems. We can use tools developed in that context to probe deeper into nature, if it is the case that mysterious nature departs from the pictures presumed in physics and instead adopts this alternative kind of order at this hypothesised deeper level."
Good luck with that, but I have not been able to understand Hankey and I doubt any other theoretical physicists will even take the time to look at it.