Aaron, I've been generally using the contest guidelines' "Evaluation Criteria" to rate essays, rather than making up my own. So I've been using a 1-6 scale (roughly 2/3) for "relevancy", which is mostly made up of the questions that they listed:
- Is it positive rather than pessimistic/dystopian?
- Is it the best state that humanity can realistically achieve?
- Is there a clear plan for getting us there?
- Does it give an idea of who implements this plan?
- Does it speak of specific technology that we can use, and include the problems and benefits of said technology?
- Does it focus on FQXi's general category of thinking/problem-solving, bringing in math, cosmology, complexity, emergence, and/or physics in some meaningful and novel way?
Then I give another 3 points (roughly 1/3) for "Quality". Which basically consists of whether or not this is something better, more original, and more novel than something I'd find in Scientific American, while also being directed at a broader audience than an academic paper. (In other words, if something is too hard to understand on one end, or too simple to understand and similar to your average blog post on LessWrong.com, I give it a 0 for Quality.)
Then I've got one point left (since the weirdos who made the rating system haven't considered the fact that you can't get whole number ratings when using thirds) to play with, and I usually use it for relevant innovation, since that's kind of a crucial element in problem solving. If I've read something similar before, in a variety of places, and/or it doesn't really answer the question of where we want to go and how we might want to get there most effectively, then it's automatically going to need to be rated lower than an essay that is novel and highly relevant.