You're welcome, Toby. Thanks in return. - You make a brave foray into the steering problem from a moral vantage, but fail to master that vantage (I think), and your own thesis.
Thank you for your honest and critical appraisal of my essay. I did find it a challenge to communicate the ideas that I thought most relevant to the essay topic. Your feedback has been helpful in prompting me to think further on the making the ideas more precise and how to communicate them more clearly. I hope you might also do me the favour of reading and replying to my responses.
The light manner in which you dismiss all prior moral philosophy (waving the wand of moral relativism, p. 2) doesn't encourage the reader to trust your judgement, or to give a fair reading to your own ideas.
My intention wasn't to dismiss all prior moral philosophy. I tried to suggest that the Markov decision process model can be use to represent both consequentialist and deontological moralities. I also found that representing a moral framework as a reward function didn't automatically bring us any closer to knowing which moral framework is the "best". This problem struck me as similar to the problem of moral relativism. I also consider this further evidence that meta-ethical moral relativism is a problem that needs addressing.
I tried, but couldn't assume with you that "each agent's morality, from a person to a nation, may be calculated as a reward function." (p. 3)
Perhaps that assumption was poorly worded and much stronger than the following argument required. In reality, agents are often irrational and are unlikely to have moralities consistent and well-defined enough to be translated into a reward function. This is especially true of agents with many cognitive biases, such as people, and those composed of many individuals with different systems of moral preferences, such as parliamentary governments. Even if an agent's morality may be expressed as a reward function it might not be possible to calculate because it could require many input variables. This may suggest that people and agents should be encouraged to adopt more explicit systems of moral preferences.
In any case, the focus may have been better placed on the assumption that systems of moral preferences have associated reward functions.
A reward function is an abstract way of looking at the moral preferences for actions and states that agents may have. A system of moral preferences may be translated into numerical values associated with states and actions, with greater value indicating greater preference. This could be an arbitrary one-to-one mapping of all combinations of states and actions to numerical values, for example, praying facing a particular direction at the right time of day might have a large positive value, or there might be ways of calculating the reward through a formula, such as summing all human happiness in that state. Either could fit into the general concept of a reward function.
I think we should be able to assume that at the very least an arbitrary one-to-one mapping of reward values exists for any agent with unchanging moral preferences. However, I think it may be more informative to try to design functions based on moral principles and attempt to calculate the reward (moral preference) of each state and action. The calculated reward could potentially be compared with our intuitions and other reward/preference functions more easily.
From here, I saw you struggle earnestly to deduce what might more convincingly have been assumed: that we value our own existence.
I don't think nihilist and non-anthropocentric view-points should be dismissed out of hand. For this reason and others, I think that there is great value in finding a logical argument for valuing ourselves and others.
These ideas were challenging to formulate and communicate. I don't think it would be effort wasted on my part to make the argument more convincing, so if there is any part of the argument in particular that you disagree with I would like to know what it was.
You then deduce that we place a supreme value on learning in all decision agents, human and non-human (p. 4), but this doesn't seem well supported by the argument, nor does it seem a moral principle in substance or form - not like those of the philosophers you dismissed earlier.
Valuing learning was a suggestion rather than something that was deduced in this essay. Perhaps I should have tried to argue how valuing learning of all agents can be the basis for any number of moral decisions.
As a moral principle, valuing learning suggests:
- Agents should be acting to learn and experience as much as they can.
- Actions and states that cause disability and death are immoral as they diminished the ability or opportunity for agents to learn.
- It's immoral to destroy sources of information and experiences, including artwork, books, historical artefacts, animals and ecosystems.
- It's immoral to deny people knowledge and experiences unless it would cause disability, death or destruction.
- Even though the negative value of other outcomes may be greater, any event or experience that results in learning has at least some value.
To put this moral principle into practice would require a weighing up immediate learning (reward) with future learning (expected value). Otherwise, I think valuing learning as a moral principle is quite an intriguing idea that has lots of intuitively positive ethical results.
Your conclusion feels equally shaky: that we should "search for good mechanisms" for steering, but "using scientific processes to decide whether a political or economic mechanism is effective". You spring this in the final two sentences without explaining it further (p. 7), as though you yourself were not quite convinced.
I can only really apologise for the shaky and unpolished conclusion. I was running up against the submission deadline so the conclusion was rushed. I would like to follow up these ideas further in future writing (and maybe try to come up with a more convincing conclusion), so I appreciate that you have taken the time to review my essay.