Yes It is my strongly held opinion that open peer-review is essential to help steer humanity away from disasters.

Many of the large scale disasters that could happen can be understood using science but the closed system of peer-review we currently use is failing to give the right answers. It is not just the review of scientific papers but also the reviews of funding and the ensuing public debate that matter here.

For example nuclear fusion is a possible new energy source that could replace our reliance on fossil fuels. The Joint European Torus is very close to providing a net gain in power and ITER is expected to succeed, yet funding for the project is very limited and there is very little public debate about it. The main problem it has is the word "nuclear" which triggers a whole cascade of biases, yet it is very different from existing nuclear power.

Peer review of climate change is also a mess with scandals over the hiding of data and methods used. The waste of money on tamiflu is another example of how easily faulty peer-review can cause problems.

As technology progresses the potential for this kind of problem is only going to increase. We desperately need good quality open peer-review so that the right policy decisions can be made based on academic research rather than the biases of public opinion.

Thank you everyone for your comments which I will try to respond to.

I request that you please do not give me any hints as to how you voted on this essay.

    Thank you for your comments and I am glad you are finding viXra useful. I look forward to reading your essay soon.

    Tommaso, thanks for your comments and questions.

    I dont think I am very optimistic that we will improve our rationalism towards peer-review. It is a hard problem. However I am optimistic that if we find away it will greatly help humanities future. Decision making (Especially at the political level) seems to be increasingly influenced by bias. At least we have taken the first steps towards recognising the significance of bias. Now we just need to figure out how to eliminate it. Perhaps clever AI systems will be able to do it.

    Your comments about experts is interesting. I think contributions from non-experts can be important as the meteoroid episode showed, but I dont undervalue expertise either. Some subjects require an expert knowledge to expose the right facts, but in most cases non-experts are just as good at evaluating the facts once presented to them. The courtroom is a good model of this where legal experts (and sometimes other experts) make all the points, but the public jury decides. For other situations the right balance and use of both expert and non-expert opinions can vary. Sometimes the experts can be prone to a bias within their discipline which may come about because of dogma or funding bias. Other times only an expert can hope to evaluate the complex logic (e.g. for a deep and complicated mathematical proof) My own experience with peer-review has shown me all the extremes.

    The FQXi contest is an interesting experiment in open peer-review itself but I did not want to mention it is the essay because that would be a bit too self-referential. I think it is excellent that anyone can enter and comment. The voting system does not really work too well so I prefer not to worry too much about the rating or prizes. This proves that straight voting is not the way to do peer-review. It seems to amplify the biases rather than eliminate them. I imagine a more structured system where people make points for or against an essay/paper and then people vote on whether those points are valid or not. You could also allow comments on comments etc, with everything rated and aggregated in some way to arrive a final score. The problem with such a system is its complexity which makes it hard to get everyone using it in the right way.

    Sorry about that! I just wanted to show a proper scale for your piece. I didn't imply that you should do the same. I know, based on last year's contest, that here is some collusion ongoing but that doesn't do any good for those involved in it.

    Pushing poor essays into the final only makes it easier for those actually good pieces to gain top positions!

    • [deleted]

    • Post #16

    Being not even a public reviewer of the claim that something will save the world, I feel somewhat amused by titles like Bee's that remind me of a self-ironic song in German "Nur mal kurz die Welt retten" (My translation: I will just save the world in a few minutes).

    I think I understand and largely support your approach. "Decisions based on academic research rather than the biases of public opinion" sounds prudent.

    Nonetheless Alfred Nobel guided me to slightly different conclusions. Is steering really always a question of decision making or did the invention of dynamite per se steer the history?

    Experienced WWII caused me to ask for what went wrong. May we blame single decisions or were patriotism, tin soldiers, revenge propaganda, etc. already before WWI among the true reasons?

    What about nuclear fusion, I wonder how many participants of the contest supported the hope for cold fusion on the basis of most likely just fabricated experimental results. I heard trustworthy comments that spoke of pathologic science. I will not invest a single Cent into something where the promise is huge for decades while there is obviously no realistic chance of true success. Instead of supporting illusions or risky for nature technologies I would prefer looking for feasible mechanisms of birth control worldwide. This will face fierce resistance by those many who are hoping to benefiting from economic growth, personal, religious, or national power, or who rely on funding.

    If groups of cherished theories or idolized experts went wrong and must be abandoned then this will also require more than a public peer-review.

    I appreciate FQXi as a forum that allows to put and pursue really foundational questions.

    Elsewhere we have some public self-declared peers.

    Sincerly,

    Eckard Blumschein

    The great thing about this essay is that it is focused and we can all agree that it is addressing an important practical problem and its potential solution which can contribute to humanity's steering. That said, it does not address the question posed directly.

    Supposing you could somehow fix the peer-review process in academia, would that solve the problem of steering the future? Is there a lack of scientific consensus which is the main reason for big problems facing humanity? Or are we imagining that "open peer review" will somehow translate to social and political consensus - and action?

    Also, I think the most telling lines of the essay are these:

    "How do you let anyone have their say while still maintaining an orderly process and arriving at an unbiased conclusion?

    The answer to that is not yet clear..."

    If I were intending to be completely snarky I would leave it at that, but the essay continues:

    ...but there is hope that letting anyone have their say is not necessarily a recipe for disaster. Wikipedia is a good example of a system that allows anyone to contribute."

    I address this also in my essay; the experience of Wikipedia has shown that, in fact, in order to maintain order it is necessary to have a hidden if not invisible clique of editors who exercise power over ordinary participants. Also, Wikipedia is not making decisions for steering humanity. Users go to it because they find it useful. On controversial issues, it aims for a balanced and reasonably comprehensive presentation of views, so partisans have no real incentive to opt out, which would only mean no representation at all.

    Also, Wikipedia is not a completely convergent process. If you don't like what's on one page about your topic, you can invent a new one that frames it differently; then maybe you will have to fight off an attempt to merge, but you might win something in that battle.

    But you are right that another key to making Wiki work is the establishment of rules and conventions. The seemingly rigid and (relative to other web content) impoverished presentation is actually very important. These are features that can work in open peer-review within academic communities with a shared interest in the advancement of knowledge, but it is less clear that it solves the problem of steering humanity where interests and strongly held viewpoints conflict and battles are fought often violently.

      John, thanks for your comments. I agree that all thoughts introduce bias, but there is an underlying unbiased truth that we must get to. It is peer-review that gets us there when it is done openly so that anyone can examine the reasoning.

      Thanks for your support. I will be reading your essay later and look forward to it.

      Bravo Philip your excellent ideas have been solidly backed by your excellent deeds. When arXiv accepted two of my papers because an academic friend had sponsored them, but then rejected my next paper because I submitted it myself, I felt demeaned and cheated . So when I discovered your viXra archive it was a godsend - no anonymous people judging my ideas because I worked independently. So again thank you for that and keep viXra alive!

      Open peer review need not be restricted to scientific research. As you suggest it can be made part of the system of governance at many level. The other day I had found that a nice wide green walkway had been decreed off- limits to cyclists. This in a city where mothers often use bicycles to carry children to their many after school cram schools. Or nurseries. Had they asked the populace to decide who uses the walkway things might have been different.

      Incidentally this fqxi system of peer rating is not open - we do not know who rated a particular essay and more importantly why....

      With best wishes

      Vladimir

      George, thanks for the comment and questions

      I think systems of government are hard to change for the simple reason that the people who get into power like the way the system worked, because it got them into power.

      What we can hope to do in a democracy is influence public opinion which then indirectly influences government. This is something that is beginning to happen on the internet where public debate on important issues is fast and powerful. The need then is to ensure that systems of public debate emphasise logic and rational thinking over more biased illogical ideals. This does not always happen.

      I like that when people comment on some news reports people can vote comments up and down. This really shows where opinion lies. I'd like to see a little more complexity in such systems to help the rational side. There is also a danger that some news media bias the results by deleting comments that are against their political views so I'd like to see independent commenting systems that cant be cheated. As an example of what I like try out http://rbutr.com/

      Dear Dr. Gibbs,

      I enjoyed your essay and agreed with many of the points: peer review as currently practiced could be greatly improved; the academic "game" is rigged against those without academic affiliations or with academic affiliations that are not from R1 universities; the idea of open access is good, but the exact way in which it will be funded is still unclear (this I think was the point of your Elsevier story on page 5 where Elsevier's stock price went up *after* open access since now they were getting paid by the government plus still getting library subscription fees (it was the government that paid right? not the individual authors).

      My only question/quibble is in the larger picture will having a better peer review system really have that big an impact on the progress of science, steering humanity to a "good" state, etc.? Aren't there more important factors to focus on? I just say this to play devil's advocate since as a theoretical physicist my main output in science is through what I publish so this is an extremely important point to me. But I am also reminded of a student-athlete at my university who once told me that the best thing to advance the goals of the university would be if there was *more* support for athletics on campus. This kind of statement may seem strange unless you spent some time at an American university which not only are academic centers, but also amateur athletic training institutions.

      Anyway I like the idea you mention of having peer review be public. Something like post the paper to some electronic archive and then people would post comments/critics of a given work. You might still need to keep the names of the people leaving critics anonymous since people, even those trained in the scientific method, often fall in love with their own ideas and have a hard time hearing even well intended negative things about a given piece of work. It doesn't help that often the criticism is in a tone that can be unnecessarily negative. Also the more people that give some comment on a piece of work the better. Something some journals have done is to list the editor in charge of the particular article along with the final published version. The idea is this then forces the editor to take some responsibility for the quality of those articles accepted. It give another layer of oversight into the quality of the paper since both the referee(s) and the editor have given the paper the OK and the editor even puts their name with the paper. Of course the reverse does not happen -- no one publishes the rejected articles with the name of the editor that rejected them. So there is some asymmetry on this point.

      Finally, you might find interesting a recent study of NIH R01 grants published in

      N. Danthi, et al. "Percentile Ranking and Citation Impact of a Large Cohort of NHLBI-Funded Cardiovascular R01 Grants", Circulation Research, 114, 302656 (2014)

      this study found that there was virtually no difference in the scientific impact of those grants which were ranked Category 1 (excellent grants according to the grant reviewers and which therefore got funded), Category 2 (very good grants according to the grant reviewers but which just missed the funding mark) and Category 3 (fair or marginal grants as judged by the reviewers which also did not get funded). Grants in all three categories published about the same number of articles and received the same number of citations. This is not necessarily the best measure of scientific worth, but it is the metric by which NIH says it judges the grants.

      Anyway and interesting and thought provoking essay.

      Best,

      Doug

        Philip,

        I overlooked in your reply "the biases of public opinion". Well, public opinion tends to be manipulated. That's why I maintain my objection to your claim "saving the world".

        Doesn't already your exemplary selection of expected catastrophes reflect the manipulated public opinion?

        Does global warming really deserve much more attention than poisoning the environment with radioactive and other waste?

        Can we expect the public opinion to abandon in time the logically untenable slogan "Earth has room and food for all"? While this naive belief might be true for now and the near future but not beyond, it protects the interests of those who are irresponsible and could misuse an open peer-review accordingly because they are strongly interested in their profit.

        I would rather appreciate if you could point to a list of viXra articles that proved at least as valuable as arXive articles. I agree on the necessity of scientific discussion on high level. To me in science, nothing must be taboo. Read my essays. Agreeing with outsiders like Nobel and Shannon rather than Einstein and Bourbaki, I do not even expect much ercy by open peers.

        Eckard

        Phil,

        Excellent job.

        Probably not a snowballs chance in a Bessemer converter of having any effect, but then a billion snowballs is an avalanche so we must start somewhere. To prove I really mean that I had this published in the 'Skeptical intelligencer' a few years ago;

        Subjugation of Skepticism in Science. (I spose I should have logged that into viXra too, whadda'you think?)

        Of course there are twofold problems; that one man's genius is another man's crackpot as all think differently, and we already have 'information overload' with the number of papers that DO get through (I have a growing pile of unread journals building up despite reading 20 papers a week). Overcome those problems and the ruling troglodyte hordes who have it all sewn up may start to worry.

        I note 'Nature' has recently be lauded for heading the other way and insisting all references are from only the landed gentry of peers. Only marriages to sisters are allowed! You may be right, the problem may now be starting to be one of mankind's very survival.

        I've had Vladimirs problems too. Some papers on arXiv, then suddenly a brick wall. Do they not also have any code of ethics? Unifying QM and SR mean nothing if nobody will publish it! (I hope you'll check out my essay).

        I greatly commend your efforts to beat Christian's record as the highest scoring community peer score to be passed over in the judging. Well done and best of luck.

        Peter

          Jonathan, Hi,

          I think the attitude of professional academics to independent researchers is very variable. Many of them are well aware of the way arXiv filters its papers and they regard that as acceptable. happily there are others who are much more open. FQXi would not be so open to outsiders if that was not the case.

          Its good to see you in the contest again with another strong contribution.

          Philip: better peer-review would surely help. I think a key element is expansion of "blind peer review" so reviewers don't know author names. The power of ad-hom prejudice is great. Meanwhile, viXra helps by providing an open forum for challenging ideas.

          Heh, clever coincidance, this was my legitimacy test:

          Important: In order to combat spam, please select the letter in this menu between 'V' and 'X':

            I agree that it would be a useful ideal if peer-review could be anonymous on the part of the author. In the present journal system papers submitted by independent authors are often rejected by editors who know their contact details.

            Perhaps a future open peer-review system could have an option for the authors to remain anonymous until initial reviews have been posted. However, for the most part authors like artists can be recognised by their work even if they dont sign it, so I dont think this ideal can be met in practice.

            The hope is that if peer-review is more open then any prejudices would be more obvious and counterbalanced.

            The letter between V and X is i :-)

            I am not the only one throwing snowballs but the avalanche has not yet begun. The traditional peer-review system is very well protected on all sides, but there are some signs that things will change eventually.

            I dont know what you are referring to about Nature. Is there a link for that?

            Good luck with your essay. I will get to it soon.

            When Wikipedia first started I thought it was hopelessly ambitious and had no chance of working. I was editor of the Physics FAQ so I knew how difficult it is to get people to write good content for free. The line between success and failure on the interwebs is a fine one but if you get the right formula your project can work.

            Wikipedia works because it has the right combination of rules and admin. It is not perfect and has its biases but smart people look at the talk pages as much as the main article if they want a more complete picture. The admin do not always get their way. They tried to delete the viXra article but failed because it fits the rules.

            Many systems for open peer-review have already failed. Look at scirate.com for example. It has been around for ages and is well known but it still gets little traction. One day someone will implement the right system, perhaps by luck. More likely they look at what works for Wikipedia, Stackexchange, Reddit etc, and adapt those features to a full-blown pee-review system. Perhaps the formula that works will look nothing like the way we imagine peer-review to work now and it may reach far beyond just academic poapers.

            Phil,

            An interesting paper here, not only showing a far higher proportion of experiments than we realise are falsely based, but then applauding treating the symptoms not the cause;

            "In 2005 John Ioannidis, an epidemiologist from Stanford University, caused a stir with a paper showing why, as a matter of statistical logic, the idea that only one such paper in 20 gives a false-positive result was hugely optimistic. Instead, he argued, "most published research findings are probably false."

            As he told the quadrennial International Congress on Peer Review and Biomedical Publication, held this September in Chicago, the problem has not gone away."

            http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble">2013 Economist article.](https://

            http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble)

            It also refers to the Nature policy.

            Peter