W

Wanpeng Tan

Donatello Dolce
Thanks a lot for your good evaluation of my essay. I have also just updated the full preprint version at
https://openreview.net/forum?id=E144GC5Vgw6
I hope that at least some of the ideas, if not all, in my proposal will be implemented in real world soon.

Donatello Dolce
Thanks a lot for your detailed response. I'll try to be concise in my reply to your points. More details can be found in my preprint posted at OpenReview.net:
https://openreview.net/forum?id=E144GC5Vgw6

1) (see the time factor in A1 on page 17, middle of page 22, etc.; also last paragraph of post #18). In essence, the proposed system is dynamically calibrated. All the rating/review activities are staged, and only the current one is used for calculating credits. This will effectively defeat such gaming behaviors. For the example you mentioned, yes, those articles regarding superluminary neutrinos could have gained more credits in the early weeks or possibly months. But late in 2012, the scientific community discovered the errors. According to the proposed system, the credits will be re-calculated for such events. No system is perfect. But the proposed one is based on the trust of the scientific community as a whole. So at least, we can reduce the blocking force from elite circles and encourage early attempts (either disruptive ideas or error-corrections).

2) Again, the system is dynamic with many built-in measures to prevent gaming (see Appendix sections in the preprint). No calculated credits are final and they all could be changed based on new findings. There might be people who like to bet on short-lived fame or credits. But in the long run, only good behaviors will be credited.

3) (see post #18 and also the preprint). The economy analogy is introduced to compare similar phenomena like monopolies and startups in both fields. But the fundamental difference makes the review process unique and extremely critical in scientific research. Again, as mentioned above, the dynamic credit system will hopefully encourage more forward-looking behaviors. I think it is consistent with our main motivation - thirst for knowledge and curiosity for truth.

*) arXiv issue. I don't think arXiv is any kind of realization of my proposed system. It is the opposite as it does not allow the whole community to participate and instead, only a few people are doing the gate-keeping moderation. It is more consistent with the "publish or perish" culture, as you just need to post traditionally accepted articles. There is no incentive at all in the arXiv service to encourage innovation. I doubt that they'll do anything at all similar to what I proposed. I think other platforms will do better as discussed above in my earlier posts.

*) I have read and commented your essay. It reads great and I rated very high. Wish you good luck in the competition.

Janko Kokosar
Yes, the proposed system is a dynamic one and need to be fine-tuned, especially at early stages. Just as elaborated in the full preprint posted at OpenReview.net:
https://openreview.net/forum?id=E144GC5Vgw6
A-1 category is reserved for truly detrimental (fraudulent or plagiaristic) works. While some in A0 could be truly garbage or useless, we hope it could be minimized due to the incentive system. As discussed in the preprint (see A.2, page 17), it is more likely to be a placeholder for fringe science or disruptive ideas that have not yet been recognized by the community. Under the new system, the good ones will eventually be credited.

Donald Palmer
Thanks. I'll be following up with them as well.

James Hoover
Thanks a lot for your interest. I also wish you good luck in the competition. My goal is not so much about winning in this essay contest, but rather about gaining needed attention to implementation of the proposed system in the real world.

Donald Palmer
Thanks again. I actually think that smaller new platforms could do better. Here are relevant paragraphs of discussion quoted from my preprint ( Sect. B2 on page 21):

Ideally, the most suitable place to implement the system would be on large preprint
service platforms that are widely used within a given field. However, due to its entrenched
dominance and inertia, the largest eprint server, arXiv.org, does not allow comments or
reviews, let alone a quantitative review system. Although this path would have been the
most efficient, it appears to be a long shot.

Conversely, emerging smaller preprint servers like bioRxiv and medRxiv are more
willing to try new ideas and could play a more significant role in the adoption of the pro-
posed system. In addition, newly established dedicated review platforms such as PRE-
review.org, ReviewCommons.org (non-profit), and ReviewerCredits.com (for-profit) could
gain increased recognition and significantly expand their user base by implementing the
new system. Interestingly, a for-profit company called ScienceOpen.com, which offers
both preprint/publishing and peer review services, has already implemented most of the
required structures except for the new credit/role system. It may soon demonstrate the
desired effect through a relatively straightforward integration of the new quantitative sys-
tem.

Any of the aforementioned platforms would be suitable for starting experiments with
the new system. There is no need to first build a national or international community
structure from scratch. Nor is it necessary to implement all aspects simultaneously. How-
ever, it is crucial to first establish the basic credit/role mechanism as proposed.

I've just got in contact with these platforms. Some immediately showed interest. Hopefully at least some of the proposed ideas will be tried out soon.

Donald Palmer
Thanks a lot for your detailed comments. You may take a look at my just updated full preprint posted at OpenReview.net:
https://openreview.net/forum?id=E144GC5Vgw6
You may find many of your questions addressed in the 2nd section of the appendix. Here is a recap.

What is the point of the analogy to a capitalist economy? The basis of the analogy is that both systems are driven by cycles of innovation. However, there are fundamental differences between the tangible products of a capitalist economy and the much less tangible outcomes of research. These differences necessitate a complex peer review process for evaluating scientific progress, which is the central focus of this article. Nonetheless, addressing this unique peer review procedure requires considering potential issues associated with start-ups and monopolies that are common to both and that could impede innovation. The point is that capitalist economies have been more successful in dealing with these issues than scientific research, which is both unfortunate and inexplicable.

How can the scientific community do better than the governmental structure that is meant to prevent monopolies in a capitalist economy, which can often be corrupted? While it is true that the capitalist economy and its regulatory government are not flawless, they do have a democratic mechanism in place that promotes fair competition and fosters healthy innovation. In contrast, the scientific community can be perceived as more authoritarian than democratic, i.e., not even up at the level of the capitalist economies. As a result, unorthodox ideas in science often face gatekeeping barriers to recognition and acceptance.

The proposed system advocates the principles of democracy and diversity that can truly preserve the freshness and vitality of the driving force of innovation. Considering the rigorous nature of scientific research, there is good reason to believe that a properly implemented system could significantly enhance the self-regulating structure within the scientific research community, making it more robust.

In scientific research, ideas embodied in preprints, publications, and proposals can be considered the ``products''. But are there more tangible products in science, comparable to the marketable goods produced in the economy? In the realm of basic science research, ideas are undoubtedly the most important output, and the proposed review system is particularly well-suited for evaluating such results. On the contrary, when it comes to research focused on applications and technology, more tangible products emerge, some of which even lead to the creation of start-up companies in the capitalist economy. We argue that applied research progresses more effectively than basic research, precisely because of the existence of such tangible products. The real challenges lie in basic science research, where progress seems to be increasingly stagnant, and cases of meaningless work or fraud have become all too common.

Simple rating systems like what FQXi has done are flawed and easy to game upon. Fortunately, the proposed system has built-in measures to counteract such attempts. In particular, the credit rewarding process is dynamic and can be continuously fine-tuned, rendering any gaming efforts ultimately ineffective. In addition, with the aid of advanced machine learning techniques, we can further enhance the system's performance and robustness as we accumulate a larger dataset of statistical information.

Vladimir Rogozhin
It is not a single project per se. The credit/role system should be implemented field-wise. So there could be many projects. Everyone is welcome to start experiments with the proposed quantitative system, including small eprint servers, dedicated peer review platforms, for-profit or non-profit. I don't have the resources to play any of the leading roles, but I can definitely be a consultant for any of these projects.

I have just updated the full preprint posted at OpenReview.net:
https://openreview.net/forum?id=E144GC5Vgw6
You may find relevant new discussions at the end of the article.

Vladimir Rogozhin
Thanks a lot for your support. I'd like to take the opportunity to elaborate it a bit more, hopefully encouraging the community to give it a try.

The proposed community-based credit/role system for peer review could be implemented either on large preprint servers or on dedicated review platforms. Most existing services and platforms, including the most recent ones, either lack incentives or do not provide the right incentives - the monetary/honorary incentive some of them have used does not really work for reviewers. The best incentive, in my opinion, is to increase their role in the community in exchange for their quality review work. This will motivate most scientists to review each other's work more.

In addition to the right incentives, such a system could actually make the platform financially self-sustaining. For example, they can receive donations or fees by helping overlay journals on peer-reviewed preprints; by assisting funding agencies, especially private foundations, in reviewing proposals; by providing academic institutions with more reliable merit evaluation of candidates for their hiring and promotion decisions.

By gradually implementing the system from paper review, proposal review, to achievement evaluation, the credit/role system will eventually make the implemented platform the most attractive one for all scientists and researchers, especially the up-and-coming young ones. The most important factor for the growth of a platform is the size of its user base. This will do it. In addition, a successful platform could further expand its scope, for example, in the business of organizing conferences (for example, determining topics and who should be invited), and reviewing proposals for experimental facilities, etc. That would be a dream come true for me.

Regarding the downsides, I have received criticism against a quantitative system that I should defend a little more. The development of science itself is an evolutionary process of becoming more and more quantitative and rigorous. If we never try to make a field more quantitative, then it stands no chance of becoming part of the science. I dare to propose an initial endeavor to quantify measures that could be applied in peer review, in the hope of establishing a first quantitative paradigm for peer review. It would be a pity if rigorous science could not be assessed in a quantitative manner.

Andrew Beckwith
Thanks for your interest. Now I see where the confusion came from. Nowhere in the essay does it suggest that the proposed credit/role system be implemented at the INSTITUTIONAL level. Rather, it must be based on the entire community of all scientists in a given field (see, for example, the 2nd paragraph of Sect. 4 on page 7). In fact, such structures are readily available in some fields. For example, arXiv.org has been used by almost all physicists in the world and will be an ideal INTERNATIONAL base for implementing the proposed credit/role system. The only obstacle is the bigotry of its top administration.

This community-based system is supposed to (largely) replace the institution-based tenure system, as part of the proposed solution. As a matter of fact, all schools, large and small, will ultimately benefit, but they all have nothing to do with the system, which should be implemented FIELD-wise rather than institution-wise.

The life sciences, as another example, may be ready for such a system. They have recently established an international platform, reviewcommons.org, supported by both the European (EMBO) and American (ASAPbio) communities, as well as many preprint and publication services in their field. They may beat our physicists to the punch and be the first to implement a similar system.

There is no need to build a national or international community structure first from scratch. For any basic science field, a widely used preprint platform would be a perfect starting point. We don't need to implement everything all at once. But the basic credit/role mechanism as proposed should be in place first. We could start with the review of preprints/publications first. Then we could add the review of grant proposals. Finally, the evaluation of individual achievements (mostly for synthetic ones, as single-paper achievements are automatically evaluated in the first step).

More details can be seen in the full version of this essay submitted as an anonymous preprint to OpenReview.net: https://openreview.net/forum?id=E144GC5Vgw6

Andrew Beckwith

You quote the whole abstract so that I am not exactly sure what you meant in your comments. But I assume that you meant the part on the evaluation of individual accomplishment up to five pieces of achievements. Then I respectfully disagree or it seems that you did not quite understand what the measures the essay proposes. Indeed, the problem for smaller schools is that they don't have ways to truly evaluate academic achievements, so in practice, you often favor big brand names instead of actual merits. What is proposed in the essay is exactly to solve this issue: it is the entire community instead of individual institution to do such evaluation. Here the community is meant to be a whole field of participating scientists in the whole nation, maybe even of the whole world, definitely not some isolated small community. That is, it should consist of all relevant scientists across all regional barriers.

In addition, such evaluation system could also prevent too much interference from other factors such as personal relations, elite circle membership, political, and many other unspoken rules.

The essay provides a very good understanding of the Kuhnian view of evolution and structure of science. Indeed, a very elegantly written article. I like the discussions of the two questions. Regarding the 2nd question, I have attempted to give an answer in my essay, titled "A Robust Community-Based Credit System to Enhance Peer Review in Scientific Research". You might want to take a look if interested.

Steven Andresen
Thanks for providing a summary by ChatGPT-4. It looks good overall. But clearly it does not have a mathematical mind. It does not really understand the credit system I proposed in the essay, in particular, for anything and its merits related to the math details, even though only very elementary math is involved. That is kind of one of the shortcomings of the new AI system.

Issues about arXiv are very well argued. This is really like a prerequisite for reading my essay titled "A Robust Community-Based Credit System to Enhance Peer Review in Scientific Research", where, between pages 2-4, I dedicated a section to talk about arXiv's monopoly. Some very interesting statistics about arXiv's reclassification and rejection are shown in my essay. Take a look if you are interested.

The goals of a complete and comprehensive solution that should invoke a revolution in the entire structure of scientific research are as follows:
• Principle of Democracy – An all-participating community is for all properly trained researchers.
• Principle of Diversity – Scientific start-ups and high-risk high-reward projects must be adequately funded.
• Quantitative Credit System – Rigorous science requires a rigorous credit system for scientific evaluation.
• Quality-Based Achievement Rewards – Quality, rather than quantity, should be emphasized in rewards for individual achievement.
• Contribution-Based Role System – Members’ contributions to the community should determine their roles in the community.
• Healthy Self-Sustaining Ecosystem – If all the above requirements are met, this is automatic.

The main merits of the credit system are summarized as follows:

• Early-Bird Encouragement – earlier contributions are credited with more points.
• High-Quality Attention – more activities are attracted to higher-quality papers/reviews/
comments.
• Robust Against Gaming – careless or irresponsible behaviors are hard to gain
credit and may result in losing it instead.
• All Member Participation – democracy and diversity are ensured by the participa-
tion of the entire community.
• Rewarding Positive Activities – a reliable role-increasing mechanism is integrated
into the self-regulating ecosystem.

The full version of this essay is submitted as an anonymous preprint to OpenReview.net: https://openreview.net/forum?id=E144GC5Vgw6
More details can be seen there.

Using an analogy with the capitalist economy, we examine the issues within modern basic science research as innovation drives both evolutionary cycles of the economy and research. In particular, we delve into the topics of peer review, academic monopolies and startups, the tenure system, and academic freedom in detail. To improve science research with a mature paradigm, a comprehensive solution is proposed, which involves implementing a credit system within a robust community structure for all scientists. Members can earn credit by contributing to the community through commenting, reviewing, and rating academic activities of submitted manuscripts, grant applications, and up to five achievements from each member. As members accumulate experience and credit, they can progress in their roles within the community, from commenter, reviewer, moderator, up to board member (serving in governing committees). High-achieving individuals are evaluated by the community for the quality, rather than the quantity, of their academic accomplishments. High-risk, high-reward projects from academic startups will be properly funded, and a healthy feedback and ecosystem will make the scientific community prosper in future innovative cycles in a self-sustaining way.

Download Essay PDF File

Download Technical Endnotes PDF File

Somehow my reply did not get posted. I am trying again.

Thank you, LC for your interest.

You are right about exact SUSY requiring zero vacuum energy. The two models of SMM2 and SMM4 for 2-d and 4-d spacetime, respectively, follow the exact N=1 gauge SUSY. On the other hand, during the phase transitions or spontaneous symmetry breaking processes, the corresponding SMM2b and SMM4b will break the exact SUSY and become pseudo-SUSY due to the emergence of new mass scales and vacuum energies. But the matching of degrees and freedom between bosons (gauge and pseudo-Goldstone) and fermions is still observed, and that is why it is called pseudo-SUSY).

Yes, there seems to be some connection with string theory (but not as a theory of everything). Please see my other preprint for more discussion on that: https://doi.org/10.31219/osf.io/8qawc

Best,

Wanpeng

Dear LC, Thanks for your interest.

You are right about exact SUSY requiring zero vacuum energy. As such, two of my models SMM2 and SMM4 for 2-d and 4-d spacetime, respectively, follow the exact N=1 gauge SUSY. On the other hand, during the phase transitions or spontaneous symmetry breaking processes, the corresponding SMM2b and SMM4b models will break the exact SUSY due to the emerging mass scales and vacuum energies. Therefore, these models are called pseudo-SUSY as non-zero mass terms leading to new energy scales but the matching of degrees of freedom between bosons (gauge and pseudo-Goldstone) and fermions is still observed.

Indeed, there is some connection with string theory (not as a theory of everything). You might find some related discussion in my preprint: https://doi.org/10.31219/osf.io/8qawc

Best,

Wanpeng