Dear community,
Here is my proposal on how I think Prolific can allocate funding in a transparent, fair, and effective manner. I highlight in […] why a certain step is important.
The problem:
Many grant schemes today make it difficult for junior scholars to apply for funding, be it because they require the researcher be a certain number of years past obtaining their PhD (e.g., ERC Starter grants - 2ys), or has a certain level of prior publishing (e.g., la Caixa - at least 1 article in Q1 journal). In many systems, researchers are rewarded for having obtained other grants in the past, creating a snowball effect for some and leaving out others who might have great ideas but have gotten unlucky.
How to submit a proposal:
Grants follow a pre-specified timeline [transparency]
Grants exist in three “levels” (and Prolific specifies how often these grant schemes take place and how much money is allocated):
→ amendment 2+3, credit to Magdalena_Kachlicka and Jantsje_Mol
Small grants, open to everyone: for conducting small-scale pilot tests of studies [ensure research is feasible and makes sense before too much money is spent]
Junior grants, open to pre-doctoral researchers and PhD students only [those who currently have the fewest opportunities]
Junior+ grants, open to pre-doctoral researchers, PhD students, post-doctoral scholars, and tenure-track faculty within 7 years of obtaining their PhD. Proposals may be for co-authored projects; in this case the co-author(s) and their contribution to the project need to be disclosed. Preference will be given to single-authored projects or those where all co-authors satisfy the junior eligibility criteria (i.e., all else equal, these projects will receive the funding over projects with senior co-authors). [do not disqualify people just because they co-author; reward good ideas] → amendment 1, credit to bethaniley: make the contribution of each scholar an evaluation criterion in order not to disadvantage PhD students who are required to (for example) add their advisors on all proposals regardless of the advisors’ involvement
Projects are not restricted time-wise (e.g., given a completion or publication deadline) but realistic timeline is part of project evaluation [prioritize quality over speed]
Submissions should contain the research question, methods, budget, and scientific contribution, and researchers will have the freedom to choose how these are presented (e.g., charts vs. text). IRB approval, pre-registration of hypotheses/methods incl. a pre-analysis plan and power calculations where applicable, and commitment to making all data and codes public after completing the study are a required part of the proposal. The researcher is free to add any other relevant information about the study (e.g., running pilots, their own expertise using the methods, …) The entire application should aim for approximately 2500 words with a limit of 3000. [commitment to scientific rigor AND open science]
Prior to the competition, Prolific specifies any legal/technical constraints the proposals need to follow (e.g., budget allocation).
How are grants allocated:
Every researcher who submits a proposal is required to be a referee for 2 other proposals in his/her discipline (broadly defined). Referees write a public (up to 1500 word) evaluation, which is posted under the proposals (and clearly flagged as such). These evaluations should point out errors (if any), and discuss whether the research is 1) scientifically sound, and 2) innovative. Prolific will provide a standardized and transparent grading scale that the referees will be instructed to use. Referees may make suggestions to the designs, which researchers may but do not have to accept. [focus on scientific rigor]
The broader Prolific community may vote or comment on proposals or referee evaluations for a period of 3 weeks afterwards. Following this input, and discussion with the researchers, the referees are allowed to change their grade of the proposal. Everybody is allowed to vote or comment, but the person’s Prolific involvement is disclosed: how long have they been active on the forum, how many studies have they run, which institution and discipline are they from. [involvement of the community without giving incentives for voting by family or friends, improving proposals together, fixes any misunderstandings]
The researcher with the best proposal as evaluated by the referees after the public discussion period will receive the funding; 10 of the most helpful referees (as voted by the community) will receive a smaller prize. [incentives for referees to help others succeed]
I love the referee system here – I think it would be a great way to encourage engagement while also ensuring adequate peer review. However, I do have two (relatively minor) suggested amendments:
Preference will be given to single-authored projects or those where all co-authors satisfy the junior eligibility criteria (i.e., all else equal, these projects will receive the funding over projects with senior co-authors).
This is an interesting idea. However, I am concerned that it would disadvantage students at universities which require a senior collaborator as part of institutional ethical approval. My suggestion here is use the CRediT framework to disclose authorship contribution as part of the grant application process. Each proposal can then be allocated a score based on the degree of involvement of junior vs. senior researchers. This score could then be added to the final number of votes.
Everybody is allowed to vote or comment, but the person’s Prolific involvement is disclosed: how long have they been active on the forum, how many studies have they run, which institution and discipline are they from.
This is a really good idea – it allows transparency in who is voting and commenting, which should encourage people to only vote for proposals they believe are high quality. I think this could also be used more directly to prevent manipulation. Proposals which have a large number of votes from users who have previously run 0 or 1 studies could be flagged. For example, if a proposal’s vote count is over two standard deviations above the mean number of votes and the majority of these votes come from low-activity users, a Prolific staff member (or a bot?) could reply to the thread to highlight this.
Thanks for your great comment! Absolutely agree, let’s have people disclose co-authorship. I think if the senior co-authors are on the paper “just for the ethical review board” (which, personally, I think is a ridiculous policy and should be scrapped, but that’s a debate for another time), the proposal could be treated as if there are no senior co-authors.
Indeed, I think Prolific could in general monitor (and flag) people who register, vote, and never use the platform for anything else. An alternative would be to introduce “verified” researcher accounts (something like the blue checkmark on twitter), allowing people to connect their Prolific account to their research (e.g., Google scholar profile). But this really depends on how far Prolific wants to and can go for the purposes of a competition…
Nice proposal! I certainly agree with the point about the restricted availability of funds to junior scholars. The situation is even worse for PhD students (like me) who cannot apply for most research grants. Most funding available to PhD students (apart from studentships that do not always include research costs!) covers only conference fees or travel expenses but not the research costs. Available resources are very scarce and highly competitive. It is a somewhat personal request, perhaps voicing the frustration of other PhD students, but it would be good to either promote one’s juniority status (as mentioned above) or consider granting awards per specific seniority level (one award for PhD students, one award for ERC up to 3 years after PhD, etc.). It seems that scholars with more experience will outrank junior researchers/students and produce better-developed proposals, which might, unfortunately, be reflected in the reports from peer reviewers and subsequent votes.
Magdalena, thanks! Let me clarify, what exactly do you mean by “promoting” your juniority status? How can this be done in the fairest way do you think?
I really like this proposal! Especially the quick but thoughtful review procedure.
I was wondering if you could also include something on pilot studies? I think pilots are super important to test experimental paradigms and but also to test the viability of a research idea. And as you point out, especially junior scholars might lack the funding for this (e.g. to test which of their ideas is most promising). Maybe an extra category where less funding is at stake, but requirements are also a bit lower (lower word limit?), and the review process even faster?
Thanks to all three of you for your wonderful suggestions, I have incorporated them in the proposal. I hope we are a shining example of how open peer review can help improve science
