#HaveYourSay - Improving the Review Process & Eliminating Underpaying Studies

[Steve_Wikeley]

Who We Are

Hi Researchers!

We’re the Trust Squad (our word for ‘team’). We’re here to facilitate a fair and trusted platform for all our users, and provide you with the highest quality data.

Our vision for 2021 is to ensure researchers can trust the quality of the data without needing to check it, and that participants can trust that they will be treated and paid fairly by researchers.

Our Objective for Q2

We’re working on:

1. An improved review process
2. Eliminating Underpaying studies

We Want Your Feedback On…

An improved review process

We want to make the submission review process pain free for both researchers and participants.

Researchers often say that the review process takes up a lot of time and effort. Specifically, the biggest problems are with the amount of admin needed to get everyone approved or rejected, and the back-and-forth messaging with rejected participants.

The biggest pain point for participants is not getting paid if they are rejected. Even if their data is unusable for researchers, participants felt it’s unfair that their time and effort isn’t being valued.

In order to address these pain points, we’re going to test out a new approach to approvals and rejections with a sub-group of researchers.

We’re running a survey at the moment and would love your input:

Please let us know your thoughts

Eliminating Underpaying studies

Although we’ve set a fair payment minimum of £5/hr or $6.50/hr, some studies pay less than this. This happens when a researcher underestimates the time it takes to complete their study.

We want our participants to know they’ll always be treated fairly and paid well for their time. We believe that the biggest reason for our high-quality participant pool is that Prolific cares about our participants’ experience.

We’ve recently released an ‘Adjustment Journey’ which makes it much easier to correct the rate if your study is underpaying.

Now that it’s easy for researchers to adjust the rate back to paying fairly, we’re starting to think about our next steps to eliminate underpaying studies.

Please let us know any thoughts you have on this subject

[Veronica]

Hi Steve!

About both points:

1. An improved review process: I’ve just completed the questionnaire and will circulate it with my colleagues. I think it’s a brilliant initiative!
2. Eliminating Underpaying studies: This is good news! However, let me just comment that what I call the “participation fee” is not the only way to reward participants. I’m used to think of studies with other types of incentive that are part of the study itself and can be transferred as “Bonus payments”. Therefore, I think it is very good to make salient the actual rate at which you are paying the subjects especially with “warnings” when it gets below the suggestet lower bound. However, I would not “eliminate” studies in which the rate does not get fixed, because they might be actually paying also the bonuses!

Let me know if it makes sense for you!
Thank you

[Steve_Wikeley]

Hi Veronica - many thanks for completing the questionnaire already and for sharing

On your second point. It’s something that we’re working towards a really clear definition of what an underpaying study is. The point you raise regarding bonuses is a tricky point we’re trying to address. The approach can leave some participants underpaid for their time and others paid more for their time based on performance.

Just to confirm: We’re pro using bonus payments as devices in studies but how we should calculate underpaying is alittle complex.

Here’s how we currently calculate whether a study is underpaying:

1. There are 10 or more completed submissions

    AND

2. More than 50% of submissions have been completed

    AND

3. Either:
    1. Study is paused, completed or awaiting review

      OR

      2. Study has had at least one approved submission

    AND

4. The average reward per hour* is less than £5 or $6.50 depending on the study currency.

* The average reward per hour is calculated 

(Rewards + Bonuses + Adjustments) / Median Submission time

The downside of this calculation is assumes the majority of participants will receive the bonus.

Would you see it fair to pay the participants not receiving the ‘bonus’ under the minimum rate per hour?

[Veronica]

Thank you for the clarification! I was not aware that in the “average reward per hour” formula you were already considering bonuses. That perfectly solves my doubt.

[ben_s]

On thought on underpaying studies from the participant perspective:
If I have a study that advertises $15/hour for a 30 minute study, as a participant I would expect to earn $7.50 for that study after spending about 25-35 minutes actually completing the study. If it now takes me an actual 45 minutes to complete the study, and I know I did it in one go, without any distractions, the hourly rate drops to $10. This is still above the minimum that prolific sets, but as the participant I would consider this “underpaying” because it is 50% less than the advertised rate based on which I agreed to the study.

I would love to see some data on median completion times compared to the advertised estimates, and how prolific is addressing researchers who consistently underestimate the time it actually takes, even if they still pay above the prolific minimum.

[Veronica]

Hi again Ben!

You can find more on Prolific’s guidelines and actions to tackle the problem on this page. The major guideline is to run a pilot to get an estimate of the completion time before running the proper experiment. Of course, if you end up being a participant of one of those pilots, it is more likley to see an incorrect “advertised” completion time! You can contact the researcher to have clarifications and ask for adjusted payments, knowing that if you took longer than the others, they may not give you any additional payments (but this should be explained to you).

Also, Prolific makes the researcher aware if his/her study does not have an appropriate pay with a warning on the reward per hour asking for adjustments in the payment. Adjustments should be paid via the “Bonus payments” tool, following a suggested procedure. As far as I know, researchers who underpay studies can be subject to the suspension of their account.

If you have any suggestion on how to eliminate underpaying studies, you can also tell them here! I’m sure the Prolific Team will be happy to take them into account.

Anyway, I have no data about what you ask but… still, hope it helps!