First time Prolific user (research-side) here! I’d like to inquire as to how many of you experience the following problem
First some context.
When reading about using MTurk for recruiting participants, I learned about Prolific and I proposed to make use of it for my dissertation. My supervisor approved and I was delighted. To get a better understanding of what participants see and experience, I also made a separate Prolific (participant) account In that time, I have come under the impression that we (as researchers) are allowed to reject participants’ submission in the event that attention checks are failed. And indeed, this is what I indicated to my research ethics board when I described what circumstances might trigger compensation (and the lackthereof).
HOWEVER, my research ethics board has responded to me indicating that I cannot withold compensation on the basis of failed attention checks! I responded and cited that withholding compensation on the basis of failed attention checks is written into Prolific’s terms of service. My wise supervisor also suggested that if researchers don’t withhold payment on the basis of failed attention checks, it might harm Prolific’s reputation for integrity.
Armed with these two points, I figured I had a good chance at convincing the REB officer in charge of my case. To my dismay, however, I have been told that since Prolific services academic and non-academic research, liberties available to some researchers (e.g., witholding compensation on the basis of attention checks) aren’t necessarily ethical for academic researchers.
Anyway - just wondering if anyone else had this issue/problem.
Conversely, if you are an academic researcher and your ethics board has approved of your study (in spite of the fact that you indicated rejecting submissions on the basis of failed attention checks) will you please, please let me know/leave a message below/contact me.
First time Prolific user (research-side) here! I’d like to inquire as to how many of you experience the following problem
First some context.
When reading about using MTurk for recruiting participants, I learned about Prolific and I proposed to make use of it for my dissertation. My supervisor approved and I was delighted. To get a better understanding of what participants see and experience, I also made a separate Prolific (participant) account In that time, I have come under the impression that we (as researchers) are allowed to reject participants’ submission in the event that attention checks are failed. And indeed, this is what I indicated to my research ethics board when I described what circumstances might trigger compensation (and the lackthereof).
HOWEVER, my research ethics board has responded to me indicating that I cannot withold compensation on the basis of failed attention checks! I responded and cited that withholding compensation on the basis of failed attention checks is written into Prolific’s terms of service. My wise supervisor also suggested that if researchers don’t withhold payment on the basis of failed attention checks, it might harm Prolific’s reputation for integrity.
Armed with these two points, I figured I had a good chance at convincing the REB officer in charge of my case. To my dismay, however, I have been told that since Prolific services academic and non-academic research, liberties available to some researchers (e.g., witholding compensation on the basis of attention checks) aren’t necessarily ethical for academic researchers.
Anyway - just wondering if anyone else had this issue/problem.
Conversely, if you are an academic researcher and your ethics board has approved of your study (in spite of the fact that you indicated rejecting submissions on the basis of failed attention checks) will you please, please let me know/leave a message below/contact me.
In general, I think attention checks are for research quality purposes, and making sure your participants are following instructions and whatnot. There’s always a chance that people won’t be paying attention for a few seconds or miss something and you can exclude them. But I don’t think that should deprive them 100% of compensation. You could, however, add a bonus and say that there are questions that will result in additional bonuses. My two cents.
It is possible to select people with a very high approval rate - thus avoiding those who might fail attention checks.
Personally I agree with your ethics board - I think it makes sense to just compensate everyone - very few people will actually not do the task properly, especially if you select based on approval rates, and those that do fail could have done so due to misunderstanding instructions etc.
In terms of your own data you can still have those attention checks as criteria for removing participants from your analysis.
Another option might be to have a way where the study stops early if the participant fails a check and then is paid for the time put in up to that point (this can be done through bonus payments) I am not sure if this is within prolific’s terms but if it is this may get round your ethics board.
I think it depends on the nature of the attention checks and if not paying participants is proportional. One can argue that we as researchers have an ethical obligation to obtain valid results and are careful with how we spend public money. Attention checks are often necessary to ensure that we get valid data from online research, because we have less control over the environment of participants. If these attention checks are very low-level, e.g. ‘select “Do not agree” here’. Then, I think it is reasonable to not pay the participant, because they clearly disregarded the basic instructions. However, if the attention check is more difficult and may depend on the participant’s ability, then not paying is arguable too harsh.
In general, I think we should apply similar standards as we would do in the lab. If I had a participants showing up for a study when they are clearly hungover, I would send them away and not pay them. If they struggle to complete the task, I would pay them and try to fix the problem on my end.
So yes my ethics board forbids me from rejecting submissions in spite of failing attention checks and the simple reason is because I am paying participants to compensate them for their time and nothing else - so to withhold payment when they spent the time would be unfair and unethical. It does not matter if they weren’t paying the full attention 100% of the time during my experiment.
Also in my mind attention checks are there to help decide (ideally based on preregistered criteria) whether to include the data in analysis or not.
A few things that might help:
Use Prolific’s approval rating pre-screen - set to at least 95%. This will result in it being less likely to get bad faith participants - I find I maybe get 1 Participants out of a sample of 80 who I would deem as not paying attention during the majority of the experiment .
Report any participants who fail you attention checks to prolific - this can help prolific remove bad faith participants
Do a Presceen study or remove participants at the moment they fail attention checks and only pay them for that amount of time - this would need to be approved by your ethics of course and be worked out with prolific how you actually go about this.
You can’t/shouldn’t choose not to compensate them (even if they fail an attention check in the computer task).
Though attention checks might be a good way for Prolific to keep track of their participant pool, I don’t know how well that would translate to withholding compensation for participation (even if the participate wasn’t paying attention the whole time, they still spent time on the study).
There are also issues beyond attention checks – participants could pay attention but still put little effort into the study.
As one general suggestion, if you want to convince your IRB of the need for rejections you need to make a case that it is an ethical thing to do. Saying that the possibility of rejection is in Prolific’s TOS and that not rejecting poor quality responses could harm Prolific’s reputation may not be convincing counterarguments to an IRB because they don’t relate very directly to concepts of ethics (e.g., avoidance of harm, beneficience, justice, autonomy, etc.).
But in practical terms… For the types of attention checks that Prolific permits as bases for rejection (i.e., very basic/easy ones), you’re unlikely to get many respondents who fail them (in my experience, maybe 1% or so?) Your easiest route here is probably just to pay those participants and report them to Prolific if you really think they need to be flagged. As others have noted, you can still then exclude these participants from your actual analyses.
(Personally I reject submissions only where there are multiple reasons to conclude the participant hasn’t made a reasonable attempt - e.g., a failed easy attention check plus a response time 1/3 of the median or less).
If most academics are unable to ethically reject participant payments and are forced to approve payments regardless of performance, I think we need another way to flag poor participants in a simple way. For example, “Approve payment but do not recommend” type feature so that payments are not implicitly tied to approval ratings.
While the analogy of bringing someone into the lab makes sense to me in some ways, I think that the lab context is a little different from the online context. If a participant takes the time to communicate with you, set up an appointment, come to your lab, speak with you personally, shows up on time, signs the consent form, etc., these are a sort of check in themselves. They have already shown a certain level of commitment-- for this, they should be compensated. They [in most cases] cannot be simultaneously working or playing a game, and they can’t be a bot.
On the other hand, I see where the ethics boards are coming from in not wanting to penalize people who have put in the time. So really this is a signal detection problem: do you accept bad data and/or pay bad actors, while also giving them legitimacy on the site, in order to make sure that every honest participant gets paid? How far are you willing to go for this principle? When does it become obvious that someone is a bad actor?
I haven’t answered these questions perfectly myself. @Mikeology has a great suggestion though.
