I ran a study, which went nice and quickly. However, I had a few participants who failed multiple attention checks. Some of them indicated they found the attention check a bit confusing, which is supported by the answers to their checks. The majority of participants (90%) have understood and answered the attention checks correctly. Only 10% had an issue with it. I am not sure what to do with their submissions. What would your advice be?
Thank you in advance. I am looking forward to your reply.
Would you mind sharing the screenshot of the attention check? If you’d prefer, you can send it to me in a private message.
But if it’s likely that the check was actually confusing, then it would be best if you paid those participants since it’s not their fault that they failed.
Here is a screenshot. The experiment is coded such that participants have to move the slider in order to continue, but the default starting position of the slider is 50. Hence the instruction.
I’m not sure it’s completely clear whether the “If 50…” instruction refers to the slider, or the number displayed (40). And I’m not sure whether the main instruction is to move it to 40, or to “move the slider away and then very close to 50”.
It seems like this attention check is a little confusing, and would be classed as unfair. Respondents who failed would have to be paid in this instance. You can read more about our attention check policy here.
Thank you for the reply. I am sorry the attention check is a bit confusing . I will approve these participants’ submissions.
I also had 2 participants who failed 1 out of 6 attention checks. Their data showed they understood the question correctly. What would your recommendation be? Or would you suggest to pay them anyway, because the check would be classed as unfair?
In this case, any rejection based on that attention check would not be allowed. But if it can be shown that they’ve provided unusable data in later questions, then that would be a valid basis for rejection.
I hope that makes sense - happy to help further if you need me
