Disagreement with a participant - who is right?

I had a bit of a disagreement with a participant and wondered who is right. Here is the case:

My study was a simple questionnaire survey. I included an instructed response item at a random location in the survey. The item was designed to mimic the other items and read “When reading this sentence, please select extremely” - extremely was one of the response options. The participant failed this attention check and I, therefore, excluded them.

The participant challenged their exclusion saying “According to Prolific’s attention check policy, participants must be given at least two chances to get a correct answer and comprehension checks must be given at the start of the study so participants are not screened out after having put in significant time and effort. In your study, none of these rules were followed”. I think that there is a difference between a comprehension question that requires actual understanding and a simple attention check that merely requires the participant to read the instructions.

The response of the participant indicate that they were clearly training to game the system. They provided some reasonable responses on the first few items and subsequently selected the same answer for all questions. The latter pattern is next to impossible for real answers on these questionnaires because half of the items are reverse-coded.

What’s your opinion: Am I right in excluding the participant or did I misunderstand Prolific’s rules?

hi @Joe_Bathelt , good question! I think it would depend on the length of your study. If less than 5 minutes, one attention check is sufficient and if 5 minute or longer, you would need two to reject participants based on Prolific policy. Here’s the help page- I think you would be usin an “Instructional Manipulation Check (IMC)”

Ok, thank you. The completion time of the participant was under 5 minutes, but the study was advertised as taking 10 minutes. I had based the inclusion of one attention check on the recommendations in Meade & Craig 2012 of one attention check per 50 questionnaire items.
I think there is a bit of a risk of participants like this gaming the system. The response pattern shows that the participants only answered the first few questions and then invested zero effort. However, in this case, I should have paid closer attention to Prolific’s rules.