I had a bit of a disagreement with a participant and wondered who is right. Here is the case:
My study was a simple questionnaire survey. I included an instructed response item at a random location in the survey. The item was designed to mimic the other items and read “When reading this sentence, please select extremely” - extremely was one of the response options. The participant failed this attention check and I, therefore, excluded them.
The participant challenged their exclusion saying “According to Prolific’s attention check policy, participants must be given at least two chances to get a correct answer and comprehension checks must be given at the start of the study so participants are not screened out after having put in significant time and effort. In your study, none of these rules were followed”. I think that there is a difference between a comprehension question that requires actual understanding and a simple attention check that merely requires the participant to read the instructions.
The response of the participant indicate that they were clearly training to game the system. They provided some reasonable responses on the first few items and subsequently selected the same answer for all questions. The latter pattern is next to impossible for real answers on these questionnaires because half of the items are reverse-coded.
What’s your opinion: Am I right in excluding the participant or did I misunderstand Prolific’s rules?