Hi there,
we are setting up a study with an early attention check (immediately following the Welcome screen) in order to screen out inattentive subject (and not waste their time). The attention check question asks subjects to select “strongly disagree” for three statements (see screenshot attached). The full study takes about 10 minutes to complete, but only 1 minute (or less) to get beyond the attention check (or fail it).
I have 2 questions regarding this:
(1) Do the 3 statements for which the participant has to provide the correct answer (“strongly disagree”) count as one attention check or three? In other words, is failure to click “strongly disagree” on all three statements enough to reject (not pay) the participant? As mentioned above, the time spent on the study when the subject is rejected would be less than 1 minute.
(2) What is the best way to deal with participants who have failed the attention check? Should we use the new redirect-function (with separate completion code) and then manually reject those participants? Or should we not provide a completion code at all to subjects who failed the attention check and ask them to “return” their submission?
Many thanks for your help!
Best,
Arno
Hello Arno
Welcome to the forum 
First of all, the question (or questions) may well fall foul of Prolific’s requirement that attention checks be in the question rather than the description of the question. I guess you could argue that there is no question in the words “I am open to new experiences,” followed by the Lickert scale unless they read the “Information” above but bearing in mind that Prolific’s attention check policy states that the following
The colour test you are about to take part in is very simple, when asked for your favourite colour you must select the second last option.
What is your favourite colour?
Red
Blue
Green
Orange
Brown
is in appropriate because, according to Prolific in the above article
- The question being asked does not reference the instructions and is therefore open to misinterpretation (i.e., a participant may just answer with their favourite colour)
- This does not test whether the participant has paid attention to the question rather than the instructions
(and another reason)
The fact that your items do not contain question marks does not imho make it sufficiently different to the above for it to be deemed an acceptable check but I am not entirely sure. You may wish to ask Support or others on the forum but my feeling is that it will not be considered acceptable.
There are are also problems with instantaneous rejection using attention checks.
However, what you can do is ask the participants to confirm their screener information. If they are not paying attention, and or replying at random, then they will get some of this different to the information that have registered with Prolific. If you use exactly the same wording as the Prolific about you questions and answers, you can instantaneously reject the participants by routing them to a dead end page where they are asked to return their submission.
Others, and importantly support=Prolific may feel differently about your question.
I am sorry. I wish I could respond with a “Yes that questions is fine” since you have clearly put a lot of thought into it.
Tim
Hi Tim,
thank you so much for your quick reply and your honest opinion about our attention check!
I see that the question in its current format may not be appropriate according to Prolific policies. I will reword the question so that the attention check is in the question itself and not in the instructions above.
I will also ask participants who have failed the attention check to (voluntarily) return their submission instead of directly rejecting them.
Best,
Arno
