Attention Checks and 5 Minute Requirement


Apologize for asking so many questions in the forums! Prolific’s strict guidelines coupled with bad past experiences with poor quality data has made me a bit paranoid.

Prolific states “If your study is 5 minutes or longer, then participants must fail at least two checks to be rejected, any shorter studies can use a single failed check to reject.” This is very reasonable IMO. I’m curious if this 5 minute cut off is for the actually time for a submission or the expected/average time for the entire study?

I’m planning on running a ~8min study in the near future. I suspect we will have a number of participants able to complete it in under 5 minutes (we observed this on MTurk previously). I do not want to bog down the experience for participants by adding more attention checks. Currently we have 2. In the case were a participant fails 1 attention check and finishes in under 5 minutes. Is that grounds for rejection? In the case that they fail both, we plan on halting survey progress and asking them to return their submission.


Hi Matthew

I guess that the 5 minutes is the actual time rather than estimated, since we could in theory estimate in our favour.

I don’t think that one is allowed to mix times with attendance check questions as grounds for rejection but if they failed two checks then a quick submission might make their rejection less likely to meet with objection as they may do if you use nonsense questions (please see below).

we plan on halting survey progress and asking them to return their submission.

That sounds fine afaik.

I recommend, by the way, using nonsense as well as IMCs. Nonsense questions require that there be no mid point in the scale and that the respondent reply positively in to the nonsense for rejection and are open to objection, of hte form “I interpreted that to mean…” but they stand out far less than “please select 5” or similar.

E.g. in a questionnaire about worry

I worry that my wrist support cauldron will require heart surgery.

I generally pay but don’t use a proportion of the data factoring this into the cost of performing and online survey, but then I was until about two months ago paid for answering these questions. I still have a considerable amount of Prolific budget left.