Since it is not possible to use prescreeners when conducting a study with a representative sample, have there been any issues with data quality when a representative sample is used? For example, if I were to use prescreeners, I might restrict the pool to participants with a high approval rate, but wouldn’t be able to do so for a representative sample. Has this been an issue at all when a representative sample is used?
Dear Dani
I don’t know how to respond really so I didn’t but it is been four days so I will say what little I know or can surmise.
The fact that it is limited to only studies without prescreeners strongly suggests that in some demographics (ages/races) Prolific is scraping the edge of the barrel so a representative sample may presumably include those with attention issues.
But, about two weeks ago it was announced that we are now allowed to use nonsense attendance checks, which I think can be used to provide very effective attendance checking.
(The previous attendance checks, such as “Please select ‘very appropriate’,” were rather easy to spot, allowing for the possibility of selective attendance. Nonsense attendances checks can however be made to look almost identical to the other items so that you can be sure who is attending.
E.g. in a self esteem survey alongside
At times I think I am no good at all.
One might include
At times I think I am really bolognese.
I am not sure if the above is sufficiently nonsense (I modified it, so I now think it is) and there is the additional rule that middle term in any Lickert response scale not be included, but with inventiveness I think that data quality should be possible to ensure.
That said, the inclusion of attendance checks affects the data, according to an article linked from the Prolific FAQ.
I am sorry I can’t be of more help – and indeed that I am not directly answering your question.
Tim
1 Like
Dear Tim,
Thank you so much for your thoughtful response! I understand there is no easy answer, but your suggestion regarding nonsense attendance checks is helpful, and I appreciate the research you pointed to on the possible issues with those checks in general.
Dani
1 Like