I recently used prolific for the first time to conduct a survey and have to say I’ve been very impressed! However, I also signed up as a participant to see the experience from the other end, and have found a couple issues that I would love to hear some thoughts on.
I’ve only taken 6 surveys so far as participant, so this is by no means a representative sample, however, as a researcher I was struck by two things:
I was not able to complete any of the surveys within the time indicated by the researcher (and I was trying). Maybe I’m a very slow survey taker, or I put in to much effort, but across all 6 surveys it took me anywhere from 30-100% more time than indicated. Combined with fairly low rewards that is not very good motivation to continue to put in real effort.
Less than ideal instructions
This is connected to issue number 1) since with unclear instructions it takes more time as a participant to figure out what the participant should be doing. This was not an issue across all surveys, but about half of the surveys so far had at least some instructions that were unclear, overly verbose or repetitive. For example: One survey outright threatened to withhold the reward if I got the questions wrong in big bold red letters above each question. This was also a survey that contained multiple attention checks, so I had to read those big bold red letters every time just to ensure that there wasn’t another attention check hidden within that warning.
As a researcher, I know that it is never possible to write 100% perfect instructions, and that estimating how long someone will take to complete a survey is tricky. Nevertheless, I take great pains to err on the side of overestimating how long people will take on average, and writing the best instructions possible, including testing and iterating on instructions multiple times before launching a survey.
All this to say that I was surprised by the experience I’ve had so far as a participant and the seeming lack of regard for participants’ time and experience with instructions that some surveys exhibit. A major concern arising out these issues is that, by having too many such surveys, we discourage a lot of people from participating that would be required to have access to a truly representative survey panel.
Would love to hear some other thoughts and experiences. Maybe I just go unlucky with the surveys I took?