Things to consider for a study with only participants from previous studies

💡 Prolific Hub
1

Dear all,

I want to ask for help and ideas about how to best implement a survey, which includes only participants from a previous study of mine. Here are the details:

I conducted a study on prolific about a year ago with ~1000 UK participants. Now I want to do the same study again with the same participants. The survey is quite short, it includes just one factorial experiment (i.e. a fictional scenario, where participants give their opinion) and a couple of other questions. Here are my specific problems and questions:

What can I expect with regard to the response rate?
Prolifics study draft calculator says that from the 1000 participants of the original study 880 are still active here. Any educated guess about what to expect as response rate would be helpful.

How much does payment matter for the response rate?
The problem is that the original study was implemented as part of a larger survey with an above average hourly rate of compensation for respondents. If I apply the same rate to the new study, which is only about 3 minutes, the relative compensation per hour looks fine, but in absolute terms participants earn only 0.45£. I could overpay even more and go up to 0.5£, but I was wondering if generally such short studies are unattractive to participants.

Any ideas on the technical implementation?
For the new study it is important that every participant receives the same experimental condition as in the previous study. I am using LimeSurvey for hosting and survey design and the only solution, which I found as of now was to ask participants for their Prolitic ID and then condition the experimental scenario for each unique ID. With 1000 participants in the sampling frame, this should be a lot of coding work and is also error prone if participant enter the wrong number. Working with ULR parameters may help, but I do not know how to implement this and there appears to be no integration guide for LimeSurvey.

Of course I am also thankful for every other idea or issue that you think is important when running a study with the same participants as in a previous study. Links or literature suggestions for methodological and statistical work that covers such study design are also very appreciated.

Best,
Felix

2

Any ideas/tips? @Community_Leaders :slight_smile:

3

Hi Felix

I am afraid I can’t help much but…

With regard to respond rate
Someone recently mentioned responses falling off after about half of the active repondents had responded. But this depends upon a lot of things, particularly your demographic. If they are students then I can imagine a lot of them would come but if they are professionals then less so.

Yes, I think think that lower paying studies are less popular even if the hourly pay is high. The email that they will recieve will tell them how much they are going to get paid and if it is only .5UKP then they may not be bothered to sign in (or go to their computer if that is a requirement).

I don’t know how much payment affects response rate. Perhaps one day Prolifc can tell us. But I think it would depend on many factors, again the demographic and opportunity costs. Students in the UK could not even get much part time work (when I was a student at least, it may be different now post Brexit) so their opportunity costs are low. Students in the US have part time jobs. Professionals have jobs and may have families, so all sorts of demographics will be attracted by the incentive in different ways.

With regard to the technical implementation…
Since presumably you know which Prolific ID was in which condition in the initial survey, rather than branching on the ID that participants input into LimeSurvey, you can set up a number of studies, equal to the number of original conditions, with a white list of Prolific IDs, which direct them to the LimeSurvey condition/survey.

People like to feel meaningful, so telling them that they are being invited back to further research, etc (to the extent that you can tell them prior to the experiment) may be more attractive than the 50p.

Tim
Expat in Japan

2 Likes
4

Dear Tim,
thank you very much for your feedback. Although you claim that you cannot help much, you actually helped me in all aspects of the problem. Your suggestion on how to improve the technical implementation is brilliant and will save me a lot of work. :trophy:

Best,
Felix

3 Likes