I’m currently running a live experiment which takes approximately 1 hour to complete. When I run each experiment, I need to spend 30 minutes training my participants individually before they go and finish the study. In order for me to prorperly train each participant, I can only run one person at a time.
The way my supervisor and I set up our study on Prolific was by opening one participant place at a time. After a Prolific participant enters their submission for the experiemnt, my study then becomes unavailable to all other potential participants. At this time, participants then join a zoom call with me (where I train them on how to comeplete the task). After they’ve completed the study, I increase the place by one again.
The problem I’m having, is it often takes over an hour to have a single participant sign up and stay for the study. I don’t know if it’s because people are weary to speak with someone in real-time while completing the study, or it’s because I’m only allowing one participant to sign up at once, but I’m not having luck getting people to sign up once it’s active. My supervisor and I are trying to brain storm other ways of setting up a live experiment on prolific (e.g., using a calander where pariticpants sign up for specific times), but there doesn’t seem to be an easy solution. Does anyone have any other methods that have been sucessful for them to run live experiments using the Prolific platform?
The problem is I think that notifications are not going out immediately after you add one more place to the study, but for some reason there is vetting going on, and that is delaying the arrival of new participants (assuming you are not attempting to recruit a limited demographic, and that you are paying okay).
I am not sure if you can get support to immediately auto approve your study but you could try asking.
If not the other way would be as you say to use
or
to schedule but then the problem is (and has been reported) that participants get busy, or forget or for other reasons do not show.
If you have money to burn you could get some sort of waiting room going so that you would open the place before you are ready for it attempt to get people to wait.
Or, brainstorming, if you could get special permission from support to ask personal information in the form of participant phone numbers so that you could sms. Or perform an initial precruitment study to get participant permission, use the anonymized prolific messaging system to contact participants, and perhaps even contact more than one participant with a first come first served basis (if you could get participant permission to do so).
There are others who have done live Zoom /Webex studies using Prolific, mainly using scheduling I think. I have not done a live study and only hear about them when something goes wrong. Perhaps someone will chip in.
I’m not fully sure I understand the issue you are having, but is it because you’re trying to have this training with them as soon as they sign up? We have studies that include interviews with folks, which isn’t a “live experiment” but does involve them finding time to sit with us for an hour (we do Zoom but I don’t think venue matters for your question). We also only release a few at a time because we don’t have the team capacity to do that many at once.
Again, with fully transparency that I’m not sure if this helps, but the way we do it is indeed with a scheduling system, in two stages. First, we have a Prolific study that’s just essentially the sign-up form, and in that form, they tell us their availability. (You could also do it with them selecting specific times through something like Calendly, but we don’t do that because managing that with our multiple team members got messy.) Prolific auto-pays them a small fee for doing the sign up (even if said sign-up results in them not having any availability during our times). Second, we schedule the interview and do the study. We then pay people the compensation for that study time via bonus payment.
Notably, this introduces some other complications (e.g., sometimes getting no-shows), but it’s worked well for us once we got a system down (including sending folks reminders).
Thank you for taking the time to respond to my inquiry! It was really helpful reading your thoughts and suggestions on this topic. It didn’t occur to me that there may be a time delay after I re-activate the study, but that seems to be most likely what is going on with my long wait times. I don’t know if support would be able to help with immediately approving my study, so we’re leaning towards using the Calendly platform you suggested for our study. Hopefully we have more success with this method, but I know that live studies done online can be pretty tricky.
@rslbliss Thank you for your help with the issues I’m having with my study. It’s been great reading about your method for recruiting participants.
So yes, I believe you correctly understood the basic set up of my study. Right now, I’m trying to have the training start as soon as they sign up on Prolific (to avoid scheduling participants and then getting no-shows). After participants finish the training component with me, they immediately go and complete the rest of the study on their own. The problem is I don’t get participants for over an hour doing it like this.
From what you’ve described, I think your method of scheduling interviews is exactly what I need to do in my study. I just have a few questions about the logistics of your sign-up study/interviews, if you don’t mind.
Just out of curiosity, how much would you recommend paying participants to sign up for a time slot? Right now, participants who complete my full 1-hour study get 9 pounds, so I’m trying to figure out how much would be a good amount for the small scheduling component beforehand.
What information did you need to collect from participants in the initial sign-up study. Also, what was the best way to make sure they were able to access your Zoom session after they signed up?
How often did you run into problems with no shows? Was there some sort of a disclaimer you used in your sign-up study that helped participants understand the importance of showing up to the time slots they selected?
I am in the U.S. so I can using dollars, we generally pay people between $0.50 to $1.00 for sign-ups.
I’ll send you a direct message with some of the language we use. Notably, we didn’t have them sign up for a time because our processes made it hard to know pre-available times (because we have multiple interviewers, etc.). So we’d find a time based on their availability overlap with our team, and then I’d send a message to them once we found a working time. I imagine it would work much easier if you only had one interviewer and used a Calendly link. We sent people the Zoom link via the Prolific messenger. (We simplified this process a lot by setting up our own internal Qualtrics form that when we enter interview information, it sends separate formatted messages to our study manager, our interviewer, and the interview participant via Prolific at the same time.)
No, we didn’t include a disclaimer about showing up - maybe we should. For context, our interviews are often about controversial topics, so we may have more no-shows than normal. In the early days, we would have up to half not showing. However, we added a process later where we’d send people reminders of their interviews about a day before the interview, and that greatly increased people’s attendance, so now probably only 1-2 out of 10 won’t show, which is certainly inconvenient but only costs us about a dollar per no-show.
