I am considering moving my originally designed for University’s ORSEE subject pool experiment to Prolific. My experiment is interactive and coded in oTree. I have run some pilots with ORSEE subjects, where I used a Zoom meeting room as the virtual lab, kept participants in mute, asked them to keep their videos on (no recording, for engagement purposes only), read out the instructions aloud to them, let them access me via Zoom chat box, put them (and myself) into two different Zoom breakout rooms to avoid opposite roles who get rematched with each other every round to see each other on the video during the experiment. I know that interactive experiments are a little tricky to conduct online, but having seen that there is a webcam prescreening feature on Prolific, I am now contemplating whether I can do the same here. So far, from what I have been reading on the “researcher-help” pages, I am leaning towards yes… but I have some questions.
Given this is an interactive study I was wondering, if similar to what we do at research labs in the University, we can slightly over-recruit and promise a show-up fee for extras?
Having the video on means they will see each other. However, I can either ask them to change their names to their Prolific IDs (as suggested on your website) or I can rename them with participant labels, which would make my life easier with monitoring their progress on oTree anyway. I guess no question here - but I just wanted to make sure having multiple people at once over Zoom is an acceptable thing to do.
Given this is an interactive study I was wondering, if similar to what we do at research labs in the University, we can slightly over-recruit and promise a show-up fee for extras?
You could do that, but we would recommend running a pre-study to schedule participants into time slots.. Participants are generally good attending these sessions so there shouldn’t be a need to over-recruit. But, of course, it’s up to you.
but I just wanted to make sure having multiple people at once over Zoom is an acceptable thing to do.
Thank you so very much for your response - this made me feel optimistic and I have spent the last couple of days trying to better understand the feasibility of my experiment on Prolific as well as the logistics. I have a couple of detailed follow-up questions to understand my options better… (Heads up: it’s a long post )
There are two kinds of potential “extras” 1. say if I slightly over-recruit and everybody shows up and 2. say the number of subjects who showed up is not equal to the numbers needed for a session (factors of 4) then I will have to send some of the subjects away - in addition to making this all very clear on the study definition, I am wondering about how to handle the situation during the session.
Specifically, I need a way that would let them participate in future sessions. Should I send them completion links and pay them a show-up fee? How much would be considered a fair show-up fee for such a situation?
For scheduling purposes, my understanding is that neither scheduling tools (Doodle and Calendly) have integration with Prolific to let me use the query string to collect participant IDs via study URL. Although not a Qualtrics user (yet), I have recently discovered that there are ways of creating a scheduling survey - which I thought may be worth trying since then I wouldn’t have to rely on subjects writing their IDs correctly. No questions here other than checking to see if there are any suggestions on that end.
Instead of creating 2 separate studies upon pre-screening for webcam usage, can I simply create 1 study for scheduling & consent explain that the study will take place in a Zoom meeting and give them the option to return? (similar to an approach also explained in that “live video interviews” article you shared)
I think I would VERY much prefer this over a 2-study setup as it would be cleaner on my end, especially because I plan to duplicate this study many times to reach high numbers of participants (each of which I will personally run over Zoom). If I can… some questions about 1 study setup (if not I have a question about a 2 study setup further below):
[1-study] Q1) My understanding is that I would put the main experiment’s estimated completion time and provide participants with a completion link at the end of the scheduling step but hold off approving until after the main session takes place. Let’s say this study is created on Monday but the provided times for the study are for the next two days, can I just put the estimated completion time (e.g. 45 mins or 1 hr) of the study once it begins?
[1-study] Q2) In this 1-study setup, if I send them participation URL with query strings as a message, would auto-recording of participation links still work?
Question about the 2-study setup:
[2-study] Q1) In 2-study setup (1st study being the pre-screening and scheduling study) am I correct to think that the first study is technically similar to the above (1-study setup) scheduling step (expressing the use of Zoom, asking them to pick a slot, getting consent, and giving them a return option) and the only thing that differs is that this study will have its own completion time and payment?
Thank you so very much for your response - this made me feel optimistic and I have spent the last couple of days trying to better understand the feasibility of my experiment on Prolific as well as the logistics.
I’m glad this has been helpful! And don’t worry about the length - I love hefty questions!
Specifically, I need a way that would let them participate in future sessions. Should I send them completion links and pay them a show-up fee? How much would be considered a fair show-up fee for such a situation?
Yes, for those who show up, but there is no space, you can send them a completion code or URL. We don’t have a policy on what would be considered fair in this scenario, so it’s really up to you.
As long as it at least compensates participants for any time they spend waiting proportional to the promised reward per hour, it should be fine. Please make sure that it’s clear to participants that there may be a chance that they don’t get a spot, but a show-up fee will be available.
No questions here other than checking to see if there are any suggestions on that end.
I believe both Calendly and Doodle allow you to add input fields to your event page. So, if you’re unable to get the Qualtrics survey to work, you can ask them to copy and paste their participant IDs. Unfortunately, there’s no automatic way to do this currently.
Instead of creating 2 separate studies upon pre-screening for webcam usage, can I simply create 1 study for scheduling & consent explain that the study will take place in a Zoom meeting and give them the option to return? (similar to an approach also explained in that “live video interviews” article you shared)
Yep, you can do this. But make sure you leave enough time between the scheduling study and actual time slots to give participants enough notice.
My understanding is that I would put the main experiment’s estimated completion time and provide participants with a completion link at the end of the scheduling step but hold off approving until after the main session takes place. Let’s say this study is created on Monday but the provided times for the study are for the next two days, can I just put the estimated completion time (e.g. 45 mins or 1 hr) of the study once it begins?
Just to be clear, are you asking whether you should put “45 mins or 1 hr” as the estimated completion time for the scheduling study?
In this 1-study setup, if I send them participation URL with query strings as a message, would auto-recording of participation links still work?
Are you aiming to send via the message system, or at the end of the scheduling study?
The only thing that differs is that this study will have its own completion time and payment?
That’s correct. But, because this first study is very short, you can reward a small amount and therefore test more participants.
Thank you Josh, these are very helpful! To answer your questions:
Yes! So this was about 1-study approach (create 1 study for scheduling, put scheduling tool’s URL as study URL, put info about study on scheduling tool and when the day&time comes send them a link for the experiment) so I was wondering if for that “scheduling” study I should put the estimated time it takes of the main experiment. (my understanding based on part 4, 5 and hints & tips here but I may be incorrect…)
I am aiming to send the link via the message system (if query string would work) so as to be able to keep track of IDs automatically…
Also wanted to mention, I am currently toying with the idea of scheduling encompassing multiple days (and send reminder message maybe a day or 12 hours before the main experiment), again to make the process as mistake-proof and frictionless as possible on my end but if there is any reason to avoid that, I can do it on a daily basis too.
Yes! So this was about 1-study approach (create 1 study for scheduling, put scheduling tool’s URL as study URL, put info about study on scheduling tool and when the day&time comes send them a link for the experiment) so I was wondering if for that “scheduling” study I should put the estimated time it takes of the main experiment. (my understanding based on part 4, 5 and hints & tips here but I may be incorrect…)
Yep, you’ve understood correctly. Just put the time for both the initial scheduling and the main study (e.g. initial survey and scheduling = 15 mins and interview = 45 mins, so the study should say 1 hour).
I am aiming to send the link via the message system (if query string would work) so as to be able to keep track of IDs automatically…
I’m just checking with our technical team on this. Let me get back to you as soon as I know
Also wanted to mention, I am currently toying with the idea of scheduling encompassing multiple days (and send reminder message maybe a day or 12 hours before the main experiment), again to make the process as mistake-proof and frictionless as possible on my end but if there is any reason to avoid that, I can do it on a daily basis too.
It’s totally fine for you to send reminder messages to the participants in advance of the studies, and you can choose the frequency at which you do it
Let me know if there’s anything else I can help with!
Looking forward to hearing back about the query string via messaging --I will edit my code accordingly-- and if that is possible, it would make the logistics so much better!
Otherwise (at least for now ) I think I know all I need - will get in touch if there is anything else. Thanks again!
Unfortunately, it’s not possible. Query strings only work when participants sign up to the study from their Prolific dashboard.
If they access the URL via a message, our system won’t recognise their ID so the {{PROLIFIC_PID}} parameter won’t be replaced in the URL.
It would work if you manually replaced the parameter with the ID of the participant who is receiving the message. But, this would be quite an involved process, so it’s far from ideal
Sounds good, thanks for checking! Sounds like I may need to stick with a 2-study approach (1 for scheduling 1 for the main experiment) then if I want to rely on auto capturing of the Prolific ID… will think about it. Thanks again!
)
) I think I know all I need - will get in touch if there is anything else. Thanks again!