Hello!
As mentioned in another post, I have been trying to run online multiplayer experiments with big groups (at least 8 at a time). However, there are a lot of complications and moving parts involved in doing so. The main aim is to get a lot of people to come in at the same time so that people donāt have to wait for the groups to slowly fill up.
It was recommended to me that I follow a process similar to that of video/audio interview. This involves creating a study, where participants are directed to a scheduling tool to schedule when they will participate. In the meantime, they have to be given a code so that they can submit the study and do other studies between now and then.
I decided to do without the scheduling tool, they can have privacy issues (even Doodle encourages - although does not force - participants to put in some sensitive information such as names and emails). Instead, I do everything in one link with my coding. Participants are sent to a page that tells them that the study will start in X amount of time and to come back here on time before it starts. It informs them about the code to be able to continue and reminds them that they have to actually complete the study to receive payment.
Nevertheless, we are left with quite a few moving parts and complications:
Participants might lose the link or not get a reminder to come back to the study in time. (sending a reminder message 15 mins before might not reach them as they wonāt get a notification in the browser add-on).
Participants might not return their participation if they didnāt complete the study. Itās never nice to have to chase participants to return their study, nor report them if they refuse to do so.
This can still seem weird to participants.
They might not plan well and start another study just before ours.
They might over-plan and wait in front of our study for a whole chunk of time before it starts.
I was wondering if Prolific might consider removing all these moving parts by creating an option do to studies where participants register for a later start. The experimenter would set when to publish the study and when to release the actual link to the study. The participants would register for it. No need to do stuff by providing a code and it wouldnāt block them from doing other studies. They get a reminder a few minutes before the start, where they can click and get to the actual study. This would make things a lot simpler for experimenters and a lot clearer for participants.
(bonus points if you can replace the scheduling process for the video and audio interviews, but I am just interested in this fix for multiplayer experiments, I donāt need people to choose amongst different time slots, just one time to come back to later in the day.)
This is a really great idea @Samuel_Dupret! We know how fiddly the current video/focus group interview process is. Any ideas on how to improve it are very welcome!
Iām working on a process to ensure that great community ideas get in front of the right team who can work on implementing them. Iāll keep you updated on where this idea could fit into our future product roadmap
I decided to do without the scheduling tool, they can have privacy issues (even Doodle encourages - although does not force - participants to put in some sensitive information such as names and emails).
Just a note on this: our policy on collecting sensitive data when arranging these types of studies
Collection of personal information is not usually permitted on Prolific, so participants would need to be assured that any personal details if at all collected (e.g. Skype usernames) will only be used to contact them for the interview, and deleted as soon as this purpose has been fulfilled.
I have been doing some testing with multiplayer scheduled experiments. I stand by my ask for a system handled by Prolific. Here are a few new things I have discovered (or rediscovered).
One thatās just general to multiplayer studies where we try to create groups (e.g., 10 participants) and we canāt run groups with fewer participants. If we cannot form a group, we send a message saying we are sorry to participants and offering a small bonus in exchange for them returning their participation. Some participants are annoyed by this and demand that they get to participate.
Some of my comments refer to stuff that a Prolific internal system will solve however:
allowing participants to enter the code early so that they can do other studies whilst waiting creates weird overall pay/time scores. Some participants only give in the code at the end, so they count as having participated for a lot longer than the actual study (say 30min wait before, and 30 more minutes of the study, instead of just 30 minutes for the study).
Despite providing the code for them to enter early, some participants refer to the waiting time as time that counts. Sometimes indirectly, such as āI waiting X amount of time, and didnāt get to participateā, etc. The fact that the countdown/time wait happens on the experimenterās side instead of through an official Prolific system means that participants might perceive the waiting time as ātime they worked forā.
Also, the auto approval feature messes up this whole system.
Iām piloting a group Zoom-call study. In case itās helpful to you, here is my current design: In my Study 1, I have participants indicate their availability for pre-set appointment times for the upcoming two weeks (this is done via Qualtrics survey). Afterwards, I send them a message via Prolific to announce the appointment time I have chosen for them (based on when most respondents are available). Then, I publish Study 2 (with Custom Allowlist) for which the study URL is the actual Zoom URL. Toward the end of our group Zoom session, I send everyone (via the Zoom chat box) a URL to another Qualtrics survey which automatically routes them to the Prolific Study Complete page when they complete that survey.
However, Iām running into the issue of participants not showing up for their session; up to 30% of participants donāt show up. (I am forming groups of 3; if I schedule 15 participants, I should have 5 groups. But if one person doesnāt show up, I have to send two people home with a prorated payment sent via the āBonusā feature after they Return the study in Prolific.)
Do you have any thoughts on how I can decrease my rate of no-shows? Currently, the day after they complete Study 1, I send an announcement of their appointment date/time. I also send a reminder message the day before their session, the morning of their session, and 1 hour before their session (all messages via Prolific). Also, I publish Study 2 immediately after announcing it (the day after Study 1). That way, Study 2 stays visible in their queue as a visual reminder (the first line of the description is an instruction to not start the study until their appointment time).
Hi Moses - This seems like a smart way to design a scheduled experiment. However, I was wondering if you had found any further improvements to get around the 30% attrition rate you experienced?
Also, in reference to Samuelās original post in this thread, has Prolific introduced any kind of proper scheduling feature onto their platform yet?
Iām glad you asked, otherwise I would have forgotten to follow up on this.
Iām not aware of any kind of scheduling feature built into Prolific yet, but the Prolific team seems really good about incorporating user-feedback into their updates. Anyway, I eventually realized I didnāt have to schedule people in advance because there are thousands of Prolific users online at any given moment. At that point, my new plan was to just launch my studies on Prolific without any scheduling (the studies had pre-screeners built into them). The whole idea of scheduling the participants was a cognitive artifact of on-campus data-collection, which naturally would require scheduling. But online, thereās no need for that because there are thousands of users at any given moment, waiting āat the doorā to enter my online ālabā.
However, when I tried that new method, I noticed I was still only getting less than a handful of participants joining per hour, and they were all spread out over the hour (my task required teams of 3 members). Eventually I discovered this page on Prolificās Help Centre, which mentions a rate-limiting mechanism that is built into Prolific (I imagine itās to help make everyoneās sample more representative, i.e., to prevent the top percentile of Prolific users from getting all of the studies). I was able to email Prolificās support team to request the rate-limiting mechanism be removed from my study, because I needed multiple people to sign up simultaneously. Once I did that, I was able to activate my studies and get as many participants as I wanted (typically I only ever needed between 6 and 15 participants per session) within 5-7 minutes of activating the study.
In short, my routine was to activate a study 10 minutes before the start-time advertised in the study title/description (after making sure to update my custom block list to disallow any prior participants from participating again) while I was waiting in my Zoom room (the āstartā button of the study linked to the Zoom room). Participants would join the Zoom waiting room and I would message everyone in the waiting room (Zoom allows me to do that) and tell them we would start shortly (of course, I had to make sure the participantsā pay rate included the time spent in the waiting room). Once I had a sufficient number of participants (this generally happened a couple minutes before the scheduled start time), I would admit people into the Zoom room and immediately change everyoneās name in Zoom to something anonymous (per my protocol) and start my session.
Weāve not implemented it yet, but if you check out our new Apps & Integrations page, youāll see that weāre laying the foundations to make it easier for you to use tools like Calendly etc, within your Prolific workflow
I attempted to run a live Public Goods Game with 18 participants on M-Turk, previously. I ran into a similar problem as you, whereby some participants joined but not enough for the game to run. I was then left with disgruntled participants who were left waiting while the game waited in vain for enough others to join.
Iāll definitely look into seeing if I can use Prolific if they are able to remove this rate-limiting function you mention. Thanks again for all of your useful advice and insight; itās been really useful!
Hello. I just wanted to say that I would also very much appreciate such a scheduling option.
I would like to announce a study on a calendar within Prolific, and people who match my criteria can reserve a spot (and cancel their signups to open slots for others). I would also put a reward.
