I would like to run a study that requires a fairly large group of people (e.g., between 10 and 20) to make decisions simultaneously. This experiment is coded in oTree.
I understand that one possibility is to schedule sessions on Zoom. After reading other threads, I still have some confusion about how non-Zoom options work. As far as I understand, these are the possibilities (I have some questions in bullets):
- Remove rate-limiting. Then, theoretically, I could get the required number of people quickly without pre-screening anyone for when they are available.
- The study is such that everyone in the group must move at a similar pace through the survey. If people start at the same time, this should happen. But, this means dropouts a few minutes into the study are a problem: I cannot just replace the dropouts, as the replacements would be too far behind the rest of the group. So, my first question is whether it’s possible on Prolific’s end to not release returned spots to new people: once a minute or so has passed, I wouldn’t want to do this anymore.
- Dropouts that happen very close to the study’s start time are ok (e.g. if a person doesn’t consent or they return the study after the first page), as long as the person is replaced quickly. When one removes rate-limiting, is it the case that returned spots get replaced quickly, or does there tend to be a lag?
- It seems for me that dropouts mostly occur during instructions. So, suppose my target is 15 people. I think one possible solution is to open the study to more than 15 people, and if >15 make it through the instructions without returning, I’d randomly choose some to ask to return. Is this allowed? I assume I would have to pay the people that I ask to return the full completion fee?
- Use the protocol for scheduling Zoom sessions, but instead use it to schedule when people will open the study link.
- What’s the most efficient way to do this? Suppose 20 people tell me they are free at 10am on May 25. Do I make a study that publishes at 9:55am on May 25, with the instruction that people should wait until 10am to start, and only invite those 20 people?
- I assume some people would not show up/forget to start the study at the right time. So, again, suppose I want 15 people. One option would be to invite more than 15 people, and only let the first 15 people who start on time complete the study. Anyone after the 15th person would be asked to immediately return it. Is this allowed, as long as I alert participants in advance that these are the rules? Would I have to pay the people who “show up” too late and are asked to return the study?
- I don’t use an initial survey to assess people’s availability. Instead, I schedule the study to be published at a certain time. Assuming people can reserve a spot as soon as I schedule the study, I could specify: the study will be published at time X, you need to click the link at that time given the nature of the study, and you shouldn’t reserve a spot if you are not available at that time.
- Can people reserve a spot for something that isn’t yet live?
Separate from the above concerns, late dropouts are a problem for me because some people’s choices depend on the composition of the group, and it’s bad if that composition changes part way through. So, I am mostly trying to figure out which of the above strategies is least likely to result in mid-study dropouts, as well as getting appropriately high initial attendance. If I’m missing an option, please let me know!