Hello, I am running a survey study where the desired population is “all active internet users who speak english” so I set up my study with very few screeners. When I opened up the survey for my first 50 participants as a test run, those slots filled in less than 10 minutes which was faster than I expected. This means that I ended up with results heavily biased towards specific countries based on time zone.
Does anyone (community or support) have suggestions for how to get better global coverage for my full sample? The simplest option I was thinking of is to progressively add more participants, at different times of the day/weekend to try and get a more global sample. This feels like a bit of a hack but might be a good way to fairly sample the entire Prolific pool.
The other option I was thinking of was to split it into multiple substudies and target specific countries, but I’m not sure how to come up with a good split based on the total pool. This also makes the integration with Qualtrics much trickier as I would have to have multiple submission return Ids. Does anyone have advice for using this method?
I think that you have thought of most of the ways to achieve a solution to your problem already.
Since there are prolific participants that are up at night, I guess I would probably go for the second option of creating multiple studies for each of the countries, or blocks of countries with a similar time zone, that you wish to recruit from.
With regard to the completion code, we are now allowed to make changes to the completion code to create completion codes of our own. Custom completion codes are usually to allow multiple types of completion from the same study (e.g. normal completion vs failed attendance check)
However since custom completion codes are now possible, I think imho afaik it would be possible to use the same completion code for multiple studies so that you do not need to create multiple Qualtrics studies.
I have used the wrong completion code in the past (from a previous study, failing to update my survey software) and I did not any difference in the overall process other than that the GUI warned me that participants were giving the wrong code.
Finally you can see how many participants there are in each country, and then aim for the same percentage (perhaps) from each by stipulating a country and then seeing how the available participants figure changes as you add (and remove) each country
With no screeners it is currently 123,185
But if I add UK nationality then it changes to 37,655 as in the screenshot below
And you could go through all the countries to see the various contributions to participant pool (bearing in mind that some will have dual nationality).
If you do, I would be interested to see the numbers of participants given for each nationality.
I am sorry I can be of much help.
Perhaps prolific will one day offer another type of simple balanced sample of as many nationalities as possible. Or even for each and every screener it might one day be possible to select “balance” so that an equal number of participants are selected from each of the choices on the screener! That would be cool.
You are too sussed for me to be of much help but perhaps someone else with have other ideas.
I tried some things out and the multiple study + country targeting solution looks like it will work. It doesn’t appear that you can manually set the completion code, but when I duplicated my existing study it reused the same code as before so that part isn’t a problem.
Here are some country estimates as of today for participants who are fluent in English (108856 total, these numbers will change). Based on these numbers, splitting into 3 studies with UK, US, and every other country would provide decent coverage of the English-speaking internet, with the notable exceptions of India, Pakistan, Nigeria, the Philipines, Uganda and Egypt:
USA: 34951
UK: 37614
Ireland: 744
Germany: 2313
France: 847
Spain: 1428
Australia: 1383
Austria: 235
Belgium: 436
Canada: 3113
Chile: 543
Czech Republic: 241
Denmark: 127
Estonia: 273
Finland: 171
Greece: 971
Hungary: 619
Israel: 316
Italy: 2288
Japan: 86
Korea: 60
Latvia: 186
Mexico: 2600
Netherlands: 1146
New Zealand: 356
Norway: 94
Poland: 3510
Portugal: 3884
South Africa: 7350
Sweden: 295
Switzerland: 175
I did some very rough math trying to estimate the number of active internet users per country, and combining that with the list above a good split for my desired 400 subjects is 200 US subjects, 50 UK subjects (UK is very over-represented here), and 150 for the rest of the world.
When I need to balance across groups like this I do what the first commenter suggested - multiple copies of the protocol for each group. (In my case this was enriching for racial diversity, but there’s on reason not to do it for geographic diversity, and i found it worked very well.)
