Did anyone else find their estimated number of eligible prolific users "active in the last 90 days" drop suddenly? From ~550 estimated data points to 151. I'm quite anxious and worried

⛑ Community Support
1

Hello,

I had a bit of anxiety between when I proposed my study to when I finally pulled myself together to complete the requests of my ethics review board.

Now that I’m ready to collect data, I signed onto my Prolific researcher account and was about to publish my study. I noticed that the number of users matching my criteria is now only 151! Several months ago when I first assessed the viability of using Prolific, the estimated number of participants was over 500!

I’m now severely worried. While in theory, if all 151 cases are good (i.e., no one shows evidence of careless responding) I might still be able to carry out some basic preliminary stats analyses (as intended for my first study) but I’m still quite worried.

Could it be because of the nature of my sample? I.e., working individuals? Particularly those who work from a central location (i.e., not working remotely from home)? Or did some sort of purge take place on prolific that resulted in the deletion of hundreds (if not thousands) of accounts?

Would really like to hear from anyone with insight into this matter. I’m very nervous and scared now.

I just published my study about 15 minutse ago and there are currently NO submissions underway. This whole time, I imagined that the moment I published my study, participants would sign up for it immediately.

Sorry if I’m ranting/rambling; I’m just quite anxious because things are not turning out the way I had expected/assumed them to and now I’m quite worried.

2

Dear Joseph

Welcome back.

I am afraid I can’t be of much help. Skip to the ★ below for the only
plausibly practical suggestion.

That is a real shame that your subject pool has gone from 500 plus
to only 151.

Prior to whether they all respond carefully, it seems to me that you
are unlikely to get all 151 to respond simply because some of them
will be busy or on holiday etc.

That said, I tend to find it is the younger less well paid subjects that
respond quickly, whereas those in paid employ take more time,
so I would be hopeful that a significant portion of those 151
will respond if you can wait a week.

(If you had been interested in people who work remotely from home
then, since in many countries in the world there has been a decrease
in Covid and people are going back to work, then I could see why your
numbers might have reduced, but unless you are screening for a country
that has had an increase in Covid, then it is difficult to see why the
numbers working centrally would have decreased. )

I doubt that Prolific has purged those that were active in the last
90 days several months ago but you’ll need to wait for Josh’s answer
on that.

★Are you sure you used identical pre-screening questions? There are
for instance three ways of screening for English speakers

Fluent in English
First language English
and
English speaking monolingual.

There are also two ways of stipulating working centrally
using the one of either of the following pre-screening questions

Remote/office work
COVID-19 Working From Home

All I can suggest is checking that you have pre-screened in exactly
the same way since it is difficult to understand why there should
be a 70% reduction in active participants over the past several months.

I am sorry I can’t be of more help.

Tim

3

Hi Tim,

Thanks for taking the time to read and respond. I think I’ve identified the major culprit/problem (I wish I identified this problem earlier) but I’ll respond to some of your points nonetheless.

Quite the opposite, actually, I’m looking specifically for people who are NOT working remotely from home.

Yes, I am sure. I triple checked. Not only did I save a draft of my study exactly as it was in Prolific (i.e., so none of my prescreen criteria changed) I also checked the protocol I wrote in MS Word and went through each prescreen item one by one.

Incidentally, I see the item that makes the biggest difference.

The ONE item “employer type” (“Please describe your work”) changes the amount of eligible participants I have from 8271 to 151.

It seems like it doesn’t matter what combination of criteria I choose from:

The options, as a reminder, are:

“Employee for a for-profit company or business or of an individual for wages, salary, or commissions”

"Employee of a not for profit, tax exempt, or charitable organization

“Local government employee”

“State Government employee”

“Fed Government Employee”

“Self-Employed in own not-incorp. business, professional practise, or farm”

“Self employed in own incorporated business, professional practice, or farm”

“Working without pay in family businses or farm”

“none of the above”

I had intended for my recruitment ad to go out ONLY to prolific users who indicated the first 5 options which I’ve bolded above (and not the latter 4).

Having said that, I was playing around with the filter last night to see how my numbers were changing on the basis of my options and I found that even if I checked off everything except, “none of the above,” the number still drop from the thousands to ~200.

I think I need to talk to my supervisor about modifying my protocol and resubmitting to ethics.

It’s been almost 12 hours and I’ve only had 7 completed responses (with 1 returned, for a total of 8 attempts). This is not the fast turnaround I was expecting or hoping for. I just hope my supervisor doesn’t kill me for me for not discovering this problem sooner…

Indeed, I have used both of these already.

At present, I’m contemplating either

(a) simply scrapping the employer type option or
(b) doing a quick custom prescreen asking individuals whether they are self-employed (since that is the purpose of my “employer type” prescreen) or
(c) doing a forked survey design where people who are self-employed can still answer a subset of my questions where their responses might still be relevant/of interest to me. Though obviously, this option would be the most risky in terms of $$$/datum.

A part of me thinks that the “work from a central place of work” + a new prescreen item “whether they work with employees” might be enough for me to not care about whether they are considered self-employed, but this will be a conversation I have with my supervisor. I really hope he doesn’t kill me for this…

1 Like
4

I’m very sorry to hear that your sample size reduced so drastically, and that this has caused you such stress

We’ve investigated, and although we did conduct a ‘clean’ of our participant pool, your sample was not affected by this. The likeliest explanation is that those participants became inactive in the time between your initial check and now.

(b) doing a quick custom prescreen asking individuals whether they are self-employed (since that is the purpose of my “employer type” prescreen) or

We would definitely recommend this option, since the employer type option reduces your sample size so much.

Alternatively, we could attempt to ‘reactivate’ the missing participants via emails. But, it’s uncertain as to whether that would work, and it may mean your data takes longer to be collected.

In the meantime, I’ve ‘refreshed’ your current study via the backend, which should speed up recruitment.

Let me know which option you’d like to go for, and I can do my best to help you with it :slight_smile:

5

Hi Josh,

Thanks for the quick response!

Ah, how frustrating. Having said that, I now have learned some new things and that learning experience in it of itself is invaluable:

  1. I should have acted sooner; that estimate of +500 eligible participants was [apparently] a volatile number and I should have respected that. Instead, I took it for granted and assumed that, “if there are +500 eligible participants today, surely that number will either stay roughyl the same or [more likely] increase over time!” - how wrong I was!

  2. I should have investigated the cause-and-effect of my prescreen criteria more carefully and aimed for a better buffer. In retrospect, ~550 eligible participants when my goal was 500 data points was cutting it quite close. Granted some of the pre-screening items I’m seeing now were not present (I don’t think) several months ago. On another note, I really appreciate that you guys are continuing to add pre-screen items that are helpful to researchers. I just submitted a suggestion last night, incidentally, for helpful pre-screen items (just a simple, “are you self-employed?” with yes vs. no response options).

That would actually be quite lovely if you don’t mind. I’d be very curious to see if the number of eligible participants shoots back up to what I saw several months ago. Would you mind doing this? It would save me a GREAT deal of time and stress if that number actually popped back up to +500. I reviewed my set of goals/timeline that I previously established with my supervisor and it appears that the goal was to have all my data for study 1 (i.e., this study) collected by the end of December so even if it means it’ll take an extra 2 or so weeks, I’d really appreciate that as it would likely be faster than waiting upon Ethics (also, I can use the more careful prescreening options [described below] for my study part 2).

Thank you! I did notice a greater rate of uptake this morning. Last night I went to bed unable to sleep, tossing and turning. In fact, I actually dreamed that I went to my computer and had somewhat of a ,“It’s a Christmas MIRACLE!” moment as I saw the data flooding in. Alas, it was just a dream.

Still, this morning, I see that the pace of responses is looking a bit more reassuring (I’m only collecting a small batch of 40 for now before assessing for data quality - e.g., are people passing my attention checks?).

My plan (I’m not sure if you’re interested but I find it helpful for myself to put into words my thoughts/plans) is this:

(1) I’m going to see how many data points I can collect in 24 hours. At which point, I will pause my study. Alternatively, if I reach my mini-goal [40 data points for the time being] then this step is irrelevant.

(2) I’m going to do a quick analysis of my these 40 data points to inspect for things like careless responding/attention checks.

(3) I’m going to draft an amendment to my protocol, selecting and justifying an alternative prescreen criterion (or criteria) that will hopefully do a more nuanced job of selecting my target demographic. I think the criterion I mentioned earlier that nuked my number of eligible participants from the thousands down to 151 is like amputating a limb when one has a wart that could have been selectively targeted with liquid nitrogen.

(4) I’m going to write an email to my supervisor expressing my contrition for not detecting this problem sooner but expressing optimism that my modified protocol should (a) yield a higher ceiling re: max number of possible participants and (b) a greater rate of data collected per day. I’ll use how much data I collect in these 24 hours as a point of reference.

(5) Hopefully, he will not be too upset, request an amendment to my study to the REB, and I’ll resume my study with an expanded pre-screen audience. I’ll likely contact you around this step to ask if I need to start a “new study” (and simply exclude participants who have previously completed my first study) or whether I can continue the “old study” so long as I update my qualtrics survey first (not sure if I can change the prescreen target audience after a study has already been initiated.

(6) hopefully at this point, I’ll get closer to my target of 500 data points on time. I MIGHT be able to get away with being happy with 200 data points, but I’d obviously rather have closer to my initial goal. I hope to be able to use Prolific for my Study Part 2 wherein the duration is more like 30 minutes instead of 10 minutes.

Cheers,

6

I’m glad to hear that you’ve been able to view this positively, and as a learning experience! It’s a great attitude to have :slight_smile:

I just submitted a suggestion last night, incidentally, for helpful pre-screen items (just a simple, “are you self-employed?” with yes vs. no response options).

Thanks for submitting, we really appreciate all the suggestions we get!

Would you mind doing this? It would save me a GREAT deal of time and stress if that number actually popped back up to +500.

We’ll do this as soon as we can. I’ve submitted a request to our technical team, and I’ll give you an update as soon as I know

Still, this morning, I see that the pace of responses is looking a bit more reassuring

Yes, I can see that you’re 90% of the way there. Hopefully, the recruitment completes soon.

My plan (I’m not sure if you’re interested but I find it helpful for myself to put into words my thoughts/plans) is this:

This plan looks comprehensive, and well-thought out. Wishing you all the best with it!

Lmk if you run into any further trouble :slight_smile:

7

Thanks! There is something you can help me with in the meantime. Assuming that I go ahead and request (from my REB) an amendment to my study, do I need to submit a new study to Prolific? Or can I unpause my current study simply with an adjusted prescreen target?

I’d just like to know which it is so I can tell the REB what my plan is.

1 Like
8

You’ll have to create a new study (or duplicate your existing one) to adjust the prescreening criteria :slight_smile:

9

Excellent! I can do that; thanks so much for letting me know! This will save me some ambiguity (and me asking again later) when I write my proposed amendment to my study! Cheerio!

1 Like
10

I’ve sent you a DM with good news :slight_smile:

1 Like
11

Hi!

Yes! I did notice; sorry I haven’t responded just yet - just some unexpected things took place (involving a broken phone t hat needed servicing which took longer than anticipated).

I’ll try to get back to your message in the new few hours!

Cheers,

1 Like