Are you confident your online experiment will get you the data quality you need?
Here are three top tips for piloting, shared by Emily Breese:
-
Pilot two groups - one including your friends and colleagues, the second including a small representative sample
The benefit of using the first pilot sample is that it’s cheap and convenient, and you can ask your participants to intentionally try and do silly things in your experiment to highlight any bugs. It’s important to also use the second sample however as it will be more representative and your participants will not have the opportunity to ask you questions so will likely throw up errors that your first sample missed. -
Be thorough - pilot every aspect of your experiment
It’s important to pilot everything in your experiment, even though it might be tempting just to pilot the main task element. Things like questionnaires and consent forms also need piloting to ensure that the instructions are clear and that things work as expected. It would be really annoying to have the perfect experiment set up only to find out that the participants weren’t able to select any of the check boxes in the questionnaire! -
Use your pilot to inform your data exclusion criteria
This is often an overlooked benefit; piloting is important for informing our decisions about exclusion criteria. Data that meets the exclusion criteria will be removed from analysis. By testing out your experiment on real people we are able to set fair criteria for removing data from analysis before continuing with your main experiment. This also allows you to pre-register this criteria, which is great for open and transparent science!
How do you pilot your experiments? Do you have a favourite strategy?