Screeners help you select the right testers, but poorly designed questions can let motivated testers slip through by guessing the desired answers. This article explains how to craft screeners that are harder to game, why it matters, and what to watch for.
Key Principles for Unbiased Screener Design
Avoid Obvious Answers
If your screener makes the desired answer too easy to spot, testers may select it just to qualify. Use several plausible options, and avoid simple yes/no choices. Rephrase questions so the correct answer is not obvious from context.
Include Plausible Distractors
Add answer choices that could credibly apply to many testers, not just the group you want. This helps disguise the qualifying response and reduces the chance of testers guessing correctly.
Phrase Questions Neutrally
Don't hint at the desired group in the way you word the question. Avoid loaded language and keep the screener focused on behavior or experience, not preferences.
Use Multiple Screeners for Cross-Validation
Ask about the same behavior in different ways across two or more questions. Inconsistent answers may indicate a tester is trying to game the system.
Examples
Poor screener: "Are you a parent? (Yes/No)". Easy to guess the qualifying answer if the test is for a parenting website.
Improved screener: "Which of the following best describes your household?"
- I live alone
- I live with adults only
- I live with children under 18
- I live with children over 18
- None of the above
For frequency-based products, use gradations: "How many times have you used a food delivery app in the past month?" with several ranges, rather than yes/no.
Recommendations
Avoid yes/no and obviously qualifying answers.
Add plausible distractors and 'none of the above' options.
Phrase screeners neutrally and focus on behavior, not intent.
Use multiple screeners to cross-check consistency.
Review responses for patterns that may indicate gaming.