What are screening questions?

In addition to demographic characteristics (age, gender, place of residence, language), you might have other requirements to filter testers. Screening questions are about adding an extra layer of filtering who gets invited, unlike a survey, which aims to categorize invited participants and can be done as part of the test.

With screening questions, you could filter for example:

  • Interests and hobbies (mountaineering, art house films, classical music, vegan diet, etc.)
  • Habits (visits to the theater, means of transport, sports, media usage, etc.)
  • Attitudes (politics, religion, behaviors, design preferences, etc.)
  • Usage or visiting of certain websites, apps, or services (Spotify, Netflix, Financial Times, etc.)

As you might guess, it is possible to end up with very few suitable testers, or none at all! In extreme cases you might be charged extra or given the option to disable your screening question. Because of this, try to create a criteria that matches at least 1 out of 10 testers.

Advantages

Just like demographics, screening questions allow you to only invite specific testers to your test. You can set up much more extensive and complex criteria and therefore target a very specific subset of testers from our pool.

Disadvantages             

  • The more testers you exclude from the test, the longer you will have to wait for results.
  • When done wrong, you might end up with too few suitable testers or none at all.
  • Setting up good screening questions takes time.
  • Testing with a homogenous group of testers will give you less different perspectives.

So before you set up screening questions, make sure that you actually need them. Always start with demographics and see if you need to further narrow them down.

Screening questions at Userbrain

When ordering testers, you can set up one or more screening questions at no charge. Enable the switch called "Screener" and start typing.

Question types

Userbrain offers two different types of questions. If you are new to screening questions, try stick to the Single Choice option.

Single Select

Testers can only select one option per question.

Multiple Select

Testers can select one or more options per question.

 

Answer validation

In order to define which answers pass the screening process and which ones don't, Userbrain offers three types of answer validation.

Approve

This option must be selected, in order to pass the screening question. Note that if you have a Single Select question with more than one answer marked 'Approve', testers who check any them will pass. In contrast, Multiple Select questions require all of them to be checked.

Reject

If this option is selected, the tester fails the screening process.

Ignore Answer (only for Multiple Select)

This option allows you to ignore the answer. You might ask yourself why you would ever need that:

  • Disguise the correct answer. Sometimes it makes sense to add one or two extra options in order to provide a more realistic spectrum. Be careful not to use too many!
  • Create disqualifying questions, such as "either option A or option B, just make sure it's not C". To do this, mark both A and B with 'Ignore Answer' and C as 'Reject'

 

Tips for successful screening questions

Let's look at some dos and don'ts to help you write coherent questions that avoid you from ending up with no participants.

Avoid "yes" or "no" answer choices

Testers are eager to test and if you ask them a yes/no type question, their answer might be biased towards a positive response. To eliminate this factor and receive honest responses and well-suited participants, provide multiple select choices instead.

Limit answer choices            

Do not provide too many answering options and make sure you have a good balance of qualifying (Approve) and disqualifying (Reject) responses. If you are off-balance, your screening process might become very hard to pass. This would make it take much longer and - in extreme cases - have higher charges.

Provide a "none of the above" option            

Now that you have limited your answer choices, a 'none of the above' type option (eg. 'Other', 'I don't know', …) is the easiest way to still be inclusive towards every participant. If such an option is not provided, testers might pick an answer randomly, possibly ending up in your study accidentally.

 

Examples