The smartest people make the dumbest mistakes: the difficult art of designing an expert survey

December 16, 2021

When conducting an expert survey, you can’t risk any ambiguity or bias

When we look at the correlation between SAT score and bias, we find that not only do smarter people (as determined by SAT score) assume that everyone else is more susceptible to thinking errors than they are, but also smarter people are in fact more vulnerable to common mental mistakes than the average person is. This “intelligence bias” has serious ramifications for designing a survey for experts – the bread and butter of the NewtonX survey team. Highly intelligent people tend to have strong beliefs. And in the case of expert surveys, where you are interviewing people with extremely niche knowledge (so niche that only 50 people in the entire world possess it), this bias can become even more exaggerated. 

This guide to designing expert surveys for some of the smartest people out there is an extract from the NewtonX survey design center playbook. The advice in this article is sourced from leading quantitative and qualitative survey experts from leading consulting firms, survey institutes, and other prominent firms. 

1. To begin your expert survey, start from the end

Ask yourself what results you are trying to get and what you wish to prove with these results. Your survey questions should be a means to an end. Don’t think “it would be interesting to ask this” — instead, you should always ask yourself, “what will I be able to demonstrate with the results of this question.”

2. Negative formulations are confusing

Jane Austen might do it, but that doesn’t mean you should. Negative formulations, such as asking “How unlikely are you to invest in a third-party cloud data center” can easily be misread as their positive counterparts — thus skewing your results. Ask questions with a positive sentence construction for more uniform and accurate results.

3. Test your questions – can you find a tester in the target audience?

Before deploying your expert survey, give your sample questions to a few testers who will try out your survey. This will accomplish two things: it will test the clarity of the questions and test the overall length of the survey. We cannot emphasize this enough — even the most advanced survey designers systematically use testers, and get extremely useful feedback out of it! Ideally, try to find testers who are as close as possible to your target audience. Even one tester is vastly better than relying only on yourself to test your survey. 

4. Give specific temporal parameters

Don’t just ask, “How much does your organization spend on customer support.” Open ended questions like this leave room for answers like, “Far too much by my standards.” Instead, give parameters around your question, such as “What was your dollar spend in 2017 on Customer Support employees and tools. Was this higher or lower than 2016.”

5. Use response scales when possible to guide respondents through your expert survey

If you’re looking for specific answers within a range on your expert survey, offer a response scale. For instance, to use the customer support example, you could give options for “over 1 million,” “between 800,000- 1M”; etc. When you do this, ensure that your ranges aren’t so large that they seem misleading (for instance, 500,000- 1M), that there isn’t a gap between options, and that the range of options is well distributed to capture responses in an even manner and avoid getting more than 20% of the responses in the extremes (ideally your responses should follow a normal distribution within the ranges provided). Rephrase Yes/No questions when possible to include phrases such as “How much,” “How often,” or “How likely” so that you can include a response scale for richer data. 

6. Long questions create bad answers. When  in doubt, split the question

Avoid garden path or run-on sentences. Splitting the question into two separate queries is the best path to clarification in 90% of use cases. Don’t risk confusing your audience with questions like “What are the biggest barriers to entry that you see in blockchain today, and what are the hottest trends?” Instead, split the question into two separate ones so that you prompt the most robust answer possible to each of your questions. 

7. Indicate desired answer length on your expert survey

If you want an answer that has a certain about of detail or elaboration, give your survey respondents an idea of how many sentences you want their responses to be. This is a more specific direction than asking for “in-depth responses,” for example, which can result in differing interpretations of what constitutes in-depth. 

8. Be careful what you wish for. Avoid leading questions!

Questions like “How beneficial do you think blockchain technology can be to your business” leads the respondent to think Blockchain is beneficial or should be beneficial to the business. If you are trying to assess what the respondent thinks of Blockchain and whether or not it can be beneficial to its business, this is not the best way to ask. Instead, opt for a neutral question such as “What is your perception of Blockchain?” 

9. Jargon is good – if it’s the jargon of your respondents

Do your due diligence before writing your expert survey. Make sure you understand key terms and are employing them correctly — this will help you meet your respondents on their level, and will elicit more in-depth, insightful responses. 

10. Make it clear what you want: an opinion is not a datapoint

People tend to want to share their opinions over facts — unless that fact backs up an opinion. Because of this human bias, you want to make sure to specify when you want a number instead of an opinion. Asking for a number and a unit usually removes all room for doubt. For instance, you could ask “In dollars, what is the total addressable market for IoT devices in the manufacturing industry?” 

11. Even complex topics can have simple questions on your expert survey

This is the most important point to remember when designing a survey for highly technical experts. These people are experts because they understand extremely complex technologies, and you will get the most out of them by allowing them to respond in-depth to a question that they interpret correctly. If a respondent struggles to answer, this means that the question is not clear and the response of the respondent will lose weight as a consequence. A good rule of thumb for this is ensuring that the respondent can answer your question after a single read. In other words, how “painful” is it to understand your question. A question should never be painful to understand. As our senior survey designers say: “All else being equal, answering your survey questions should be enjoyable.”

Sign up for our newsletter, NewtonX Insights:

Your playbook to making confident business decisions enabled by B2B research. Expect market research trends, tools, and case studies with leading enterprises, delivered monthly.

Related Content

B2B market research template

Businesses conduct B2B market research to answer a wide variety of questions. However, these questions generally fall into three broad categories, or B2B market research use cases. It’s important to know which use case fits

read more

The top questions to ask on your B2B market research survey

The utility of a survey is heavily dependent on the quality of the B2B market research questions it poses. The professionals who take B2B surveys don’t have the time to complete surveys full of redundant

read more

B2B Panels are passé: NewtonX and the future of B2B market research

Panels are not designed to answer industry-specific B2B questions. In truth, there’s no such thing as a B2B panel.

read more