The Smartest People Make The Dumbest Mistakes: The Difficult Art of Designing Expert Surveys

Google+ Pinterest LinkedIn Tumblr +

A bat and ball cost a dollar and ten cents. The bat costs a dollar more than the ball. How much does the ball cost?

Over 50% of students at Harvard, Princeton, and M.I.T. gave the incorrect answer to this question, confidently responding that the bat is a dollar, and the ball is 10 cents (this answer is, of course incorrect; the ball is five cents, and the bat is a dollar and five cents). When we look at the correlation between SAT score and bias we found that not only do smarter people (as determined by SAT score) assume that everyone else is more susceptible to thinking errors than they are, but also smarter people are in fact more vulnerable to common mental mistakes than the average person is.

Smarter people are more vulnerable to common mental mistakes than the average person is. Click To Tweet

This “intelligence bias” has serious ramifications for designing a survey for experts – the bread and butter of the NewtonX survey team. Highly intelligent people tend to have strong beliefs. And in the case of expert surveys, where you are interviewing people with extremely niche knowledge (so niche that only 50 people in the entire world possess it), this bias can become even more exaggerated.

This guide to designing expert surveys for some of the smartest people out there is an extract from the NewtonX survey design center playbook. The advice in this article is sourced from leading quantitative and qualitative survey experts from leading consulting firms, survey institutes, and other prominent firms.

1. To Begin, Start From The End

Ask yourself what results you are trying to get and what you wish to prove with these results. Your survey questions should be a means to an end. Don’t think “it would be interesting to ask this” — instead, you should always ask yourself, “what will I be able to demonstrate with the results of this question.”

2. Negative Formulations are Confusing — Especially Double Negatives!

Jane Austen might do it, but that doesn’t mean you should. Negative formulations, such as asking “How unlikely are you to invest in a third-party cloud data center” can easily be misread as their positive counterparts — thus skewing your results. Ask questions with a positive sentence construction for more uniform and accurate results.

3. Test Your Questions – Can You Find a Tester in the Target Audience?

Before deploying your survey give your sample questions to a few testers who will try out your survey. This will accomplish two things: it will test the clarity of the questions and test the overall length of the survey. We cannot emphasize this enough — even the most advanced survey designers systematically use testers, and get extremely useful feedback out of it! Ideally, try to find testers who are as close as possible to your target audience. Even one tester is vastly better than relying only on yourself to test your survey.

4. Give Specific Temporal Parameters

Don’t just ask, “How much does your organization spend on customer support.” Open ended questions like this leave room for answers like, “Far too much by my standards.” Instead, give parameters around your question, such as “What was your dollar spend in 2017 on Customer Support employees and tools. Was this higher or lower than 2016.”

5. Use Response Scales When Possible to Guide Your Experts

If you’re looking for specific answers within a range, offer a response scale. For instance, to use the customer support example, you could give options for “over 1 million,” “between 800,000- 1M”; etc. When you do this, ensure that your ranges aren’t so large that they seem misleading (for instance, 500,000- 1M), that there isn’t a gap between options, and that the range of options is well distributed to capture responses in an even manner and avoid getting more than 20% of the responses in the extremes (ideally your responses should follow a normal distribution within the ranges provided). Rephrase Yes/No questions when possible to include phrases such as “How much,” “How often,” or “How likely” so that you can include a response scale for richer data.

6. Long Questions Create Bad Answers. When In Doubt, Split The Question

Avoid garden path or run-on sentences. Splitting the question into two separate queries is the best path to clarification in 90% of use cases. Don’t risk confusing your audience with questions like “What are the biggest barriers to entry that you see in blockchain today, and what are the hottest trends?” Instead, split the question into two separate ones so that you prompt the most robust answer possible to each of your questions.

7. Do You Need a Word or a Paragraph? Indicate Desired Answer Length!

If you want an answer that has a certain about of detail or elaboration, give your survey respondents an idea of how many sentences you want their responses to be. This is a more specific direction than asking for “in-depth responses,” for example, which can result in differing interpretations of what constitutes in-depth.

8. Be Careful What You Wish For. Avoid Leading Questions!

Questions like “How beneficial do you think blockchain technology can be to your business” leads the respondent to think Blockchain is beneficial or should be beneficial to the business. If you are trying to assess what the respondent thinks of Blockchain and whether or not it can be beneficial to its business, this is not the best way to ask. Instead, opt for a neutral question such as “What is your perception of Blockchain?”

9. Jargon is Good – if it’s the Jargon Of Your Respondents

Do your due diligence before writing your survey. Make sure you understand key terms and are employing them correctly — this will help you meet your respondents on their level, and will elicit more in-depth, insightful responses.

10. Make It Clear What You Want – An Opinion Is Not A Datapoint

People tend to want to share their opinions over facts — unless that fact backs up an opinion. Because of this human bias, you want to make sure to specify when you want a number instead of an opinion. Asking for a number and a unit usually removes all room for doubt. For instance, you could ask “In dollars, what is the total addressable market for IoT devices in the manufacturing industry?”

11. Even Complex Topics Deserve Simple Questions

This is the most important point to remember when designing a survey for highly technical experts. These people are experts because they understand extremely complex technologies, and you will get the most out of them by allowing them to respond in-depth to a question that they interpret correctly. If a respondent struggles to answer, this means that the question is not clear and the response of the respondent will lose weight as a consequence. A good rule of thumb for this is ensuring that the respondent can answer your question after a single read. In other words, how “painful” is it to understand your question. A question should never be painful to understand. As our senior survey designers say: “All else being equal, answering your survey questions should be enjoyable.”

Design Your Survey So That The Smartest People Give The Smartest Answers

Let’s return to the bat and the ball question. Say we had asked it like this: “A bat costs one dollar and five cents, and together with a ball the total adds up to one dollar and ten cents. How much did the ball cost?”

The answer is the same. And it’s likely that 100% of students at Harvard, M.I.T., and Princeton would have immediately gotten it correct.

The way you design your survey informs how well your experts will be able to showcase their knowledge. And when that knowledge is as rare and niche as it is in the case of expert consulting, you can’t risk any ambiguity or bias — the stakes are higher than they are in crowdsourced surveys.


To request an expert survey or learn more about how to design expert surveys, contact


About Author

Comments are closed.