Paper Reading: The Fundamentals of Writing Questions

NOTE: This is a Paper Reading for Advanced Software Engineering. The original paper can be found here.

This part of the book addresses the problem of crafting survey questions that respondents are willing to answer and respond to accurately. It first discusses issues to consider when designing survey questions, then presents the structure of a survey question and different question formats, before providing specific guidelines on wording survey questions.

Issues to consider when designing survey questions

• What concepts to measure
  • Recommended: Adopt established measures from existing surveys
• What data to collect
  • Factual information: precise, readily available
  • Opinion: requires time to formulate, strongly influenced by context
  • Behavior: better memory of recent, memorable events compared with distant, mundane events
• What question format to use
  • Different cognitive information processing for aural and visual surveys
• What mode to adopt
  • The presence of an interviewer may speed up surveys, but may induce social desirability and acquiescence, leading to interviewer bias.
  • Lack of standardization among different interviewers may lead to interviewer variance.
• What to modify (from existing surveys)
  • no changes or only minimal changes when replicating or comparing results
    • questions should also be asked in a similar fashion
• How to motivate respondents
  • think about the cognitive process respondents go through
  • pay attention to the context and wording

The structure of a survey question

• Question stem
• Additional instructions
• Answer spaces or choices

Different question formats

• Open-ended
  • rich, detailed
  • more prone to skipping
  • requires lengthy data processing
• Closed-ended
  • nominal or ordinal categories
  • set of answer choices known in advance
  • easy to analyze
• Partially closed-ended
  • closed-ended with "other" response
  • respondents more likely to select the options instead of "other"

Specific guidelines on wording survey questions

• Choose the appropriate question format.
• Make sure the question applies to the respondent.
• Ask one question at a time.
• Make sure the question is technically accurate.
• Use simple, familiar and specific words.
• Use short, simple sentences that take a question form.
• Avoid double negatives.
• Organize questions in a more straightforward, comprehensible way.

Also sprinkled throughout the section is the notion that the crafter should get into a respondent's state of mind when crafting survey questions, and also test the survey questions to evaluate their quality.

This section is very comprehensive and convincing, as the author supports his arguments by analyzing specific examples from actual surveys, and also frequently quoting previous work on the topic. From such a chapter we can gain a deep understanding of the nature of survey questions, especially the underlying cognitive, psychology and sociology problems, as well as the best practices within the domain, and we can also refer to this chapter as a guide and checklist when we craft survey questions ourselves.