Paper Reading: Qualitative Methods in Empirical Studies of Software Engineering

NOTE: This is a Paper Reading for Advanced Software Engineering. The original paper can be found here.

What were the primary contributions of the paper as the author sees it? How does this work move the research forward?

With empirical studies of software engineering beginning to address the human aspects of software development, the author presents and reviews a number of different methods for the collection and analysis of qualitative data, and describes them in terms of how they might be incorporated into empirical studies of software engineering, in particular how they might be combines with quantitative methods.

• Collecting Qualitative Data
  • Participant Observation
  • Interviewing
• Extracting Quantitative Values from Qualitative Data for Quantitative Analysis (Coding)
• Analyzing Qualitative Data
  • Theory Generation: extract from a set of field notes a statement or preposition that is supported in multiple ways by the data.
  • Theory Confirmation: confirming a preposition after it has been generated from the data.

What were the main contributions of the paper as you (the reader) see it? How could this research be applied in practice?

Aside from the primary contributions of the paper as the author sees it, in my opinion, another major contribution of the paper is identifying the four main categories of empirical studies, and explaining in detail how combinations of quantitative and qualitative methods can be designed for each category.

The four main categories of empirical studies: - Blocked subject-project study: - Several projects, several subjects. - Reduces bias, but increases the cost of the experiment. - Replicated project study: - One project, several subjects. - Isolates the effect of differences between subjects. - Multiproject variation: - Several projects, one subject. - Observes the performance of the subject on a project before some treatment is applied, and on a different project after that treatment is applied. - Single project study: - One project, one subject. - Similar to a case study. - Certain attributes are examined and possibly compared to some baseline.

How combinations of quantitative and qualitative methods can be designed for each category: - Blocked subject-project study, Replicated project study: - When testing hypotheses and finding casual relationships between variables, use qualitative data to illuminate the statistical results. - Multiproject variation study: - Qualitative analysis: revealing new issues and tracking changes relative to other issues. - Quantitative analysis: looking more closely at the issues suggested by the qualitative analysis. - Single project study: - First, data is collected qualitatively through interviews. - A taxonomy of the question under research is generated. - Part of the interview data is coded to yield quantitative variables. - Any relationships found between quantitative variables are checked against qualitative data.

How was the work validated?

Examples, interviews, quotes from experts, and paper citations are used to validate the points presented when reviewing a number of different methods for the collection and analysis of qualitative data, identifying the four main categories of empirical studies, and explaining in detail how combinations of quantitative and qualitative methods can be designed for each category.

How could this research be extended?

In the last paragraph, the author points out that "we must exploit to the fullest every opportunity we do have, by collecting and analyzing as much data of as many different types as possible". Aside from the examples presented in the paper, what other types of data can be collected, and how they can be analyzed, is a future direction of research.