Paper Reading: Selection and Presentation Practices for Code Example Summarization

NOTE: This is a Paper Reading for Advanced Software Engineering. The original paper can be found here.

What were the primary contributions of the paper as the author sees it? How does this work move the research forward? How could this research be applied in practice?

Code examples are important in modern software development. As part of the first steps toward automatic source-to-source summarization, the authors studied how humans summarize examples to understand how to automate the process, and propose empirically-supported hypotheses justifying the use of specific practices.

Selection Practices - Practices Related to Language Constructs - Practices Based on Query Term - Practices Considering the Human Reader Presentation Practices - Trimming a Line When Needed - Compressing a Large Amount of Code - Truncating Code - Formatting Code for Readability - Improving Code

The results provide a grounded basis for the development of code example summarization and presentation technology.

How was the work validated?

We chose a well-defined corpus of programming documents, The Official Android API Guides, which contains a mix of natural-language text and code fragments.

We collected 156 pairs of code examples and their summaries from 16 participants, along with over 26 hours of think-aloud verbalizations detailing the decisions of the participants during their summarization activities. We analyzed common practices behind these decisions across the hand-generated representations, as well as the rationale behind the practices.

What were the main contributions of the paper as you (the reader) see it?

In my opinion, aside from the obvious contributions of the paper presented by the author, there is a lot to learn from the study set-up and the conceptual framework for interpreting the results.

1. To understand the rationale behind the practices, we instructed the participants to verbalize their thought process using the think-aloud protocol.
2. We distinguished practices concerning the type of content selected and the way the content was presented in a summary, because even summaries with content associated with the same part of the original fragment could vary on how to present the summary.
3. To make hypotheses justifying the use of different practices, we relied on a quantitative analysis of the distribution of each practice across code fragments and participants. In-lined histograms presents the distribution of observations of a given practice for the participants over the code fragments. This provides a convenient and compact assessment of the amount of evidence for a practice.

Furthermore, the authors have borrowed a lot from related domains of research, including natural language generation, natural language summarization of code, etc. Some examples:

1. The separation of content selection from presentation is typical in a natural language generation system.
2. The comments demonstrated a number of different ways to abstract content, including aggregating lexically and aggregating semantically - natural language generation terminology.
3. Seven participants injected additional natural language into the code summaries. This motivates a novel type of transformations that mix code and text. The only work we know of in this area is the natural summaries generated by Rastkar et al.

This gives revelations on exploiting knowledge from related domains when doing our own research.

How could this research be extended?

The goal of the study was to inform the design of concise representations of source code and automatic summarization algorithms. A natural future direction is to implement these representations and algorithms, and conduct empirical studies assessing their usefulness in summarizing source code.