Paper Reading: Exploiting Dynamic Information in IDEs Improves Speed and Correctness of Software Maintenance Tasks

NOTE: This is a Paper Reading for Advanced Software Engineering. The original paper can be found here.

Summary

The pervasive use of inheritance, interfaces, and runtime polymorphism in object-oriented software systems leads to it being unclear which concrete method is invoked at a call site. Modern IDEs such as Eclipse offer static views of the source code, but provide little help in revealing the execution paths an object-oriented software system actually takes at runtime.

In this situation, developers usually resort to debuggers or profilers. However, the information extracted by such tools are volatile, and developers cannot continuously benefit from dynamic information integrated in the static source views in the IDE.

To solve this problem, the authors propose Senseo, an Eclipse plugin that collects dynamic information by running unit and/or system tests of the project with a customized JVM, that enriches the source views of Eclipse with dynamic information, such as:

• which concrete methods a particular method invokes, and how often
• which methods invoke this particular method
• how many objects are allocated in methods
• the dynamic collaborations between different source artifacts
• a visualization of the system's Calling Context Tree

These are displayed in tooltips, ruler columns, the Package Explorer, and a dedicated Collaboration Overview.

The authors conducted an experiment with 30 professional Java developers solving five typical software maintenance tasks in JEdit, an unfamiliar, medium-sized software system, measured the time and correctness of the tasks, and conducted statistical tests on the measurements. Senseo yields a significant decrease in time of 17.5 percent and a significant increase in correctness of 33.5 percent, which validates the practical usefulness of Senseo.

Personal Thoughts

There is no doubt that the idea of enriching the source views of an IDE with dynamic information, as well as its implementation Senseo, is of great practical value to developers writing object-oriented software systems. However, I do have a few concerns after reading the paper.

• To enrich the source views of Eclipse with dynamic information, Senseo runs unit and/or system tests of the project with a customized JVM. There are several concerns here.
  • The project should have unit and/or system tests that thoroughly exercise all units in a manner resembling an actual execution of the project in production, otherwise, the dynamic information for some units may be missing and/or inaccurate.
  • The unit and/or system tests should be self-contained and not rely on interacting with the environment, such as getting input from the user, using OS services, etc. If so, a possible remedy would be to carve unit tests from such executions.
  • There is significant overhead in the process of collecting dynamic information. As the authors have reported: "On average (geometric mean), CCT creation alone causes an overhead of factor 2.68. CCT creation and collection of dynamic information result in an overhead of factor 9.07. The total overhead, including serialization/transmission, is of factor 9.47." Although the authors claim that "even though the overall overhead is high when gathering dynamic information, we do not consider this a major issue as the application does not need to run at productive speed while analyzing it", this could be a problem for lengthy system tests, especially if units in the system tests are frequently modified, and new dynamic information has to be reacquired. Carving unit tests from such system tests would also be a possible remedy.

Furthermore, aside from the idea and implementation of the tool, something else I appreciate and have learned from this paper is the experimental study, in which two measures, the time and correctness of the tasks, are selected, and statistical tests on the measurements are conducted. This convincingly proves the effectiveness of Senseo.