Forensic Testing: Uncovering Quality Issues Using Your Organization’s Code Repository

By Andrea Giugliano

A Luring Lead

Imagine for a minute that you’re a detective in an old Hollywood movie instead of a software tester in a cubicle. Someone just sent you an email with two lines: a link to a repository and a question: "They told me we are ready to release. Should we?" This is the first time you see this software, but you can read code and you are up to exploring things. However, you’re missing the most important item: a lead on where to find the quality issues.

Where should you begin? As a movie detective, you would start looking for witnesses. But, returning to reality, you are a tester and your most likely witnesses, the developers on your team, are already working on something else. So you have to take another tack.

The next best alternative? Go into reports, find the history of what happened from some objective source, be a bookworm. And here we are onto something: the link in the mysterious email points to a GitHub repository! The repository contains the history you need, and now your lead is just a few command entries away.

This article will explain how you can guide your quality investigations with analyses of GitHub repositories. You will see how to identify areas of the code that may contain issues and how to display that information in a readable format. Then you will learn how to go from code to a user flow (the graphical user interface that exercises the code), so you can decide where to invest time and effort in exploratory testing. And you will learn how to create interactive visualizations to discover which developer to ask for clarification.  

A Word About The Visualizations Created For This Article

I generated the visualizations in this article with a tool I built myself: code-compass. It builds upon the work of Adam Tornhill. 

At the moment code-compass requires users to be familiar with Emacs. If you’d like to get started with similar visualizations but you aren’t an Emacs user, see References and Alternate Tools below.

Finding Quality Issues With Hotspots

You now have the link to your organization’s GitHub repository. But where do you start looking?  

Assessing code quality is an exploration. There are common code antipatterns, defined by the Agile Alliance as

"a common practice that initially looks like an appropriate solution but ends up having bad consequences that outweigh any benefits. Meanwhile, there’s another solution that is known, repeatable, and effective."

You can examine your organization’s code for antipatterns and other common issues that affect quality adversely.  

Here,  the quality exploration is similar to an investigation: you want to separate noise from signal, random events from clues, vulnerable functionality from well-established one. And a good investigator looks first for clues that are still fresh. They would check the places where the crime occurred and speak with people that may have seen or heard something relevant. For them time is crucial: the more time that goes by, the more information they lose. How does this translate to a code base?

Unlike the Hollywood detective, version control means that evidence is preserved indefinitely.  As a result, we as testers may have the opposite problem: too many possible clues that could lead nowhere. 

However, if we want to prioritise finding defects, I think the most dangerous bugs result from  miscommunication and omission.

Frequently, multiple people change the same code. Sometimes whole teams overlap their changes because of different requirements and expectations. Each individual has their own outlook and background. This leads to everybody having a slightly different understanding of the same thing. Too many people assume that they understand a concept in lieu of asking clarifying questions, for fear of looking incompetent. Over time the misunderstandings may compound to the point where the software becomes brittle and bug-prone. In the worst cases software testers get overloaded by urgent work.

Too much information can also cause quality problems. On average our brains can keep track of only a very few things at a time. Although developers are trained to handle many complex concepts within a few lines of code, their mental capacity is still finite. If you give them hundreds of lines of code, the complexity may become overwhelming, and the misunderstandings discussed above may become even worse.

So, how do we find the parts of the code that may have suffered from miscommunication and confusion? We can translate these aspects into metrics:

• likelihood of miscommunication ~ number of authors and number of changes

• difficulty of understanding ~ number of lines of code

Can we find these in a code base? Sure we can! 

Length of code is easy to measure, and GitHub repositories can easily tell us the number of authors and changes for any given file. Large files that have been changed many times by several authors are one of the first places to look for “clues.” Adam Tornhill calls this type of  file a “hotspot.”In his book “Your Code as a Crime Scene,” he shows how to create interactive visualizations of hotspots in the code base.

To give an example, let's say that you have been asked to look at the repository for Selenium. This code base is large and complex; the command line tool cloc shows the following information for Selenium:

       

Instead of opening any of these thousands of files at random to find potential quality issues, you can create a hotspot graph:

The large circle above represents the organization of your code base. Each smaller circle represents a directory in your repository. The smallest circles are the files in your repository. The size of the circles represents the amount of code in a directory or file. 

A few aspects of the diagram can be confusing at first. For example, the cloc results told you that Selenium is mostly JavaScript, while the hotspot diagram indicates that the java directory is larger than that for javascript. This apparent contradiction results from the fact that JavaScript can be found in several directories, including javascript and thirdparty. So Java seems to dominate the Selenium repository, but its presence in a single directory makes that directory look larger. In fact, the predominant code language in Selenium is JavaScript.

This visualization is interactive, so you can dive further in to find hotspots. The visualization algorithm identifies hotspots by color (red) and size. Since the java circle is so big, it’s a good choice for expansion:

There are some colored dots in the client module, so let's expand that next:

At this point the remote module stands out with a big red circle, so we expand this:

The hotspot above is RemoteWebDriver.java. This file has about 30 different contributors and a thousand lines of code.

If we are looking for bugs, this could be a good place to start. We can do better though: we can restrict the history to the most recent changes only. 

The git command line tool provides options to define a starting and ending date to select the changes you want to inspect. The terminal command you would use to do that looks like:

git log --since <start-date> --until <end-date>

code-compass exploits this information to collect all the changes that happened over the repository in the last six months.

When you do that, the diagram looks different:

Now you can see that RemoteWebDriver.java is much hotter (or redder) ! And it is not the only one.

You can see that there are some more reddish dots on the picture now. Each of these may be a clue as to where to find other quality issues. As the size and red color of any circle increases, it’s more likely that you’ll find an issue there.

Using Your Technical Knowledge: Going From Code To User Flow

Your hotspot analysis has identified several modules that look like promising places to start. 

What is the next step? White-box testing of the hotspot’s source code is a great idea. See “Digging In: Getting Familiar With The Code To Be a Better Tester”. However, you will likely want to do integration and end to end tests involving the module as well. 

Since we are investigating the quality of this software repository as detectives, exploration is needed! The question is: how to move from a module to a user flow?

From the directory structure we can see that the Selenium repository has a clear nomenclature. This can give us clues as to which code is potentially problematic. For example, we saw that RemoteWebDriver.java was a hotspot. Just looking at the package name suggests that we need to use the Java client in a remote fashion to exercise this code.

The package name suggests something about its functionality. In Selenium a WebDriver runs actions in a browser to automate a user interaction. The name RemoteWebDriver suggests that we are running actions over a remote browser. A remote browser means that the browser is running on a machine different from our own. In Selenium naming conventions the machine with the browser you want to run remotely is a server while the machine from which you want to send commands is the client. So the package name is telling us that the file RemoteWebDriver.java is the Java implementation of the remote client.

Commit history is a great place to find out how to exercise the code. The quickest way is to consult the interface of the service that hosts the Selenium repository:
https://github.com/SeleniumHQ/selenium/commits/trunk/java/client/src/org/openqa/selenium/remote/RemoteWebDriver.java

Several of the commit messages contain interesting information:

If a remote end does not explicitly state JavaScript support we suppose it's a "normal" browser that supports JavaScript
...
adding TakesScreenshot to WebElement, not going down the Augmentor route right now since that is setup just for WebDriver

Via the command line, you can see all the commit messages by changing directory to selenium/java/client/src/org/openqa/selenium/remote/ and running:

git log --pretty=format:"%s" RemoteWebDriver.java

From this you can get a rough summary of the commit semantics. 

When data scientists analyze text, they pay special attention to both the most frequent and least frequent words. The least frequently used terms in the commit messages are:

word,occurrences
'#3489',1
'#401',1
'#4555',1
'#4565',1
'#4669',1
'#4781',1
'#5809',1
'(#2130)',1
'(#2141)',1
'(#2142)',1
'(#2143)',1
'(#7222)',1
'(#7760)',1
'(#7842)',1
'(#7925)',1
'(#8141)',1
'(#8265)',1
'(#8344)',1
'(#8367)',1
'(#8991)',1
'(6a126ab3782deb7dd0cc99c6e3785c72d636959b)',1
'(and',1
'(but',1
'(command',1
'(correct',1
'(diamond',1
...
'command',20
'[java]',21
'is',22
'simonstewart:',24
''',26
'"',28
'remotewebdriver',33
'and',36
'for',45
',',48
'in',49
'of',52
'a',56
'to',83
'the',135
'.',159

The least frequently used “words” in this repository are actually issue ID numbers. From here, you can navigate over to the corresponding issue reports in the GitHub repository, where there often are tips on running the code. As an example, see  https://github.com/SeleniumHQ/selenium/issues/4565. 

Old issues quickly bring us to user flows. In other code bases you can find other tickets or incident identifiers which will most likely lead you to an user flow. Once you find that lead, it is just a matter of following it to finally start your flow exploration.

Let's take the issue above as an example. The reporter of the issue wants to retrieve the operating system name from the RemoteWebDriver. That is a user flow of Selenium. The user in our case is a Java developer using the RemoteWebDriver client to find the platform on which the machine is running. The issue shows us how to use an instance of RemoteWebDriver with the following code:

package demo;


import java.util.Map;


import org.openqa.selenium.Capabilities;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.firefox.FirefoxDriver;

import org.openqa.selenium.remote.RemoteWebDriver;


public class WEBDRIVER_Platform

{

  public static void main(String[] args)

    {

      String attribute = "platform";

      System.setProperty("webdriver.gecko.driver", "C:\\Utility\\BrowserDrivers\\geckodriver.exe");

      WebDriver driver = new FirefoxDriver();

      Capabilities cap = ((RemoteWebDriver) driver).getCapabilities();

      System.out.println("Platform is : "+cap.getPlatform().toString());

      Map<String, ?> map = cap.asMap();

      Object value = map.get(attribute);

      System.out.println("Key : " + attribute + " Value : "+ value);

    }

}

Now we know how to use RemoteWebDriver and we can look into testing more of it. In the issue, the RemoteWebDriver capabilities were returning different values from two interfaces operating on the same data. Guided by that, we could check if the interfaces we can access via the RemoteWebDriver produces other inconsistent values.

Numerical IDs may also correspond to requirements, stories, or bug tickets. Generally, software engineers follow a standard way of committing to a repository. For example, they prefix the commit name or the branch name with a story identifier. When they link to a project management application like JIRA, you may find the corresponding requirement or story. 

Alternately, if these identifiers point to branches, you will need slightly different command syntax:

# print git commits hashes similarly to above different formatting https://stackoverflow.com/questions/1441010/the-shortest-possible-output-from-git-log-containing-author-and-date
# then https://stackoverflow.com/questions/2706797/finding-what-branch-a-git-commit-came-from (git name-rev <sha>)

for i in `git log --pretty=format':"%h" RemoteWebDriver.java`;
  do git name-rev $i;
done;

Now that you have identified some flows to exercise, you can look for people who know the flow to help give you more context. 

Your Informants: A Knowledge Map

Detectives need informants to help guide their search for the truth. And as a tester you know that a five-minute conversation on an issue can add essential detail. Fortunately, the repository history keeps track of the developers who modified the code.

After you identify your target modules, you can query git for a sorted list of contributors:

git shortlog HEAD -n -s -- RemoteWebDriver.java

The most frequent contributors are likely to know more than the others.

However, if these contributors are unavailable, what do you do? Often you can find some of their collaborators by looking at the history of other modules. How does this work? If your top contributors worked with others on different modules in the repo, it may be that those others have pertinent information about the module you want to test. 

For example:

This visualization represents communication information for the last two months of the repository history. It represents how contributors may have communicated among them during the last two months of the repository history. The data for this visualization comes again from git; we check what files each contributor modified and look for work intersections with other contributors. Note that there are several people other than the main contributors whom we can ask for help.

You can also create visualizations of how knowledge is distributed in the code base and who are the rightful owners of the repository. The knowledge map for Selenium looks like this at the moment:

The top contributors in the legend are the ones that worked most on the repository as a whole. Those people are your most valuable informants. 

We can also focus onRemoteWebDriver.java and see how knowledge is distributed there:

Here, you can see that the most knowledgeable person is also the main contributor of the project. This implies that if we were to find a bug in this hotspot, it would likely be significant, because even the most knowledgeable person in this context missed it!

Now you know who to ask for help during your investigation. If you manage to find quality issues, you also know who is likely to provide the resolution. 

And you discovered all of this independently, without disrupting others’ work. This is a major win for you, your busy colleagues, and your organisation.

More Sleuthing To Come

There are analyses I could not mention here due to lack of space. Let me know if you are interested in more and I will write follow-up articles! 

As I do more work on code-compass, I plan to automate matching of code and user flows so that the user could jump with a click between code and its corresponding user flow.

If you want to learn more about this, feel free to contact me. Happy forensics testing!

References and Associated Tools

Selenium Docs

• Selenium RemoteWebDriver

For Further Reading

• How to Identify Risk in a Large Code Base with Noah Sussman

• Your Code as a Crime Scene

• Software Design X-Rays

Tools

• CodeScene: this tool, created by Adam Tornhill, can be used to manage software and organizational complexity. code-compass builds upon CodeScene.

• code-risk: this is a set of scripts Noah Sussman uses to find quality issues in repositories. code-compass includes these.

• code-forensics: this repository makes available code-maat analyses in a node application. code-compass offers a subset of these for now and focuses more on supporting you while you edit your project.

Author Bio

Andrea Giugliano is a software engineer on a DevOps team at ZorgDomein. He tests in a multidimensional way (unit/system/integration/e2e/performance/exploratory) to help guarantee a great user experience. He holds a Ph.D. in software verification and enjoys refining testing tools and sharing solid, novel ideas to help prepare the testing community for new challenges. 

If you want to catch up with him:

https://club.ministryoftesting.com/u/testthisout

http://ag91.github.io/blog/