Java code coverage tools

Java code coverage tools are of two types: first, tools that add statements to the Java source code and require its recompilation. Second, tools that instrument the bytecode, either before or during execution. The goal is to find out which parts of the code are tested by registering the lines of code executed when running a test.

JaCoCo

JaCoCo

Developer(s)	Marc Hoffmann, Brock Janiczak, Evgeny Mandrikov, Mirko Friedenhagen
Stable release	0.8.12 / March 31, 2024;4 months ago (2024-03-31)
Type	Code coverage
License	EPL
Website	www.jacoco.org/jacoco

JaCoCo is an open-source toolkit for measuring and reporting Java code coverage. JaCoCo is distributed under the terms of the Eclipse Public License. It was developed as a replacement for EMMA, ^[1] under the umbrella of the EclEmma plug-in for Eclipse.

Features

JaCoCo offers instructions, line and branch coverage.

In contrast to Atlassian Clover and OpenClover, which require instrumenting the source code, JaCoCo can instrument Java bytecode using two different approaches:

• like JCov on the fly while running the code with a Java agent ^[2]
• like Cobertura and JCov prior to execution (offline)

And can be configured to store the collected data in a file, or send it via TCP. Files from multiple runs or code parts can be merged easily. ^[3] Unlike Cobertura and EMMA it fully supports Java 7, Java 8, ^[4] Java 9, Java 10, Java 11, Java 12, Java 13, Java 14, Java 15, Java 16, Java 17, Java 18, Java 19 and Java 20.

Tools using or including JaCoCo

• SonarQube JaCoCo plugin — one of the defaults for coverage analyses within the code quality management platform SonarQube
• EclEmma Eclipse (software) Code Coverage Plugin, was formerly EMMA based ^[5]
• Jenkins JaCoCo Plugin ^[6]
• Netbeans JaCoCo support ^[7]
• IntelliJ IDEA since v11 ^[4]
• Gradle JaCoCo Plugin ^{[8] [9]}
• Maven JaCoCo Plugin ^{[10] [11]}
• Visual Studio Team Services ^[12]
• TeamCity ^[13]
• STAMP (https://www.stamp-project.eu/%5B%5D) ^[14]

JCov

JCov

Developer(s)	Leonid Arbouzov, Alexander Petrov, Vladimir Generalov, Serguei Chukhontsev, Oleg Uliankin, Gregory Steuck, Pavel Ozhdikhin, Konstantin Bobrovsky, Robert Field, Alexander Kuzmin, Leonid Mesnik, Sergey Borodin, Andrey Titov, Dmitry Fazunenko, Alexey Fedorchenko, Leonid Kuskov
Stable release	3.0 / September 1, 2014;9 years ago (2014-09-01)
Type	Code coverage
License	GPL v2 (with the Classpath Exception)
Website	wiki.openjdk.java.net/display/CodeTools/jcov

JCov is the tool which has been developed and used with Sun JDK (and later Oracle JDK) from the very beginning of Java: from the version 1.1. JCov is capable of measuring and reporting Java code coverage. JCov is distributed under the terms of the GNU General Public License (version 2, with the Classpath Exception). JCov has become open-source as a part of OpenJDK code tools project in 2014.

Features

JCov is capable of reporting the following types of code coverage:

• Block coverage
• Line coverage
• Branch coverage
• Method coverage

JCov implements two different ways to save the collected data:

• Into a file on the filesystem
• Onto a server (a.k.a. "network grabber")

JCov works by instrumenting Java bytecode using two different approaches:

• Static instrumentation which is done upfront, changing the tested code
• Dynamic instrumentation which is done on the fly by means of Java agent

JCov has a few more distinctive features which include, but are not limited to:

• Field coverage
• Abstract API coverage
• Direct/indirect coverage
• Per-test coverage information (a.k.a. "test scales")
• Public API and SPI which makes it possible to implement custom filtering and/or mining the coverage data

Tools using JCov

• Oracle JDK (SE and ME)
• JCK (the Java Compatibility Kit)
• Various Java SE and Java ME TCKs
• Java FX SDK
• Java FX Scene Builder

OpenClover

OpenClover

Developer(s)	Marek Parfianowicz, Grzegorz Lewandowski
Stable release	4.5.2 / January 31, 2024;6 months ago (2024-01-31)
Type	Code coverage
License	Apache License 2.0
Website	openclover.org

OpenClover is a free and open-source successor of Atlassian Clover, created as a fork from the Clover code base published by Atlassian in 2017. It contains all features of the original Clover (the server edition). The OpenClover project is led by developers who maintained Clover in years 2012–2017. ^[15]

OpenClover uses source code instrumentation technique and handles Java, Groovy and AspectJ languages. Some of its features include: fine control over scope of coverage measurement, test optimisation and sophisticated reports.

OpenClover integrates with Ant, Maven, Gradle, Grails, Eclipse, IntelliJ IDEA, Bamboo, Jenkins, Hudson, Griffon, SonarQube and AspectJ.

IntelliJ IDEA Code Coverage Agent

IntelliJ IDEA Code Coverage Agent

Developer(s)	JetBrains
Type	Code coverage
License	Apache 2.0
Website	github.com/JetBrains/intellij-coverage

IntelliJ IDEA Code Coverage Agent is a code coverage tool integrated in IntelliJ IDEA IDE and TeamCity CI server. It supports branch coverage and per-test coverage tracking.

Testwell CTC++ for Java

Testwell CTC++

Developer(s)	Verifysoft Technology
Stable release	8.0. / June 30, 2016;8 years ago (2016-06-30)
Type	Code coverage
License	Proprietary
Website	verifysoft.com/en_ctcpp.html

Testwell CTC++ is a code coverage tool for C, C++, Java and C#. The development of this tool started in 1989 at Testwell in Finland. Since 2013 support and development has been continued by Verifysoft Technology, a company from Offenburg, Germany. Testwell CTC++ analyses for all code coverage levels up to Modified condition/decision coverage and Multicondition Coverage. ^[16] The tool works with all compilers. ^[17]

Notable historic tools

Clover

"Clover (software)" redirects here. For the dating app, see Clover (mobile app).

Clover

Developer(s)	Atlassian
Stable release	4.1.2 / October 11, 2016;7 years ago (2016-10-11)
Type	Code coverage
License	Apache 2.0
Website	atlassian.com

Clover is a Java code coverage analysis utility bought and further developed by Atlassian. In April 2017 Atlassian announced end-of-life of Clover and at the same time open-sourced it under Apache 2.0 license.

Clover uses a source code instrumentation technique (as opposed to Cobertura and JaCoCo, which use byte code instrumentation), which has its advantages (such as an ability to collect code metrics) and disadvantages (re-compilation of sources is necessary). ^[18] Some of its features include historical reporting, huge control over the coverage gathering process, command line toolset and API for legacy integration and more.

Clover also allows testing time to be reduced by only running the tests that cover the application code that was modified since the previous build. This is called Test Optimization ^[19] and can lead to huge drops in the amount of time spent waiting for automated tests to complete.

Clover comes with a number of integrations both developed by Atlassian (Ant, Maven, Grails, Eclipse, IDEA, Bamboo) and by open source community (Gradle, Griffon, Jenkins, Hudson, Sonar).

In April 2017, Atlassian announced that they would no longer release new versions of Clover after version 4.1.2, and its code was made available as open-source software hosted on Bitbucket. ^{[20] [21]}

Cobertura

Cobertura

Developer(s)	Steven Christou
Stable release	2.1.1 / February 26, 2015;9 years ago (2015-02-26)
Repository	github.com/cobertura/cobertura
Type	Code coverage
License	GPL 2.0
Website	cobertura.github.io/cobertura/

Cobertura is an open-source tool for measuring code coverage. It does so by instrumenting the byte code. It was the predecessor to JaCoCo.

EMMA

EMMA

Developer(s)	Vlad Roubtsov
Stable release	2.1 / May 13, 2005;19 years ago (2005-05-13)
Repository	sourceforge.net/projects/emma/
Type	Code coverage
License	Common Public License 1.0
Website	emma.sourceforge.net

EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA is distributed under the terms of Common Public License v1.0.

EMMA is not currently under active development; the last stable release took place in mid-2005. As replacement, JaCoCo was developed. ^[22] EMMA works by wrapping each line of code and each condition with a flag, which is set when that line is executed. ^[23]

Features

• instrument classes for coverage either offline (before they are loaded) or on the fly (using an instrumenting application classloader).
• Supported coverage types: class, method, line, basic block. EMMA can detect when a single source code line is covered only partially.
• Coverage stats are aggregated at method, class, package, and "all classes" levels.
• Output report types: plain text, HTML, XML. All report types support drill-down, to a user-controlled detail depth. The HTML report supports source code linking.
• Output reports can highlight items with coverage levels below user-provided thresholds.
• Coverage data obtained in different instrumentation or test runs can be merged.
• it is possible to dump or reset coverage data remotely and without a JVM exit.
• does not require access to the source code and degrades gracefully with decreasing amount of debug information available in the input classes.
• can instrument individual .class files or entire .jars (in place, if desired). Efficient coverage subset filtering is possible, too.
• Makefile and ANT build integration are supported on equal footing.
• The runtime overhead of added instrumentation is small (5–20%) and the bytecode instrumentor itself is very fast (mostly limited by file I/O speed). Memory overhead is a few hundred bytes per Java class.
• EMMA is 100% pure Java, has no external library dependencies, and works in any Java 2 JVM (even 1.2.x).

Serenity

Serenity

Developer(s)	Michael Couck
Stable release	1.0 / December 8, 2013;10 years ago (2013-12-08)
Type	Code coverage
License	Apache Software License version 2.0
Website	wiki.jenkins-ci.org/display/JENKINS/Serenity+Plugin

Serenity is an open-source tool creating better-automated software acceptance tests in less time. It and measures and reports Java code coverage. It also generates easy-to-understand reports that describe what the application does and how it works, including which tests were run and what requirements were met. It works with Selenium WebDriver, Appium, and BDD tools.

Major code metrics such as cyclometric complexity, stability, abstractness, and distance from main are measured. The report data is persisted to an object database and made available via Jenkins/Hudson. The interface visually replicates the Eclipse IDE interface.

Serenity dynamically enhances the byte code, making a post-compile step unnecessary. Ant and Maven projects are supported. Configuration is done in xml, an Ant example would be:

<!-- Serenity system properties. --><syspropertykey="included.packages"value="your.package.name.here"/><syspropertykey="included.adapters"value="coverage,complexity,dependency"/><!-- Serenity JVM command line. --><jvmargline="-javaagent:serenity/serenity.jar"/>

And a Maven configuration example would be:

<properties><included.packages>-Dincluded.packages=your.package.name.here</included.packages><included.adapters>-Dincluded.adapters=coverage,complexity,dependency</included.adapters></properties><argLine>-javaagent:serenity/serenity.jar-Xms512m-Xmx1024m${included.packages}${included.adapters}</argLine>

For a full example of a configuration please refer to the Jenkins wiki at https://wiki.jenkins-ci.org/display/JENKINS/Serenity+Plugin.

Jenkins slaves as well as Maven multi module projects are supported.

References

1. JaCoCo Mission
2. Patroklos Papapetrou (19 December 2012). "Code Coverage Tools (JaCoCo, Cobertura, Emma) Comparison in Sonar". Only Software matters. Retrieved 3 March 2013.
3. "Measure Coverage by Integration Tests with Sonar – Updated". Archived from the original on 2013-02-23. Retrieved 2013-03-01.
4. "Code Coverage". IntelliJ IDEA 12.0 Web Help. JetBrains. Archived from the original on 26 April 2013. Retrieved 3 March 2013.
5. EclEmma, Eclipse code coverage plugin
6. Jenkins JaCoCo Plugin
7. "NetBeans JaCoCo support". Archived from the original on 2012-05-31. Retrieved 2013-02-22.
8. Gradle JaCoCo Plugin
9. Gulati & Sharma 2017, pp. 99–103, Chapter §6 Integrating Tools - Build Tools - Gradle.
10. Maven JaCoCo Plugin
11. Gulati & Sharma 2017, p. 115, Chapter §6 Integrating Tools - Build Tools - Maven Extension.
12. JaCoCo integration in Visual Studio Team Services
13. JaCoCo integration in TeamCity
14. Jacoco is used by STAMP DSpot tool as a test selector (seen as a fitness) to select tests that increase the coverage and have unique executed path
15. OpenClover - About us
16. Testwell CTC++ supports all coverage levels
17. Testwell CTC++ supports all compilers
18. "Why does Clover use source code instrumentation?"
19. Test Optimization
20. "Atlassian Clover is now open source". atlassian.com. 11 April 2017.
21. "atlassian/clover". bitbucket.org.
22. EMMA code coverage files on SourceForge.net
23. Expert Spring MVC and Web Flow; By Seth Ladd, Darren Davison, Steven Devijver, Colin Yates, p. 289

Citations

• Gulati, Shekhar; Sharma, Rahul (2017). Java Unit Testing with JUnit 5. Berkeley, CA: Apress. doi:10.1007/978-1-4842-3015-2. ISBN   978-1-4842-3014-5.

External links

• Comparison of OpenClover, Clover, Cobertura, JaCoCo, JCov, CodeCover and PIT
• JaCoCo vs Clover2
• Pick your code coverage tool in Sonar 2.2 Archived 2013-02-23 at the Wayback Machine , a little outdated, as JaCoCo meanwhile supports branch coverage
• Testing Java Applications with Visual Studio Team Services