It is not a secret that writing code can sometimes take less time than making sure it works correctly. Running tests is crucial to guarantee the new build of your software has a successful release. Automating this task can save a lot of time for your developers and reduce time to market. But how can you be sure the tests you run are efficient enough?
Code coverage
It is logical to strive to test as much of your code and as thoroughly as possible. That is why, for example, unit testing is used; it allows testing of smaller, logically isolated pieces of code like a property or a function. Also, that is why automated testing is so popular, it saves time and increases efficiency (learn more about test automation frameworks.) It is important, though, to understand exactly how much of your code is covered by your testing, to make sure there are no significant blind spots.
Code coverage measures how many of your lines of code are executed when you run automated tests. This metric shows the parts of the code that are covered or not covered by testing, which allows you to further improve your tests and potentially discover more bugs. Code coverage is usually calculated as a percentage. Besides lines or statements, many tools also allow you to vary the method of coverage, branches, etc.
You can use code coverage analysis as prerequisites in your CI/CD pipeline or deployment process, for example, when running security checks. Depending on the project, you might decide to only merge pull requests when code coverage on the latest changes, or overall, is higher than 70%.
There are code coverage tools for many different programming languages. However, Java is one of the most popular languages today, so let’s take a look at some of the code coverage tools that work for Java.
Check what we can offer in terms of Java software development services.
Java code coverage tools
There are two types of code coverage tools for Java. The first uses the source code instrumentation technique, which means these tools gather information about which lines of code are executed by adding statements into the source code. This type of tool requires recompiling the code first.
The other type of Java code coverage tool uses bytecode instrumentation. Bytecode, as opposed to the source code that is readable by humans, is a binary machine code that is readable by computer processors and represents a set of instructions for Java Virtual Machine. Bytecode is one of the reasons Java is so popular; it makes Java platform-independent, since any machine with JVM can execute the bytecode generated after the program is compiled.
So, let’s take a look at the best code coverage tools that offer both types of code coverage. Let’s start our review with some solutions that have been around for a while.
Clover
Atlassian originally introduced Clover as a commercial code coverage tool but its support ended in 2017. Since then, Clover has been an open-source solution and has remained popular due to a number of available integrations with Java build tools, like Maven and Ant, or CI solutions, like Bamboo and Jenkins.
Clover instruments the source code which requires source re-compilation but it has a lot of advantages too. Thanks to code coverage measuring, you can collect and analyze more than 20 different code metrics. Clover offers reports in multiple formats and optimizes testing to ensure only the recently modified parts of the code are tested instead of the whole source.
The flexibility of this tool is rather impressive, you can adjust the coverage scope as required by excluding either single statements, methods, classes or even whole files and packages.
Since it has become an open-source project, this tool is known as OpenClover. The same developers that provided maintenance for Atlassian Clover continue to work on OpenClover as well. Besides Java, it can also handle AspectJ and Groovy.
Check out Clover here 
.
Emma
It is rare to see a toolkit that has not been actively developed since 2005 still remain widely used, but it appears Emma has managed to achieve that. This open-source solution uses bytecode instrumentation and has a long list of impressive features, but it is most notable for its speed of operation and support of enterprise development on a large scale.
Emma can output reports in multiple formats, and the HTML version even allows it to link to the source code. This toolkit supports basic block, method, and class coverage types but can’t always detect the coverage of a single line in the code. It also supports Ant and Makefile build integration.
The main issue with Emma, of course, is that the latest version of Java is not supported. A heightened demand for its replacement led to the same team developing a new Java code coverage toolkit, JaCoCo.
Check out Emma here .
JaCoCo
This free code coverage tool is the very embodiment of Java code coverage, since that is what its name means. Just like Emma, it is a free, open-source solution. A lot of features are similar to what was available in Emma too, but JaCoCo is compatible with the latest versions of Java and is still being actively developed.
JaCoCo, like Emma before it, can instrument bytecode both offline and when a Java agent runs the code (on the fly). It can send collected code coverage data via TCP or store it in HTML, XML, or CSV format. JaCoCo is integrated with many popular solutions, like Jenkins, Visual Studio, Gradle, Maven, Ant, and others. Most notably, JaCoCo has replaced Emma in the EclEmma code coverage plugin developed by the Eclipse Foundation.
Among other advantages of JaCoCo, it is worth mentioning that it covers not only lines, branches, methods, and instructions, but also cyclomatic complexity. This toolkit shows high performance with a minimum runtime for large-scale projects. Its modular design also simplifies the development of custom plugins. JaCoCo supports both functional and regression testing but only if test cases are based on JUnit (the Java framework for unit testing).
Check out JaCoCo here .
Cobertura
Some consider Cobertura one of the code coverage tools that is the easiest to manage. That might explain why it is still in use, even though it was introduced in 2004. This open-source tool is often compared to JaCoCo, while it seems it should actually be compared to Emma, as Cobertura is also no longer actively supported and Java 10 is the last version compatible with it. However, there are multiple QA tools integrated with Cobertura, and if you use Maven or Eclipse for your Java project build automation, then using it might still be viable.
One of the advantages of using Cobertura is its configurable memory usage which could be a significant benefit for larger projects. In general, it is similar to JaCoCo in that it also instruments bytecode, but it offers fewer features overall. You can generate convenient XML or HTML reports with information on line and branch coverage either for the whole project or each package/class separately.
Check out Cobertura here .
JCov
This is another tool that has been around for a long time, ever since Java 1.1. JCov instruments bytecode both statically and dynamically, which means it can function both by changing the code that is tested upfront and on the fly. JCov used to be a part of Oracle JDK.
It has been an open-source solution since 2014 and thanks to public API it is easy to customize its coverage, data mining and/or filtering. It provides branch, method, block, and line coverage, as well as abstract API and field coverage. With JCov it is possible to gather per-test coverage information.
It is also one of the tools compatible with a more modern QA solution, Codecov, which allows you to plug in various coverage tools as long as they output data in compatible formats.
Check out JCov here .
Codecov
If you want to get the most out of your coverage metrics, Codecov can help with that. This solution connects directly to GitLab, GitHub or Bitbucket and aims to make your workflow healthier by promoting pull requests code coverage. Codecov integrates into your CI/CD process and conveniently merges the code coverage data from various tools in different languages into a single report.
Codecov is compatible with OpenClover, JaCoCo and JCov but it also supports multiple tools for other languages. It allows you to motivate your developers to increase code coverage by inserting the coverage metrics directly into the workflow and thus making them more visible. It is also an easy way to track coverage percentage across multiple commits, and discover and find the cause of unexpected coverage changes.
There are multiple versions of Codecov available, including both free and paid options. You can choose the open-source or enterprise solution, whichever suits your project needs best.
Check out Codecove here .
Parasoft JTest
The testing suite from Parasoft includes multiple modern QA tools that cover all the testing needs of your project. JTest is the part of this suite that, among other things, allows measurement of code coverage, analyzing it statically and dynamically. It supports code coverage by condition, path, branch, block, statement or line. It also helps with the JUnit testing itself, since it can create unit tests automatically.
Another advantage of Parasoft JTest is the in-built test impact analysis. When you change something in the code, JTest can automatically determine which tests are impacted by these changes and then run only this test subset. You can also quickly get detailed feedback on the testing, including suggestions on fixing the most significant vulnerabilities and bugs.
JTest makes it easy to demonstrate your code compliance with multiple established security standards, since not only does it run the necessary checks, it also automatically creates the respective documentation. Reporting possibilities include a web dashboard where you can review a combination of multiple test results.
Since only custom pricing on demand is available for Parasoft JTest, the solution might turn out to be rather expensive. It might be a better fit for larger enterprise projects rather than startups.
Check out Parasoft JTest here .
Conclusions
A higher percentage of code coverage means improving the quality of the testing and, consequently, the quality of the code in your project. However, it doesn’t mean you have to set your code coverage target at 100% (even if that percentage is reached, it might not mean 100% of cases get checked).
There are multiple factors that will define your target coverage. For example, covering as much code as possible with tests isn’t something on which you should focus. If your coverage is very high but the tests are aimed at simply covering most of the source code instead of actually discovering issues, such testing might turn out to be useless. Another thing to consider is that some features might be free of bugs but not work as intended; improving code coverage won’t help with that issue. Which is why it is also vital to raise the quality of unit tests. The better your unit tests, the easier it will be for developers to move on to really important work on other features. They essentially provide the current product state data so when unit tests are good enough, developers won’t have to check with the product owner or documentation for that information.
Finally, the project itself can determine what your coverage target should be. When you work on a long-term project with legacy functionality and a large codebase not previously covered by tests, it is reasonable to aim to create tests for existing features when you update or modify them but cover new functionality as soon as it is added. The code coverage for such a project might be lower than 50% but it would still be a viable result.
There is a wide range of Java code coverage tools available these days, some of them have emerged recently, while others have withstood the test of time. You can choose the one with the integration capabilities and reporting formats that would work best for your unique project requirements.
