On this page… (hide)

Design of the JUnit tests

JUnit is a framework for testing java classes. It is an open source project that has strong support within Ant. JUnit is an “optional task” within Ant, so it is not distributed as part of Ant. You can find out more, and download a binary distribution from http://www.junit.org. The JUnit FAQ is very helpful for beginners (and experts, too).

JUnit is a powerful tool for testing java classes. If used well, it can make two extremely important contributions to the quality of a project:

all available JUnit tests should be executed automatically at the end of the build process. Their successful completion proves that the new source code and distribution continues to fulfill the contract defined by the previous release. Of course, if the test suite is incomplete in any way, then the success of the tests cannot say anything about the untested logic, but what it has to say is still very valuable. It allows a project such as SIP Communicator to retrofit JUnit tests and immediately begin to improve the confidence level of a new release. In effect, the growing set of tests become a definition of the contract.
each single JUnit test is a powerful debugging harness. In practice, the diagnosis of a new bug should begin with writing a new JUnit test that fails because it demonstrates the existence of the bug. Once a test has been written that fails in the predicted manner, it can then be used to debug and fix the defective logic. Once the bug has been fixed, the test remains to prove there is never a case of regression in the remaining life of the project.

Of course, a comprehensive set of JUnit tests can be a source of frustration when you are confronted with the need to change the public signature of a method or constructor. Many tests will be broken by the change, because the “contract” with the external world has been re-written. Rather than curse JUnit, the developer ought to become more respectful of the consequences of inadequate design in a public API. After all, the pain of rewriting the test suite is likely to be less than the pain experienced by a user of the API who unexpectedly discovers his or her application failing catastrophically when upgrading to a new release of the project distribution! A good set of JUnit tests will immediately show the external developer how to use the refactored API successfully.

Design of a new JUnit test suite involves some subtle issues that need to be understood before it can be used effectively as the “policeman” of a project.

What method signatures to test?

Java methods can be private, public, protected or package-private. Although you would think it best to test all kinds of method signatures, this objective will cause problems. In general, it is better to test only the public methods of a class.

If you are prepared to only test the public methods, you immediately gain two advantages:

your JUnit suite will test the contract between the class and any user of the class. You might think your tests would be “more comprehensive” if you test other signatures, but it will blur your focus on the important design of your public API. With a stable, well-designed public API, you won’t be tempted to exploit “back door” methods that rely on visibility of the non-public methods and fields of the class. Your class will be more useful, and more stable over the life of the project. Also, if your protected methods are used at all, they will be exercised by public methods in other classes… provided you have tests for those classes!
your test classes will not need to be part of the packages they are testing. In other words, you can physically separate your working classes from your test classes (source as well as compiled code), and that makes it simpler to distribute a jar that is uncluttered by your tests. For example, the JUnit test for a class called “xxx.yyy.zzz” should be called “test.xxx.yyy.zzz”. (It is possible to achieve a test-free jar with the JUnit tests residing in the same package as the worker classes, but the build.xml logic becomes considerably more complex).

Note: the SIP Communicator project has been defined with its initial unit test classes in a separate directory tree, but within the same java package as the main classes. This allows developers to write unit tests for the protected API of a class. At this stage of the project, it was felt that developers would find this less restrictive packaging helpful when debugging their code.

What methods to test and what not to test?

Well, I’ll assume I’ve convinced you to test only the public methods already! Most JUnit programmers never bother to test simple accessor methods such as setXXX(), getXXX(), isXXX() and hasXXX(). These methods ought to be one-liners that manipulate single protected or (preferably) private fields. If they compile, they ought to be too trivial to be worth testing explicitly.

However, you ought to be using these methods extensively within your other unit tests, so these accessors will be tested to a significant extent in passing. The point is that you ought not to worry about exhaustively testing accessors unless you write deliberately obscure code!

Of course, you ought to be validating the arguments passed to your setXXX() methods. If your validation amounts only to explicitly throwing NullPointerException, ClassCastException, IllegalArgumentException, etc, then you should not exhaustively and explicitly write unit tests for these conditions either.

On the other hand, any method that performs non-trivial validation or processing should be exhaustively tested.

… and you would never have non-final public fields that can be set from outside your class, so you wouldn’t want any tests for those cases either, would you?

How should you test many classes?

You should write one (or more) test classes for a given worker class. Generally speaking, a single test class will be sufficient, but if you find the list of tests getting too big to view, or the setup/teardown logic has too many decisions, then you should refactor your test class into a “basic” and a “complex” test class.

How do you get many tests run within a class?

JUnit has a very useful naming convention. If you put an entire test class into a test suite, JUnit’s test runners will assume you want to run every method that starts with the substring “test”. In other words, it will run a method called “testX” and ignore one called “notestX”.

This mechanism should encourage you to write smaller test classes. If a test is bigger than a screen, you ought to consider refactoring it into several smaller tests.

How do you execute all your test classes?

JUnit’s test runners allow you to “nest” several test classes into a test suite. You can also put test suites inside test suites. A commonly-used scheme is to create a special test class called something like “AllJUnitTests” and explicitly execute it with a JUnit testrunner class. Within an Ant target, you can use the junit task to run this test class.

The “AllJUnitTests” class should have a suite() method. You can then explicitly add your test classes to the current test suite and they will be executed in turn.

In what order will my tests be executed?

You must assume you have no control over the running order of your tests - either over methods within a class, or even over the running order of the test classes.

If you find you must perform initialization and termination to make each individual test method completely self-contained, then use the setup() and teardown() methods, because these are guaranteed to be invoked around each of your tests.

Note: be very careful of using static fields within your test classes. You run the risk that different classloaders will lead to different results when you run the same test suites under different environments.

But what about test methods with names such as “test10xxx” and “test11xxx”?

You should choose method names to help someone unfamiliar with your system to understand the contract of the class under test. By using a numeric order, most IDE’s will sort the methods and so you can quickly decide which tests are logically simpler, and also which tests are logically related. The remainder of the name should be as helpful as possible, e.g. test27sortThreeWithDuplicates. Oh, and don’t forget your tests could execute in any order… but if several fail within a class, you would be well advised to start your problem determination with the test containing the smallest index number!

What if JUnit is not available on the build machine?

Then it is about time you installed it! JUnit ought to be as important to your development environment as Ant (you have got Ant installed, haven’t you?) Anyway, the SIP Communicator build.xml file includes a conditional test for the JUnit classes. If JUnit is not available, a warning message will be printed and the build will be attempted without running the unit tests.

Note: if you choose a target that needs to run any tests, be careful to ensure that Ant can find JUnit when it runs. This is because out build.xml uses the Ant optional task called junit to run unit tests. Ant has some very subtle requirements because of the way it uses classloaders and you should have the JUnit jar on the classpath when you execute Ant. (It is possible to have the build detect your junit jar, but Ant might not find it!)

Summary of JUnit design philosophy

ALL public fields should be static final. All mutable fields should be inaccessible to API users (i.e. outside the package), except via accessor methods. This means the fields do not need their own unit tests.
ALL public methods (except trivial accessors) should have unit tests for every single logic path. If a particular method needs more than (say) 6 tests, it should be refactored so that its logic is simpler to understand and consequently needs fewer tests.
ALL non-public methods should not be tested explicitly. This allows flexibility in the implementation of the public api. You can refactor without breaking all your unit tests. Also, if your unit tests suddenly get compilation errors, then you KNOW you have broken your api contract and need to think hard about the consequences, especially to users outside the project.
Most non-public methods will be called by classes that extend the base class. Sooner or later, the non-public methods will become testable via a higher-level classes public methods.
In practice, rule 4 sometimes violates rule 2. When it does, I write a new trivial public method in the base class and flag it “for JUnit testing ONLY - use this at your peril!”.

Author: Brian Burch