Quarkus has always been focused on developer experience, and Quarkus 2.0 has taken this to the next level with support for a feature we are calling continuous testing.
From its inception Quarkus has supported live reload in development mode, where changes to Java files take effect immediately. For testing though our approach has been the same as any other Java application, namely either run them through a build tool such as Maven or Gradle, or run them manually through an IDE. Quarkus 2.0 changes this, and now tests can be run continuously as part of dev mode, so that test results are available immediately.
The Inception
Continuous testing came about due to a post on the Quarkus mailing list back at the beginning of the year from a user who mainly used scripting languages as part of their work. They said that they loved the live reload offered by development mode, but found the testing experience clunky and slow compared to the scripting language world, where tools like jester run tests as you type. This lead to the team thinking that things could definitely be improved, and continuous testing was born.
Seeing it in Action
To show this in action I am going to use the Quarkus Hibernate ORM Quickstart.
In the application directory we run mvn quarkus:dev to start development mode. As this application requires a database and it has not been configured, Quarkus will start one for you in a docker container. Once this has started you will notice that there is now a new status display down the bottom of the screen:
You may have noticed that during startup this same status display was used to show the status of the PostgreSQL database as it was starting. This stops the logs being polluted with docker and testcontainers output, but still allows you to see what is going on.
NOTE: If you are using Gradle you should use the quarkus CLI to launch dev mode to get the status display, instead of running ./gradlew quarkusDev directly. This is because Gradle has its own status display that interferes with the Quarkus one. You can still use continuous testing, but the results will not look as nice and you need to press ‘enter’ for your commands to be read.
This status will display test results once we enable testing, but can also be used to control some other features of Quarkus. If we press ‘h’ we get a full list of options:
For now we are interested in continuous testing, so press ‘r’ to resume. You should see the status display change through a few different states. First a new PostgreSQL database is started (Dev Services starts a new dedicated test database, so tests cannot interfere with development mode and vice versa. This is only started once, so subsequent runs will be faster). Once this has completed you should see the tests running and then the results displayed on the status line:
You can press ‘r‘ again to re-run the tests, and you should see them complete much faster on the second run, as there is no need to wait for the database to start.
NOTE: If you want continuous testing to automatically run on startup without the need to press ‘r’ you can set ‘quarkus.test.continuous-testing=enabled’ in ‘application.properties’.
My code is broken…
Our one test is passing, so let’s break some stuff and see what happens. For the sake of the example I am going to change the path that this is served at, from ‘/fruits’ too ‘/fruit’. As soon as I make this change in FruitResource the tests will immediately run again, and less than a second later I have the following output telling me my test has failed:
If I then go to my test and revert my change I should see the test status change back to everything passing. Note that you will not see any output from the test (log messages or System.out) unless you press ‘o’ to enable test output. This is so that logging from tests won’t overwhelm the console with log messages.
TESTS ONLY PLEASE
So far this has talked about testing from within dev mode, where you can both use your application manually and run the tests at the same time, but what if you just want to run the tests without dev mode? In that case you can run mvn quarkus:test and Quarkus will start in continuous testing mode, but without starting the application.
How about a UI?
Seeing the results on the command line is great, but what if I want more information about the test failures? Quarkus has you covered there as well, as the test runner is integrated into Dev UI, our new web based developer tooling. To access Dev UI press ‘d‘ in the console, and a browser will open to http://localhost:8080/q/dev/. At the bottom of the screen you should see a status bar showing test results:
Click on the clipboard icon and you will be taken to the results page where you can see more details about any failing tests:
You can also control continuous testing from here, and can do many of the same things you can do from the console with keyboard commands.
But my test suite is huge
Well this is cool, but this project has only a single test, so of course it is quick to test. The big question is how this works on a real project with hundreds of tests? If you run the full test suite every time you could be waiting minutes for results.
The answer is that Quarkus will only run the tests that touch changed code. This means that only tests that are directly affected by the changed code will be run (and any already failing tests). Despite this the status at the bottom will always display the full testsuite status even if the last run only tested a subset of the tests.
If you keep making changes while tests are running all these changes are batched together, and when the current run is finished a new run will be started that will incorporate all the changed you have made in the meantime.
How it works
To figure out which code is tested by a given test we use Quarkus ability to modify bytecode. When we load any code from the current project (i.e. code that might be changed) we make a slight modification to the code. Say we have the following method:
This will get transformed into:
This added tracing call allows Quarkus to tell when a specific code is being invoked. When Quarkus runs your tests for the first time it stores all this information, so at the end of the initial run it has an internal map of which code is used by which tests. When code is modified we only run the tests that we know touch this code, and no others. The only real limitation of this approach is that we need to run all the tests at the start to determine this information, however after this initial run we can run targeted tests with almost 100% accuracy.
This is one of the key innovations behind the Quarkus version of continuous testing, as it allows for testing to be very fine grained, without relying on any particular setup or conventions being followed. If you modify some code you can be very confident that all tests related to that code have been run. There are some cases where this won’t be 100% accurate, such as if you generated a random number and ran different code based on the result, but in practice such situations are rare.
Conclusion
Hopefully this article has whetted your interest to try continuous testing out for yourself. If you are already a Quarkus user it’s only a key press away, otherwise head to https://quarkus.io/ and try it out for yourself.
steinhauer.software