1. Overview
In this tutorial, we’ll learn about property-based testing and understand this approach by comparing it to the more traditional example-based testing.
After that, we’ll dive into Jqwik, a library dedicated to this approach for unit testing. We’ll explore Jqwik’s API and use it to write our first property-based test.
Finally, we’ll discuss a few helpful features that enable us to verify our code with a comprehensive set of input parameters and cover all the edge cases.
2. Examples vs. Properties
The traditional and widely adopted approach to unit testing is known as “example-based” testing. In this context, examples are just pairs of input parameters and expected output values.
On the other hand, property-based testing verifies software based on general rules it should conform to. To verify these properties, we use large amounts of randomly generated inputs and try to cover all the edge cases of our software.
Let’s start exploring the differences between the two approaches using JUnit5 parameterized tests.
2.1. Example-Based Testing
Example-based testing involves verifying a function’s output against expected results using specific predefined input cases. These inputs – or “examples” – can be defined separately in individual tests, or specified as a collection and processed by a parameterized test.
Let’s imagine we want to test a class that converts integers to Roman numerals. We can write a simple JUnit5 @ParameterizedTest that supplies a set of examples through the @MethodSource annotation:
class JUnit5_ExampleVsPropertyBasedUnitTest {
val converter = RomanNumeralsConverter()
companion object {
@JvmStatic
fun intsToRomanNumerals() = listOf(
Arguments.of(3, "III"),
Arguments.of(4, "IV"),
Arguments.of(58, "LVIII"),
Arguments.of(1234, "MMMCMXCIX")
);
}
@ParameterizedTest
@MethodSource("intsToRomanNumerals")
fun `should converted integer to roman numeral`(integer: Int, roman: String) {
assertEquals(roman, converter.intToRoman(integer))
}
}
Furthermore, we can now re-use the examples for the opposite method, romanNumeralToInt(), we only need to adjust the assertion:
@ParameterizedTest
@MethodSource("intsToRomanNumerals")
fun `should converted roman numeral to integer`(integer: Int, roman: String) {
assertEquals(integer, converter.romanToInt(roman))
}
As we can notice, our test only covers four input-output pairs. Consequently, we cannot guarantee the converter functions correctly for other inputs and special cases. For instance, none of the “examples” covers a Roman numeral containing the character “D”, equivalent to 500.
2.2. Property-Based Testing
Property-based testing relies on validating the tested component against a large set of generated inputs. Moreover, when writing the test, we don’t know the specific input parameters that will be generated. Consequently, predicting the expected return value and directly asserting against it will be impossible.
As a result, we’ll need to change our mindset and identify some generic rules – or “properties” – that our software should adhere to across a wide range of input values.
If we look at the RomanNumeralConverter from the previous example, we can identify an interesting property. If we convert an integer to a Roman numeral and then back to an integer, the function should yield the original number. Therefore, we can use JUnit5 to write our first property-based test and execute it for a list of 300 random integers between zero and 4.000.
Firstly, let’s create a function that generates the test inputs:
companion object {
// ...
@JvmStatic
fun randomIntsInRangeInRangeZeroTo4k() = (0..<4000).shuffled().take(300)
}
Now, let’s use this method as an argument source for our test:
@ParameterizedTest
@MethodSource("randomIntsInRangeInRangeZeroTo4k")
fun `should converted integer to roman numeral and back`(integer: Int) {
val roman = converter.intToRoman(integer)
assertEquals(integer, converter.romanToInt(roman))
}
As we can see, the property-based test covers many different code paths and it can potentially identify edge cases. On the other hand, our simplistic approach that uses randomly generated inputs also has some downsides:
- We cannot re-run the test with the same inputs, to see if an issue was fixed
- We cannot make sure we include special values, such as interval limits
- In case of failures, we don’t have a clear description of the cause of the issue
- Creating argument sources for a wide variety of data types can be a tedious task
Luckily, there are libraries dedicated to property-based testing, such as Jqwik, which tackle these issues. Moreover, they offer additional features and a declarative API that allows us to craft expressive tests.
3. Getting Started with Jqwik
Firstly, we need to add the jqwik and jqwik-kotlin dependencies to our pom.xml:
<dependency>
<groupId>net.jqwik</groupId>
<artifactId>jqwik</artifactId>
<version>1.8.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>net.jqwik</groupId>
<artifactId>jqwik-kotlin</artifactId>
<version>1.8.5</version>
<scope>test</scope>
</dependency>
Additionally, we’ll add a few compiler options that will make debugging easy:
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<version>1.9.4</version>
<configuration>
<args>
<arg>-java-parameters</arg> <!-- Get correct parameter names in jqwik reporting -->
<arg>-Xjsr305=strict</arg> <!-- Strict interpretation of nullability annotations in jqwik API -->
<arg>-Xemit-jvm-type-annotations</arg> <!-- Enable annotations on type variables -->
</args>
</configuration>
</plugin>
Now, let’s create a new class and start using Jqwik’s API. The first change we’ll notice is that our tests will be annotated with @Property. Additionally, we can use @ForAll to indicate the fields where we need randomly generated values.
Let’s write a simple test that supplies random integers and logs them to the console. As an assertion, we can check that the argument is of type Int:
@Property
fun `should generate input values`(@ForAll arg: Int) {
println("$arg")
assertTrue { arg is Int }
}
If we execute the test and check the console, we’ll find 1000 arguments used as test input. Following that, we’ll see a brief report that indicates, among others, the number of executions, the number of edge cases, and the seed that was used for generating the random values:
timestamp = 2024-06-25T11:54:56.857321, Jqwik PropertyBasedUnitTest:should generate input values =
|-----------------------jqwik-----------------------
tries = 1000 | # of calls to property
checks = 1000 | # of not rejected calls
generation = RANDOMIZED | parameters are randomly generated
after-failure = SAMPLE_FIRST | try previously failed sample, then previous seed
when-fixed-seed = ALLOW | fixing the random seed is allowed
edge-cases#mode = MIXIN | edge cases are mixed in
edge-cases#total = 9 | # of all combined edge cases
edge-cases#tried = 9 | # of edge cases tried in current run
seed = -1329617974163688825 | random seed to reproduce generated values
The @Property annotation enables us to customize these generated parameters. We can now write another test that verifies only 100 inputs using the same seed. Additionally, Jqwik allows us to skip the assertion and simply return a boolean value instead, let’s use this feature as well:
@Property(tries = 100, seed = "-1329617974163688825")
fun `should generate input values for a given seed`(@ForAll arg: Int): Boolean {
println("$arg")
return arg is Int
}
4. Generating Arguments Through Annotations
At this point, we should know enough to test the RomanNumeralsConverter with Jqwik:
@Property
fun `should convert integer to roman and back`(@ForAll originalInt: Int): Boolean {
val roman = converter.intToRoman(originalInt)
val finalInt = converter.romanToInt(roman)
return originalInt == finalInt
}
Right now, the test will fail because our implementation only supports positive integers smaller than 4.000. However, we can fix it by using Jqwik annotations to constrain the parameters that need to be generated.
4.1. Generating Integers
For integers, we can use annotations that help us declaratively describe the arguments to be generated. For our code example, we’ll need @Positive integers from a given @IntRange:
@Property
fun `should convert integer to roman and back`(
@ForAll @Positive @IntRange(max = 3999) originalInt: Int
): Boolean { /* ... */ }
Needless to say, we can avoid using @Positive by defining zero as the minimum value of the range, @IntRange(min = 0, max = 3999), and the test will have the same outcome. Furthermore, if the number of arguments is smaller than the configured number of tries, Jqwik will perform exhaustive testing, validating all possible values.
We can achieve this by setting the maximum number of tries to 4.000:
@Property(tries = 4000)
fun `should convert integer to roman and back for all valid inputs`(
@ForAll @IntRange(min = 0, max = 3999) originalInt: Int,
): Boolean { /* ... */ }
If we re-run the test, we’ll notice that the generation property of the report changed from “RANDOMIZED” to “EXHAUSTIVE“, indicating that all the possible combinations were verified.
4.2. Generating Strings
We can generate String arguments just by marking String arguments with the @ForAll annotation. Moreover, for debugging, we can use the method-level annotation @Report which will log the arguments of each execution:
@Report(Reporting.GENERATED)
@Property(tries = 10)
fun `should generate Strings`(
@ForAll firstName: String,
@ForAll lastName: String
): Boolean { /* ... */ }
Let’s run the test and inspect a few of the test executions:
ඈⷅ㷋唸쌌晿 蘖菾Ḩ塂遃덐瘻鬤timestamp = 2024-06-25T15:11:08.981301700, generated =
arg0: "捧뺢跿澌㨴ް뽳"
arg1: "ë果"
捧뺢跿澌㨴ް뽳 ë果timestamp = 2024-06-25T15:11:08.986314500, generated =
arg0: "�"
arg1: "�"
By default, Jqwik runs the test with as many special arguments as possible, trying to find all the edge cases. In the case of Strings, we can find arguments passed as empty Strings and unprintable characters. Of course, we have constraints dedicated to Strings, that can be chained together to meet our test’s needs. Here are some of the options we have at our disposal:
- @AlphaChars – uses alphanumeric characters
- @UniqueChars – avoids duplicating characters within the String argument
- @NotEmpty – avoids Strings with length equal to zero
- @NotBlank – avoids Strings consisting only of whitespace
- @Whitespace – includes whitespace
- @CharRange – uses a defined range of characters
- @StringLength – uses a certain String length
Let’s use them to configure the first argument as an alphanumeric, non-blank string, with unique characters, and the second one as a String containing special characters and whitespace, which is between three and five characters long:
@Property
fun `should generate Strings`(
@ForAll @AlphaChars @UniqueChars @NotBlank foo: String,
@ForAll @Whitespace @CharRange(from = '!', to = '&') @StringLength(min = 3, max = 5) bar: String,
): Boolean { /* ... */}
4.3. Generating Other Types
The library exposes helpful annotations that enable us to generate specific collections as arguments. Depending on the collection’s implementation, Jqwik can also try to supply collections containing duplicate elements. If this behavior is not needed for our test, we can avoid it via @UniqueElements. Additionally, we can use annotations like @NotEmpty or @Size to restrict the number of elements for the generated test inputs:
@Property(tries = 10)
fun `should generate Lists of Doubles`(
@ForAll @NotEmpty @Size(max = 10) numbers: @UniqueElements List<Double>
): Boolean { /* ... */ }
There are out-of-the-box features for other data types with a finite number of possible values, such as enums and booleans. To generate arguments from enums, booleans, or nullable values, we simply need to annotate these arguments with @ForAll, same as before:
@Property
fun `should generate Enums and Booleans`(
@ForAll month: Month,
@ForAll isLeapYear: Boolean?,
@ForAll @IntRange(min = 2000, max = 2024) year: Int,
): Boolean { /* ... */ }
Furthermore, there is a finite number of combinations when generating this test’s input, namely: 12 * 3 * 25 = 900. Since this number is smaller than the default number of tries, 1.000, we can expect Jqwik to perform exhaustive testing in this case.
5. Generating Arguments Programmatically
So far, we have only generated arguments for generic types and constrained them using the various annotations. However, in some cases, we might want to reuse certain generators or create customized arguments.
Let’s imagine we find ourselves duplicating the same group of annotations throughout various tests:
@Property
fun `should be able to sign up`(
@ForAll @AlphaChars @NumericChars @UniqueChars @StringLength(min = 3, max = 8) username: String
): Boolean { /* ... */ }
@Property
fun `should be able to login`(
@ForAll @AlphaChars @NumericChars @UniqueChars @StringLength(min = 3, max = 8) username: String
): Boolean { /* ... */ }
5.1. Argument Providers
To tackle the duplication, Jqwik exposes the Arbitraries API and enables us to create and re-use our custom data providers, tailor-made for our use cases. So, instead of the annotations, we’ll create a function and annotate it with @Provide. This function will use Jqwik’s fluent API to build an Arbitrary object of a specific type:
@Provide
fun usernames(): Arbitrary<String> = String.any()
.alpha()
.numeric()
.uniqueChars()
.ofLength(3..8)
After that, we can use the @ForAll annotation to point to the appropriate provider:
@Property
fun `should be able to sign up`(@ForAll("usernames") username: String): Boolean { /* ... */ }
@Property
fun `should be able to login`(@ForAll("usernames") username: String): Boolean { /* ... */ }
On the other hand, we can combine Arbitraries to create generators for custom objects, specific to our application. For instance, let’s assume we have an Account class as part of our domain model:
data class Account(val id: Long, val username: String, val dateOfBirth: LocalDate)
Firstly, we’ll create an Arbitrary instance for each field we want to generate randomly. After that, we’ll use combine() and provide a lambda to show how to merge the three parameters into an Arbitrary
@Provide
fun accounts(): Arbitrary<Account> {
val ids: Arbitrary<Long> = Long.any().greaterOrEqual(100)
val usernames: Arbitrary<String> = String.any().alpha().numeric().repeatChars(15.0).ofLength(3..8)
val datesOfBirth: Arbitrary<LocalDate> = Dates.dates().yearBetween(1900, 2000)
return combine(ids, usernames, datesOfBirth) { id, name, dob -> Account(id, name, dob) }
}
Needless to say, we can reuse the usernames() provider function from the previous example, and use a method reference for instantiating the Account class. Let’s refactor the code to simplify it:
@Provide
fun accounts(): Arbitrary<Account> {
val ids = Long.any().greaterOrEqual(100)
val datesOfBirth = Dates.dates().yearBetween(1900, 2000)
return combine(ids, usernames(), datesOfBirth, combinator = ::Account)
}
As a result, we can now use @ForAll(“accounts”) to verify our logic with a wide variety of generated Account objects:
@Property
fun `should validate account`(@ForAll("accounts") account: Account): Boolean { /* ... */ }
6. Conclusion
In this article, we learned the differences between example-based and property-based testing. We discover that the latter implies a different way of thinking about the tested components, focusing on their properties and effects.
After that, we delved into Jqwik, a library that offers support for this approach to software testing. We discussed its main features that enable us to automatically generate huge amounts of test inputs, trying to find all the edge cases of our software. After that, we learned how to create our own data providers using Arbitrary APIs.
As always, the complete code for this article is available over on GitHub.