Testing with `zazr-test`¶

dev.zazr:zazr-test checks properties against generated values. You state what must hold for every value; it generates a few hundred values and reports the first one that breaks the rule.

A check is a method call that fails the test itself when a value breaks the rule, so it runs in any test framework, such as JUnit. Check.evaluate returns the result instead, for code that looks at it.

zazr-test follows the design of ZIO Test, the testing library of ZIO: one generator type whose values grow over a run, assertions that explain why they failed, and seeds that replay a run.

MavenGradle (Kotlin)

<dependency>
    <groupId>dev.zazr</groupId>
    <artifactId>zazr-test</artifactId>
    <version>0.5.0</version>
    <scope>test</scope>
</dependency>

dependencies {
    testImplementation("dev.zazr:zazr-test:0.5.0")
}

Running in your test suite¶

zazr-test is not a test runner and not a JUnit engine. A check is a method call inside an ordinary test method.

class ListReverseTest {

    @Test
    void reversingTwiceGivesTheListBack() {
        var lists = Gen.list(Gen.integers()); // Gen<List<Integer>>
        Check.check(lists, list -> assertThat(list.reverse().reverse(), equalTo(list)));
    }
}

mvn test or gradle test runs it through your test framework, like any other test. A check that passes is a green test.

When a sample breaks the property, Check.check throws an AssertionError. The framework reports it as a normal test failure, with the counterexample, the sample number and the seed in the message. The Surefire summary, its XML reports and your CI show it like any other failure.

For a test everyListIsShort whose property is list.size() < 5, run with the seed 42, Maven prints:

[ERROR] Failures:
[ERROR]   ListShortTest.everyListIsShort:13 falsified at sample 14 by (List(1064429137, -1, 2147483646, -499641955, 2147483647)) (seed 42, replay with -Dzazr.check.seed=42)

The example uses JUnit 5. JUnit 4, TestNG and Spock work the same way: the test method calls the check, and the framework reports the AssertionError.

zazr-test depends on no test framework, only on zazr-core.

The types¶

Six types, all in dev.zazr.test:

Type	Role
`Gen<A>`	a generator of values of type `A`: scalars, combinators, and generators for the Zazr types
`Check`	runs a property for 1 to 8 generators: `check`, `checkN` and `checkAll` fail the test when it breaks; `evaluate`, `evaluateN` and `evaluateAll` return the result
`Assertion<A>`	a condition on an `A` that explains why a value breaks it: `equalTo`, `isGreaterThan`, `hasSize`, `isSome`, ...
`TestResult`	the outcome of `assertThat(value, assertions...)`: a success, or a failure with its explanation
`CheckResult`	the outcome of a check: `Satisfied`, `Falsified` or `Erroneous`
`CheckConfig`	the number of samples, the size, the seed and the discard budget

If your tests use Java modules, add requires dev.zazr.test; to their module-info.java: it reads dev.zazr too.

A first property¶

Check.check takes the generators and the property. The property returns the result of assertThat, which says why a value breaks it.

var lists = Gen.list(Gen.integers()); // Gen<List<Integer>>
Check.check(lists, list -> assertThat(list.reverse().reverse(), equalTo(list)));

The check runs 200 samples. It returns when every sample passed, and throws an AssertionError otherwise, so the test fails.

assertThat and the assertions are static methods of Assertion (import static dev.zazr.test.Assertion.*;). A property may also return a plain boolean: true when it holds.

What a failure prints¶

The error names the sample that broke the property, its number, the seed of the run, and why the value breaks the assertion.

var config = CheckConfig.defaults().withSeed(42);                       // CheckConfig
Check.check(config, Gen.list(Gen.integers()), hasSize(isLessThan(5)));  // throws an AssertionError

falsified at sample 14 by (List(1064429137, -1, 2147483646, -499641955, 2147483647)) (seed 42, replay with -Dzazr.check.seed=42):
  List(1064429137, -1, 2147483646, -499641955, 2147483647) has size 5:
    5 is not less than 5

The sample is shown as a tuple, one value per generator. check(gen, assertions...) is the shortcut for a property that only asserts on the generated value.

Small counterexamples, no shrinking¶

zazr-test does not shrink a failing value into a smaller one. It finds small values first instead.

The size grows over a run, from 0 for the first sample to 100 for the last. Collections and strings are at most as long as the size, so the first samples are empty or short, and the first failure is usually a small one.

In the example above, the first list that breaks hasSize(isLessThan(5)) has exactly 5 elements.

Reading a result¶

Check.evaluate runs the same check and returns its CheckResult instead of throwing. A CheckResult is one of three records, so pattern matching over it covers every outcome:

Satisfied(samples): every sample passed.
Falsified(sampleNumber, seed, counterexample, message): the property returned false or a failed TestResult, or threw an AssertionError; message holds the explanation.
Erroneous(sampleNumber, seed, cause, sample): a generator threw, or the property threw another exception.

var result = Check.evaluate(CheckConfig.defaults().withSeed(42), Gen.integers(0, 1000), n -> n < 500); // CheckResult
var summary = switch (result) {
    case CheckResult.Satisfied(var samples) -> "passed " + samples + " samples";
    case CheckResult.Falsified(var sampleNumber, var _, var counterexample, var _) -> "broken at sample " + sampleNumber + " by " + counterexample;
    case CheckResult.Erroneous(var sampleNumber, var _, var cause, var _) -> "failed at sample " + sampleNumber + " with " + cause;
};
// "broken at sample 3 by (1000)"

isSatisfied(), isFalsified() and isErroneous() answer the same question without pattern matching, and assertIsSatisfied() throws the AssertionError that Check.check would.

Assertions¶

An Assertion<A> tests a value and, when the value breaks it, explains why. assertThat(value, assertion) returns a TestResult: a success, or a failure whose explanation follows the failing part of a nested assertion.

var result = assertThat(Option.some(4), isSome(isGreaterThan(5))); // TestResult
// Failure: "Some(4) holds 4:\n  4 is not greater than 5"

The assertions are static methods of Assertion. If your test also imports AssertJ's assertThat, import Assertion.assertThat by name too: Java then picks one by its arguments.

The catalogue¶

Values	Assertions
any value	`equalTo`, `anything`, `isTrue`, `isFalse`
comparable values	`isGreaterThan`, `isGreaterThanOrEqualTo`, `isLessThan`, `isLessThanOrEqualTo`, `isWithin(min, max)`
strings	`startsWithString`, `endsWithString`, `containsString`, `matchesRegex`
any `Iterable`	`isEmpty`, `isNonEmpty`, `hasSize`, `contains`, `exists`, `forall`, `hasFirst`, `hasLast`, `hasAt`, `hasSameElements`, `isSorted`
Zazr types	`isSome`, `isNone`, `isLeft`, `isRight`, `isSuccess`, `isFailure`, `isValid`, `isInvalid`
code	`throwsA(type)`, `throwsWith(assertion)`

Assertion.of(name, value -> ...) builds an assertion of your own from a function that returns a TestResult.

Combining assertions¶

and, or and not combine assertions, and label names one in its explanation. or tests its right side only when its left side fails.

var digit  = isGreaterThanOrEqualTo(0).and(isLessThan(10)).label("a digit");  // Assertion<Integer>
var result = assertThat(12, digit);                                           // TestResult
// Failure: "a digit: 12 is not less than 10"

Several assertions at once¶

assertThat and check take several assertions. A failure lists every one that fails, not only the first.

Every assertion is tested, even after one failed. So isNonEmpty() does not guard the assertions after it: one that throws on an empty list makes the check Erroneous. Put such an assertion behind isEmpty().or(...) instead.

var result = assertThat(15, isLessThan(10), not(equalTo(15))); // TestResult
// Failure: "15 is not less than 10\n15 satisfies equalTo(15), but must not"
Check.check(Gen.integers(0, 9), isGreaterThanOrEqualTo(0), isLessThan(10));

zio-test's assertTrue(a, b, c) shows the source of each failing boolean expression, which Java cannot do, so there is no assertTrue(boolean...): pass several assertions instead.

With several generators, the results combine with and and or.

var ints = Gen.integers(); // Gen<Integer>
Check.check(ints, ints, (a, b) -> assertThat(a + b, equalTo(b + a)).and(assertThat(a * b, equalTo(b * a))));

Code that must throw¶

var result = assertThat(() -> Integer.parseInt("x"), throwsA(NumberFormatException.class)); // TestResult
// Success

JUnit and AssertJ assertions¶

The property may also use JUnit or AssertJ assertions. A failed assertion falsifies the sample, and its message is kept in the result.

A property that ends with an AssertJ chain works as it is: a body that returns something other than a boolean or a TestResult passes when it completes without throwing. A null result makes the check Erroneous.

var digits = Gen.vector(Gen.integers(0, 9));       // Gen<Vector<Integer>>
var config = CheckConfig.defaults().withSeed(42);  // CheckConfig
// CheckResult
var result  = Check.evaluate(config, digits, vector -> assertThat(vector.distinct()).isEqualTo(vector));
var message = result.message(); // Option<String>
// Some("expected: Vector(9, 1, 2, 9, 8, 9) but was: Vector(9, 1, 2, 8)"), AssertJ's message on three lines

Any other exception thrown by the property makes the result Erroneous, with the exception as its cause.

Generators¶

Scalars¶

Values	Generators
numbers	`integers()`, `integers(min, max)`, `longs()`, `longs(min, max)`, `doubles()`, `doubles(min, max)`
booleans	`booleans()`
characters	`chars()`, `chars(min, max)`, `alphaChars()`, `numericChars()`, `alphaNumericChars()`, `asciiChars()`, `printableChars()`, `unicodeChars()`
strings	`strings()`, `strings(chars)`, `stringsN(n, chars)`, `alphaNumericStrings()`
dates	`localDateTimes()`, `localDateTimes(min, max)`

The scalar generators have plural names, such as integers() and strings(): each one generates many values.

A range favours its edges. Half of the draws of integers(min, max) are the bounds, their neighbours, -1, 0 or 1; the others are uniform. Off-by-one mistakes show up in a few samples.

Combining generators¶

map, flatMap, zip, zipWith and filter build a generator from others. constant, elements, oneOf and weighted choose among values or generators.

var dice    = Gen.integers(1, 6);                // Gen<Integer>
var twoDice = dice.zipWith(dice, Integer::sum);  // Gen<Integer>
var coin    = Gen.elements("heads", "tails");    // Gen<String>
// Gen<String>
var loadedCoin = Gen.weighted(Tuple.of(Gen.constant("heads"), 9.0), Tuple.of(Gen.constant("tails"), 1.0));
Check.check(twoDice, isWithin(2, 12));

Gen.zip(g1, ..., g8) and Gen.zipWith combine up to eight generators at once.

Null values¶

constant, elements and fromIterable accept null. map, flatMap, filter, zip, the tuples and lazy pass it through, so a property can test code that takes null.

With JSpecify annotations, the type says so: a code base checked by NullAway accepts Gen<@Nullable String> maybe = Gen.elements("a", null). A failing null sample prints as (null).

The generators of collections, option, some, either, tryOf and validation take generators of non-null values only: those types reject null.

Finite generators and `checkAll`¶

fromIterable and constant are finite: they give their values in order, then stop. zip of two finite generators gives every combination.

var sizes    = Gen.fromIterable(Vector.of("S", "M"));       // Gen<String>
var colours  = Gen.fromIterable(Vector.of("red", "blue"));  // Gen<String>
var variants = sizes.zip(colours).runCollect();             // List<Tuple2<String, String>>
// List((S, red), (S, blue), (M, red), (M, blue))

checkAll checks every value of finite generators exactly once, instead of 200 samples; evaluateAll shows it.

var days   = Gen.fromIterable(EnumSet.allOf(DayOfWeek.class));    // Gen<DayOfWeek>
var result = Check.evaluateAll(days, day -> day.plus(7) == day);  // CheckResult
// Satisfied[samples=7]

elements is random, not finite: each sample is one of its values, drawn at random.

Size¶

The size bounds what a generator produces, such as the length of a collection. These generators read or change it:

Gen.sized(size -> ...) builds a generator from the current size;
Gen.small(size -> ...) gives a random size between 0 and the current size, most of them small;
Gen.large(size -> ...) gives a random size between 0 and the current size, all as likely;
withSize(n) runs a generator at a fixed size.

var depth = Gen.sized(size -> Gen.integers(0, size));                 // Gen<Integer>
var words = Gen.small(size -> Gen.stringsN(size, Gen.alphaChars()));  // Gen<String>
// Gen<List<Integer>>, at most 3 elements
var shortLists = Gen.list(Gen.integers()).withSize(3);

A check grows the size evenly over its samples, from 0 to the configured size:

var sizes = Gen.size().runCollectN(5, CheckConfig.defaults().withSize(100)); // List<Integer>
// List(0, 25, 50, 75, 100)

Filtering¶

filter keeps the values that satisfy a predicate. A random generator is run again until it gives one.

When no small value can pass, such as with a filter that rejects the empty list, the check tries a larger size.

Filtering has a budget: 1,000 discards per sample, shared by every filter of the check. A rejected value is a discard, and so is a try that gave no value. Past the budget, the result is Erroneous. A predicate that rejects most values is better written as a map that builds the wanted values.

var evens = Gen.integers(-1000, 1000).filter(n -> n % 2 == 0); // Gen<Integer>
// Gen<Integer>, with no rejected value
var alsoEvens = Gen.integers(-500, 500).map(n -> n * 2);
var nonEmpty  = Gen.list(Gen.integers()).filter(list -> !list.isEmpty()); // Gen<List<Integer>>
// CheckResult
var impossible = Check.evaluate(Gen.integers().filter(n -> false), n -> true);
// Erroneous: Gen.filter rejected too many values: 1001 discards since the last sample, more than the discard budget of 1000; ...

Configuration¶

CheckConfig.defaults() is 200 samples, a size of 100, a random seed and a budget of 1,000 discards. Each value has a with method, and check, checkAll and their evaluate forms take a configuration as their first argument.

var config = CheckConfig.defaults().withSamples(1_000).withSeed(42); // CheckConfig
Check.check(config, Gen.integers(), n -> assertThat(Integer.parseInt(Integer.toString(n)), equalTo(n)));
Check.checkN(50, Gen.alphaNumericStrings(), s -> s.strip().equals(s));

checkN(n, ...) is check with n samples.

System properties of the test JVM change the defaults for every check, without touching the code:

Property	Default
`zazr.check.samples`	200
`zazr.check.size`	100
`zazr.check.seed`	a random seed
`zazr.check.maxDiscards`	1000

Replaying a failure¶

The same seed gives the same values, so a failure replays from the seed it prints. Put that seed in the code with CheckConfig.defaults().withSeed(42), and the check fails at the same sample again.

The failure also prints -Dzazr.check.seed=42: that system property replays it without touching the code, when the build passes it to the test JVM. Maven does; a Gradle build needs systemProperty in its test task. A seed set with withSeed takes precedence over the property.

Generators for Zazr types¶

Gen has a generator for each Zazr collection, control type and tuple from Tuple2 to Tuple8. Pass it the generators of the elements.

Kind	Generators
Control types	`option`, `some`, `none`, `either`, `tryOf`, `validation`, `lazy`
Tuples	`tuple2` to `tuple8`
Sequences	`vector`, `vectorN`, `nonEmptyVector`, `list`, `queue`, `lazyList`
Sets	`hashSet`, `linkedHashSet`, `treeSet`, `nonEmptySet`, `nonEmptySortedSet`
Maps	`hashMap`, `linkedHashMap`, `treeMap`, `nonEmptyMap`, `nonEmptySortedMap`

var checks = Gen.validation(Gen.elements("too short", "no digit"), Gen.integers()); // Gen<Validation<String, Integer>>
Check.check(checks, checks, (a, b) -> assertThat(a.zip(b).isValid(), equalTo(a.isValid() && b.isValid())));

A collection has up to the current size elements. Half of the lengths are 0, 1, the size or the size minus one, so empty, single-element and full collections come up often.

A non-empty collection always has at least one element, even when the element generator gives the same value every time. Its lengths go from 1 to the size, and about half of them are 1, 2, the size or the size minus one. The sorted sets and maps use the natural order of their elements or keys.

Tuple0 and Tuple1 have no generator of their own: Gen.constant(Tuple.empty()) and gen.map(Tuple::of) give them.

Unusual layouts included¶

A collection can hold the same elements in different internal layouts, and a bug may hide in only one of them. The collection generators build each value in several ways, so your properties meet the less common layouts too:

a Vector built by dropping a prefix, by prepending, or as a slice, not only by ofAll;
a Queue whose elements sit in both of its internal lists;
a LazyList whose tail is not evaluated yet;
sets and maps that went through removals, or keys overwritten with new values.

Laws¶

A law is a named rule that every value of a type must satisfy, such as mapIdentity: mapping the identity function changes nothing. The package dev.zazr.test.laws states each law once, and you check it against any type.

A law set groups laws. MapLaws.all() is mapIdentity and mapComposition; and joins two laws or two sets.

Laws	What they check
`MapLaws`	`mapIdentity`, `mapComposition`
`FlatMapLaws`	`flatMapAssociativity`, `flatMapLeftIdentity`, `flatMapRightIdentity`, `mapIsFlatMapSucceed`
`ZipLaws`	`zipAssociativity`, `zipLeftIdentity`, `zipRightIdentity`, and `zipLeft`/`zipRight` agreeing with `zip`
`EqualityLaws`	`equalsHashCodeConsistency`, and `equalsAgreesWithModel`: `equals` agrees with a simple model of the value, such as its elements as a JDK list
`CollectionLaws`	`size`, `toList` and `equals` agreeing with the elements, the iteration order (input, insertion or sorted), and equality across collection types
`BuilderLaws`	a builder or a collector gives the same collection as `ofAll`
`ValidationLaws`	`zip` of two `Validation`s keeps the errors of both sides, in order
`NonEmptyVectorLaws`	`head` never fails, and `equals` between a `NonEmptyVector` and its `Vector` gives the same answer both ways

Law.of(name, (subject, config) -> ...) states a law of your own; its check is usually a Check.evaluate, which returns the result the law reports.

Checking your own type¶

A law needs two things from your type: a generator of its values, and the operation under test. You give them in a subject, such as a MapSubject for the map laws. The elements are integers; the laws see them as Object.

record Box(Vector<Object> items) {
    Box map(Function<Object, Object> f) { return new Box(items.map(f)); }
}
var boxes = new MapSubject<Box>() {
    public Gen<Box> values() { return Gen.vector(Gen.integers(-100, 100)).map(v -> new Box(v.map(x -> (Object) x))); }
    public Box map(Box box, Function<Object, Object> f) { return box.map(f); }
}; // MapSubject<Box>
MapLaws.<Box>all().assertSatisfied(boxes);

The other subjects work the same way: FlatMapSubject adds succeed and flatMap, ZipSubject adds zip, and EqualitySubject and CollectionSubject describe values and collections.

When a law fails¶

assertSatisfied checks every law of the set, then throws one AssertionError that names each failing law, the value that broke it, and the two sides of the rule that differed. Here, a map that drops the last element:

var broken = new MapSubject<Box>() {
    public Gen<Box> values() { return boxes.values(); }
    public Box map(Box box, Function<Object, Object> f) { return new Box(box.map(f).items().dropRight(1)); }
}; // MapSubject<Box>
MapLaws.<Box>all().assertSatisfied(broken, CheckConfig.defaults().withSeed(42)); // throws an AssertionError

2 law(s) failed:
mapIdentity: falsified at sample 5 by (Box[items=Vector(6)]): left = Box[items=Vector()], right = Box[items=Vector(6)] (seed 42, replay with -Dzazr.check.seed=42)
mapComposition: falsified at sample 7 by (Box[items=Vector(99, -6)], x -> -1 * x + -9, x -> -10 * x + 80): left = Box[items=Vector()], right = Box[items=Vector(1160)] (seed 42, replay with -Dzazr.check.seed=42)

The laws of a set run with the same seed, so one -Dzazr.check.seed=42 replays every failure.

Testing with zazr-test¶