It's easy as pie: before checking in, your test suite should always be green. Or should it? What if your tests are all green but you forgot to check one important edge case? What if your underlying system environment lets you down, but only under rare conditions that you didn't cover in your tests?
This talk introduces randomised testing as used by projects like Apache Lucene and Elasticsearch based on the Carrotsearch Randomised Testing framework. It has helped uncover (and ultimately fix) a huge number of bugs not only in these project’s source code, but also in the JVM itself which those projects rely on.
Writing unit and integration tests can be tricky: assumptions about your code may not always be true as any number of "this should never happen" log entries in production systems show. When implementing a system that will be integrated in all sorts of expected, unexpected, and outright weird ways by downstream users, testing all possible code paths, configurations and deployment environments gets complicated.
With the Carrotsearch Randomised Testing framework, projects like Apache Lucene and Elasticsearch have introduced a new level to their unit and integration tests. Input values are no longer statically pre-defined but are generated based on developer defined constraints, meaning The test suite is no longer re-run with a static set of input data each time. Instead, every continuous integration run adds to the search space covered. Though generated at random, tests are still reproducible as all configurations are based on specific test seeds that can be used to re-run the test with the exact same configuration.
Add to this randomising the runtime environment by executing tests with various JVM versions and configurations,and you are bound to find cases where your application runs into limitations and bugs in the JVM.
This talk introduces randomised testing as a concept, shows examples of how the Carrotsearch Randomised Testing framework helps with making your test cases more interesting, and provides some insight into how randomising your execution environment can help save downstream users from surprises. All without putting too much strain on your continuous integration resources.