Previous in the series: https://itnext.io/quickcheck-testing-in-golang-772e820f0bd5
“A good programmer is someone who always looks both ways before crossing a one-way street.” — Doug Linder
In my last testing article I went through the fuzz testing available in the golang “testing/quick” package but in the process showed that, while useful, the package does not replicate one of the most powerful features of QuickCheck. QuickCheck not only produces random inputs but performs a “shrink” operation on any failure cases before reporting them. From the perspective of a developer who not only wants assurances in the correctness of their algorithms but the tools to correct any errors that are found this “shrink” stage is crucial.
Shrinking is done by applying a “shrinking” function that takes as its input a given type and returns one or more values of the same type that are in some metric “smaller” than the original. While in some cases smaller is hard to quantify many are intuitive. For instance a string shrinker could take the input string “abcdefg” and produce a list of (“abcdef”, “bcdefg”). These values are “smaller” in the sense that they contain fewer characters. For objects like slices and string a length operation is a logical one to apply to determine a “best” solution.
Once a solution is shrunk, each of the possible shorter values can be fed back into the property test to determine if any or all of them produce a failure. If any do, they could be “shrunk” once again and this process should repeat until the “shortest” possible failure case if found. This process is the familiar graph traversal problem where it isn’t simply enough to show that a bug is present due to an input but once it is found a minimal sized version of that input should be produced.
Thanks to redditor zombiecalypse to pointing out that rather than having to write this ourselves (which was my initial fear) there is already a package available that allows us to take our generative testing to the next level: https://github.com/leanovate/gopter/ and among other features it produces shrunk results. In addition to the crucial shrink step, this package also provides a number of powerful generators, “command” based testing that I eluded to in the last article and integration with goConvey.
Time and Time Again
With the basics covered in the last article we can repeat the last test with some small and welcome changes. First I wrote a goConvey assertion that is the opposite of the built in ShouldSucceedForAll because for the sake of this demonstration a failure is exactly what we want and a success is a failure.
Now all that is left is to build the same test we had last time but use gopter:
Our parsing functions remain the same but the generators in gopter are functionally composable. Instead of running the two functions in parallel we can take a character generator (size is determined by the MinSize and MaxSize in the parameters) and both map our prepend function across the random values but also pre-check that the values are reasonable by directly filtering with the time.Parse based filter. While this is not the most interesting example it becomes immediately clear that with a map and a filter operation we can produce complex generators that shape and filter random data to fit reasonable constraints. As hoped the property tester quickly found our bug:
With the elapsed time and the case that generated the error given in the failure description.
While in the last article a basic framework for command or sequence testing was laid out it left a great deal to the reader. Creating a sequence of keys that map to commands definitely gets the job done but the test was obtuse and it required some post processing to make sense of the results. With gopter command testing this is more straightforward. Similar to the last fake API we can simulate an API that fails after a sequence of commands:
Rather than using random numbers as before, it is possible to directly name and define the expected behavior of these functions. Here are simple wrappers that give names to the functions decorating the class and allow gopter to call them:
Next a function that is designed to catch the failure:
The final step is to wrap all these commands up nicely into a commands property and run the test:
Running the tests we get a much better “failure” than when simply using testing/quick:
Not only is the code more readable, the test produces a shorter failure path that could be easily used as part of the debugging process for the API.
Fuzz testing is typically better than nothing and “testing/quick” has already won accolades for finding bugs even within the core language that no programmer would have immediately expected. If the intention is to run background testing over a near infinite space, much like password cracking, then “testing/quick” with continuous iteration can make sure you spend your spare cpu cycles well. In the case, however, where you don’t need to dump entropy at a problem but you have software that you want assurances of correctness on — or worse yet you have stubborn and hard to reproduce bugs in existing systems — QuickCheck and by proxy gopter is a more economical choice. The ability to shrink failure cases into easily reproducible sets with no extra steps should allow developers to fix bugs quickly in cases where setup and testing with brute force could bog down time and resources indefinitely.
You can find the code mentioned here at https://github.com/weberr13/Kata/blob/master/gogen/gopter_test.go
And as always is available via the MIT license.