Chaining generation

Sometimes, it is useful to be able to generate a finite set of "special values", before throwing the full brunt of possible values of a type at your function. At other times, you may want to test a few special distributions of values whose generation isn't finite, before having PropCheck.jl try a generic type-based fuzzing approach. In these cases, it's helpful to try one of the subtypes of PropCheck.FiniteIntegrated.

IntegratedOnce

PropCheck.IntegratedOnce is the "Give me a value, but please only one value" version of PropCheck.ival. It is semantically the same as PropCheck.IntegratedLengthBounded(ival(x), 1) (which we'll look at in detail later), but with a more efficient implementation.

Here's an example:

julia> using PropCheck
julia> using PropCheck: IntegratedOnce
julia> gen = IntegratedOnce(5)IntegratedOnce(5, shrink, false)
julia> tree = generate(gen)Tree(5)
julia> subtrees(tree) |> collect # The shrink tree is unfolded as usual9-element Vector{PropCheck.Tree{Int64}}:
 Tree(-2)
 Tree(2)
 Tree(-4)
 Tree(-3)
 Tree(-1)
 Tree(4)
 Tree(3)
 Tree(0)
 Tree(1)
julia> generate(gen) isa Nothing # but subsequent `generate` calls don't produce any valuetrue
julia> generate(gen) isa Nothingtrue

If you know that you're only going to require the value to be tested exactly once, while still being able to test its shrinks in case the property fails, IntegratedOnce can be a very good choice. An example use case is for regression testing of known previous failures.

Of course, this kind of integrated shrinker can be mapped and filtered just the same as regular infinite generators:

julia> gen = filter(iseven, IntegratedOnce(5));
julia> tree = generate(gen)Tree(0)
julia> root(tree)0
julia> subtrees(tree) |> collectPropCheck.Tree{Int64}[]
julia> gen = map(x -> 2x, IntegratedOnce(5));
julia> tree = generate(gen)Tree(10)
julia> root(tree)10
julia> subtrees(tree) |> collect9-element Vector{PropCheck.Tree{Int64}}:
 Tree(-6)
 Tree(2)
 Tree(0)
 Tree(4)
 Tree(8)
 Tree(-2)
 Tree(-8)
 Tree(6)
 Tree(-4)

Copying & exhaustion

Keep in mind that all finite generators can only be exhausted once. So be sure to deepcopy the finite generators if you want to reuse them in multiple places. This may later be relaxed to only copy for some finite generators, in order to reuse as many reusable generators as possible.

IntegratedFiniteIterator

PropCheck.IntegratedFiniteIterator can be used to produce the values of a given finite iterator, one after the other, before suspending generation of new values.

This is useful when you have a known set of special values that you want to try, which are likely to lead to issues. IntegratedOnce is similar to this integrated shrinker, with the difference being that IntegratedFiniteIterator can take any arbitrary iterable (except other AbstractIntegrated) to produce their values in exactly the order they were originally produced in from the iterator.

julia> using PropCheck
julia> using PropCheck: IntegratedFiniteIterator
julia> iter = 1:2:211:2:21
julia> gen = IntegratedFiniteIterator(iter); # all odd values between 1 and 21, inclusive
julia> length(gen) == length(iter)true
julia> all(zip(gen, iter)) do (gval, ival)
           root(gval) == ival
       endtrue
julia> generate(gen) isa Nothing # and of course, once it's exhausted that's ittrue

IntegratedLengthBounded

PropCheck.IntegratedLengthBounded can be used to limit an PropCheck.AbstractIntegrated to a an upperbound in the count of generated values, before generation is suspended.

This can be useful for only wanting to generate a finite number of elements from some other infinite generator before switching to another one, as mentioned earlier. The basic usage is passing an AbstractIntegrated as well as the desired maximum length. If a FiniteIntegrated is passed, the resulting integrated shrinker has as its length the min of the given FiniteIntegrated and the given upper bound.

julia> using PropCheck
julia> using PropCheck: IntegratedLengthBounded, IntegratedOnce
julia> gen = IntegratedLengthBounded(itype(Int8), 7);
julia> collect(gen) # 7 `Tree{Int8}`7-element Vector{PropCheck.Tree{Int8}}:
 Tree(-105)
 Tree(-33)
 Tree(81)
 Tree(-44)
 Tree(87)
 Tree(21)
 Tree(114)
julia> gen = IntegratedLengthBounded(IntegratedOnce(42), 99);
julia> length(gen) # still only one `Tree{Int}`1
julia> collect(gen)1-element Vector{PropCheck.Tree{Int64}}:
 Tree(42)

IntegratedChain

While itself not guaranteed to be finite, PropCheck.IntegratedChain is the most useful tool when combining finite generators in this fashion. Its constructor takes any number of AbstractIntegrated, though all but the last one are required to subtype FiniteIntegrated. The last integrated shrinker may be truly AbstractIntegrated, though being FiniteIntegrated is also ok.

This allows IntegratedChain to be a building block for grouping special values together, or for preparing a known set of previous failures into a regression test, while still allowing the values to shrink according to the shrinking function used during the original generation, if available.

julia> using PropCheck
julia> using PropCheck: IntegratedChain, IntegratedOnce, IntegratedFiniteIterator
julia> using Test
julia> function myAdd(a::Float64, b::Float64)
           # whoops, this isn't `add` at all!
           1.0 <= a && return NaN
           a + b
       endmyAdd (generic function with 1 method)
julia> previousFailure = interleave(IntegratedOnce(0.5), IntegratedOnce(0.2));
julia> specialCases = IntegratedFiniteIterator((NaN, -1.0, 1.0, 0.0));
julia> specialInput = interleave(specialCases, deepcopy(specialCases)); # take care to not consume too early
julia> addGen = IntegratedChain(previousFailure, # fail early if this doesn't work
                                specialInput, # some known special cases
                                PropCheck.tuple(ival(2), itype(Float64))); # finally, fuzzing on arbitrary stuff!
julia> function nanProp(a, b)
           res = myAdd(a,b)
           # only `NaN` should produce `NaN`
           (isnan(a) || isnan(b)) == isnan(res)
       endnanProp (generic function with 1 method)
julia> @testset "myAdd" begin # We find a failure past our first chain of special cases
           @test check(splat(nanProp), addGen)
       end┌ Info: Found counterexample for 'splat(nanProp)', beginning shrinking...
└   Counterexample = (1.0, 1.0)
[ Info: 12 counterexamples found for splat(nanProp)
myAdd: Error During Test at REPL[10]:2
  Expression evaluated to non-Boolean
  Expression: check(splat(nanProp), addGen)
       Value: (1.0, 0.0)
Test Summary: | Error  Total  Time
myAdd         |     1      1  1.5s
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.