Composing Generators

I've mentioned in the last section how we need tuple and vector to generate tuples and vectors of values respectively. In this chapter, we'll use these to build a slightly more complex example.

Let's take a look at a more complicated object and an associated function, like this one for example:

struct Student
   name::String
   age::Int
   grades::Dict{String,Int} # subject => points
end

"""
    passes(s::Student)

Checks whether a student passes this grade. At most one subject may have a failing grade with less than 51 points.
"""
function passes(s::Student)
    count(<(51), values(s.grades)) <= 1
end

Main.passes

First, we're going to need a custom generator for our grades:

grade = map(Base.splat(Pair), PropCheck.interleave(itype(String), isample(0:100))) # random subject, with points in 0:100
gradegen = map(Base.splat(Dict{String,Int}), PropCheck.tuple(isample(1:10), grade))

Integrated{PropCheck.Tree{Dict{String, Int64}}, PropCheck.var"#genF#92"{Base.Splat{Type{Dict{String, Int64}}}, Integrated{PropCheck.Tree{Tuple{Vararg{Pair{String, Int64}, N}} where N}, PropCheck.var"#genF#106"{PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}, Integrated{PropCheck.Tree{Pair{String, Int64}}, PropCheck.var"#genF#92"{Base.Splat{Type{Pair}}, Integrated{PropCheck.Tree{Tuple{String, Int64}}, PropCheck.var"#gen#109"{Tuple{Integrated{PropCheck.Tree{String}, PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}}, PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}}}}}}}}}(PropCheck.var"#genF#92"{Base.Splat{Type{Dict{String, Int64}}}, Integrated{PropCheck.Tree{Tuple{Vararg{Pair{String, Int64}, N}} where N}, PropCheck.var"#genF#106"{PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}, Integrated{PropCheck.Tree{Pair{String, Int64}}, PropCheck.var"#genF#92"{Base.Splat{Type{Pair}}, Integrated{PropCheck.Tree{Tuple{String, Int64}}, PropCheck.var"#gen#109"{Tuple{Integrated{PropCheck.Tree{String}, PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}}, PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}}}}}}}}(splat(Dict{String, Int64}), Integrated{PropCheck.Tree{Tuple{Vararg{Pair{String, Int64}, N}} where N}, PropCheck.var"#genF#106"{PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}, Integrated{PropCheck.Tree{Pair{String, Int64}}, PropCheck.var"#genF#92"{Base.Splat{Type{Pair}}, Integrated{PropCheck.Tree{Tuple{String, Int64}}, PropCheck.var"#gen#109"{Tuple{Integrated{PropCheck.Tree{String}, PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}}, PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}}}}}}}(PropCheck.var"#genF#106"{PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}, Integrated{PropCheck.Tree{Pair{String, Int64}}, PropCheck.var"#genF#92"{Base.Splat{Type{Pair}}, Integrated{PropCheck.Tree{Tuple{String, Int64}}, PropCheck.var"#gen#109"{Tuple{Integrated{PropCheck.Tree{String}, PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}}, PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}}}}}}(PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}(1:10, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}(PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}(Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}(PropCheck.var"#119#120"{UnitRange{Int64}}(1:10)), PropCheck.var"#65#66"{Int64, Int64}(1)))), Integrated{PropCheck.Tree{Pair{String, Int64}}, PropCheck.var"#genF#92"{Base.Splat{Type{Pair}}, Integrated{PropCheck.Tree{Tuple{String, Int64}}, PropCheck.var"#gen#109"{Tuple{Integrated{PropCheck.Tree{String}, PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}}, PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}}}}}(PropCheck.var"#genF#92"{Base.Splat{Type{Pair}}, Integrated{PropCheck.Tree{Tuple{String, Int64}}, PropCheck.var"#gen#109"{Tuple{Integrated{PropCheck.Tree{String}, PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}}, PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}}}}(splat(Pair), Integrated{PropCheck.Tree{Tuple{String, Int64}}, PropCheck.var"#gen#109"{Tuple{Integrated{PropCheck.Tree{String}, PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}}, PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}}}(PropCheck.var"#gen#109"{Tuple{Integrated{PropCheck.Tree{String}, PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}}, PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}}((Integrated{PropCheck.Tree{String}, PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}}(PropCheck.var"#gen#83"{Generator{String, PropCheck.var"#9#10"{String}}, typeof(shrink)}(Generator{String, PropCheck.var"#9#10"{String}}(PropCheck.var"#9#10"{String}()), PropCheck.shrink)), PropCheck.IntegratedRange{PropCheck.Tree{Int64}, UnitRange{Int64}, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}(0:100, Integrated{PropCheck.Tree{Int64}, PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}}(PropCheck.var"#gen#83"{Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}, PropCheck.var"#65#66"{Int64, Int64}}(Generator{Int64, PropCheck.var"#119#120"{UnitRange{Int64}}}(PropCheck.var"#119#120"{UnitRange{Int64}}(0:100)), PropCheck.var"#65#66"{Int64, Int64}(0)))))))))))))

Different ranges

isample(2:10, PropCheck.shrinkTowards(2)) gives us an integrated shrinker producing elements in the range 2:10. They'll shrink towards 2.

Now to our student:

students = map(Base.splat(Student), PropCheck.interleave(itype(String), itype(Int), gradegen))
check(passes, students)

Main.Student("", 0, Dict("\0" => 0, "" => 0))

And we can see that just generic shrinking produced the minimal student that doesn't pass. A nameless, ageless student who received no points on two subjects. Note that due to us using a dictionary (which forces unique keys), the two subjects have different names!

PropCheck tries to be fast when it can, so this reduction barely took any time:

pairs(@timed @time check(passes, students))

pairs(::NamedTuple) with 5 entries:
  :value   => Student("", 0, Dict("\0"=>0, ""=>0))
  :time    => 2.11252
  :bytes   => 83377769
  :gctime  => 0.0128325
  :gcstats => GC_Diff(83377769, 0, 0, 1030537, 789, 19, 12832467, 1, 0)

Let's say now that we expect our students to be between 10-18 years old, have a name consisting of 5-20 lowercase ASCII letters and having between 5 and 10 subjects of 5-15 lowercase ASCII letters. We could build them like this:

subj_name = PropCheck.str(isample(5:15), isample('a':'z'))
grade = map(Base.splat(Pair), PropCheck.interleave(subj_name, isample(0:100))) # random subject, with points in 0:100
gradegen = map(Base.splat(Dict{String,Int}), PropCheck.vector(isample(5:10), grade))
stud_name = PropCheck.str(isample(5:20), isample('a':'z')) # we don't want names shorter than 5 characters
stud_age = isample(10:18) # our youngest student can only be 10 years old
students = map(Base.splat(Student), PropCheck.interleave(stud_name, stud_age, gradegen))
collect(Iterators.take(students, 5))

5-element Vector{PropCheck.Tree{Main.Student}}:
 Tree(Main.Student("plyeghuauqrjevhda", 15, Dict("ppjwllq" => 19, "yidhte" => 5, "mwdziuovmtwbe" => 32, "pjdwjbuvbyzwxyw" => 66, "xjctmdsrdyuxvpw" => 60, "yjwcqk" => 1, "dddpeefdys" => 96, "wxwjcud" => 61)))
 Tree(Main.Student("sklgcnw", 10, Dict("gzzbeltnungwnv" => 6, "sqymerbnbheg" => 25, "vslkxuffqtca" => 54, "pjypgtvrsr" => 6, "asqhyrqq" => 57, "jhtpihfzhzacak" => 82, "tctrnrvn" => 27, "hhacogvrf" => 5, "rtzen" => 74, "uheszuiiummoi" => 30…)))
 Tree(Main.Student("mhzffcmmud", 17, Dict("auwoxoxnwv" => 16, "tiorafrgfckwmgb" => 37, "hzefdrycp" => 73, "theymwp" => 69, "yvwbjemwxyhi" => 71, "ascmbayocntsvcz" => 46, "jmztodtqoget" => 84)))
 Tree(Main.Student("novctfdaksvbmcbliruu", 10, Dict("irzacbbsjxpeqvi" => 33, "kkwbshh" => 29, "jotqlgkgdvf" => 27, "jhkgfaslt" => 61, "yqtkgw" => 12)))
 Tree(Main.Student("ivbidtxfedple", 18, Dict("klxikmosngsqabx" => 1, "aqiouzqdaxvcox" => 74, "xdsdryjpe" => 5, "vlzqsjrqanusds" => 36, "xnvppcve" => 71, "sayxwdcrsefhi" => 57, "gaknkn" => 47)))

which will preserve the invariants described during generation when shrinking:

check(passes, students)

Main.Student("aaaaa", 10, Dict("aaaaa" => 0, "aaaab" => 0))

The student returned has a name with 5 characters, is 10 years old, has taken two distinct subjects and received 0 points in both of them. We can do much better if we modify our generators a bit, at the cost of having a smaller pool of possible tests:

# sample their classes
subj_name = isample(["Geography", "Mathematics", "English", "Arts & Crafts", "Music", "Science"], PropCheck.noshrink)

# random subject, with points in 0:100
grade = map(Base.splat(Pair), PropCheck.interleave(subj_name, isample(0:100)))

# generate their grades
gradegen = map(Base.splat(Dict{String,Int}), PropCheck.vector(isample(5:10), grade))

# give them a name that doesn't vanish
stud_name = isample(["Alice", "Bob", "Claire", "Devon"], PropCheck.noshrink)

# our youngest student can only be 10 years old
stud_age = isample(10:18)

# create our students
students = map(Base.splat(Student), PropCheck.interleave(stud_name, stud_age, gradegen))

# and check that not all students pass
using Test
@testset "All students pass" begin
    @test check(passes, students)
end

┌ Info: Found counterexample for 'passes', beginning shrinking...
└   Counterexample = Main.Student("Alice", 15, Dict("Mathematics" => 63, "English" => 23, "Geography" => 40, "Science" => 6, "Music" => 74, "Arts & Crafts" => 30))
[ Info: 32 counterexamples found for passes
All students pass: Error During Test at properties.md:99
  Expression evaluated to non-Boolean
  Expression: check(passes, students)
       Value: Main.Student("Alice", 10, Dict("English" => 0, "Science" => 0, "Arts & Crafts" => 0))
Test Summary:     | Error  Total  Time
All students pass |     1      1  0.6s

Dictionaries

While this example directly splats a vector into the Dict{String,Int} constructor, this is in general not optimal. Dict will delete previously set values if a key is duplicated, so it's usually better to generate a list of unique keys first, which is then combined with a seperately generated list of values. In order to generate that list of unique keys, you can use iunique.

Test stdlib and `@test`

Currently, check returns the minimized failing testcase, so that @test displays that the test has evaluated to a non-Boolean. This is suboptimal and misuses the @test macro. In the future, this may be replaced by a @check macro, which creates a custom TestSet for recording what kind of failure was experienced.