Documentation Reference
This section contains a complete reference of everything Supposition.jl contains, on one page. This is not a devdocs section, but a reference, for quick lookups of what something does, without having to hunt for the exact definition in the source code. A proper devdocs section with a high level introduction will be added at a later date.
The entries written on this page are automatically generated and DO NOT represent the currently supported API surface. Feel free to use anything you can find here, but be aware that just because it's listed here, does not mean it's covered under semver (though it may be - check Userfacing API if you're unsure).
Data reference
The Data
module contains most everyday objects you're going to use when writing property based tests with Supposition.jl. For example, the basic generators for integers, strings, floating point values etc. are defined here. Everything listed in this section is considered supported under semver.
Supposition.Data.bind
Supposition.Data.postype
Supposition.Data.postype
Supposition.Data.produce!
Supposition.Data.recursive
Supposition.Data.AsciiCharacters
Supposition.Data.Bind
Supposition.Data.BitIntegers
Supposition.Data.Booleans
Supposition.Data.Characters
Supposition.Data.Dicts
Supposition.Data.Floats
Supposition.Data.Floats
Supposition.Data.Integers
Supposition.Data.Just
Supposition.Data.Map
Supposition.Data.OneOf
Supposition.Data.Pairs
Supposition.Data.Possibility
Supposition.Data.Recursive
Supposition.Data.SampledFrom
Supposition.Data.Satisfying
Supposition.Data.Text
Supposition.Data.UnicodeCharacters
Supposition.Data.Vectors
Supposition.Data.WeightedNumbers
Supposition.Data.WeightedSample
Functions
Base.:|
— Method|(::Possibility{T}, ::Possibility{S}) where {T,S} -> OneOf{Union{T,S}}
Combine two Possibility
into one, sampling uniformly from either.
If either of the two arguments is a OneOf
, the resulting object acts as if all original non-OneOf
had be given to OneOf
instead. That is, e.g. OneOf(a, b) | c
will act like OneOf(a,b,c)
.
See also OneOf
.
Base.filter
— Methodfilter(f, pos::Possibility)
Filter the output of produce!
on pos
by applying the predicate f
.
In order not to stall generation of values, this will not try to produce a value from pos
forever, but reject the testcase after some attempts.
Base.map
— Methodmap(f, pos::Possibility)
map(f, pos::Vararg{Possibility})
Apply f
to the result of calling produce!
on pos
(lazy mapping).
Equivalent to calling Map(pos, f)
, or MultiMap(pos, f)
for the Vararg
case (where pos
is the tuple of Possibility
). In the Vararg
case, f
is expected to take as many arguments as Possibility
are passed to map
.
See also Map
.
Supposition.Data.bind
— Methodbind(f, pos::Possibility)
Maps the output of produce!
on pos
through f
, and calls produce!
on the result again. f
is expected to take a value and return a Possibility
.
Equivalent to calling Bind(pos, f)
.
See also Bind
.
Supposition.Data.postype
— Methodpostype(::P) where P <: Possibility
Gives the type of objects this Possibility
object will generate.
Supposition.Data.postype
— Methodpostype(::Type{P<:Possibility})
Gives the type of objects this Possibility
type will generate.
Supposition.Data.produce!
— Functionproduce!(tc::TestCase, pos::Possibility{T}) -> T
Produces a value from the given Possibility
, recording the required choices in the TestCase
tc
.
This needs to be implemented for custom Possibility
objects, passing the given tc
to any inner requirements directly.
See also Supposition.produce!
You should not call this function when you have a Possibility
and want to inspect what an object produced by that Possibility
looks like - use example
for that instead.
Supposition.Data.recursive
— Methodrecursive(f, pos::Possibility; max_layers=5)
Recursively expand pos
into deeper nested Possibility
by repeatedly passing pos
itself to f
. f
returns a new Possibility
, which is then passed into f
again until the maximum depth is achieved.
Equivalent to calling Recursive(pos, f)
.
See also Recursive
.
Supposition.Data.AsciiCharacters
— TypeAsciiCharacters() <: Possibility{Char}
A Possibility
of producing arbitrary Char
instances that are isascii
. More efficient than filtering Characters
.
julia> using Supposition
julia> ascii = Data.AsciiCharacters()
julia> example(ascii, 5)
5-element Vector{Char}:
'8': ASCII/Unicode U+0038 (category Nd: Number, decimal digit)
'i': ASCII/Unicode U+0069 (category Ll: Letter, lowercase)
'R': ASCII/Unicode U+0052 (category Lu: Letter, uppercase)
'\f': ASCII/Unicode U+000C (category Cc: Other, control)
'>': ASCII/Unicode U+003E (category Sm: Symbol, math)
Supposition.Data.Bind
— TypeBind(source::Possibility, f)
Binds f
to source
, i.e., on produce!(::Bind, ::TestCase)
this calls produce!
on source
, the result of which is passed to f
, the output of which will be used as input to produce!
again.
In other words, f
takes a value produce!
d by source
and gives back a Possibility
that is then immediately produce!
d from.
Equivalent to bind(f, source)
.
Supposition.Data.BitIntegers
— TypeBitIntegers() <: Possibility{Union{Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}}
A Possibility
for generating all possible bitintegers with fixed size.
Supposition.Data.Booleans
— TypeBooleans() <: Possibility{Bool}
A Possibility
for sampling boolean values.
julia> using Supposition
julia> bools = Data.Booleans()
julia> example(bools, 4)
4-element Vector{Bool}:
0
1
0
1
Supposition.Data.Characters
— TypeCharacters(;valid::Bool = false) <: Possibility{Char}
A Possibility
of producing arbitrary well-formed Char
instances.
This will produce!
ANY well-formed Char
by default, not just valid unicode codepoints! Notably, this includes overlong Char
. To only produce valid unicode codepoints, pass valid=true
as a keyword argument.
Keyword arguments:
valid
: Whether the producedChar
must be valid, i.e. not malformed and not have Unicode categoryInvalid
.
julia> using Supposition
julia> chars = Data.Characters()
julia> example(chars, 5)
5-element Vector{Char}:
'⠺': Unicode U+283A (category So: Symbol, other)
'𰳍': Unicode U+30CCD (category Lo: Letter, other)
'\U6ec9c': Unicode U+6EC9C (category Cn: Other, not assigned)
'\U1a05c5': Unicode U+1A05C5 (category In: Invalid, too high)
'𓂫': Unicode U+130AB (category Lo: Letter, other)
Supposition.Data.Dicts
— TypeDicts(keys::Possibility, values::Possibility; min_size=0, max_size=10_000)
A Possibility
for generating Dict
objects. The keys are drawn from keys
, while the values are drawn from values
. min_size
/max_size
control the number of objects placed into the resulting Dict
, respectively.
julia> dicts = Data.Dicts(Data.Integers{UInt8}(), Data.Integers{Int8}(); max_size=3);
julia> example(dicts)
Dict{UInt8, Int8} with 2 entries:
0x54 => -29
0x1f => -28
Supposition.Data.Floats
— MethodFloats(;nans=true, infs=true) <: Possibility{Union{Float64,Float32,Float16}}
A catch-all for generating instances of all three IEEE floating point types.
Supposition.Data.Floats
— TypeFloats{T <: Union{Float16,Float32,Float64}}(;infs=true, nans=true, minimum=-T(Inf), maximum=+T(Inf)) <: Possibility{T}
A Possibility
for sampling floating point values.
The keyword infs
controls whether infinities can be generated. nans
controls whether any NaN
(signaling & quiet) will be generated.
This possibility will generate any valid instance, including positive and negative infinities, signaling and quiet NaNs and every possible float.
julia> using Supposition
julia> floats = Data.Floats{Float16}()
julia> example(floats, 5)
5-element Vector{Float16}:
-8.3e-6
1.459e4
3.277
NaN
-0.0001688
Supposition.Data.Integers
— TypeIntegers(minimum::T, maximum::T) <: Possibility{T <: Integer}
Integers{T}() <: Possibility{T <: Integer}
A Possibility
representing drawing integers from [minimum, maximum]
. The second constructors draws from the entirety of T
.
Produced values are of type T
.
julia> using Supposition
julia> is = Data.Integers{Int}()
julia> example(is, 5)
5-element Vector{Int64}:
-5854403925308846160
4430062772779972974
-9995351034504801
2894734754389242339
-6640496903289665416
Supposition.Data.Just
— TypeJust(value::T) <: Possibility{T}
A Possibility
that always produces value
.
The source object given to this Just
is not copied when produce!
is called. Be careful with mutable data!
julia> using Supposition
julia> three = Data.Just(3)
julia> example(three, 3)
3-element Vector{Int64}:
3
3
3
Supposition.Data.Map
— TypeMap(source::Possibility, f) <: Possibility
A Possibility
representing mapping values from source
through f
.
Equivalent to calling map(f, source)
.
The pre-calculated return type of Map
is a best effort and may be wider than necessary.
julia> using Supposition
julia> makeeven(x) = (x ÷ 2) * 2
julia> pos = map(makeeven, Data.Integers{Int8}())
julia> all(iseven, example(pos, 10_000))
true
Supposition.Data.OneOf
— TypeOneOf(pos::Possibility...) <: Possibility
A Possibility
able to generate any of the examples one of the given Possibility
can produce. The given Possibility
are sampled from uniformly.
At least one Possibility
needs to be given to OneOf
.
postype(::OneOf)
is inferred as a best effort, and may be wider than necessary.
OneOf
can also be constructed through use of a | b
on Possibility
. Constructed in this way, if either a
or b
is already a OneOf
, the resulting OneOf
acts as if it had been given the original Possibility
objects in the first place. That is, OneOf(a, b) | c
acts like OneOf(a, b, c)
.
See also WeightedNumbers
and WeightedSample
.
julia> of = Data.OneOf(Data.Integers{Int8}(), Data.Integers{UInt8}());
julia> Data.postype(of)
Union{Int8, UInt8}
julia> ex = map(of) do i
(i, typeof(i))
end;
julia> example(ex)
(-83, Int8)
julia> example(ex)
(0x9f, UInt8)
Supposition.Data.Pairs
— TypePairs(first::Possibility{T}, second::Possibility{S}) where {T,S} <: Possibility{Pair{T,S}}
A Possibility
for producing a => b
pairs. a
is produced by first
, b
is produced by second
.
julia> p = Data.Pairs(Data.Integers{UInt8}(), Data.Floats{Float64}());
julia> example(p, 4)
4-element Vector{Pair{UInt8, Float64}}:
0x41 => 4.1183566661848205e-230
0x48 => -2.2653631095108555e-119
0x2a => -6.564396855333643e224
0xec => 1.9330751262581671e-53
Supposition.Data.Recursive
— TypeRecursive(base::Possibility, extend; max_layers::Int=5) <: Possibility{T}
A Possibility
for generating recursive data structures. base
is the basecase of the recursion. extend
is a function returning a new Possibility
when given a Possibility
, called to recursively expand a tree starting from base
. The returned Possibility
is fed back into extend
again, expanding the recursion by one layer.
max_layers
designates the maximum layers Recursive
should keep track of. This must be at least 1
, so that at least the base case can always be generated. Note that this implies extend
will be used at most max_layers-1
times, since the base case of the recursion will not be wrapped.
Equivalent to calling recursive(extend, base)
.
Examples
julia> base = Data.Integers{UInt8}()
julia> wrap(pos) = Data.Vectors(pos; min_size=2, max_size=3)
julia> rec = Data.recursive(wrap, base; max_layers=3);
julia> Data.postype(rec) # the result is formatted here for legibility
Union{UInt8,
Vector{UInt8},
Vector{Union{UInt8, Vector{UInt8}}}
}
julia> example(rec)
0x31
julia> example(rec)
2-element Vector{Union{UInt8, Vector{UInt8}}}:
UInt8[0xa9, 0xb4]
0x9b
julia> example(rec)
2-element Vector{UInt8}:
0xbd
0x25
Supposition.Data.SampledFrom
— TypeSampledFrom(collection) <: Possibility{eltype(collection)}
A Possibility
for sampling uniformly from collection
.
collection
, as well as its eachindex
, is assumed to be indexable.
The source objects from the collection given to this SampledFrom
is not copied when produce!
is called. Be careful with mutable data!
To sample from a String
, you can collect
the string first to get a Vector{Char}
. This is necessary because String
s use the variable-length UTF-8 encoding, which isn't arbitrarily indexable in constant time.
julia> using Supposition
julia> sampler = Data.SampledFrom([1, 1, 1, 2])
julia> example(sampler, 4)
4-element Vector{Int64}:
1
1
2
1
Supposition.Data.Satisfying
— TypeSatisfying(source::Possibility, pred) <: Possibility
A Possibility
representing values from source
fulfilling pred
.
Equivalent to calling filter(f, source)
.
julia> using Supposition
julia> pos = filter(iseven, Data.Integers{Int8}())
julia> all(iseven, example(pos, 10_000))
true
Supposition.Data.Text
— TypeText(alphabet::Possibility{Char}; min_len=0, max_len=10_000) <: Possibility{String}
A Possibility
for producing String
s containing Char
s of a given alphabet.
julia> using Supposition
julia> text = Data.Text(Data.AsciiCharacters(); max_len=15)
julia> example(text, 5)
5-element Vector{String}:
"U\x127lxf"
"hm\x172SJ-("
"h`\x03\0\x01[[il"
"\x0ep4"
"9+Hk3 ii\x1eT"
Supposition.Data.UnicodeCharacters
— TypeUnicodeCharacters(;valid::Bool = false, malformed = true) <: Possibility{Char}
A Possibility
of producing arbitrary Char
instances.
This will produce!
ANY Char
by default, not just valid or well-formed unicode codepoints! To only produce valid unicode codepoints, pass valid=true
as a keyword argument. To produce well-formed unicode codepoints, pass malformed=false
as a keyword argument.
Keyword arguments:
malformed
: Whether producedChar
are allowed to be malformed. This only has an effect whenvalid=false
.valid
: Whether the producedChar
must be valid, i.e. not malformed and not have Unicode categoryInvalid
.
julia> using Supposition
julia> chars = Data.UnicodeCharacters()
julia> example(chars, 5)
5-element Vector{Char}:
'\xd2\x5a\x96\x07': Malformed UTF-8 (category Ma: Malformed, bad data)
'\x44\x45\xc5\x64': Malformed UTF-8 (category Ma: Malformed, bad data)
'\x04': Unicode U+0004 (category Cc: Other, control)
'\x2b\xe0\x6a\x89': Malformed UTF-8 (category Ma: Malformed, bad data)
'\xf5\x9b\x63\x05': Malformed UTF-8 (category Ma: Malformed, bad data)
Supposition.Data.Vectors
— TypeVectors(elements::Possibility{T}; min_size=0, max_size=10_000) <: Possibility{Vector{T}}
A Possibility
representing drawing vectors with length l
in min_size <= l <= max_size
, holding elements of type T
.
min_size
and max_size
must be positive numbers, with min_size <= max_size
.
julia> using Supposition
julia> vs = Data.Vectors(Data.Floats{Float16}(); max_size=5)
julia> example(vs, 3)
3-element Vector{Vector{Float16}}:
[9.64e-5, 9.03e3, 0.04172, -0.0003352]
[9.793e-5, -2.893, 62.62, 0.0001961]
[-0.007023, NaN, 3.805, 0.1943]
Supposition.Data.WeightedNumbers
— TypeWeightedNumbers(weights::Vector{Float64}) <: Possibility{Int}
Sample the numbers from 1:length(weights)
, each with a weight of weights[i]
.
The weights may be any number > 0.0.
See also OneOf
.
julia> using Supposition
julia> bn = Data.WeightedNumbers([1.0, 1.0, 3.0]);
julia> example(Data.Vectors(bn; min_size=3, max_size=15), 5)
5-element Vector{Vector{Int64}}:
[3, 2, 3, 3, 2, 3, 3]
[1, 1, 1, 2, 1, 3, 1, 3]
[2, 3, 3, 3, 3, 3, 3, 1, 1, 3, 3, 3]
[3, 3, 2, 3, 3]
[1, 3, 3, 3, 2, 2]
Supposition.Data.WeightedSample
— TypeWeightedSample{T}(colllection, weights::Vector{Float64}) <: Possibility{T}
Draw an element from the indexable collection
, biasing the drawing process by assigning each index i
of col
the weight at weights[i]
.
length
of col
and weights
must be equal and eachindex(col)
must be indexable.
See also OneOf
.
To sample from a String
, you can collect
the string first to get a Vector{Char}
. This is necessary because String
s use the variable-length UTF-8 encoding, which isn't arbitrarily indexable in constant time.
julia> bs = Data.WeightedSample(["foo", "bar", "baz"], [3.0, 1.0, 1.0]);
julia> example(bs, 10)
10-element Vector{String}:
"foo"
"foo"
"bar"
"baz"
"baz"
"foo"
"bar"
"foo"
"foo"
"bar"
Supposition reference
Supposition.@check
Supposition.@composed
Supposition.assume!
Supposition.err_less
Supposition.event!
Supposition.reject!
Supposition.target!
Supposition.ExampleDB
Supposition.DEFAULT_CONFIG
Supposition.@check
— Macro@check [key=val]... function...
The main way to declare & run a property based test. Called like so:
julia> using Supposition, Supposition.Data
julia> Supposition.@check [options...] function foo(a = Data.Text(Data.Characters(); max_len=10))
length(a) > 8
end
Supported options, passed as key=value
:
rng::Random.AbstractRNG
: Pass an RNG to use. Defaults toRandom.Xoshiro(rand(Random.RandomDevice(), UInt))
.max_examples::Int
: The maximum number of generated examples that are passed to the property.broken::Bool
: Mark a property that should pass but doesn't as broken, so that failures are not counted.record::Bool
: Whether the result of the invocation should be recorded with any parent testsets.db
: Either a Boolean (true
uses a fallback database,false
stops recording examples) or anExampleDB
.config
: ACheckConfig
object that will be used as a default for all previous options. Options that are passed explicitly to@check
will override whatever is provided throughconfig
.
The arguments to the given function are expected to be generator strategies. The names they are bound to are the names the generated object will have in the test. These arguments will be shown should the property fail!
Extended help
Reusing existing properties
If you already have a predicate defined, you can also use the calling syntax in @check
. Here, the generator is passed purely positionally to the given function; no argument name is necessary.
julia> using Supposition, Supposition.Data
julia> isuint8(x) = x isa UInt8
julia> intgen = Data.Integers{UInt8}()
julia> Supposition.@check isuint8(intgen)
Passing a custom RNG
It is possible to optionally give a custom RNG object that will be used for random data generation. If none is given, Xoshiro(rand(Random.RandomDevice(), UInt))
is used instead.
julia> using Supposition, Supposition.Data, Random
# use a custom Xoshiro instance
julia> Supposition.@check rng=Xoshiro(1234) function foo(a = Data.Text(Data.Characters(); max_len=10))
length(a) > 8
end
Be aware that you cannot pass a hardware RNG to @check
directly. If you want to randomize based on hardware entropy, seed a copyable RNG like Xoshiro
from your hardware RNG and pass that to @check
instead. The RNG needs to be copyable for reproducibility.
Additional Syntax
In addition to passing a whole function
like above, the following syntax are also supported:
text = Data.Text(Data.AsciiCharacters(); max_len=10)
# If no name is needed, use an anonymous function
@check (a = text) -> a*a
@check (a = text,) -> "foo: "*a
@check (a = text, num = Data.Integers(0,10)) -> a^num
# ..or give the anonymous function a name too - works with all three of the above
@check build_sentence(a = text, num = Data.Floats{Float16}()) -> "The $a is $num!"
build_sentence("foo", 0.5) # returns "The foo is 0.5!"
While you can pass an anonymous function to @check
, be aware that doing so may hinder replayability of found test cases when surrounding invocations of @check
are moved. Only named functions are resistant to this.
Supposition.@composed
— Macro@composed
A way to compose multiple Possibility
into one, by applying a function.
The return type is inferred as a best-effort!
Used like so:
julia> using Supposition, Supposition.Data
julia> text = Data.Text(Data.AsciiCharacters(); max_len=10)
julia> gen = Supposition.@composed function foo(a = text, num=Data.Integers(0, 10))
lpad(num, 2) * ": " * a
end
julia> example(gen)
" 8: giR2YL\rl"
In addition to passing a whole function
like above, the following syntax are also supported:
# If no name is needed, use an anonymous function
double_up = @composed (a = text) -> a*a
prepend_foo = @composed (a = text,) -> "foo: "*a
expo_str = @composed (a = text, num = Data.Integers(0,10)) -> a^num
# ..or give the anonymous function a name too - works with all three of the above
sentence = @composed build_sentence(a = text, num = Data.Floats{Float16}()) -> "The $a is $num!"
build_sentence("foo", 0.5) # returns "The foo is 0.5!"
# or compose a new generator out of an existing function
my_func(str, number) = number * "? " * str
ask_number = @composed my_func(text, num)
Supposition.Data.produce!
— Methodproduce!(p::Possibility{T}) -> T
Produces a value from the given Possibility
, recording the required choices in the currently active TestCase
.
This can only be called while a testcase is currently being examined or an example for a Possibility
is being actively generated. It is ok to call this inside of @composed
or @check
, as well as any functions only intended to be called from one of those places.
Supposition.adjust
— Methodadjust(ts::TestState, attempt)
Adjust ts
by testing for the choices given by attempt
.
Returns whether attempt
was by some measure better than the previously best attempt.
Supposition.assemble
— Methodassemble(::T, sign::I, expo::I, frac::I) where {I, T <: Union{Float16, Float32, Float64}} -> T
Assembles sign
, expo
and frac
arguments into the floating point number of type T
it represents. sizeof(T)
must match sizeof(I)
.
Supposition.assume!
— Methodassume!(precondition::Bool)
If this precondition is not met, abort the test and mark the currently running testcase as invalid.
This can only be called while a testcase is currently being examined or an example for a Possibility
is being actively generated. It is ok to call this inside of @composed
or @check
, as well as any functions only intended to be called from one of those places.
Supposition.assume!
— Methodassume!(::TestCase, precondition::Bool)
Reject this TestCase
if precondition
is false
.
Supposition.choice!
— Methodchoice!(tc::TestCase, n)
Force a number of choices to occur, taking from the existing prefix first. If the prefix is exhausted, draw from [zero(n), n]
instead.
Supposition.consider
— Methodconsider(ts::TestState, attempt::Attempt) -> Bool
Returns whether the given choices are a conceivable example for the testcase given by ts
.
Supposition.err_choices
— Methoderr_choices
Return the choices that led to the recorded error, if any. If none, return Nothing
.
Supposition.err_less
— Functionerr_less(e1::E, e2::E) where E
A comparison function for exceptions, used when encountering an error in a property. Returns true
if e1
is considered to be "easier" or "simpler" than e2
. Only definable when both e1
and e2
have the same type.
This is optional to implement, but may be beneficial for shrinking counterexamples leading to an error with rich metadata, in which case err_less
will be used to compare errors of the same type from different counterexamples. In particular, this function will likely be helpful for errors with metadata that is far removed from the input that caused the error itself, but would nevertheless be helpful when investigating the failure.
There may also be situations where defining err_less
won't help to find a smaller counterexample if the cause of the error is unrelated to the choices taken during generation. For instance, this is the case when there is no network connection and a Sockets.DNSError
is thrown during the test, or there is a network connection but the host your program is trying to connect to does not have an entry in DNS.
Supposition.event!
— Functionevent!(obj)
event!(label::AbstractString, obj)
Record obj
as an event in the current testcase that occured while running your property. If no label
is given, a default one will be chosen.
This can only be called while a testcase is currently being examined or an example for a Possibility
is being actively generated. It is ok to call this inside of @composed
or @check
, as well as any functions only intended to be called from one of those places.
Supposition.example
— Methodexample(pos::Possibility; tries=100_000, generation::Int)
Generate an example for the given Possibility
.
example
tries to have pos
produce an example tries
times and throws an error if pos
doesn't produce one in that timeframe. generation
indicates how "late" in a usual run of @check
the example might have been generated.
Usage:
julia> using Supposition, Supposition.Data
julia> example(Data.Integers(0, 10))
7
Supposition.example
— Methodexample(gen::Possibility, n::Integer; tries=100_000)
Generate n
examples for the given Possibility
. Each example is given tries
attempts to generate. If any fail, the entire process is aborted.
julia> using Supposition, Supposition.Data
julia> is = Data.Integers(0, 10);
julia> example(is, 10)
10-element Vector{Int64}:
9
1
4
4
7
4
6
10
1
8
Supposition.find_user_error_frame
— Functionfind_user_error_frame(err, trace)
Try to heuristically guess where an error was actually coming from.
For example, ErrorException
is (generally) thrown from the error
function, which would always report the same location if we'd naively take the first frame of the trace. This tries to be a bit smarter (but still fairly conservative) and return something other than the first frame for a small number of known error-throwing functions.
Supposition.find_user_stack_depth
— Methodfind_user_stack_depth
Return a heuristic guess for how many functions deep in user code an error was thrown. Falls back to the full length of the stacktrace.
Supposition.for_choices
— Methodfor_choices(prefix; rng=Random.default_rng())
Create a TestCase
for a given set of known choices.
Supposition.forced_choice!
— Methodforced_choice(tc::TestCase, n::UInt64)
Insert a definite choice in the choice sequence.
Note that all integrity checks happen here!
Supposition.generate!
— Methodgenerate(ts::TestState)
Try to generate an example that falsifies the property given to ts
.
Supposition.num_testcases
— Methodnum_testcases(sr::SuppositionReport)
Returns the number of valid TestCase
s that were attempted during this run.
Supposition.reject!
— Methodreject!()
Reject the current testcase as invalid, meaning the generated example should not be considered as producing a valid counterexample.
This can only be called while a testcase is currently being examined or an example for a Possibility
is being actively generated. It is ok to call this inside of @composed
or @check
, as well as any functions only intended to be called from one of those places.
Supposition.reject
— Methodreject(::TestCase)
Mark this test case as invalid.
Supposition.run
— Methodrun(ts::TestState)
Run the checking algorithm on ts
, generating values until we should stop, targeting the score we want to target on and finally shrinking the result.
Supposition.should_keep_generating
— Methodshould_keep_generating(ts::TestState)
Whether ts
should keep generating new test cases, or whether ts
is finished.
true
returned here means that the given property is not trivial, there is no result yet and we have room for more examples.
Supposition.shrink_redistribute
— Methodshrink_redistribute(ts::TestState, attempt::Attempt, k::UInt)
Try to shrink attempt
by redistributing value between two elements length k
apart.
Supposition.shrink_reduce
— Methodshrink_reduce(::TestState, attempt::Attempt)
Try to shrink attempt
by making the elements smaller.
Supposition.shrink_remove
— Methodshrink_remove(ts::TestState, attempt::Attempt, k::UInt)
Try to shrink attempt
by removing k
elements at a time
Supposition.shrink_sort
— Methodshrink_sort(::TestState, attempt::Attempt, k::UInt)
Try to shrink attempt
by sorting k
contiguous elements at a time.
Supposition.shrink_swap
— Methodshrink_swap(::TestState, attempt::Attempt, k::UInt)
Try to shrink attempt
by swapping two elements length k
apart.
Supposition.shrink_zeros
— Methodshrink_zeros(::TestSTate, attempt::Attempt, k::UInt)
Try to shrink attempt
by setting k
elements at a time to zero.
Supposition.target!
— Methodtarget!(score)
Update the currently running testcase to track the given score as its target.
score
must be convert
ible to a Float64
.
This score can only be set once! Repeated calls will be ignored.
This can only be called while a testcase is currently being examined or an example for a Possibility
is being actively generated. It is ok to call this inside of @composed
or @check
, as well as any functions only intended to be called from one of those places.
Supposition.target!
— Methodtarget!(tc::TestCase, score::Float64)
Update tc
to use score
as the score this TestCase
achieves during optimization.
This score can only be set once! Repeated calls will be ignored.
Supposition.target!
— Methodtarget!(ts::TestState)
If ts
has a target to go towards set, this will try to climb towards that target by adjusting the choice sequence until ts
shouldn't generate anymore.
If ts
is currently tracking an error it encountered, it will try to minimize the stacktrace there instead.
Supposition.tear
— Methodtear(x::T) where T <: Union{Float16, Float32, Float64} -> Tuple{I, I, I}
Returns the sign, exponent and fractional parts of a floating point number. The returned tuple consists of three unsigned integer types I
of the same bitwidth as T
.
Supposition.test_function
— Methodtest_function(ts::TestState, tc::TestCase)
Test the function given to ts
on the test case tc
.
Returns a NTuple{Bool, 2}
indicating whether tc
is interesting and whether it is "better" than the previously best recorded example in ts
.
Supposition.weighted!
— Methodweighted(tc::TestCase, p::Float64)
Return true
with probability p
, false
otherwise.
Supposition.windows
— Methodwindows(array, a, b)
Split array
into three windows, with split points at a
and b
. The split points belong to the middle window.
Supposition.CheckConfig
— TypeCheckConfig(;options...)
A struct holding the initial configuration for an invocation of @check
.
Options:
rng
: The initial RNG object given to@check
. Defaults to a copyableRandom.AbstractRNG
.max_examples
: The maximum number of examples allowed to be drawn with this config.-1
means infinite drawing (careful!). Defaults to10_000
.record
: Whether the result should be recorded in the parent testset, if there is one. Defaults totrue
.verbose
: Whether the printing should be verbose, i.e. print even if it's aPass
. Defaults tofalse
.broken
: Whether the invocation is expected to fail. Defaults tofalse
.db
: AnExampleDB
for recording failure cases for later replaying. Defaults todefault_directory_db()
.buffer_size
: The default maximum buffer size to use for a test case. Defaults to100_000
.
At any one point, there may be more than one active buffer being worked on. You can try to increase this value when you encounter a lot of Overrun
. Do not set this too large, or you're very likely to run out of memory; the default results in ~800kB worth of choices being possible, which should be plenty for most fuzzing tasks. It's generally unlikely that failures only occur with very large values here, and not with smaller ones.
Supposition.Composed
— TypeComposed{S,T} <: Possibility{T}
A Possibility
composed from multiple different Possibility
through @composed
. A tiny bit more fancy/convenient compared to map
if multiple Possibility
are required to be mapped over at the same time.
Should not be instantiated manually; keep the object returned by @composed
around instead.
Supposition.DirectoryDB
— TypeDirectoryDB <: ExampleDB
An ExampleDB
that records examples as files in a directory.
Supposition.Error
— TypeError
A result indicating that an error was encountered while generating or shrinking.
Supposition.Fail
— TypeFail
A result indicating that a counterexample was found.
Supposition.NoRecordDB
— TypeNoRecordDB <: ExambleDB
An ExampleDB
that doesn't record anything, and won't retrieve anything.
If you're wondering why this exists, I can recommend "If you're just going to sit there doing nothing, at least do nothing correctly" by the ever insightful Raymond Chen!
Supposition.Pass
— TypePass
A result indicating that no counterexample was found.
Supposition.Result
— TypeResult
An abstract type representing the ultimate result a TestState
ended up at.
Supposition.SuppositionReport
— TypeSuppositionReport <: AbstractTestSet
An AbstractTestSet
, for recording the final result of @check
in the context of @testset
Supposition.TestCase
— TypeTestCase
A struct representing a single (ongoing) test case.
prefix
: A fixed set of choices that must be made first.rng
: The RNG this testcase ultimately uses to draw from. This is used to seed the task-local RNG object before generating begins.generation
: The "generation" thisTestCase
was made in. Can be used for determining how far along in the generation process we are (higher is further).max_generation
: The maximum "generation" thisTestCase
could have been made in. Does not necessarily exist.max_size
: The maximum number of choices thisTestCase
is allowed to make.choices
: The binary choices made so far.targeting_score
: The score thisTestCase
attempts to maximize.
Supposition.TestState
— TypeTestState
config
: The configuration thisTestState
is running withis_interesting
: The user given property to investigaterng
: The currently used RNGvalid_test_cases
: The count of (so far) valid encountered testcasescalls
: The number of timesis_interesting
was called in totalresult
: The choice sequence leading to a non-throwing counterexamplebest_scoring
: The best scoring result that was encountered during targetingtarget_err
: The error this test has previously encountered and the smallest choice sequence leading to iterror_cache
: A cache of errors encountered during shrinking that were not of the same type as the first found one, or are from a different locationtest_is_trivial
: Whetheris_interesting
is trivial, i.e. led to no choices being requiredprevious_example
: The previously recorded attempt (if any).
Supposition.UnsetDB
— TypeUnsetDB
An ExampleDB
that is only used by the default CheckConfig
, to mark as "no config has been set". If this is the database given in a config to @check
and no other explicit database has been given, @check
will choose the default_directory_db()
instead.
Cannot be used during testing.
Supposition.CURRENT_TESTCASE
— ConstantCURRENT_TESTCASE
A ScopedValue
containing the currently active test case. Intended for use in user-facing functions like target!
or assume!
that need access to the current testcase, but shouldn't require it as an argument to make the API more user friendly.
Not intended for user-side access, thus considered internal and not supported under semver.
Supposition.DEFAULT_CONFIG
— ConstantDEFAULT_CONFIG
A ScopedValue
holding the CheckConfig
that will be used by default & as a fallback.
Currently uses these values:
rng
:Random.Xoshiro(rand(Random.RandomDevice(), UInt))
max_examples
:10_000
record
:true
verbose
:false
broken
:false
db
:UnsetDB()
buffer_size
:100_000
@check
will use a new instance of Random.Xoshiro
by itself.
Supposition.MESSAGE_BASED_ERROR
— TypeMESSAGE_BASED_ERROR
A Union
of some some in Base that are known to contain only the field :msg
.
If you're using one of these errors and require specialized shrinking on them, define a custom exception type and throw that instead of overriding err_less
. The definition of err_less
for these types is written for the most generality, not perfect accuracy.
This heavily relies on internals of Base, and may break & change in future versions. THIS IS NOT SUPPORTED API.
Supposition.Option
— TypeOption{T}
A utility alias for Union{Some{T}, Nothing}
.