Concepts

Describing decisions

There are multiple situations where you need to describe how a submission should be evaluated and what is correct behaviour and what not. This can include the following:

  • Specify the success condition of a test case

  • Trigger actions like showing warnings depending on the submission

  • Idea: Provide a formula for the allowed number of blocks (per type) a submission may use (requires selectors that are not implemented yet)

This library thinks of decisions in the following way:

  1. You select values that are currently unknown (as you don’t know what submissions will be handed in in the future), …

  2. … transform these values with multiple operations, …

  3. … and finally give them to a criterion that decides if the condition is satisfied.

By using that model, this library only provides some predefined selectors, transformations and criteria that can be plugged together in multiple ways.

Placeholders live in the future

The created tree of multiple different selector-, transformation- and criterion-nodes is then transformed to a JSON representation that can be stored or sent to a server.

Warning

As this tree can only contain predefined stages plugged together, you can’t define new functionality for evaluating submissions that can not be expressed by combining the predefined stages. The generated JSON will be interpreted at a later point in time by another program that won’t know anything about your modifications and deviations from the normal specification format which will likely just cause errors.

Available decision components

Selectors

Transformations

Criteria

Catchable errors

There are multiple situations in which decision entities can encounter unexpected situations or failures. One of the simplest can occur when selecting the first OUTPUT during a decision as there can be submissions that don’t produce any output at all. As the term Catchable errors implies, one can still work with those errors to procede the evaluation by catching them.

Normally, selectors, transformations and criteria work on the happy path:

                                  v catch
(error path)       +--------------+
                  /                \
                 /                  \
(happy path) ---+--------------------+----->
                ^ error

When errors – that aren’t severe enough to cancel the whole execution – occur, the values move to the error path and further (normal) transformations and criteria will be skipped to pass the error up. By using a special method to catch the error, the execution can come back to the happy path when the error is handled.

Uncatched errors have different effects depending on the context in which they occur.

See err for a list of available error kinds that can be catched.

Actions

In addition to specifying if a test should pass or fail, Criteria can be used to trigger actions if there conditions are satisfied:

Test specification

The top-level part of a specification is Bast3StSpec which contains all categories and meta information about the point of the whole exercise. A Bast3StSpec object can contain multiple Category objects that group tests by logical relation. Each Category has multiple MainTest instances each of them being able to hold AlternativeTest objects. These objects (aside from Bast3StSpec) should mostly not be instantiated directly but through special functions:

See the functions for required arguments. All levels support extra hook conditions to run actions if some criterion is suffied:

There are also those convenience functions which call the above with ALWAYS_FULFILLED:

Each of those functions specifies an order which decides when the condition should be checked and the action be run. The following shows a typical flow

Execution order of a test specification

  1. run hooks of specification that should be run before all categories

  2. process each Category:

    1. run hooks of Category that should be run before all tests of this category

    2. process each MainTest:

      1. run hooks of MainTest that should be run before the actual test

      2. evaluate the main test’s Criterion to decide if the test should be passed

      3. run hooks of MainTest that should be run after the actual test

      4. If the main test failed, run hooks of MainTest that should be run after the main test but before the alternative tests

      5. If the main test failed, process each AlternativeTest in order until one passes or all were tried:

        1. run hooks of AlternativeTest that should be run before the alternative test

        2. evaluate the alternative test’s Criterion to decide if the test should be passed. (if so, run 12 but proceed to 13 without trying 9 again)

        3. run hooks of AlternativeTest that should be run after the alternative test

      1. run hooks of MainTest that should be run after the main test and after the alternative tests

    1. run hooks of Category that should be run after all tests of this category

  1. run hooks of specification that should be run after all categories

Note

This order of (hook) execution is important if you rely on results from other stages being present, like when you set flags or change the global state in other means.

Categories of a specification and main tests of a category are computed independently from each other (maybe even in parallel).

Setting and reading flags

Flags are a way for the author of a test specification to store values across stages and communicate them to external tools. If you set the same flag in multiple stages or want to read a flag that was set in another stage, you should consider the following about flag visibility to ensure that the changes are really visible when you expect them to be available:

  1. Flag changes will never be visible during the same “atomic” step in which they were made, where “atomic” step means a single evaluation like checking if the criterion of a hook is valid, evaluating the action of a hook, …

  2. Different steps within a sequentially treated process can see all flag changes of their predecessors in the order they are specified in source.

    1. Different hooks of the same kind and entity (e. g. all before main hooks of a main test) are run sequentially. So every hook will see the flag-state its predecessor hook produced.

    2. The action of a hook is executed strictly after the criterion of the same hook, so the criterion can set flags that are then visible to the action

    3. Inside a main test (the main tests hooks, alternative tests and their hooks) are run sequentially in the order Execution order of a test specification describes.

  3. Parallel stages are all started with the same initial flag-view and operate independently from the flag-setting of others till they end. As soon as they have finished, their flags are merged in the order in which they occur in the specification file:

    1. All categories of a specification are run with the flag-state after the last before all categories hook. When they finished, their final flag values are merged sequentially. Afterwards this state is used as input to the first after all categories hook.

    2. All tests of a category are run with the flag-state after the last before all tests hook. When they finished, their final flag values are merged sequentially. Afterwards this state is used as input to the first after all tests hook.

How flags are merged

Flag-merging is non-commutative, so the order matters if there are conflicting keys. The left or old flags name in the following the flags from a predecessor step and right or new the flags from the current step that need to be merged into the existing ones.

Flags allow only primitive values (texts and numbers), so no mappings or arrays can be set here, but as flags do allow nested keys, one can set multiple keys with the same prefix and then read them at a later point as a mapping.

A simple example:

  1. set_flag("points", "test1", value=4) (after: {"points": {"test1": 4}})

  2. set_flag("points", "test2", value=0) (after: {"points": {"test1": 4, "test2": 0}})

  3. set_flag("points", "test3", value=2) (after: {"points": {"test1": 4, "test2": 0, "test3": 2}})

  4. set_flag("points", value=FLAGS["points"].sum()) (after: {"points": 6})

The merging process can be described with pattern matching and recursion:

match (old, new):
(Mapping o, Mapping n) =>

items with keys that occur in exactly one of the mappings are taken as they are, for conflicting keys the values are merged in the same order to get the key’s new value.

(Primitive o, Mapping n) =>

the old value is treated as the mapping {"": o} (empty string to normal value) and this value is then merged with the mapping n, so it survives as long as n doesn’t contain "" as a key.

(Mapping o, Primitive n) =>

the new value (n) wins and is kept instead of the whole mapping

(Primitive o, Primitive n) =>

the new value (n) wins and is kept

Parameters

Parameters are similar to flags, but they are readonly for specifications and can only be set from the outside. They are accessible through PARAM and are grouped into multi-level scopes:

{
    "doc": {
        "blockcount": {
            "total": NUMBER,
            "opcode": {
                opcode1: NUMBER,
                opcode2: NUMBER,
                ...
            }
        }
    },
}

The opcode mapping contains a value for every block-kind the submission used. See the Scratch wiki for a list of official opcode-names but be aware that only the ones from scratch-test-model/BlockKindUnit and co are detected by the application. (detected doesn’t mean supported and unsupported blocks will just be skipped during execution of a program if possible or cancel execution if not.) If you try to read an item from PARAM["doc", "blockcount", "opcode"] that doesn’t exist, you will receive 0 instead of an error.

PARAM["doc", "blockcount", "total"] will be the same value as PARAM["doc", "blockcount", "opcode"].sum() or PARAM["doc", "blockcount", "opcode"].values().sum().