Did you know ... Search Documentation:
Packs (add-ons) for SWI-Prolog

Package "mutalog"

Title:Mutation testing for SWI-Prolog
Rating:Not rated. Create the first rating!
Latest version:0.3.2
SHA1 sum:49ad13888e79f0d10e9ce72d85a4728e5a995df0
Author:Piotr Franczyk https://github.com/Looted
Home page:https://github.com/Looted/Mutalog
Download URL:https://github.com/Looted/mutalog/archive/v*.zip

Reviews

No reviews. Create the first review!.

Details by download location

VersionSHA1#DownloadsURL
0.3.249ad13888e79f0d10e9ce72d85a4728e5a995df01https://github.com/Looted/mutalog/archive/v0.3.2.zip
0.3.116af08a925984ea0b8f16a1ab16c559f14aa21241https://github.com/Looted/mutalog/archive/v0.3.1.zip
0.3.0f7fa61ef800a83c304f5a8186022bee651a21f031https://github.com/Looted/mutalog/archive/v0.3.0.zip
0.2.0deef6c6cbca6bc578bb002d416e29187469d88531https://github.com/Looted/mutalog/archive/v0.2.0.zip
0.1.294c5b42cdbe15781788111177caf9f976a767fd51https://github.com/Looted/mutalog/archive/v0.1.2.zip

<h1> <picture> <source media="(prefers-color-scheme: dark)" srcset="assets/logo/mutalog-logo-dark.svg"> <img src="assets/logo/mutalog-logo.svg" alt="Mutalog" height="64"> </picture> </h1>

![Kibi requirement health](https://looted.github.io/Mutalog/kibi-report/)

Mutation testing for SWI-Prolog.

Mutalog makes small changes to your Prolog code, such as a flipped comparison, a removed cut or a dropped clause, and checks that your plunit tests notice. A change the tests do not notice is a surviving mutant: a behaviour no test pins down.

Status: released, see CHANGELOG.md. Mutalog generates mutants, runs your plunit tests against each one on parallel workers, and reports a mutation score in the terminal, as HTML and as JSON. See Roadmap for what is in it.

Requirements

SWI-Prolog 9.0 or later. Mutalog has no dependencies outside the SWI-Prolog distribution.

Install

As a pack:

?- pack_install(mutalog).

Then, from your project root:

swipl mutalog run                     # SWI-Prolog 9.1.22 or later
swipl -g "use_module(library(mutalog))" -g mutalog:main -t 'halt(2)' -- run   # any 9.x

Or from a checkout, with no install:

git clone https://github.com/Looted/Mutalog.git
Mutalog/bin/mutalog run

The examples below use mutalog for whichever of these you use.

Usage

Run from your project root:

mutalog run                                   # mutate, run the tests, report the score
mutalog run --only 'src/eval.pl:42:9:compare:1'   # re-run chosen mutants while fixing tests
mutalog dry-run                               # run the suite once: per-test times and coverage
mutalog list                                  # print the mutants without running anything
mutalog list --format json --check            # machine-readable; verify every mutant re-reads

Every command takes `--mutate GLOB and --tests GLOB` (both repeatable), `--operators a,b`, `--timeout-ms N`, `--config FILE and --root DIR`. run also takes `-c N or --concurrency N` for the number of workers, `--reporters a,b` to pick the outputs, and `--verbose` to show the project's own output.

`mutalog run` prints each mutant as it finishes, then the survivors with their source change, and a summary:

Survived (12):
  src/calc.pl:16:18:compare:1
      X >= Y  ->  X > Y
      covered by 1 test(s)

File                                       Score  Killed   Surv. Timeout No cov.  Errors
src/calc.pl                                67.3%      35      12       2       6       0

Mutation score: 67.3% (covered code: 75.5%)

It also writes two report files to `reports/mutation/`:

  • mutation.html is one self-contained page with no network access. It shows the score per file and the source with every mutant marked. You can filter by status and click a mutant to see the change and the tests that cover it.
  • mutation.json follows the open mutation-testing-report-schema format, schema version 2, so other tools that read the format can use it. A survivor is a change no test notices: add a test that pins the behaviour down. If the change is equivalent, so that no test could tell it apart, suppress it with a comment on the line above it, or list its id in ignore_mutants: 
    size(node(L, R), N) :-
    % mutalog-disable-next-line cut_insert: size/2 has one solution
    size(L, NL),
    size(R, NR),
    N is NL + NR + 1.

    `% mutalog-disable-next-line` suppresses the mutants that start on the next line. Operator names after it, separated by commas, limit it to those operators, and text after a colon is a free-form reason. The comment moves with the code, so it survives edits elsewhere in the file.

    Ids, as `--only and ignore_mutants` take them, have the form file:line:column:operator:n. They stay stable as long as the code above the mutant doesn't move. When they do move, `--only` stops with an error naming the ids it didn't find, and ignore_mutants entries that no longer match print a warning.

Statuses and score

StatusMeaning
killedA test that covers the mutated clause failed
survivedEvery covering test passed
timeoutThe tests ran past the time limit, usually because the mutant loops
no coverageNo test entered the mutated clause, so nothing was run
compile errorThe mutated file did not load
runtime errorThe worker process crashed, on every retry

The mutation score is the share of mutants detected, counting killed and timeout, out of killed, timeout, survived and no coverage. The covered-code score leaves out no coverage. `mutalog run` exits with 1 when the score is below thresholds(_, _, break(N)), or when N is above 0 and either no mutant was scored or more than half of the mutants did not load. Generated mutants always parse, so many compile errors mean the run itself is broken; the clear-text output shows the first load error of each file, and the JSON report puts it in statusReason. A mutate or tests glob that matches no files is an error.

Configuration

Put a mutalog.config.pl in the project root. Every key is optional.

Globs and report_dir are resolved against the project root, which is the directory that holds the config file. So `--config FILE` also moves the root to the directory of FILE. To keep configs elsewhere, for example one per module, set root(Dir) in the file (relative to the file's directory) or pass `--root DIR` (relative to the current directory).

mutate(["src/**/*.pl"]).              % files to mutate (globs, ** for any depth)
ignore(["src/generated/**"]).         % files to skip
tests(["test/**/*.plt"]).             % plunit files to load and run
load(["load.pl"]).                    % files to load before the tests, if they need it
operators([arith, compare, unify, negate, cut, cut_insert,
           goal_true, goal_fail, clause_drop, literal]).
ignore_mutants(["src/eval.pl:42:9:compare:1"]).
timeout_ms(5000).                     % time limit per mutant is
timeout_factor(1.5).                  %   factor * covering tests' time + timeout_ms
stack_limit('1g').                    % for each worker process
concurrency(0).                       % workers; 0 means CPU count - 1
retries(1).                           % extra attempts for a mutant that crashes a worker
worker_reuse(0).                      % replace a worker after N mutants; 0 never
thresholds(high(80), low(60), break(0)). % report colours; exit 1 below break
reporters([clear_text, progress, html, json]).
report_dir("reports/mutation").
root("..").                           % project root, relative to this file

In the reports a score at or above high shows green, at or above low amber, and below low red. Set break above 0 to fail CI when the score drops below it.

Mutation operators

OperatorWhat it changesExample
arith+ and -, * and /, // mod rem to *, min and max, << and >>; unary minus removedX is A + B to X is A - B
compareArithmetic and standard-order comparisons, across the boundary and flippedX > 0 to X >= 0, X =< 0
unify= and \=, == and \==, =@= and \=@=X == Y to X \== Y
negateRemoves a \+; negates the condition of an if-then-else\+ member(X, L) to member(X, L)
cutRemoves a cutp :- q, !, r. to p :- q, true, r.
cut_insertInserts a cut after the first goal of a non-last clause that has none. When that goal has only one solution the mutant is equivalent; drop the operator or suppress those mutantsp :- q, r. to p :- q, !, r.
goal_trueReplaces a body goal with truep :- q, r. to p :- true, r.
goal_failReplaces a body goal with failp :- q, r. to p :- q, fail.
clause_dropRemoves one clause of a multi-clause predicatethe second of p(a). p(b).
literalNumbers (0 to 1, N to N+1 and -N), strings (to and from ""), true and false, [] to [_]limit(10) to limit(11)

Directives, DCG rules, CHR rules and anything inside begin_tests/end_tests are never mutated.

Operators are clauses of the multifile hooks operator/2 and mutate/4, so a project can add its own; see prolog/mutalog/operators.pl.

How it works

  1. Read. Each file is read with SWI-Prolog's own reader (prolog_read_source_term/4), so operators, quoting, dicts and strings are handled exactly as the compiler handles them. That includes operators imported from libraries. Each clause comes with its subterm positions, so a mutant can be reported as a character range and a replacement.
  2. Dry run. A child swipl loads the tests and runs each one under SWI-Prolog's coverage instrumentation. It records how long each test took and which clauses of the mutated files it entered. A failing test stops the run, because mutants can only be judged against a green suite.
  3. Mutants. A pool of child processes, one per worker, applies each mutant by reloading its file from the source text with the replacement spliced in. A worker then runs only the tests that entered the mutated clause, stopping at the first failure. Mutants with no covering test are settled without running anything.
  4. Time limits. Each mutant gets a limit of timeout_factor times its covering tests' dry-run time, plus timeout_ms. The worker enforces it with call_with_time_limit/2. Code under test can catch that exception and loop on, so the controller also enforces a hard limit 10 seconds later. It kills a worker that misses it and starts a fresh one.
  5. Scheduling and retries. Mutants with the longest limits start first, so a slow mutant doesn't leave the other workers idle at the end. A mutant that crashes its worker is tried again on a fresh worker, up to retries times. Timeouts aren't retried, because a timeout already counts as detected. Results are reported in source order whatever the order they finished in. `mutalog list --check` re-reads every spliced mutant and compares it with the clause the runner will load.

Roadmap

  1. Read and generate: `mutalog list`. Done.
  2. Dry run and worker: `mutalog run and mutalog dry-run` with one worker. Done.
  3. Pool, scheduler, timeouts: parallel workers, longest-first scheduling, hard time limits, retries for crashes. Done.
  4. Reports and thresholds: clear text, JSON in the open mutation-testing report schema, a self-contained HTML report, break threshold exit code. Done.
  5. Release: BSD-2-Clause license, installable as a pack with a `swipl mutalog` app, versioned releases. Done in 0.1.2, see CHANGELOG.md.

Development

swipl test/run.pl                                    # tests
bin/mutalog list --mutate 'prolog/**/*.pl' --check   # Mutalog's own mutants re-read
(cd test/project && ../../bin/mutalog run)           # the example project

Releasing

  1. In a pull request, bump version('X.Y.Z') in pack.pl and add a `## X.Y.Z` section to CHANGELOG.md. CI fails if the two disagree.
  2. Merge it, then run the Release workflow on master from the Actions tab. The workflow refuses a version that is already tagged. It creates the vX.Y.Z tag and GitHub release with the changelog section as notes, and publishes the pack. Don't create tags or releases by hand: a tag on a commit whose pack.pl disagrees can't be reused.

License

BSD-2-Clause. See [LICENSE](LICENSE).

Contents of pack "mutalog"

Pack contains 335 files holding a total of 624K bytes.