Test Tool

Canonical source: docs/guide/tools/test-tool.md at locked revision e99b712ecf658a527b1fe82c8beba2c1286d8dc9. This rendered page is non-normative; the applicable Protos specification remains authoritative.

The Test Tool is the bundled Protos Tool behind protos test.

It owns higher-level testing policy in Protos: loading the retained corpus plans, interpreting expectations, deciding bounded outer test-case admission, aggregating results in a stable order, and producing Test Tool progress/results. The host runtime supplies lower mechanisms such as executing one already-selected source in an isolated semantic Process.

For the general Tool/Core/Standard-Library/runtime boundary, start with Bundled Tools. The durable implementation/work record is TOOL002.

Current-surface note: this chapter documents the executable Test Tool that exists now. The repository suite graph that orders its plans is an internal routing structure, not a public CLI selection surface.

First use

The ordinary command is:

protos test

The Test Tool currently executes the Protos test corpora selected by the installed/repository toolchain. It does not recursively discover arbitrary *.protos files from the current working directory, and the current CLI does not take a user-supplied test-root or manifest path.

Today the bundled entry point executes the leaves of the repository suite graph (RepositorySuite.protos) in their fixed logical order. Each leaf is reported under a phase label such as main, actor, group or package-toml; the complete label list lives in Main.protos. The first phase, main, is the primary language/runtime/library conformance corpus rooted at protos/tests/conformance.

This routing is a current implementation boundary, not a design recommendation for future corpus composition.

The corpus is manifest-driven

Manifest-backed plans, such as main, have a manifest.tsv. For the ordinary conformance-style plans, every non-empty, non-comment line has three tab-separated fields:

path<TAB>expectation<TAB>expected

Blank lines and lines beginning with # are ignored.

The path is a confined relative test path. The current parser rejects empty path components, . and .. components, backslashes and NUL. A manifest therefore selects an exact file inside the corpus rather than giving the Tool an ambient host path to resolve freely.

The primary retained manifest can be inspected at protos/tests/conformance/manifest.tsv. The manifest parser and inert plan representation live in Manifest.protos.

The Package Tool TOML syntax corpus retains a smaller two-field format:

path<TAB>retained-expectation

where the current retained values are true and error. The Test Tool maps those rows into its ordinary internal expectation model before execution.

These manifest formats are the current repository corpus contract. They should not be read as a promise that all future third-party/project tests will be registered through exactly this file format.

A real minimal retained case

The repository contains this exact source at integer/add-small.protos:

1 + 1

Its manifest row is:

integer/add-small.protos  integer  2

Read that row as:

1. execute exactly integer/add-small.protos in the case’s isolated execution;
2. interpret the resulting observation using the integer expectation policy;
3. pass only if the completed value is the integer 2.

A test case is therefore not required to call a special assertion function. For this retained corpus model, the program result plus the manifest expectation define success or failure.

Expectation model

The Test Tool compares each supported manifest expectation against the captured execution observation. Common retained forms include:

Expectation	Meaning of expected
boolean	true or false
null	no payload (- is retained convention)
integer	decimal ordinary Integer text
float-bits	exact 16-hex-digit binary64 bit pattern
float-nan	NaN result; payload/sign are not made portable semantics
error	execution must fail with an ordinary guest Error
error-parent	guest Error whose parent matches the retained prototype name

The current retained corpus also has specialized expectation families for Futures and error-identity/liveness checks, including resolved Future values, Future cancellation/error states and selected inspection-based cases. Those are implemented by the same runner rather than by a second test framework.

The current expectation vocabulary is defined by Runner.protos. When adding retained corpus rows, use an expectation that the current runner already recognizes; documentation is not authority to invent a new expectation kind.

One fresh semantic Process per test execution

Each selected test execution runs in a fresh semantic Protos Process. The source is parsed/executed inside that fresh Process context, and the Process is terminated after the execution observation is obtained.

That is the isolation boundary that matters publicly. One case must not depend on mutable Process/RootActor state left by a previous case.

The lower exact-execution mechanism is deliberately test-neutral; it receives an already-selected source and returns captured execution evidence. The Test Tool, not that host mechanism, owns manifests, expectations, scheduling and reporting.

Current mechanism references:

• ProtosFreshProcessExecutor creates one fresh semantic Process for an exact source;
• ProtosCapturedProcessExecution captures the execution outcome and private standard streams.

Those Java class names explain the current implementation. They are not a portable Test Tool API.

Standard output is private to the case execution

A test execution receives private standard-stream backends. Its stdout and stderr bytes are captured as part of that execution’s evidence instead of sharing one writable stream with concurrently running cases.

This matters for parallel testing: two tests may physically overlap without interleaving their guest output into one semantic test stream.

The Test Tool has its own progress/reporting channel. Do not use guest output timing as a synchronization or result-order contract.

--jobs: bounded outer test-case parallelism

By default, the Test Tool uses logical capacity 1:

protos test

To allow up to four outer test-case executions to be in flight in one scheduling window:

protos test --jobs 4

--jobs N controls Test Tool case admission, not the concurrency mechanisms inside one test. A test remains free to use ordinary Protos Futures, Actors, Actor Groups or other published concurrency semantics inside its own fresh Process.

The current option contract is:

• N is required after the exact separate token --jobs;
• N must contain only decimal digits and represent an Integer greater than zero;
• if --jobs is absent, the value is 1;
• the selected option may occur at most once; duplicate or missing selected values fail option parsing;
• there is currently no separate documented numeric maximum below the ordinary Integer domain; practical admitted concurrency may be lower when other Test Tool constraints apply.

For compatibility with the existing TOOL002 argument policy, only the exact separate-token form is selected. In particular:

--jobs 4     # selected
--jobs=4     # currently ignored, not an alias for --jobs 4

Other unknown tokens are currently ignored by the Test Tool projection layer, including --. That permissive compatibility behavior should not be used as a namespace for scripts. Prefer only documented exact options.

The current parser is Options.protos.

Determinism with --jobs > 1

Increasing --jobs changes how many independent test cases may execute concurrently; it does not make physical completion order the public result order.

The runner admits bounded windows, waits for the admitted executions, then projects their observations in the original TestPlan/manifest order. The stable case order therefore comes from the plan, even when physical executions finish in another order.

Conceptually:

manifest / TestPlan order
    case A   case B   case C   case D
       \       |        /       /
        \--- physical overlap --/
                 |
                 v
       observations projected back
          in TestPlan order

This is why CI or tooling that consumes Test Tool results should not infer meaning from which physical case happened to finish first.

The outer scheduler belongs to bundled Protos policy in Runner.protos. Current host code only transports already-admitted exact executions; host thread topology is not part of the portable --jobs contract.

Progress is separate from case completion order

The current Test Tool reports phase progress independently from guest case streams. Phases use the suite-graph labels described in First use, and the progress layer can report failed cases plus phase/invocation summaries.

Detailed failure classes, final diagnostics and public exit-status meaning are covered by the DOC005 results/diagnostics slice. The important fundamental rule here is simpler:

parallel execution may change physical timing, but it must not redefine the plan’s logical ordering or make guest stream interleaving the reporting model.

The current progress policy is implemented in Progress.protos.

--file: exact file-backed focal selection

To run only the tests that the Test Tool already associates with one local source file:

bin/protos test --file protos/tests/library/uri/parse.protos

The option uses the exact separate-token form --file FILE. As with --jobs, an attached form such as --file=FILE is not an alias and is currently ignored.

FILE is a locator for this invocation only. A relative path is interpreted against the working directory of the protos test invocation; an absolute path is also accepted. The path is not a test’s identity: it is not a Case identity or --case reference, not a corpus or suite identity, not an execution-requirement identity, and it is not persisted anywhere as a name for the selected tests.

Selection is a filter over the authoritative test plans the Test Tool has already built, not a discovery mechanism. An existing *.protos file does not become a test because it was named on the command line, and --file never executes an arbitrary source that the plans do not already contain. The file can correspond to:

Authoritative Cases for FILE	Result
none (unregistered, missing, outside every corpus, or a directory)	the selection fails as a Test Tool configuration error before any test runs
one	that Case is selected
several	every matching Case is retained

The example above is the “several” case: the file is one physical source, but its protos/corpus/library/uri plan entry contributes several logical Cases (parse components, parse invalid, parse valid, …). All of them run, each still a distinct logical Case.

--file FILE may be repeated. Each occurrence must independently match at least one authoritative Case, otherwise the invocation fails before scheduling. The matches of all occurrences are combined as a union: a Case selected more than once runs once, and selected Cases keep the canonical plan order rather than the order of the --file arguments.

Selection happens after the authoritative plans are materialized and before progress counting and scheduling. Each selected Case keeps its existing metadata, expectations, resource requirements and execution authority and then follows the ordinary Test Tool pipeline. --file does not change the meaning of --jobs or --resource-catalog, fresh-Process isolation, result classification, progress or exit status; it only reduces which Cases are executed.

--directory, --list-cases, and --case

Three further options refine the same authoritative plans. Like --file, they use separate-token forms only; attached forms are ignored.

--directory DIR (repeatable) selects every authoritative Case whose source lies under DIR, recursively. It is a filter, not filesystem discovery: files under DIR that no plan contains are not run. Each --directory must match at least one Case. --file and --directory combine as a union, a Case selected twice runs once, and plan order is kept.

bin/protos test --directory protos/tests/library/semver

--list-cases builds the normal plans, applies any --file, --directory, and --case selection, prints one JSON document on stdout, and stops before progress, scheduling, or running any Test body:

{"schema":"protos.test.cases/v1","cases":[{"ref":"v1.…","display":"…"}, …]}

Cases appear in authoritative plan order. ref is the stable Case reference; display is presentation-only text for people and must not be parsed.

--case CASE_REF (repeatable) restricts execution, or a listing, to the discovered Cases whose ref matches exactly. A ref is opaque: copy it from --list-cases output rather than building it. A malformed, repeated, unknown, stale, or out-of-scope ref fails the invocation before any Case runs. Selected Cases still run in plan order, regardless of argument order.

bin/protos test --directory protos/tests/library/semver --list-cases
bin/protos test --case v1.… --case v1.…

None of these options changes --jobs, isolation, result classification, or exit status; they only reduce which Cases run.

Suite-native Cases

Many current corpora, including the Standard Library suites under protos/tests/library/, are suite-native: each source declares its own Cases with std:test/Test and std:test/Assertions, and the Test Tool discovers them from the source’s tests Array:

Assertions: import("std:test/Assertions")
Test: import("std:test/Test")
SemVer: import("std:semver/SemVer")

tests: [
    Test("build metadata is ignored by precedence", () => {
        a: SemVer.parse("1.2.3+linux")
        b: SemVer.parse("1.2.3+windows")
        Assertions.require(SemVer.comparePrecedence(a, b) == 0)
    })
]

Each Test(name, body) is one logical Case. Assertions.require(condition) signals Assertions.AssertionFailure when condition is false, and Assertions.signals(errorPrototype, body) requires body to signal a matching Error. A suite-native source runs only when it belongs to a registered corpus; writing such a file elsewhere does not make protos test find it.

What protos test does not mean yet

With the current published executable surface:

• it does not recursively discover every repository *.protos test;
• it does not select arbitrary current-directory test roots;
• it does not expose the D122 suite graph as a public CLI selection surface;
• --jobs does not mean “number of Actors” or “number of host threads”;
• manifest ordering remains semantically relevant even when cases overlap.

Results and exit status

protos test has more than one failure lane. The final process status is useful for automation, but it should be interpreted together with the diagnostic lane that produced it.

For a normally completed Test Tool invocation, bundled Protos aggregates every owned plan that actually executed. The primary conformance plan remains the compatibility payload, but it is not the only plan that can make the command fail.

Outcome	Meaning	Current CLI exit
completed; every executed owned plan healthy	ordinary successful test run	0
completed; at least one ordinary guest/test failure	tests ran normally, but one or more tests failed	1
infrastructure-aborted	the Test Tool could not complete normal execution because an admitted resource/provider attempt reached infrastructure failure	3
unexpected Test Tool/runtime failure	internal/tool implementation failure rather than a structured Test Tool outcome	70

The completed classification is invocation-wide. A failure in actor, group or package-toml affects the command just as a failure in main does. An ordinary guest/test failure does not by itself stop later owned plans from running.

Infrastructure abort is different. D108 fail-stop prevents later work from being admitted once the invocation reaches that condition, and exit 3 has precedence even if ordinary guest/test failures were already produced earlier.

Configuration and Tool-policy errors are not completed test failures

There is one important current CLI distinction around exit 1.

Some failures happen before the Test Tool can return a completed TestRunOutcome. For example, an invalid selected --jobs value or an invalid Tool-owned configuration/schema can signal an ordinary Test Tool Error. The bundled-tool host reports that lane as:

Test tool error: ...

and currently returns exit 1.

That is not the same thing as:

TestRunOutcome.status = "completed"
exit = 1

from an ordinary failed test run.

Therefore automation that needs to distinguish “a test failed” from “the Test Tool rejected its configuration” must also inspect the diagnostic class/text; the numeric exit alone is not sufficient for that distinction today.

The driver retains exit 2 for its CLI usage-error lane, but the Test Tool’s historical argument projection is intentionally permissive: unknown Test Tool tokens are currently ignored, while invalid values of selected Tool options are Tool Errors rather than a new generic usage classification.

Guest failure and infrastructure failure are orthogonal

A guest program can fail its expected test without anything being wrong with the Test Tool infrastructure.

Conceptually:

test executes
    |
    +-- observation matches expectation
    |       -> guest/test PASS
    |
    `-- observation does not match expectation
            -> guest/test FAIL

That is ordinary completed Test Tool policy.

Infrastructure evidence belongs to a separate lane. A provider/provisioning, transport, terminalization, cleanup or unsafe-capacity failure must not be invented as a failed guest test.

The stable infrastructure-abort payload preserves these categories separately:

executedCaseRuns
ordinaryUnsupportedCount
infrastructureAttempts
cutoverNotAdmittedCases
retainedUnsafeReservationCount

Already-produced guest CaseRuns remain real guest evidence even if a later cleanup/infrastructure failure changes the invocation status to infrastructure-aborted.

Likewise, a case that was never admitted after fail-stop is not fabricated as a failed test or an ordinary skipped test.

Current infrastructure-abort CLI diagnostic

The current host-side reporter consumes the already-decided Test Tool outcome. It does not recompute guest pass/fail policy.

For infrastructure-aborted, the current CLI writes a bounded summary to standard error with this shape:

Test infrastructure aborted
infrastructure attempts: N
cutover not admitted: N
retained unsafe reservations: N

The internal Test Tool outcome retains richer ordered evidence than these three counts. The current human-readable CLI output is deliberately not a frozen machine-readable report schema.

In particular, do not parse those prose lines as a durable JSON/JUnit-style API. A future structured reporter can project the same Test Tool evidence without changing the 0 / 1 / 3 classification.

Progress output

Test Tool progress is separate from guest standard output.

Guest stdout and stderr remain private captured streams for the individual case execution. Test Tool progress is written on the Tool’s own standard-error channel.

Current progress records have shapes such as:

[main] 0/N
[main] FAIL some/case.protos
[main] K/N
[main] N/N passed

or, for infrastructure failure:

[main] INFRA some/case.protos
[main] K/N infrastructure-aborted

The final invocation summary reports aggregate passed/failed counts for normal completion, or completed/failed counts plus infrastructure-aborted for the infrastructure lane.

These records are useful to humans and CI logs, but physical case completion order is still not the result-order contract. Logical TestPlan order remains the stable evidence order.

CI recipes

Simple fail-closed CI

If the CI system only needs success versus non-success, use the command directly:

bin/protos test --jobs 2

The command returns non-zero for ordinary completed test failure, structured infrastructure abort, or an unexpected Tool/runtime failure.

The repository’s TOOL002 closure uses this Java-first / Protos-tool-second model: Java/runtime validation is one stage, then the real public Test Tool command is a separate fail-closed stage.

Distinguish the major exit lanes

When CI needs to react differently to infrastructure abort, inspect the exit code without trying to infer semantics from physical completion order:

set +e
bin/protos test --jobs 2
status=$?
set -e

case "$status" in
  0)
    echo "Protos tests passed"
    ;;
  1)
    echo "Protos tests failed or the Test Tool rejected configuration/policy" >&2
    exit 1
    ;;
  2)
    echo "Protos CLI usage error" >&2
    exit 2
    ;;
  3)
    echo "Protos Test Tool infrastructure aborted" >&2
    exit 3
    ;;
  70)
    echo "Unexpected Test Tool/runtime failure" >&2
    exit 70
    ;;
  *)
    echo "Unexpected protos test exit: $status" >&2
    exit "$status"
    ;;
esac

If your CI needs to distinguish ordinary failed tests from Tool configuration errors within exit 1, retain and classify the Test Tool diagnostic text as well. Do not silently treat every 1 as an assertion/test mismatch.

Do not key automation to completion timing

With --jobs > 1, independent cases may finish physically in a different order from their manifest order. CI should use the final exit classification and stable Test Tool evidence/reporting rather than treating “the first line that finished” as authoritative.

What is deliberately not a public reporting contract yet

The current Test Tool does not publish a stable JSON result schema, JUnit XML mapping, retry/flaky classification, remote-worker protocol, hard timeout recovery model or event-stream API.

Those can be added later without changing the current core classification:

0  completed and healthy
1  completed with ordinary guest/test failure
3  infrastructure-aborted

Nor should a future richer reporter erase the distinction between guest evidence and infrastructure evidence.

Resource-guide status

For manifest-backed and package-toml plans, the Test Tool still applies the ratified resource-requirements.toml join while planning. True absence of the sidecar leaves the plan without added requirements; a present sidecar remains fail-closed under the existing D094 acquisition, schema and D091 referential-join rules. This is retained planning/validation behavior only.

The current Test Tool does not execute resource-backed Cases. Every scheduled Case runs through the logical Case runner, which fails closed if a Case carries resource requirements. A parsed --resource-catalog PATH does not feed that execution path, and no public resource provider, binding or reservation path exists. This chapter therefore documents no resource-aware recipe.

Where to go next

This chapter now covers the current Test Tool fundamentals plus the published result/diagnostic/exit-status and CI surface.