Test Matrix

The test surface is broader than a single make test because Formulon ships multiple packages from one source tree. The core, Oracle, packaging, and cross-surface agreement checks catch different categories of regression.

Glossary: CTest labels

CTest groups tests with text labels: SLOW, BENCH, TSAN, PARITY, and VARIANT. Unlabeled tests form the fast tier. Targets include or exclude labels, so a fast pre-commit run can skip slow or diagnostic tests while CI still runs them.

Parity compares Formulon surfaces with each other; only Oracle tests compare it with Excel.

Core tests

make build
make test
make test-slow
make test-all

make test excludes SLOW, BENCH, and TSAN labels. make test-slow includes SLOW while still excluding BENCH and TSAN. make test-all runs every test enabled in the build configuration. Run make test before every commit and make test-all before opening a PR that touches the core.

Oracle tests

make oracle-verify

Oracle verification compares Formulon output with committed goldens generated from Excel. It does not launch Excel and is safe for CI.

Variant oracle tests are opt-in:

cmake -B build-variants -DCMAKE_BUILD_TYPE=Debug -DFORMULON_ORACLE_VARIANTS=ON
cmake --build build-variants --target formulon_oracle_variant_tests formulon_workbook_oracle_variant_tests --parallel
ctest --test-dir build-variants -L VARIANT --output-on-failure

Only provenance-approved variant directories are included. Enabling the option does not activate every captured target. Use this check when investigating profile-specific differences or before adding a supported profile.

The current profile catalog contains 28 ids across 14 locales. The formula Oracle gates cover 14 Mac profiles; win-365-ja_JP is the measured Windows workbook target, and the other 13 Windows profiles are estimated from the Mac measurements plus the Windows host rules. Locale switching, formula translation, and locale-fact access are covered by the core and binding tests even when a Windows Excel capture is not available.

Packaging smoke tests

Surface	Commands
WASM	make wasm, make test-wasm, make npm-test
Python	make python-test
Native Node	make node-test
CLI	CTest target under tests/cli

These verify each binding's load → mutate → recalc → save loop, including host-side value translation and resource lifetime behavior, after a core change.

Cross-surface parity

make parity-test

The runner compares the CLI, npm WASM, and Python channels that are available locally. Unavailable channels are skipped. At least two active channels are required; otherwise the runner exits with skip code 77 and make reports a nonzero result. It fails on disagreements, evaluation failures, or incorrect fixture expectations. Native Node, the pthread npm entry, MCP, and the cell UI require separate tests.

Parity and Oracle

Parity checks whether the available runtime channels agree with each other. Oracle checks whether the core agrees with Excel. Both signals are needed before a release.

Diagnostics

Command	Purpose
ctest -R RegistryCatalog.CoverageReport -V --output-on-failure (run from the build directory, after make build)	Runtime function-registration status against the canonical catalog
make function-status	Recognized, implemented, stub, and environment-bound counts per category, read from the catalog files without a build
make behavior-status	Status of the fine-grained behavior catalog
make coverage	Local coverage diagnostic
make mutation	Local mutation-testing diagnostic

RegistryCatalog.CoverageReport is a diagnostic-only gtest case — it always passes and prints its coverage percentage to stdout, which only surfaces with ctest -V. It is not a gate; treat it as a way to read the current number, not a pass/fail check.

Extended diagnostics

Target	Purpose
make bench	Run the BENCH-labelled benchmark checks. It reconfigures build/ as Release
make fuzz / make fuzz-long	Run the parser, evaluator, file-format, and print-settings fuzz harnesses
bash tools/ci/run_tsan.sh	Run the thread-sanitizer suite used by CI

The fuzz harnesses exist only in a build configured with -DFM_BUILD_FUZZ=ON and carry the SLOW label there; neither the default build nor CI includes them. They need a Clang that ships the libFuzzer runtime, which the Apple toolchain does not, so macOS needs a separate LLVM. AddressSanitizer is off by default on macOS because it deadlocks in its own shadow-memory setup against recent dynamic linkers, so a macOS fuzz run catches crashes, timeouts and undefined behavior but not heap corruption.

Read next

• Build from source — what the test targets compile against.
• Oracle contribution — what feeds oracle-verify.
• Release checklist — when each test runs in the release flow.