SOTER Testing Strategy & Code Quality Guidelines

Purpose

This specification defines the unified testing approach, code quality rules, and automated validation systems (Pyrefly) that ensure the reliability and consistency of the SOTER platform. It exists so that developers and AI agents follow one shared contract for layer integrity, type strictness, test coverage, and pre-commit/CI enforcement instead of ad hoc conventions.

Scope

This document covers:

• layer segregation rules across the compilation pipeline (.model -> Parser -> AST -> Validation -> Semantic Model -> .view/.sheet -> Render Plan -> Output);
• strict type specification rules enforced by Pyrefly, including phase-specific typings and identifier separation;
• the three SOTER test levels (unit, integration, end-to-end/regression) and the tools that execute them (pytest, Playwright);
• the developer "vibecoding" workflow and the local quality-check commands required before a commit;
• test data management conventions (tests/data/ vs. soter-workspace/);
• pre-commit rules and the CI/CD verification pipeline;
• stakeholder-driven E2E scenarios used to validate real-world workflows;
• the Pyrefly validation error code reference ("Pyrefly Codex").

Non-Goals

• This document does not define the Logos language grammar itself; see LogosLanguageSpec.md.
• This document does not define platform or compiler architecture; see ArchitectureOverview.md and TechnicalArchitectureSpec.md.
• This document does not configure the CI/CD pipeline or pre-commit hooks themselves; it describes the rules those systems must enforce.
• The Pyrefly Codex is explicitly a living reference: it does not attempt to enumerate every validation error the Pyrefly engine may eventually implement.

Terminology

Model or Contract

SOTER relies on strict architectural boundaries. The system acts as a pipeline that must never be bypassed:

.model ──> Parser ──> AST ──> Validation ──> Semantic Model ──> .view/.sheet ──> Render Plan ──> Output

Each phase of this pipeline requires its own distinct domain type, so that a type from one phase can never leak into another:

RawNode ──> ParsedNode ──> ValidatedNode ──> SemanticElement ──> SheetSpec ──> RenderPlan ──> RendererConfig

Verification of this pipeline is partitioned into three concentric circles of testing, executed via pytest and Playwright:

• Unit Testing (pytest): individual, stateless module validation with mocked dependencies — Parser grammar validation, Linter syntax/semantic checks, Graph Engine (NetworkX) construction correctness, and utilities (i18n helpers, custom indenters, config parsers).
• Integration Testing: flow correctness between integrated layers — pipeline stages (Loader -> Graph -> Validator -> Store), the database layer (SQLite ledger writes and in-memory KuzuDB graphs), and the Web API (pytest-django blueprint and route handler verification).
• End-to-End (E2E) & Regression Testing: the complete execution loop under real-world scenarios — CLI regression against stored "golden" files, Web UI Playwright E2E (diagram rendering without asset 404s, navigation, ledger execution/Git integration interactions), and workspace integration checks between SOTER and soter-workspace.

Rules

• No mixing of levels: raw AST nodes must never be passed directly to the renderer, and semantic models must not be used directly as renderer configurations.
• Separation of concerns: always keep .model (physics/ontology) concepts isolated from .view/.sheet (projections/UI) concepts.
• Term precision: Element, Diagram, Sheet, and Projection are distinct system boundaries and must never be used as synonyms.
• No implicit derivation: vague terms and implicit defaults are prohibited; every message flow and constraint must be explicitly declared in Logos.
• Signature annotations: every public function must have complete, explicit type annotations, e.g. def transform_element(node: ValidatedNode) -> SemanticElement:.
• Banned loose types: dict[str, Any], list[Any], and object are forbidden, except inside the raw parser (input parsing boundary) or as explicitly documented and approved exceptions. Prefer dataclasses, TypedDict, or dedicated domain models.
• Phase-specific typings: never reuse a type (RawNode, ParsedNode, ValidatedNode, SemanticElement, SheetSpec, RenderPlan, RendererConfig) across multiple pipeline phases.
• Identifier separation: do not mix or swap UUID, Name, Alias, and Internal ID; each identifier must have a distinct wrapper class or type definition.
• Test data management: use small, mock Logos models in tests/data/ for deterministic parser edge-case tests; do not rely on large files there. Use soter-workspace/ only for large-scale, end-to-end integration flows (multi-tenant setups, full event ledger replays).
• Definition of done: a task is not considered done if there are unresolved lints, failing unit tests, or Pyrefly check failures.

Examples

Stakeholder-driven E2E scenarios validate real-world workflows aligned with specific roles, preventing regression in user-facing components without the maintenance overhead of fragile view-level unit tests:

graph TD
    Tenant["1. Tenant Domain Authentication"] --> Project["2. Switch/Select Project"]
    Project --> Model["3. Edit/Upload Logos Model"]
    Model --> Compile["4. Compile & Generate Artifacts"]
    Compile --> Explore["5. Interactive Exploration"]

• Business Analyst (BA) & Domain Architect: log in under a tenant subdomain (e.g. acme.localhost:8000), select the active project, edit .model files under /input/models, trigger compilation, and verify that output artifacts (BPMN diagrams, D2 charts, event ledgers) are generated without errors and are explorable in the Web UI.
• Developer: edit SOTER logic or templates locally, run ruff and pyrefly, run the pytest suite, and query local JSON APIs (e.g. /api/v1/models, /api/workflows/boards) to confirm schema stability.
• DevOps Engineer: access SOTER via non-standard local ports or tenant subdomains to verify middleware routing, verify tenant database migrations (tenants.sqlite3) and .ledger audit trails sync correctly, and assert that workspace sandbox file permissions and diagnostic logs (/log or 2_output/6_logs) are correct.
• Management & Auditor: browse the Ledger Scroll to inspect historical commit logs and model modifications, verify that every compiled element points to its origin file, git commit hash, and source line, and run cryptographic check subcommands (auditor / ledger check) to confirm the append-only audit trail has not been compromised.

Validation

Tooling Suite

• Linting & Formatting: ruff, pre-commit
• Static Verification: pyrefly
• Test Runners: pytest, pytest-mock, pytest-cov
• UI Testing: Playwright (Python wrapper)

Local Developer Workflow (Vibecoding)

To prevent compiling broken states, developers and AIs must execute tests iteratively:

[Generate Code] ──> [Run Local Quality Checks] ──> [Fix Issues] ──> [Verify Green] ──> [Commit]

Before attempting a Git commit, verify changes locally using:

ruff check .
pytest
pyrefly check

Pre-Commit Rules & CI/CD Pipeline

To protect the shared ledger from contamination, quality checks are enforced automatically:

1. Trim Trailing Whitespace / End of File Fixer: formats file syntax.
2. Commit Isolation Guard: checks for mixed branch tasks.
3. Pyrefly Guard Phases:
4. Phase 1 (Soft): the Pyrefly hook warns but does not block.
5. Phase 2 (Hard): zero Pyrefly type warnings allowed; commit blocks on any failure.
6. CI/CD Build: executes the complete verification sweep (ruff check ., pytest, pyrefly check) on every Pull Request.

Pyrefly Codex — Validation Error Codes

This table is the cheat-sheet reference for Pyrefly compilation and lint validation error codes.

Compatibility

The Pyrefly Guard rolls out in two enforcement phases: Phase 1 (Soft) warns without blocking commits, and Phase 2 (Hard) blocks on any type warning. The Pyrefly Codex is a living document and will be expanded as new validation rules are implemented into the Pyrefly engine; new codes must be added to the table without removing or renumbering existing ones.

Open Questions

• No open questions are currently tracked for this specification.

Related Documents

• DocumentationStyleGuide.md - writing profile and document-type rules.
• TechnicalSpecTemplate.md - technical document template.
• ArchitectureOverview.md - technical documentation entry point.
• TechnicalArchitectureSpec.md - implementation architecture referencing this spec.
• PyreflyLinterSetup.md - Pyrefly linter installation and setup guide.
• LogosOverview.md - language and model semantics.