ss-tools/.opencode/command/speckit.test.md

description, handoffs

description	handoffs
Execute mocking discipline audit, semantic verification, and native testing for the active superset-tools feature batch (pytest + vitest). Read-only audit first, then auto-fix violations.	label agent prompt send Orchestration Control swarm-master Review tester feedback and coordinate next steps. true

User Input

$ARGUMENTS

You MUST consider the user input before proceeding (if not empty). User may specify a subset of files or a specific scope override.

Goal

Run the full verification loop for the touched superset-tools scope:

1. Mocking discipline audit — scan every test file in scope, classify every mock/spy/stub/patch, flag violations
2. Auto-fix violations — correct SUT mocks and Logic Mirrors (no flag needed; fix by default)
3. Semantic audit — contract density, belief runtime, rejected-path regression
4. Executable tests — run pytest + vitest + lint
5. Documentation — mock audit report + coverage summary + ADR guardrail status

Operating Constraints

Golden Rules (from semantics-testing skill)

1. Mock only [EXT:...] — external boundaries (DB drivers, HTTP clients, file I/O, third-party APIs).
2. NEVER mock the SUT — the production #region contract you are actively verifying.
3. Anti-Tautology (Logic Mirror) is forbidden — never compute expected_result by repeating the production algorithm inside the test.
4. Global DOM mocks are infrastructure, not logic — ResizeObserver, scrollTo, IntersectionObserver in vitest.setup.ts or setupTests.ts are not violations.

Additional Constraints

5. NEVER delete existing tests unless the user explicitly requests removal.
6. NEVER duplicate tests when existing test coverage already validates the same contract.
7. Decision-memory regression guard: tests and audits must not silently normalize any path documented as rejected (@REJECTED, ADR guardrails).
8. Project-native structure: prefer existing test organization — backend/tests/ for Python, frontend/src/lib/**/__tests__/ for Svelte.

Mandatory Skills

Before scanning any test file, load:

• skill({name="semantics-testing"})
• skill({name="semantics-core"})
• skill({name="semantics-contracts"})
• skill({name="semantics-python"}) (for backend tests)
• skill({name="semantics-svelte"}) (for frontend tests)

---

Execution Steps

1. Analyze Context

Run .specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks and determine:

• FEATURE_DIR
• touched implementation tasks from tasks.md
• affected .py and .svelte files
• relevant ADRs, @RATIONALE, and @REJECTED guardrails

All test documentation emitted by this workflow belongs under FEATURE_DIR/tests/ or other files inside specs/<feature>/..., never under .kilo/plans/.

Scope discovery: If the user provided $ARGUMENTS specifying files or directories, narrow the audit scope accordingly. Otherwise, derive scope from the active feature's touched files.

2. Load Relevant Artifacts

Load only the necessary portions of:

• tasks.md
• plan.md
• contracts/modules.md when present
• quickstart.md when present
• .specify/memory/constitution.md
• README.md
• relevant docs/adr/*.md

3. Mocking Discipline Audit (NEW — Primary Step)

This is a systematic, read-only scan of every test file in scope. The audit classifies every mock, spy, stub, patch, and fake against the golden rules.

3a. Discover Test Files

For the scoped feature (or user-specified scope), discover:

Layer	Patterns
Backend unit	backend/tests/**/*.py
Backend integration	backend/tests/integration/**/*.py
Frontend unit	frontend/src/**/*.test.ts, frontend/src/**/__tests__/*.ts
Frontend integration	frontend/src/**/*.integration.test.ts
Frontend UX	frontend/src/**/*.ux.test.ts
Frontend component	frontend/src/**/__tests__/*.svelte.js

3b. Extract Per-Test Metadata

For each test file:

• Which production #region contracts it references — look for @RELATION BINDS_TO, @TEST_INVARIANT, or import paths to production modules. Discover contract IDs via axiom_semantic_discovery read_outline on production files.
• All mock/patch/stub/spy declarations (unittest.mock.patch, unittest.mock.MagicMock, pytest.monkeypatch, vi.mock, vi.fn, vi.spyOn, mockResolvedValue, etc.)
• Whether the file is a global setup file (conftest.py, vitest.setup.ts, setupTests.ts)

3c. Classify Every Mock

Apply this classification table to every mock found:

Mock target	Verdict	Rule
[EXT:Database], [EXT:HTTP], [EXT:File], [EXT:ThirdParty]	✅ VALID	External boundary — allowed
localStorage, fetch, fs.readFileSync, os.environ	✅ VALID	External API / I/O — allowed
Date.now, Math.random, uuid.v4	✅ VALID	Non-deterministic input — allowed
ResizeObserver, IntersectionObserver, scrollTo, matchMedia in global setup	✅ VALID	DOM infrastructure — allowed
ResizeObserver, IntersectionObserver in individual test file (not setup)	✅ VALID	DOM environment polyfill — allowed
AuthService (the #region production contract under test)	❌ VIOLATION	Mocking SUT — forbidden
GitPlugin (the #region production contract under test)	❌ VIOLATION	Mocking SUT — forbidden
MigrationEngine (the #region production contract under test)	❌ VIOLATION	Mocking SUT — forbidden
Database session/repo when it IS the integration boundary under test	❌ VIOLATION	Mocking SUT in integration test
Test computes expected = a + b to test add(a, b)	❌ VIOLATION	Logic Mirror — tautology
Test computes expected = production_fn(x) to test production_fn	❌ VIOLATION	Logic Mirror — tautology
Something unclear, ambiguous ownership	⚠️ UNCERTAIN	Flag for human review

Do NOT flag as violations:

• @vi.fn or vi.spyOn on callback handlers that are NOT the SUT
• Mocks in conftest.py, vitest.setup.ts, setupTests.ts that provide shared test infrastructure (DB stubs, browser API stubs, auth fixtures)
• MagicMock / AsyncMock used as placeholder arguments that are NOT the SUT
• monkeypatch.setenv for environment configuration (infrastructure, not logic)

3d. Integration Test Special Handling

Integration tests have different mock boundaries than unit tests. Apply these additional rules:

Pattern	Classification	Rationale
TestClient (FastAPI) / test_client fixture	✅ INFRASTRUCTURE	Test harness, not a mock
Real test database (SQLite :memory:, testcontainers PostgreSQL)	✅ INFRASTRUCTURE	Real dependency for integration fidelity
conftest.py DB session fixtures	✅ INFRASTRUCTURE	Shared test infrastructure
Mocking an external HTTP API (e.g., Superset API, Git service) in an integration test	✅ VALID	External boundary — allowed
Mocking the application's own router/endpoint in an integration test	❌ VIOLATION	Mocking SUT
Mocking the database layer in an integration test	❌ VIOLATION	Defeats purpose of integration test
Full-stack test that mocks the frontend API client	✅ VALID	External boundary from backend perspective
File I/O via tmp_path / tmpdir fixtures	✅ INFRASTRUCTURE	Real filesystem, not a mock

Integration test file size limit: Per semantics-testing skill §II.5, integration test files using Testcontainers may be up to 800 lines. Flag files exceeding this as ⚠️ SIZE with a recommendation to split.

3e. Logic Mirror Detection

For each test assertion, check if the expected value is computed algorithmically by mirroring the production code:

Python example violation:

# Production: def add(a, b): return a + b
# Test VIOLATION: expected = a + b   ← algorithmic mirror of production

JavaScript example violation:

// Production: export const formatDate = (d) => d.toISOString().split('T')[0]
// Test VIOLATION: expect(result).toBe(date.toISOString().split('T')[0])  ← mirror

Correct approach: use a hardcoded fixture value.

expected = 5              # hardcoded, not computed
expected = "2025-01-15"   # hardcoded, not calling toISOString

4. Coverage Matrix

Build a compact matrix enriched by audit findings:

Module / Flow	File	Existing Tests	Complexity	Mock Violations	Guardrails	Needed Verification

5. Semantic Audit and Logic Review

Before executing tests, perform a semantic audit of the touched scope:

1. Reject malformed or pseudo-semantic markup.
2. Verify contract density matches effective complexity.
3. Verify C4/C5 Python flows account for belief runtime markers (reason, reflect, explore with JSON structured logging).
4. Verify C4/C5 Svelte components account for console markers ([ComponentID][MARKER]).
5. Verify no touched code silently restores an ADR- or contract-rejected path.
6. Emulate the algorithm mentally to ensure @PRE, @POST, @INVARIANT, and declared side effects remain coherent.
7. Cross-reference with mock audit: violations found in step 3 that intersect with semantic contracts must be prioritized.

If audit fails, emit [AUDIT_FAIL: semantic_noncompliance | contract_mismatch | logic_mismatch | rejected_path_regression] with concrete file-based reasons.

6. Fix Violations (Auto-Fix by Default)

Every VIOLATION and Logic Mirror found in step 3 MUST be fixed. No opt-in flag required — this is the default behavior.

Fixing SUT Mock Violations

• Replace the mock of the SUT with a real instantiation of the production contract
• If the SUT depends on [EXT:...] boundaries, mock ONLY those boundaries, not the SUT itself
• If instantiation is complex, extract the mocked logic to a separate #region contract and test that independently

Fixing Logic Mirror Violations

• Replace algorithmic expected-value computation with a hardcoded fixture
• Use @TEST_FIXTURE to document the fixture source
• If multiple scenarios need different values, use a parameterized table, not a loop that re-computes

Fixing Integration Test Violations

• If an integration test mocks the application's database layer, replace with a real test database (SQLite :memory: or testcontainers)
• If an integration test mocks the application's own router, rewrite as a true integration test using TestClient

Uncertain Cases

For ⚠️ UNCERTAIN flags:

• Leave the mock in place
• Add a comment # AUDIT_NOTE: [YYYY-MM-DD] Flagged as UNCERTAIN — [brief reason]. Review at next test cycle.
• List in the report under "Uncertain — Requires Human Review"

7. Test Writing / Updating

When test additions are needed (beyond fixing violations):

• Python: prefer backend/tests/test_*.py with pytest
• Svelte: prefer __tests__/*.test.ts with vitest + @testing-library/svelte
• Use deterministic fixtures rather than logic mirrors (see Anti-Tautology rules)
• Trace tests back to semantic contracts (@TEST_INVARIANT) and ADR guardrails
• Add explicit rejected-path regression coverage when the touched scope has a forbidden alternative (@REJECTED)
• For every C4/C5 flow: include belief-runtime verification (assert reason/reflect/explore log events)

For non-UI backend features, UX verification means validating API envelopes, error responses, and recovery messaging promised by ux_reference.md. For UI features, use browser validation via chrome-devtools MCP.

8. Execute Verifiers

Run the full verification stack for the touched scope:

# Backend
cd backend && source .venv/bin/activate && python -m pytest -v
python -m ruff check backend/src/ backend/tests/

# Frontend
cd frontend && npm run test
npm run lint
npm run build

Use narrower test runs when sufficient, then widen verification when finalizing.

9. Test Documentation

Create or update specs/<feature>/tests/ documentation using .specify/templates/test-docs-template.md.

Document:

• Mocking audit report (see Output format below)
• Coverage summary
• Semantic audit verdict
• Commands run
• Failing or waived cases
• Decision-memory regression coverage
• Integration test boundaries verified

10. Update Tasks

Mark test tasks complete only after:

• Mocking audit is clean (0 remaining VIOLATIONS; UNCERTAIN items documented)
• Semantic audit passes
• All verifiers pass (pytest + vitest + lint + build)

---

Integration Test Boundaries (Reference)

What Integration Tests SHOULD Use (Real)

Layer	Real Infrastructure
Database	SQLite :memory:, testcontainers PostgreSQL, or dedicated test DB
Application Router	TestClient (FastAPI), real SvelteKit app.render()
File System	tmp_path / tmpdir fixtures (pytest), real temp directories
Environment	monkeypatch.setenv (infrastructure), .env.test files
Auth Tokens	Real JWT generation with test secret, or TestClient auth headers

What Integration Tests SHOULD Mock (External)

Layer	Mock Strategy
External HTTP APIs	responses, httpx.MockTransport, vi.mock('./api')
Third-party services (Superset, Git service, LLM providers)	MagicMock / vi.fn for the client wrapper
WebSocket servers (external)	Mock the connection, not the app's WS handler
Email / notification services	Mock the transport layer

File Size Limit

• 600 lines for unit test files
• 800 lines for integration test files (due to longer setup/teardown)
• Files exceeding these limits SHOULD be split by domain or test class

---

Output

Produce a single Markdown test report containing all of the following sections:

1. Mocking Audit Report

## Mocking Audit Report

### Summary
| Total tests scanned | Total mocks found | Valid mocks | Violations | Logic Mirrors | Uncertain |
|---------------------|-------------------|-------------|------------|---------------|-----------|
| N                   | N                 | N           | N          | N             | N         |

### Violations
| File | Line | Contract under test | Mock target | Why it's wrong | Fix applied |
|------|------|---------------------|-------------|----------------|-------------|
| ... | ... | ... | ... | ... | ... |

### Logic Mirrors
| File | Production code | Test code | Hardcoded fixture applied |
|------|-----------------|-----------|---------------------------|
| ... | ... | ... | ... |

### Integration Test Boundaries
| File | Type | Real deps | Mocked deps | Verdict |
|------|------|-----------|-------------|---------|
| ... | integration | DB, Router | External API | ✅ CLEAN |

### Clean tests (no violations)
- [list of files that are fully compliant]

### Global setup (not violations)
- [list of infrastructure mocks in conftest.py, setupTests.ts, vitest.setup.ts]

### Uncertain (requires human review)
| File | Line | Mock target | Why uncertain |
|------|------|-------------|---------------|
| ... | ... | ... | ... |

2. Coverage Summary

• Commands executed
• Pass/fail counts per layer
• Coverage percentage (if available)

3. Semantic Audit Verdict

• Contract density check results
• Belief runtime instrumentation status (C4/C5 flows)
• ADR / rejected-path coverage status

4. Issues Found and Resolutions

• All violations found and how they were fixed
• Any remaining technical debt

5. Remaining Risk or Debt

• UNCERTAIN items pending human review
• Files flagged for size split
• Known coverage gaps