busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
604b3dc706007acba84bff64d2e80a60835d5a99
ss-tools/.kilo/command/speckit.test.md
busya 39aa4a7e0c chore(kilo): consolidate agent skills, commands and workflows 
- remove legacy .ai/ knowledge shots and reports, semantic skills
  invariant assessment, and obsolete .kilo/workflows/
- update agent model selection (omniroute/terra) for qa-tester,
  security-auditor, svelte-coder
- add swarm-master agent and speckit openapi/prototype/resume/validate
  plus test.* commands; update speckit plan/ux/implement docs
- refresh skill SKILL.md files (semantics core/testing/svelte/belief,
  molecular-cot-logging, semantic-frontend)
- add semantic curation report
2026-08-02 23:54:30 +07:00

367 lines
17 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
description: Execute mocking discipline audit, semantic verification, and native testing for the active superset-tools feature batch (pytest + vitest). Read-only audit first, then auto-fix violations.
handoffs:
- label: Orchestration Control
agent: swarm-master
prompt: Review tester feedback and coordinate next steps.
send: true
---
## User Input
```text
$ARGUMENTS
```
You **MUST** consider the user input before proceeding (if not empty). User may specify a subset of files or a specific scope override.
## Goal
Run the full verification loop for the touched superset-tools scope:
1. **Mocking discipline audit** — scan every test file in scope, classify every mock/spy/stub/patch, flag violations
2. **Auto-fix violations** — correct SUT mocks and Logic Mirrors (no flag needed; fix by default)
3. **Semantic audit** — contract density, belief runtime, rejected-path regression
4. **Executable tests** — run pytest + vitest + lint via `make test` (tiered, timeout-protected)
5. **Documentation** — mock audit report + coverage summary + ADR guardrail status
**When to use `/speckit.test` vs `/test.*`:** Use this command for COMPREHENSIVE audit (mocking + semantic + execution) on a feature batch. For quick verify loops during development (just run tests, no audit), use the lightweight alternatives:
- `/test.unit` — fast unit tests (backend + frontend, <30s)
- `/test.related` — only tests linked to a changed file via `@RELATION BINDS_TO`
- `/test.coverage` — coverage reports only
- `/test.all` — full suite + coverage
## Operating Constraints
### Golden Rules (from `semantics-testing` skill)
1. **Mock only `[EXT:...]`** — external boundaries (DB drivers, HTTP clients, file I/O, third-party APIs).
2. **NEVER mock the SUT** — the production `#region` contract you are actively verifying.
3. **Anti-Tautology (Logic Mirror) is forbidden** — never compute `expected_result` by repeating the production algorithm inside the test.
4. **Global DOM mocks are infrastructure, not logic** — `ResizeObserver`, `scrollTo`, `IntersectionObserver` in `vitest.setup.ts` or `setupTests.ts` are **not violations**.
### Additional Constraints
5. **NEVER delete existing tests** unless the user explicitly requests removal.
6. **NEVER duplicate tests** when existing test coverage already validates the same contract.
7. **Decision-memory regression guard**: tests and audits must not silently normalize any path documented as rejected (`@REJECTED`, ADR guardrails).
8. **Project-native structure**: prefer existing test organization — `backend/tests/` for Python, `frontend/src/lib/**/__tests__/` for Svelte.
## Mandatory Skills
Before scanning any test file, load:
- `skill({name="semantics-testing"})`
- `skill({name="semantics-core"})`
- `skill({name="semantics-contracts"})`
- `skill({name="semantics-python"})` (for backend tests)
- `skill({name="semantics-svelte"})` (for frontend tests)
---
## Execution Steps
### 1. Analyze Context
Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --include-tasks` and determine:
- `FEATURE_DIR`
- touched implementation tasks from `tasks.md`
- affected `.py` and `.svelte` files
- relevant ADRs, `@RATIONALE`, and `@REJECTED` guardrails
All test documentation emitted by this workflow belongs under `FEATURE_DIR/tests/` or other files inside `specs/<feature>/...`, never under `.kilo/plans/`.
**Scope discovery**: If the user provided `$ARGUMENTS` specifying files or directories, narrow the audit scope accordingly. Otherwise, derive scope from the active feature's touched files.
### 2. Load Relevant Artifacts
Load only the necessary portions of:
- `tasks.md`
- `plan.md`
- `contracts/modules.md` when present
- `quickstart.md` when present
- `.specify/memory/constitution.md`
- `README.md`
- relevant `docs/adr/*.md`
### 3. Mocking Discipline Audit (NEW — Primary Step)
**This is a systematic, read-only scan of every test file in scope.** The audit classifies every mock, spy, stub, patch, and fake against the golden rules.
#### 3a. Discover Test Files
For the scoped feature (or user-specified scope), discover:
| Layer | Patterns |
|-------|----------|
| Backend unit | `backend/tests/**/*.py` |
| Backend integration | `backend/tests/integration/**/*.py` |
| Frontend unit | `frontend/src/**/*.test.ts`, `frontend/src/**/__tests__/*.ts` |
| Frontend integration | `frontend/src/**/*.integration.test.ts` |
| Frontend UX | `frontend/src/**/*.ux.test.ts` |
| Frontend component | `frontend/src/**/__tests__/*.svelte.js` |
#### 3b. Extract Per-Test Metadata
For each test file:
- Which production `#region` contracts it references — look for `@RELATION BINDS_TO`, `@TEST_INVARIANT`, or import paths to production modules. Discover contract IDs via `axiom_semantic_discovery read_outline` on production files.
- All mock/patch/stub/spy declarations (`unittest.mock.patch`, `unittest.mock.MagicMock`, `pytest.monkeypatch`, `vi.mock`, `vi.fn`, `vi.spyOn`, `mockResolvedValue`, etc.)
- Whether the file is a **global setup** file (`conftest.py`, `vitest.setup.ts`, `setupTests.ts`)
#### 3c. Classify Every Mock
Apply this classification table **to every mock found**:
| Mock target | Verdict | Rule |
|------------|---------|------|
| `[EXT:Database]`, `[EXT:HTTP]`, `[EXT:File]`, `[EXT:ThirdParty]` | ✅ VALID | External boundary — allowed |
| `localStorage`, `fetch`, `fs.readFileSync`, `os.environ` | ✅ VALID | External API / I/O — allowed |
| `Date.now`, `Math.random`, `uuid.v4` | ✅ VALID | Non-deterministic input — allowed |
| `ResizeObserver`, `IntersectionObserver`, `scrollTo`, `matchMedia` in global setup | ✅ VALID | DOM infrastructure — allowed |
| `ResizeObserver`, `IntersectionObserver` in individual test file (not setup) | ✅ VALID | DOM environment polyfill — allowed |
| `AuthService` (the `#region` production contract under test) | ❌ VIOLATION | Mocking SUT — forbidden |
| `GitPlugin` (the `#region` production contract under test) | ❌ VIOLATION | Mocking SUT — forbidden |
| `MigrationEngine` (the `#region` production contract under test) | ❌ VIOLATION | Mocking SUT — forbidden |
| Database session/repo when it IS the integration boundary under test | ❌ VIOLATION | Mocking SUT in integration test |
| Test computes `expected = a + b` to test `add(a, b)` | ❌ VIOLATION | Logic Mirror — tautology |
| Test computes `expected = production_fn(x)` to test `production_fn` | ❌ VIOLATION | Logic Mirror — tautology |
| Something unclear, ambiguous ownership | ⚠️ UNCERTAIN | Flag for human review |
**Do NOT flag as violations**:
- `@vi.fn` or `vi.spyOn` on callback handlers that are NOT the SUT
- Mocks in `conftest.py`, `vitest.setup.ts`, `setupTests.ts` that provide shared test infrastructure (DB stubs, browser API stubs, auth fixtures)
- `MagicMock` / `AsyncMock` used as placeholder arguments that are NOT the SUT
- `monkeypatch.setenv` for environment configuration (infrastructure, not logic)
#### 3d. Integration Test Special Handling
Integration tests have **different mock boundaries** than unit tests. Apply these additional rules:
| Pattern | Classification | Rationale |
|---------|---------------|-----------|
| `TestClient` (FastAPI) / `test_client` fixture | ✅ INFRASTRUCTURE | Test harness, not a mock |
| Real test database (SQLite `:memory:`, testcontainers PostgreSQL) | ✅ INFRASTRUCTURE | Real dependency for integration fidelity |
| `conftest.py` DB session fixtures | ✅ INFRASTRUCTURE | Shared test infrastructure |
| Mocking an **external HTTP API** (e.g., Superset API, Git service) in an integration test | ✅ VALID | External boundary — allowed |
| Mocking the **application's own router/endpoint** in an integration test | ❌ VIOLATION | Mocking SUT |
| Mocking the **database layer** in an integration test | ❌ VIOLATION | Defeats purpose of integration test |
| Full-stack test that mocks the **frontend API client** | ✅ VALID | External boundary from backend perspective |
| File I/O via `tmp_path` / `tmpdir` fixtures | ✅ INFRASTRUCTURE | Real filesystem, not a mock |
**Integration test file size limit**: Per `semantics-testing` skill §II.5, integration test files using Testcontainers may be up to **800 lines**. Flag files exceeding this as `⚠️ SIZE` with a recommendation to split.
#### 3e. Logic Mirror Detection
For each test assertion, check if the expected value is **computed algorithmically** by mirroring the production code:
**Python example violation:**
```
# Production: def add(a, b): return a + b
# Test VIOLATION: expected = a + b ← algorithmic mirror of production
```
**JavaScript example violation:**
```
// Production: export const formatDate = (d) => d.toISOString().split('T')[0]
// Test VIOLATION: expect(result).toBe(date.toISOString().split('T')[0]) ← mirror
```
Correct approach: use a **hardcoded fixture** value.
```
expected = 5 # hardcoded, not computed
expected = "2025-01-15" # hardcoded, not calling toISOString
```
### 4. Coverage Matrix
Build a compact matrix enriched by audit findings:
| Module / Flow | File | Existing Tests | Complexity | Mock Violations | Guardrails | Needed Verification |
|---------------|------|----------------|------------|-----------------|------------|---------------------|
### 5. Semantic Audit and Logic Review
Before executing tests, perform a semantic audit of the touched scope:
1. Reject malformed or pseudo-semantic markup.
2. Verify contract density matches effective complexity.
3. Verify C4/C5 Python flows account for belief runtime markers (`reason`, `reflect`, `explore` with JSON structured logging).
4. Verify C4/C5 Svelte components account for console markers (`[ComponentID][MARKER]`).
5. Verify no touched code silently restores an ADR- or contract-rejected path.
6. Emulate the algorithm mentally to ensure `@PRE`, `@POST`, `@INVARIANT`, and declared side effects remain coherent.
7. **Cross-reference with mock audit**: violations found in step 3 that intersect with semantic contracts must be prioritized.
If audit fails, emit `[AUDIT_FAIL: semantic_noncompliance | contract_mismatch | logic_mismatch | rejected_path_regression]` with concrete file-based reasons.
### 6. Fix Violations (Auto-Fix by Default)
**Every VIOLATION and Logic Mirror found in step 3 MUST be fixed.** No opt-in flag required — this is the default behavior.
#### Fixing SUT Mock Violations
- Replace the mock of the SUT with a **real instantiation** of the production contract
- If the SUT depends on `[EXT:...]` boundaries, mock ONLY those boundaries, not the SUT itself
- If instantiation is complex, extract the mocked logic to a separate `#region` contract and test that independently
#### Fixing Logic Mirror Violations
- Replace algorithmic expected-value computation with a **hardcoded fixture**
- Use `@TEST_FIXTURE` to document the fixture source
- If multiple scenarios need different values, use a parameterized table, not a loop that re-computes
#### Fixing Integration Test Violations
- If an integration test mocks the application's database layer, replace with a real test database (SQLite `:memory:` or testcontainers)
- If an integration test mocks the application's own router, rewrite as a true integration test using `TestClient`
#### Uncertain Cases
For `⚠️ UNCERTAIN` flags:
- Leave the mock in place
- Add a comment `# AUDIT_NOTE: [YYYY-MM-DD] Flagged as UNCERTAIN — [brief reason]. Review at next test cycle.`
- List in the report under "Uncertain — Requires Human Review"
### 7. Test Writing / Updating
When test additions are needed (beyond fixing violations):
- Python: prefer `backend/tests/test_*.py` with pytest
- Svelte: prefer `__tests__/*.test.ts` with vitest + @testing-library/svelte
- Use deterministic fixtures rather than logic mirrors (see Anti-Tautology rules)
- Trace tests back to semantic contracts (`@TEST_INVARIANT`) and ADR guardrails
- Add explicit rejected-path regression coverage when the touched scope has a forbidden alternative (`@REJECTED`)
- **For every C4/C5 flow**: include belief-runtime verification (assert `reason`/`reflect`/`explore` log events)
For non-UI backend features, UX verification means validating API envelopes, error responses, and recovery messaging promised by `ux_reference.md`.
For UI features, use browser validation via `chrome-devtools` MCP.
### 8. Execute Verifiers
Run the full verification stack for the touched scope. The project Makefile provides tiered targets with built-in timeout protection:
```bash
# Tier 1: Fast unit tests (no Docker, <120s timeout)
make test-unit # backend SQLite tests
make test-frontend # frontend vitest tests
# Tier 1 alt: Smart test selection (only tests related to changed files)
make test-related F=backend/src/path/to/changed_file.py
# Linting (ruff + eslint)
make lint
# Coverage (optional — run after tests pass)
make coverage
# Tier 2: Integration tests (requires Docker, <600s timeout)
# Only when the scope includes integration boundaries
make test-integration
# Full suite (unit + integration + coverage)
make test-all
```
**Timeout safety**: All `make test-*` targets are wrapped with `timeout N` shell guards. Unit tests have 120s; integration tests have 600s. The agent NEVER hangs on a hung test.
**Narrow-first principle**: Start with `make test-unit` for backend changes, `make test-frontend` for frontend changes. Use `make test-related F=<file>` to run only semantically-linked tests. Widen to `make test` (both layers) when finalizing.
**When to run integration tests**: Only when the scope includes files under `backend/tests/integration/` or when the change touches Docker/testcontainers fixtures. Otherwise, skip.
### 9. Test Documentation
Create or update `specs/<feature>/tests/` documentation using `.specify/templates/test-docs-template.md`.
Document:
- **Mocking audit report** (see Output format below)
- Coverage summary
- Semantic audit verdict
- Commands run
- Failing or waived cases
- Decision-memory regression coverage
- Integration test boundaries verified
### 10. Update Tasks
Mark test tasks complete only after:
- Mocking audit is clean (0 remaining VIOLATIONS; UNCERTAIN items documented)
- Semantic audit passes
- All verifiers pass (pytest + vitest + lint + build)
---
## Integration Test Boundaries (Reference)
### What Integration Tests SHOULD Use (Real)
| Layer | Real Infrastructure |
|-------|--------------------|
| Database | SQLite `:memory:`, testcontainers PostgreSQL, or dedicated test DB |
| Application Router | `TestClient` (FastAPI), real SvelteKit `app.render()` |
| File System | `tmp_path` / `tmpdir` fixtures (pytest), real temp directories |
| Environment | `monkeypatch.setenv` (infrastructure), `.env.test` files |
| Auth Tokens | Real JWT generation with test secret, or `TestClient` auth headers |
### What Integration Tests SHOULD Mock (External)
| Layer | Mock Strategy |
|-------|--------------|
| External HTTP APIs | `responses`, `httpx.MockTransport`, `vi.mock('./api')` |
| Third-party services (Superset, Git service, LLM providers) | `MagicMock` / `vi.fn` for the client wrapper |
| WebSocket servers (external) | Mock the connection, not the app's WS handler |
| Email / notification services | Mock the transport layer |
### File Size Limit
- **600 lines** for unit test files
- **800 lines** for integration test files (due to longer setup/teardown)
- Files exceeding these limits SHOULD be split by domain or test class
---
## Output
Produce a single Markdown test report containing all of the following sections:
### 1. Mocking Audit Report
```markdown
## Mocking Audit Report
### Summary
| Total tests scanned | Total mocks found | Valid mocks | Violations | Logic Mirrors | Uncertain |
|---------------------|-------------------|-------------|------------|---------------|-----------|
| N | N | N | N | N | N |
### Violations
| File | Line | Contract under test | Mock target | Why it's wrong | Fix applied |
|------|------|---------------------|-------------|----------------|-------------|
| ... | ... | ... | ... | ... | ... |
### Logic Mirrors
| File | Production code | Test code | Hardcoded fixture applied |
|------|-----------------|-----------|---------------------------|
| ... | ... | ... | ... |
### Integration Test Boundaries
| File | Type | Real deps | Mocked deps | Verdict |
|------|------|-----------|-------------|---------|
| ... | integration | DB, Router | External API | ✅ CLEAN |
### Clean tests (no violations)
- [list of files that are fully compliant]
### Global setup (not violations)
- [list of infrastructure mocks in conftest.py, setupTests.ts, vitest.setup.ts]
### Uncertain (requires human review)
| File | Line | Mock target | Why uncertain |
|------|------|-------------|---------------|
| ... | ... | ... | ... |
```
### 2. Coverage Summary
- Commands executed: `make coverage` (backend pytest-cov + frontend vitest v8)
- Pass/fail counts per layer
- Coverage percentage: backend statement/line %, frontend statement/line/function/branch % with threshold comparison
### 3. Semantic Audit Verdict
- Contract density check results
- Belief runtime instrumentation status (C4/C5 flows)
- ADR / rejected-path coverage status
### 4. Issues Found and Resolutions
- All violations found and how they were fixed
- Any remaining technical debt
### 5. Remaining Risk or Debt
- UNCERTAIN items pending human review
- Files flagged for size split
- Known coverage gaps
Reference in New Issue View Git Blame Copy Permalink