busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(agent): extend coverage — agent handler, confirmations, conversation API, tools

Browse Source 
- test_agent_handler: additional edge cases for streaming, HITL, file upload
- test_confirmations: HITL confirm/deny lifecycle coverage
- test_conversation_api: conversation save/load persistence tests
- test_langchain_tools: tool registration, dual-auth header propagation
- ConversationList.test.ts (frontend): conversation list component tests
- conftest: shared fixtures for agent tests
- task_manager/manager: minor fixes from test coverage
- tasks.md/test-documentation.md: spec and test documentation updates
- speckit.test.md: speckit workflow documentation update
This commit is contained in:
busya
2026-06-10 16:38:06 +03:00
parent 1fd5e55db7
commit 7760214170
12 changed files with 1385 additions and 138 deletions
Download Patch File Download Diff File Expand all files Collapse all files

299
.opencode/command/speckit.test.md
View File

@@ -1,5 +1,10 @@
---
description: Execute semantic audit and native testing for the active ss-tools feature batch (pytest + vitest).
description: Execute mocking discipline audit, semantic verification, and native testing for the active ss-tools feature batch (pytest + vitest). Read-only audit first, then auto-fix violations.
handoffs:
- label: Orchestration Control
agent: swarm-master
prompt: Review tester feedback and coordinate next steps.
send: true
---
## User Input
@@ -8,18 +13,41 @@ description: Execute semantic audit and native testing for the active ss-tools f
$ARGUMENTS
```
You **MUST** consider the user input before proceeding (if not empty).
You **MUST** consider the user input before proceeding (if not empty). User may specify a subset of files or a specific scope override.
## Goal
Run the verification loop for the touched ss-tools scope: semantic audit, decision-memory audit, executable tests, logic review, and documentation of coverage/results.
Run the full verification loop for the touched ss-tools scope:
1. **Mocking discipline audit** — scan every test file in scope, classify every mock/spy/stub/patch, flag violations
2. **Auto-fix violations** — correct SUT mocks and Logic Mirrors (no flag needed; fix by default)
3. **Semantic audit** — contract density, belief runtime, rejected-path regression
4. **Executable tests** — run pytest + vitest + lint
5. **Documentation** — mock audit report + coverage summary + ADR guardrail status
## Operating Constraints
1. **NEVER delete existing tests** unless the user explicitly requests removal.
2. **NEVER duplicate tests** when existing test coverage already validates the same contract.
3. **Decision-memory regression guard**: tests and audits must not silently normalize any path documented as rejected.
4. **Project-native structure**: prefer existing test organization — `backend/tests/` for Python, `frontend/src/lib/**/__tests__/` for Svelte.
### Golden Rules (from `semantics-testing` skill)
1. **Mock only `[EXT:...]`** — external boundaries (DB drivers, HTTP clients, file I/O, third-party APIs).
2. **NEVER mock the SUT** — the production `#region` contract you are actively verifying.
3. **Anti-Tautology (Logic Mirror) is forbidden** — never compute `expected_result` by repeating the production algorithm inside the test.
4. **Global DOM mocks are infrastructure, not logic** — `ResizeObserver`, `scrollTo`, `IntersectionObserver` in `vitest.setup.ts` or `setupTests.ts` are **not violations**.
### Additional Constraints
5. **NEVER delete existing tests** unless the user explicitly requests removal.
6. **NEVER duplicate tests** when existing test coverage already validates the same contract.
7. **Decision-memory regression guard**: tests and audits must not silently normalize any path documented as rejected (`@REJECTED`, ADR guardrails).
8. **Project-native structure**: prefer existing test organization — `backend/tests/` for Python, `frontend/src/lib/**/__tests__/` for Svelte.
## Mandatory Skills
Before scanning any test file, load:
- `skill({name="semantics-testing"})`
- `skill({name="semantics-core"})`
- `skill({name="semantics-contracts"})`
- `skill({name="semantics-python"})` (for backend tests)
- `skill({name="semantics-svelte"})` (for frontend tests)
---
## Execution Steps
@@ -33,6 +61,8 @@ Run `.specify/scripts/bash/check-prerequisites.sh --json --require-tasks --inclu
All test documentation emitted by this workflow belongs under `FEATURE_DIR/tests/` or other files inside `specs/<feature>/...`, never under `.kilo/plans/`.
**Scope discovery**: If the user provided `$ARGUMENTS` specifying files or directories, narrow the audit scope accordingly. Otherwise, derive scope from the active feature's touched files.
### 2. Load Relevant Artifacts
Load only the necessary portions of:
@@ -44,75 +74,272 @@ Load only the necessary portions of:
- `README.md`
- relevant `docs/adr/*.md`
### 3. Coverage Matrix
### 3. Mocking Discipline Audit (NEW — Primary Step)
Build a compact matrix:
**This is a systematic, read-only scan of every test file in scope.** The audit classifies every mock, spy, stub, patch, and fake against the golden rules.
| Module / Flow | File | Existing Tests | Complexity | Guardrails | Needed Verification |
|---------------|------|----------------|------------|------------|---------------------|
#### 3a. Discover Test Files
### 4. Semantic Audit and Logic Review
For the scoped feature (or user-specified scope), discover:
Before writing or executing tests, perform a semantic audit of the touched scope:
| Layer | Patterns |
|-------|----------|
| Backend unit | `backend/tests/**/*.py` |
| Backend integration | `backend/tests/integration/**/*.py` |
| Frontend unit | `frontend/src/**/*.test.ts`, `frontend/src/**/__tests__/*.ts` |
| Frontend integration | `frontend/src/**/*.integration.test.ts` |
| Frontend UX | `frontend/src/**/*.ux.test.ts` |
| Frontend component | `frontend/src/**/__tests__/*.svelte.js` |
#### 3b. Extract Per-Test Metadata
For each test file:
- Which production `#region` contracts it references — look for `@RELATION BINDS_TO`, `@TEST_INVARIANT`, or import paths to production modules. Discover contract IDs via `axiom_semantic_discovery read_outline` on production files.
- All mock/patch/stub/spy declarations (`unittest.mock.patch`, `unittest.mock.MagicMock`, `pytest.monkeypatch`, `vi.mock`, `vi.fn`, `vi.spyOn`, `mockResolvedValue`, etc.)
- Whether the file is a **global setup** file (`conftest.py`, `vitest.setup.ts`, `setupTests.ts`)
#### 3c. Classify Every Mock
Apply this classification table **to every mock found**:
| Mock target | Verdict | Rule |
|------------|---------|------|
| `[EXT:Database]`, `[EXT:HTTP]`, `[EXT:File]`, `[EXT:ThirdParty]` | ✅ VALID | External boundary — allowed |
| `localStorage`, `fetch`, `fs.readFileSync`, `os.environ` | ✅ VALID | External API / I/O — allowed |
| `Date.now`, `Math.random`, `uuid.v4` | ✅ VALID | Non-deterministic input — allowed |
| `ResizeObserver`, `IntersectionObserver`, `scrollTo`, `matchMedia` in global setup | ✅ VALID | DOM infrastructure — allowed |
| `ResizeObserver`, `IntersectionObserver` in individual test file (not setup) | ✅ VALID | DOM environment polyfill — allowed |
| `AuthService` (the `#region` production contract under test) | ❌ VIOLATION | Mocking SUT — forbidden |
| `GitPlugin` (the `#region` production contract under test) | ❌ VIOLATION | Mocking SUT — forbidden |
| `MigrationEngine` (the `#region` production contract under test) | ❌ VIOLATION | Mocking SUT — forbidden |
| Database session/repo when it IS the integration boundary under test | ❌ VIOLATION | Mocking SUT in integration test |
| Test computes `expected = a + b` to test `add(a, b)` | ❌ VIOLATION | Logic Mirror — tautology |
| Test computes `expected = production_fn(x)` to test `production_fn` | ❌ VIOLATION | Logic Mirror — tautology |
| Something unclear, ambiguous ownership | ⚠️ UNCERTAIN | Flag for human review |
**Do NOT flag as violations**:
- `@vi.fn` or `vi.spyOn` on callback handlers that are NOT the SUT
- Mocks in `conftest.py`, `vitest.setup.ts`, `setupTests.ts` that provide shared test infrastructure (DB stubs, browser API stubs, auth fixtures)
- `MagicMock` / `AsyncMock` used as placeholder arguments that are NOT the SUT
- `monkeypatch.setenv` for environment configuration (infrastructure, not logic)
#### 3d. Integration Test Special Handling
Integration tests have **different mock boundaries** than unit tests. Apply these additional rules:
| Pattern | Classification | Rationale |
|---------|---------------|-----------|
| `TestClient` (FastAPI) / `test_client` fixture | ✅ INFRASTRUCTURE | Test harness, not a mock |
| Real test database (SQLite `:memory:`, testcontainers PostgreSQL) | ✅ INFRASTRUCTURE | Real dependency for integration fidelity |
| `conftest.py` DB session fixtures | ✅ INFRASTRUCTURE | Shared test infrastructure |
| Mocking an **external HTTP API** (e.g., Superset API, Git service) in an integration test | ✅ VALID | External boundary — allowed |
| Mocking the **application's own router/endpoint** in an integration test | ❌ VIOLATION | Mocking SUT |
| Mocking the **database layer** in an integration test | ❌ VIOLATION | Defeats purpose of integration test |
| Full-stack test that mocks the **frontend API client** | ✅ VALID | External boundary from backend perspective |
| File I/O via `tmp_path` / `tmpdir` fixtures | ✅ INFRASTRUCTURE | Real filesystem, not a mock |
**Integration test file size limit**: Per `semantics-testing` skill §II.5, integration test files using Testcontainers may be up to **800 lines**. Flag files exceeding this as `⚠️ SIZE` with a recommendation to split.
#### 3e. Logic Mirror Detection
For each test assertion, check if the expected value is **computed algorithmically** by mirroring the production code:
**Python example violation:**
```
# Production: def add(a, b): return a + b
# Test VIOLATION: expected = a + b ← algorithmic mirror of production
```
**JavaScript example violation:**
```
// Production: export const formatDate = (d) => d.toISOString().split('T')[0]
// Test VIOLATION: expect(result).toBe(date.toISOString().split('T')[0]) ← mirror
```
Correct approach: use a **hardcoded fixture** value.
```
expected = 5 # hardcoded, not computed
expected = "2025-01-15" # hardcoded, not calling toISOString
```
### 4. Coverage Matrix
Build a compact matrix enriched by audit findings:
| Module / Flow | File | Existing Tests | Complexity | Mock Violations | Guardrails | Needed Verification |
|---------------|------|----------------|------------|-----------------|------------|---------------------|
### 5. Semantic Audit and Logic Review
Before executing tests, perform a semantic audit of the touched scope:
1. Reject malformed or pseudo-semantic markup.
2. Verify contract density matches effective complexity.
3. Verify C4/C5 Python flows account for belief runtime markers (`reason`, `reflect`, `explore` with JSON structured logging).
4. Verify C4/C5 Svelte components account for console markers (`[ComponentID][MARKER]`).
5. Verify no touched code silently restores an ADR- or contract-rejected path.
6. Emulate the algorithm mentally to ensure `@PRE`, `@POST`, `@INVARIANT`, and declared side effects remain coherent.
7. **Cross-reference with mock audit**: violations found in step 3 that intersect with semantic contracts must be prioritized.
If audit fails, emit `[AUDIT_FAIL: semantic_noncompliance | contract_mismatch | logic_mismatch | rejected_path_regression]` with concrete file-based reasons.
### 5. Test Writing / Updating
### 6. Fix Violations (Auto-Fix by Default)
When test additions are needed:
**Every VIOLATION and Logic Mirror found in step 3 MUST be fixed.** No opt-in flag required — this is the default behavior.
#### Fixing SUT Mock Violations
- Replace the mock of the SUT with a **real instantiation** of the production contract
- If the SUT depends on `[EXT:...]` boundaries, mock ONLY those boundaries, not the SUT itself
- If instantiation is complex, extract the mocked logic to a separate `#region` contract and test that independently
#### Fixing Logic Mirror Violations
- Replace algorithmic expected-value computation with a **hardcoded fixture**
- Use `@TEST_FIXTURE` to document the fixture source
- If multiple scenarios need different values, use a parameterized table, not a loop that re-computes
#### Fixing Integration Test Violations
- If an integration test mocks the application's database layer, replace with a real test database (SQLite `:memory:` or testcontainers)
- If an integration test mocks the application's own router, rewrite as a true integration test using `TestClient`
#### Uncertain Cases
For `⚠️ UNCERTAIN` flags:
- Leave the mock in place
- Add a comment `# AUDIT_NOTE: [YYYY-MM-DD] Flagged as UNCERTAIN — [brief reason]. Review at next test cycle.`
- List in the report under "Uncertain — Requires Human Review"
### 7. Test Writing / Updating
When test additions are needed (beyond fixing violations):
- Python: prefer `backend/tests/test_*.py` with pytest
- Svelte: prefer `__tests__/*.test.js` with vitest + @testing-library/svelte
- use deterministic fixtures rather than logic mirrors
- trace tests back to semantic contracts and ADR guardrails
- add explicit rejected-path regression coverage when the touched scope has a forbidden alternative
- Svelte: prefer `__tests__/*.test.ts` with vitest + @testing-library/svelte
- Use deterministic fixtures rather than logic mirrors (see Anti-Tautology rules)
- Trace tests back to semantic contracts (`@TEST_INVARIANT`) and ADR guardrails
- Add explicit rejected-path regression coverage when the touched scope has a forbidden alternative (`@REJECTED`)
- **For every C4/C5 flow**: include belief-runtime verification (assert `reason`/`reflect`/`explore` log events)
For non-UI backend features, UX verification means validating API envelopes, error responses, and recovery messaging promised by `ux_reference.md`.
For UI features, use browser validation via `chrome-devtools` MCP.
### 6. Execute Verifiers
### 8. Execute Verifiers
Run the smallest truthful verifier set for the touched scope:
Run the full verification stack for the touched scope:
```bash
# Backend
cd backend && source .venv/bin/activate && python -m pytest -v
python -m ruff check backend/src/ backend/tests/
cd frontend && npm run lint
# Frontend
cd frontend && npm run test
npm run lint
npm run build
```
Use narrower test runs when sufficient, then widen verification when finalizing.
### 7. Test Documentation
### 9. Test Documentation
Create or update `specs/<feature>/tests/` documentation using `.specify/templates/test-docs-template.md`.
Document:
- coverage summary
- semantic audit verdict
- commands run
- failing or waived cases
- decision-memory regression coverage
- **Mocking audit report** (see Output format below)
- Coverage summary
- Semantic audit verdict
- Commands run
- Failing or waived cases
- Decision-memory regression coverage
- Integration test boundaries verified
### 8. Update Tasks
### 10. Update Tasks
Mark test tasks complete only after semantic audit and executable verification succeed.
Mark test tasks complete only after:
- Mocking audit is clean (0 remaining VIOLATIONS; UNCERTAIN items documented)
- Semantic audit passes
- All verifiers pass (pytest + vitest + lint + build)
---
## Integration Test Boundaries (Reference)
### What Integration Tests SHOULD Use (Real)
| Layer | Real Infrastructure |
|-------|--------------------|
| Database | SQLite `:memory:`, testcontainers PostgreSQL, or dedicated test DB |
| Application Router | `TestClient` (FastAPI), real SvelteKit `app.render()` |
| File System | `tmp_path` / `tmpdir` fixtures (pytest), real temp directories |
| Environment | `monkeypatch.setenv` (infrastructure), `.env.test` files |
| Auth Tokens | Real JWT generation with test secret, or `TestClient` auth headers |
### What Integration Tests SHOULD Mock (External)
| Layer | Mock Strategy |
|-------|--------------|
| External HTTP APIs | `responses`, `httpx.MockTransport`, `vi.mock('./api')` |
| Third-party services (Superset, Git service, LLM providers) | `MagicMock` / `vi.fn` for the client wrapper |
| WebSocket servers (external) | Mock the connection, not the app's WS handler |
| Email / notification services | Mock the transport layer |
### File Size Limit
- **600 lines** for unit test files
- **800 lines** for integration test files (due to longer setup/teardown)
- Files exceeding these limits SHOULD be split by domain or test class
---
## Output
Produce a Markdown test report containing:
- coverage summary
- commands executed
- semantic audit verdict
Produce a single Markdown test report containing all of the following sections:
### 1. Mocking Audit Report
```markdown
## Mocking Audit Report
### Summary
| Total tests scanned | Total mocks found | Valid mocks | Violations | Logic Mirrors | Uncertain |
|---------------------|-------------------|-------------|------------|---------------|-----------|
| N | N | N | N | N | N |
### Violations
| File | Line | Contract under test | Mock target | Why it's wrong | Fix applied |
|------|------|---------------------|-------------|----------------|-------------|
| ... | ... | ... | ... | ... | ... |
### Logic Mirrors
| File | Production code | Test code | Hardcoded fixture applied |
|------|-----------------|-----------|---------------------------|
| ... | ... | ... | ... |
### Integration Test Boundaries
| File | Type | Real deps | Mocked deps | Verdict |
|------|------|-----------|-------------|---------|
| ... | integration | DB, Router | External API | ✅ CLEAN |
### Clean tests (no violations)
- [list of files that are fully compliant]
### Global setup (not violations)
- [list of infrastructure mocks in conftest.py, setupTests.ts, vitest.setup.ts]
### Uncertain (requires human review)
| File | Line | Mock target | Why uncertain |
|------|------|-------------|---------------|
| ... | ... | ... | ... |
```
### 2. Coverage Summary
- Commands executed
- Pass/fail counts per layer
- Coverage percentage (if available)
### 3. Semantic Audit Verdict
- Contract density check results
- Belief runtime instrumentation status (C4/C5 flows)
- ADR / rejected-path coverage status
- issues found and resolutions
- remaining risk or debt
### 4. Issues Found and Resolutions
- All violations found and how they were fixed
- Any remaining technical debt
### 5. Remaining Risk or Debt
- UNCERTAIN items pending human review
- Files flagged for size split
- Known coverage gaps