40 lines
3.9 KiB
Markdown
40 lines
3.9 KiB
Markdown
# Checklist: Reliability & Security Requirements Quality v2
|
||
|
||
**Purpose**: Validate requirement quality for LLM integration reliability and security.
|
||
**Feature**: `017-llm-analysis-plugin`
|
||
**Created**: 2026-01-28 | **Updated**: 2026-05-31 — v2
|
||
|
||
## Requirement Completeness
|
||
|
||
- [x] CHK001 Are retry strategies (count, backoff) defined for all external LLM API calls? [Completeness, Spec §FR-018] — 5 attempts, 5-60s exponential
|
||
- [x] CHK002 Are timeout thresholds specified for long-running validation tasks? [Completeness, Gap] — addressed: Path B ≤30s, Path A ≤120s
|
||
- [x] CHK003 Are encryption requirements defined for storing API keys at rest? [Completeness, Spec §FR-002] — AES-256, unchanged
|
||
- [x] CHK004 Are masking requirements defined for displaying API keys in the UI? [Completeness, Spec §FR-028] — unchanged
|
||
- [x] CHK005 Is the fallback behavior defined when the primary screenshot method (Headless) fails? [Completeness, Spec v2: screenshot is optional; user chooses path]
|
||
- [x] CHK006 Are requirements defined for handling rate limits from LLM providers? [Completeness, Gap] — addressed: retryDelay extraction + sleep (service.py L1048-1081)
|
||
- [x] CHK007 Are data privacy requirements specified regarding what dashboard data (screenshots, logs) is sent to the LLM? [Completeness, Spec §FR-029] — unchanged
|
||
|
||
## v2 Reliability Items
|
||
|
||
- [ ] CHK-V2-R01 Are requirements defined for Path B when chart data execution overloads Superset? [Completeness, Spec: execute_chart_data is off by default; concurrency_limit controls parallelism]
|
||
- [ ] CHK-V2-R02 Is fallback behavior defined when dataset health check fails (API unreachable, timeout)? [Completeness, Spec §FR-044]
|
||
- [ ] CHK-V2-R03 Are requirements defined for handling deleted dashboards referenced by ValidationSource? [Completeness, Spec §Edge Cases v2]
|
||
- [x] CHK-V2-R04 Is concurrent task execution safety defined (two scheduled tasks overlapping)? [Completeness, Spec §Edge Cases v2: same task → toast "already running", no queue. Different task → FIFO queue with concurrency_limit.]
|
||
- [x] CHK-V2-R05 Are requirements defined for API key rotation — what happens to existing tasks when a provider's key changes? [Completeness, Spec §Edge Cases v2: tasks fetch current key at execution time from LLMProviderService; no key caching.]
|
||
|
||
## Requirement Clarity
|
||
|
||
- [x] CHK008 Is "securely store" quantified with specific encryption standards? [Clarity, Spec §FR-002] — AES-256
|
||
- [x] CHK009 Are "recent execution logs" defined by specific time window or line count? [Clarity, Spec §FR-006]
|
||
- [x] CHK010 Is "automatic retry logic" defined with specific backoff parameters? [Clarity, Spec §FR-018]
|
||
- [x] CHK-V2-R06 Is dataset health check defined — which Superset API endpoints, what constitutes «healthy»? [Clarity, Spec §FR-044]
|
||
- [x] CHK-V2-R07 Are Path B timing expectations defined (dashboard fetch + chart fetch + dataset fetch → total)? [Clarity, Spec §SC-007]
|
||
|
||
## Edge Case Coverage
|
||
|
||
- [x] CHK011 Are requirements defined for scenarios where the LLM provider is completely unreachable? [Edge Case, Gap] — addressed: retry 5× then fail with clear error
|
||
- [x] CHK012 Are requirements defined for handling "token limit exceeded" errors with large logs? [Edge Case, Gap] — addressed: logs limited to 100 lines/24h; images compressed to 1024px JPEG
|
||
- [x] CHK013 Are requirements defined for invalid/expired API keys during task execution? [Edge Case, Gap] — addressed: auth errors are NOT retried (service.py L992-993)
|
||
- [x] CHK-V2-R08 What happens when multi-chunk screenshots produce 15+ images exceeding LLM token limit? [Edge Case, Spec §Edge Cases v2: estimated payload calculation → progressive quality reduction (30%) + width reduction (800px) → Path B fallback. Logged via molecular-cot: REASON/REFLECT.]
|
||
- [x] CHK-V2-R09 What happens when Path B topology parsing encounters unexpected position_json format? [Edge Case, gap — needs clarification]
|