ss-tools/specs/017-llm-analysis-plugin/checklists/reliability-security.md

Checklist: Reliability & Security Requirements Quality v2

Purpose: Validate requirement quality for LLM integration reliability and security. Feature: 017-llm-analysis-plugin Created: 2026-01-28 | Updated: 2026-05-31 — v2

Requirement Completeness

• CHK001 Are retry strategies (count, backoff) defined for all external LLM API calls? [Completeness, Spec §FR-018] — 5 attempts, 5-60s exponential
• CHK002 Are timeout thresholds specified for long-running validation tasks? [Completeness, Gap] — addressed: Path B ≤30s, Path A ≤120s
• CHK003 Are encryption requirements defined for storing API keys at rest? [Completeness, Spec §FR-002] — AES-256, unchanged
• CHK004 Are masking requirements defined for displaying API keys in the UI? [Completeness, Spec §FR-028] — unchanged
• CHK005 Is the fallback behavior defined when the primary screenshot method (Headless) fails? [Completeness, Spec v2: screenshot is optional; user chooses path]
• CHK006 Are requirements defined for handling rate limits from LLM providers? [Completeness, Gap] — addressed: retryDelay extraction + sleep (service.py L1048-1081)
• CHK007 Are data privacy requirements specified regarding what dashboard data (screenshots, logs) is sent to the LLM? [Completeness, Spec §FR-029] — unchanged

v2 Reliability Items

• CHK-V2-R01 Are requirements defined for Path B when chart data execution overloads Superset? [Completeness, Spec: execute_chart_data is off by default; concurrency_limit controls parallelism]
• CHK-V2-R02 Is fallback behavior defined when dataset health check fails (API unreachable, timeout)? [Completeness, Spec §FR-044]
• CHK-V2-R03 Are requirements defined for handling deleted dashboards referenced by ValidationSource? [Completeness, Spec §Edge Cases v2]
• CHK-V2-R04 Is concurrent task execution safety defined (two scheduled tasks overlapping)? [Completeness, Spec §Edge Cases v2: same task → toast "already running", no queue. Different task → FIFO queue with concurrency_limit.]
• CHK-V2-R05 Are requirements defined for API key rotation — what happens to existing tasks when a provider's key changes? [Completeness, Spec §Edge Cases v2: tasks fetch current key at execution time from LLMProviderService; no key caching.]

Requirement Clarity

• CHK008 Is "securely store" quantified with specific encryption standards? [Clarity, Spec §FR-002] — AES-256
• CHK009 Are "recent execution logs" defined by specific time window or line count? [Clarity, Spec §FR-006]
• CHK010 Is "automatic retry logic" defined with specific backoff parameters? [Clarity, Spec §FR-018]
• CHK-V2-R06 Is dataset health check defined — which Superset API endpoints, what constitutes «healthy»? [Clarity, Spec §FR-044]
• CHK-V2-R07 Are Path B timing expectations defined (dashboard fetch + chart fetch + dataset fetch → total)? [Clarity, Spec §SC-007]

Edge Case Coverage

• CHK011 Are requirements defined for scenarios where the LLM provider is completely unreachable? [Edge Case, Gap] — addressed: retry 5× then fail with clear error
• CHK012 Are requirements defined for handling "token limit exceeded" errors with large logs? [Edge Case, Gap] — addressed: logs limited to 100 lines/24h; images compressed to 1024px JPEG
• CHK013 Are requirements defined for invalid/expired API keys during task execution? [Edge Case, Gap] — addressed: auth errors are NOT retried (service.py L992-993)
• CHK-V2-R08 What happens when multi-chunk screenshots produce 15+ images exceeding LLM token limit? [Edge Case, Spec §Edge Cases v2: estimated payload calculation → progressive quality reduction (30%) + width reduction (800px) → Path B fallback. Logged via molecular-cot: REASON/REFLECT.]
• CHK-V2-R09 What happens when Path B topology parsing encounters unexpected position_json format? [Edge Case, gap — needs clarification]