Instructions

Coordinate the end-to-end news monitoring pipeline that scans sources, analyzes content for authoritarian markers, aggregates threat scores, and updates political figure profiles.

Pipeline Flow

The orchestrator manages four sequential stages:

Scan → Analyze → Score → Update

1. Scan (Source Scanner): Fetch new content from RSS feeds and Tavily search, deduplicate, normalize
2. Analyze (Speech Analyzer): Run each content item through Claude for marker detection and severity assessment
3. Score (Aggregator): Combine individual analysis results into per-figure threat scores
4. Update (Profile Writer): Write updated scores and evidence to political figure profiles

Scheduling

Stage Dependencies

• Scan must complete before Analyze begins — the analyzer needs deduplicated content
• Analyze can process items in parallel — each content item is independent
• Score waits for all analysis results from the current batch
• Update writes atomically — no partial profile updates

Error Handling

• If Scan fails: retry twice with exponential backoff, then skip the cycle and alert
• If Analyze fails on a single item: log the failure, continue with remaining items
• If Score calculation produces an outlier (>2 standard deviations from rolling average): flag for human review before writing
• If Update fails: queue the update for the next cycle, do not retry immediately

Coordination Protocol

• Each stage emits a completion event with item count and error count
• The orchestrator logs stage transitions with timestamps for audit
• Pipeline state is persisted so a crashed run can resume from the last completed stage
• Concurrent pipeline runs are prevented — a lock ensures only one scan-analyze-score-update cycle runs at a time

Output Artifacts

• Per-run summary: items scanned, items analyzed, figures updated, errors
• Updated figure profiles with timestamped score history
• Flagged items requiring human review (outlier scores, low-confidence analyses)

Examples

Normal daily cycle: Scan retrieves 47 new items → Analyze processes all 47 (2 failures logged) → Score updates 12 figures → Update writes 12 profiles. Run time: ~15 minutes.

Breaking scan with high-severity detection: Scan retrieves 8 items from wire services → Analyze flags 1 item with severity +1 on multiple markers → Score recalculates the figure’s threat score → Outlier detected (score jumped >2 SD) → Flagged for human review before profile update.