# Advanced Scraper V3.5 Design

## Goal
Make scraper ingestion resilient so transient failures, unusual page structures, pagination, lazy images, and metadata variations do not silently cause missed news.

## Scope
Backend only, extending V3.4 District Source Network. Existing RSS, Hybrid AI classification, district mappings, deduplication, and Local News API must remain backward compatible.

## Architecture
1. Add a focused `includes/advanced_scraper.php` module for URL normalization, retry-aware fetching, multi-strategy listing discovery, article extraction, retry queue helpers, and diagnostics.
2. Keep `run_link_scrape_collection()` as orchestration but delegate discovery/extraction/retry behavior to the module.
3. Persist per-source scraper controls/statistics and a retry queue in additive DB migrations.
4. Do not mark transiently failed article URLs as seen. Mark only successfully saved, duplicate-confirmed, too-old, or permanent-invalid URLs as processed.

## Discovery strategies
- Anchor URL heuristics.
- `article`, common card/post/news containers.
- JSON-LD `ItemList` / `NewsArticle` URLs.
- Canonical and structured links where relevant.
- Same-host internal link scoring.
- Per-source include/exclude regex.
- Pagination via rel=next, common next labels and configurable page depth.

## Article extraction fallback
Priority:
1. JSON-LD NewsArticle/Article.
2. OpenGraph/Twitter/meta tags.
3. Semantic DOM (`article`, `main`, h1, time).
4. Common content class/id fallbacks.
5. Paragraph fallback.

Images support `src`, `data-src`, `data-lazy-src`, `data-original`, and `srcset`.

## Retry behavior
- Retry network/HTTP 408/425/429/500/502/503/504 failures with bounded backoff.
- Persist unresolved transient failures in `scrape_retry_queue` with attempt count and `next_retry_at`.
- Permanent 404/410 or invalid/non-article pages may be marked processed to avoid endless retries.
- Ingestion continues when one article fails.

## Source controls
Add fields for pagination depth, discovery limit, include/exclude URL regex, minimum title length, minimum description length and retry limit. Defaults must preserve existing behavior as closely as possible.

## Diagnostics
Per source track discovered, attempted, saved, duplicate, failed, retry pending, old-skipped, last extraction rate, last error, and last run time. Admin page exposes these metrics and a Test Scraper action that performs read-only diagnostics.

## Data safety
Migration is additive. No destructive schema changes. Existing scrape source rows receive safe defaults.

## Verification
- TDD regression for failed-link-not-seen behavior.
- Tests for URL normalization, structured-data extraction, lazy images, retry classification, pagination discovery and diagnostics contracts.
- Existing V3.4 tests must continue to pass.
- PHP lint and ZIP integrity must pass.