# BD News Control Center V3.5 — Advanced Scraper

## What changed
- Hybrid listing discovery: article/card anchors, JSON-LD ItemList/Article links, internal-link scoring.
- Pagination support with `rel=next` / next-link detection (1–5 pages configurable per source).
- Hybrid article extraction: JSON-LD → OpenGraph/Twitter/meta → semantic DOM → content/paragraph fallback.
- Lazy image support: `src`, `data-src`, `data-lazy-src`, `data-original`, `srcset`.
- URL normalization removes common tracking parameters and fragments before deduplication.
- Immediate HTTP retries plus persistent retry queue for transient failures.
- HTTP 401/403/408/425/429/5xx and extraction failures remain retryable; 404/410 are terminal.
- Transiently failed links are NOT inserted into `scraped_link_index`.
- Per-source include/exclude regex, pagination depth, discovery limit, minimum title/description lengths and retry limit.
- Per-source diagnostics: found, attempted, saved, duplicate, failed, retry pending, old skipped, extraction rate.
- Admin `Test Scraper` action shows discovery and sample extraction without saving news.
- DOM-less PHP fallback parser for shared hosting where ext-dom is unavailable.

## Upgrade
For an existing V3.4 database, import once:

`database/migration_advanced_scraper_v35.sql`

Then open Admin → Auto Link Scrape Sources and tune individual sources only when needed. Defaults are safe for existing sources.

## Recommended defaults
- Pagination depth: 1 (use 2–3 for district archive pages if needed)
- Discovery limit: 100
- Retry limit: 2
- Min title length: 8
- Min description length: 0

## Important limitation
No generic scraper can guarantee 100% capture from JavaScript-only pages, CAPTCHAs, paywalls, or sites that aggressively block automated clients. V3.5 avoids permanently losing transient failures and gives diagnostics/retry controls, but a site requiring a real browser may still need a publisher-specific connector or headless-browser service.