# Hybrid AI Local News Classification Design

## Goal
Unify RSS and scraping ingestion so every incoming article can receive a controlled category and Bangladesh division/district/upazila classification, using fixed source mappings first, deterministic rules second, and the existing Gemini integration only as a quota-safe fallback. The system must be switchable globally from Admin and configurable per source without blocking news ingestion when AI is unavailable.

## Current Codebase Baseline
- `includes/functions.php` already contains `resolve_category()` with a fixed category map plus dynamic category fallback.
- `includes/functions.php` already contains `detect_geo_from_text()` backed by `config/bd_geo.php` and a small `get_all_upazilas()` list.
- RSS sources already expose district/upazila fields through `admin/sources.php` and the source persistence helpers.
- Scrape sources currently have category but no equivalent district/upazila/location-mode controls.
- The existing AI provider settings support Gemini, Anthropic and OpenAI; Gemini is the default and is already used for summary/translation.
- `api/local_news.php` currently requires GPS lat/lon and returns radius-based local news.
- Existing logs have shown Gemini quota exhaustion, so AI must never be mandatory for ingestion.

## Design Principles
1. Source mapping outranks AI.
2. Deterministic classification outranks AI when confidence is adequate.
3. One AI request may classify both category and geography; never send two separate calls for the same article.
4. AI failure, timeout, invalid JSON, quota exhaustion, or disabled AI must never prevent article storage.
5. Category output is restricted to the existing approved category whitelist.
6. District output is restricted to keys in `config/bd_geo.php`; unknown free-form district names are rejected.
7. Existing APIs and source behavior remain backward compatible.

## Classification Pipeline
For every article before final insert/update:

1. **Source Policy**
   - Read source category policy and location policy.
   - A fixed category is accepted immediately.
   - A `strict` district mapping is accepted immediately and cannot be overridden.
   - A `smart` district mapping becomes the default candidate but may be overridden by a higher-confidence deterministic/AI result.
   - `auto` has no fixed default.

2. **Deterministic Category Rule**
   - Run existing category keywords against source/page name, article title and description.
   - Do not create dynamic `cat-*` categories during automatic article classification.
   - Automatic classification may only return the approved whitelist.

3. **Deterministic Geography Rule**
   - Run location matching against title first, then description/lead text.
   - Exact upazila match maps to its parent district/division.
   - Exact district match maps to its division.
   - Source district in `smart` mode remains the default if no stronger text evidence exists.

4. **Gemini Fallback / Verification**
   - Only run if the global AI classifier is enabled, an AI key exists, source-level AI is not disabled, and either category or geography remains ambiguous.
   - One structured request receives title, short description/lead, source/page name, current deterministic candidates and allowed category/district identifiers.
   - The model must return JSON only.
   - Server validates every returned value against allowlists before accepting it.

5. **Final Decision and Audit**
   - Resolve final category and geo using precedence rules.
   - Store classification origin and confidence metadata.
   - Continue inserting article even if AI could not run.

## Source-Level Modes
Each RSS and Scrape source gets the same controls:

### Category Mode
- `fixed`: always use selected category.
- `auto`: rules may classify; Gemini can be fallback if enabled.

### Location Mode
- `strict`: every article from the source is forced to selected district/upazila.
- `smart`: selected district/upazila is the default, but strong article evidence can override it.
- `auto`: no fixed location; rules/Gemini classify.

### AI Per-Source
- `inherit`: use global AI classifier setting.
- `off`: never send articles from this source to AI.

This supports district-specific pages such as a Sunamganj page: set `district=sunamganj`, `location_mode=strict` or `smart`.

## Admin Settings
Add a dedicated "AI Classification" section in `admin/settings.php` with:
- Master `ai_classification_enabled` switch.
- `ai_classification_min_confidence` default 80.
- `ai_classification_daily_limit` default 200.
- `ai_classification_cooldown_seconds` default 2.
- `ai_classification_fail_open` fixed true (documented, not user-disableable).
- Existing provider/model/API key are reused; no duplicate key field.

The switch affects only category/location classification. Existing summary/translation toggles remain independent.

## Category Whitelist
Automatic classification may return only:
- `national`
- `international`
- `sports`
- `entertainment`
- `technology`
- `business`
- `politics`
- `opinion`
- `lifestyle`
- `education`
- `health`
- `english`

Dynamic custom categories remain available for manual/admin use, but automatic AI/rule classification does not create new `cat-*` categories.

## Geography Model
`config/bd_geo.php` remains the source of truth for divisions/districts.

V1 guarantees reliable district-level classification for all 64 districts. Upazila matching is best-effort and may be expanded incrementally. An article is still valid Local News if district is known and upazila is null.

Every accepted geo decision carries:
- `division_key`
- `district_key`
- `district`
- `upazila`
- `geo_lat`
- `geo_lon`

## Database Changes
### RSS sources
Add/normalize fields:
- `category_mode VARCHAR(20) DEFAULT 'fixed'`
- `division_key VARCHAR(64) NULL`
- `district_key VARCHAR(64) NULL`
- `upazila VARCHAR(120) NULL`
- `location_mode VARCHAR(20) DEFAULT 'auto'`
- `ai_classification_mode VARCHAR(20) DEFAULT 'inherit'`

### Scrape sources
Add the same fields so RSS and scraping use identical policy semantics.

### News
Add audit fields:
- `division_key VARCHAR(64) NULL`
- `district_key VARCHAR(64) NULL`
- `category_source VARCHAR(20) NULL` (`fixed`, `rule`, `ai`, `legacy`)
- `location_source VARCHAR(20) NULL` (`source`, `rule`, `ai`, `legacy`)
- `classification_confidence TINYINT UNSIGNED NULL`
- `classified_at DATETIME NULL`
- `classification_meta TEXT NULL` for compact JSON audit context.

Existing `district`, `upazila`, `geo_lat`, `geo_lon`, and `category` remain in place for compatibility.

### AI Usage Log
Create `ai_classification_usage`:
- `id`
- `news_id` nullable
- `source_id`
- `provider`
- `model`
- `status` (`success`, `quota`, `timeout`, `invalid`, `error`)
- `confidence`
- `created_at`

This supports daily limits and operational visibility without storing API secrets.

## Core PHP Interfaces
Create `includes/news_classifier.php` with focused responsibilities:

- `classification_category_whitelist(): array`
- `classification_settings(array $settings): array`
- `classification_source_policy(array $source, string $sourceType): array`
- `classification_rule_category(string $title, string $description, array $policy): ?array`
- `classification_rule_geo(string $title, string $description, array $policy): ?array`
- `classification_ai_allowed(array $settings, array $policy): bool`
- `classification_ai_request(array $article, array $policy, array $settings): ?array`
- `classification_validate_ai_result(array $raw): ?array`
- `classify_news_article(array $article, array $source, string $sourceType, array $settings): array`
- `classification_log_usage(array $entry): void`

Return shape from `classify_news_article()`:
```php
[
  'category' => 'national',
  'category_source' => 'fixed|rule|ai|legacy',
  'division_key' => 'sylhet',
  'district_key' => 'sunamganj',
  'district' => 'সুনামগঞ্জ',
  'upazila' => null,
  'geo_lat' => 25.0667,
  'geo_lon' => 91.3950,
  'location_source' => 'source|rule|ai|legacy',
  'confidence' => 95,
  'meta' => [...],
]
```

## Gemini Contract
Reuse the existing provider/model/API key and HTTP helper. Do not introduce a second Gemini client.

Prompt contract requires JSON only:
```json
{
  "category": "national",
  "is_local": true,
  "division_key": "sylhet",
  "district_key": "sunamganj",
  "upazila": "জগন্নাথপুর",
  "confidence": 94
}
```

Validation rules:
- category must be in the fixed whitelist.
- district_key must exist in `bd_geo()`.
- division_key must match the selected district's configured division.
- confidence must be integer 0..100.
- below configured minimum confidence, AI geography/category is ignored in favor of rule/source fallback.

## Quota and Failure Protection
- Daily classification call cap defaults to 200.
- Per-source `off` mode bypasses AI.
- AI is skipped when fixed/strict policy already resolves the article.
- If provider responds with HTTP 429, mark usage as `quota` and suppress additional classification calls for the remainder of the current cron execution.
- Network timeout, malformed JSON, or provider error falls back to deterministic/source classification.
- No retry loop inside the same article ingestion.
- AI errors are logged without API key or raw secret headers.

## RSS Integration
Update `run_news_collection()` path:
- Build article candidate from parsed RSS item.
- Call classifier before insert.
- Insert final category and geo fields together with audit metadata.
- Preserve existing deduplication, age filters, breaking detection, summary/translation and health logic.

## Scraper Integration
Update `run_auto_scrape()` path:
- After article content extraction and before insert, call the same classifier.
- Scrape sources gain the same category/location modes as RSS.
- Preserve existing deduplication, max-per-run, delays, breaking behavior and source health logic.

## Local News API Expansion
Keep current GPS mode fully backward compatible.

Support three mutually compatible query styles:
- `GET /api/local_news.php?lat=...&lon=...&radius=50`
- `GET /api/local_news.php?district=sunamganj`
- `GET /api/local_news.php?division=sylhet`
- `GET /api/local_news.php?districts=sunamganj,sylhet` (up to 10 keys)

Precedence:
1. explicit `districts`
2. explicit `district`
3. explicit `division`
4. GPS lat/lon

The response keeps `success`, `data`, and `meta` structure. Add `division_key`, `district_key`, and `classification_confidence` to item payloads where available.

## Admin UI Changes
### `admin/sources.php`
Add:
- Category mode selector (`Fixed`, `Auto`).
- Division and district selectors.
- Optional upazila text/select.
- Location mode selector (`Strict`, `Smart`, `Auto`).
- AI selector (`Inherit`, `Off`).
- Table badges indicating source policy.

### `admin/scrape_sources.php`
Add exactly the same classification controls so scraped category pages can be bound to a district.

### `admin/settings.php`
Add global AI classification controls and current daily usage/status summary.

### Optional operational view
Show recent AI classification failures/quota events in existing Automation/Monitoring area rather than creating another large dashboard unless needed.

## Migration and Backward Compatibility
Create `database/migration_hybrid_local_classification.sql`.

Migration must:
- add missing columns idempotently where practical for supported MySQL version,
- create `ai_classification_usage`,
- preserve all existing rows,
- backfill `district_key`/`division_key` from existing `district` values when safely matched,
- default old RSS/scrape sources to behavior equivalent to current category/location behavior.

Existing installations import this migration once. Fresh schema also receives the same columns/tables.

## Logging and Observability
Use existing `app_log()` infrastructure.

Log only operational events:
- AI quota reached,
- invalid AI result,
- provider failure,
- source-policy conflict resolved,
- classification migration/backfill problems.

Do not log API keys, Authorization headers, or full provider responses containing sensitive context.

## Testing
Add PHP tests for:
1. fixed category beats rules/AI,
2. strict source district beats article text,
3. smart source district is used when article has no stronger location,
4. deterministic district text can override smart default,
5. auto mode can resolve district from text,
6. AI result rejected for unknown category,
7. AI result rejected for unknown district,
8. low-confidence AI result ignored,
9. AI disabled path makes zero provider calls,
10. quota condition fails open,
11. RSS and scrape source policies normalize identically,
12. Local News API filters by district/division while retaining GPS compatibility.

Static/lint verification:
- `php -l` over every PHP file,
- migration/schema presence checks,
- API contract regression tests,
- existing Automation Center, Live, Feed Hub tests unchanged and passing.

## Out of Scope for This Version
- Android UI for choosing/following districts.
- Push notifications per followed district.
- Complete official upazila dataset if not already available in the project.
- AI-generated category creation.
- GPS reverse geocoding in the app.

These can be added after the backend classification and Local News API are stable.

## Acceptance Criteria
- Admin can globally enable/disable AI category/location classification.
- RSS and scraping sources expose identical classification controls.
- A district-specific page can force or default every article to its district.
- Ambiguous generic feeds can use existing Gemini as a single fallback classifier.
- Gemini outage/quota never blocks news collection.
- Auto category never creates unknown `cat-*` categories.
- District-key Local News API works without GPS and existing GPS mode still works.
- Every classified article has enough metadata to explain whether source, rule or AI chose its category/location.