Task 5: demand-aware ingestion funnel

[mircealungu]

Makes the crawler's expensive LLM simplification demand-driven instead of supply-driven, and gives newly-demanded buckets an immediate head start. All 6 phases of the Task 5 design.

Problem

Simplification ran until a static per-language cap (50/day/topic for da/fr/de, 20 otherwise), then stopped. So dormant languages were simplified at the same rate as busy ones, and whichever articles appeared first in feed order won the quota (first-N, not best-N). Evidence from the week-28 report: ~370 crawled per 1 opened.

What changed

New zeeguu/core/content_retriever/funnel.py

• compute_demand_surface() — the distinct (language, topic) buckets that active readers (last_seen ≤ 30d) actually want, deduped across the cohort. A reader with no topic subscription reads the whole language, so they lend demand to every topic in it.
• FunnelBudget — per-(language, topic) quota that scales 20→50 with reader count, plus a pilot-light floor (≤5 distinct topics/day) for dormant supported languages so a brand-new learner (or the Romanian kiosk) never opens an empty feed.
• Pre-download title triage — before paying any readability/download cost: skip a feed entirely when its language is already satisfied for the day, or keep only the best-N titles when headroom is limited. Cheap LLM ranker with a safe first-N fallback (only runs when candidates exceed the keep count), so triage can only reduce waste, never drop a must-have article.

article_downloader.py + crawl.py thread a FunnelBudget instead of the old static-cap counts dict. Legacy download_from_feed callers pass no budget → behaviour unchanged. No-topic articles are now simplified only when the language has all-topic readers (old code always simplified them = pure waste).

Promotion-on-join — backfill.py
Subscribing to a topic or switching learned language fires a background backfill that simplifies a few existing recent raw articles for the newly-demanded bucket. Dormant buckets already hold raw, un-simplified articles (the crawl stores them, only skips the LLM), so this is fast and download-free — no mini-crawl. No-ops if the bucket already has fresh inventory. Wired into the subscribe and learned-language endpoints via run_in_background.

Phase mapping

Phase	Status
1. Demand surface + quota	done
2. Two-tier pilot-light floor	done
3. Title-triage best-N	done (pre-download LLM triage)
4. Pre-simplify cache	already satisfied (crawl-time simplification = the cache; /simplify_article = the tail)
5. Promotion-on-join	done
6. Romanian floor	satisfied by the floor covering every supported language

Tests

31 tests in zeeguu/core/test/test_funnel.py (in-memory SQLite): quota scaling, demand surface, gate/floor/no-topic paths, headroom, triage selection + fallback, backfill inventory queries. Regression-checked test_retrieve_and_compute / test_article / test_feed / test_endpoint_names.

Reviewer notes / conscious choices

• Active reader = last_seen ≤ 30d (chosen over open-based). Implication: German (many recently-seen users) stays "in demand" — this does not reproduce the open-based "German → 0" reduction; topic-dedup and dormant-language savings still apply. The definition is isolated in one function (_active_reader_rows) for an easy switch later.
• Not yet validated on a real crawl / real subscribe flow — logic is unit-tested with the LLM mocked; the LLM ranker and background backfill need a live run.
• crawl_stats.py dashboard still displays the now-superseded static "cap" (legacy constants kept alive only for it) — follow-up.
• Backfill has no dedup guard against concurrent triggers (conservative caps mitigate); a brand-new never-crawled language finds no raw inventory and relies on the floor + next crawl.

🤖 Generated with Claude Code

[github-actions]

ArchLens detected architectural changes in the following views: