Skip to content

Vahter: gated text-repeat ML feature behind ML_REPEAT_COUNT_ENABLED#19

Open
Szer wants to merge 1 commit into
mainfrom
vahter-ml-repeat-feature
Open

Vahter: gated text-repeat ML feature behind ML_REPEAT_COUNT_ENABLED#19
Szer wants to merge 1 commit into
mainfrom
vahter-ml-repeat-feature

Conversation

@Szer

@Szer Szer commented May 3, 2026

Copy link
Copy Markdown
Owner

Summary

Adds an optional text_repeat_count ML feature, gated by a new bot_setting key ML_REPEAT_COUNT_ENABLED (default false). When enabled, the SDCA model conditions on how often the exact text has been seen in the training window — a precision-improving prior that survives campaign dormancy cycles.

Motivated by experiments documented offline:

  • 54% of the top-50 spam campaigns have a max dormant gap >30 days, so the production 30-day window forgets them between bursts. Adding repeats helps the model recognize a returning campaign even when the previous instance has rotated out.
  • On a shared 120-day evaluation set, longWindow+repeats had marginally higher precision than longWindow alone across every age bucket and the best LogLoss — i.e. fewer false-positive bans.

The window-bump itself (ML_TRAIN_INTERVAL_DAYS=30→120) is a separate bot_setting change you can roll out independently of this PR.

Mechanism

  • SQLDB.MlData always returns text_repeat_count via a new text_repeat_counts CTE keyed on the existing event.msg_text_md5 (V32).
  • Feature flagMlRepeatCountEnabled. Read once on startup, refreshable via POST /reload-settings.
  • Training — only includes repeatCountF = log1p(text_repeat_count) in the Concatenate when the flag is on. log1p compresses the long tail (max ~200 → ~5.3) so SDCA doesn't over-weight a single very-repeated campaign.
  • InferenceBot.GetMlRepeatCount looks up the count via DB.GetTextRepeatCount (also gated by the FF). Skipped when the flag is off so production hot-path latency is unchanged.
  • Window — inference window matches MlTrainInterval, so the inference distribution lines up with training.

Toggling guidance

Behaviour after toggling the FF takes effect on the next retrain (daily, or via your usual retrain trigger). Until then the loaded model still has the old pipeline:

  • FF off → on: until retrain, old model ignores the new column even though we'd populate it. Pipeline still correct, repeats simply unused.
  • FF on → off: until retrain, old model expects the column but inference now passes 0. Predictions are temporarily slightly worse for repeat-heavy spam. Trigger a retrain right after the toggle.

Migration

V35__event_msg_text_md5_index.sql adds a partial index event(msg_text_md5) WHERE event_type='MessageReceived' AND msg_text_md5 IS NOT NULL. Without it, DB.GetTextRepeatCount would seq-scan the ~700k-row event log on every spam check.

Test plan

  • CI green: dotnet test tests/VahterBanBot.Tests -c Release. The existing MLTrainingPipelineTests fixture now seeds ML_REPEAT_COUNT_ENABLED=true, so the smoke test exercises FF=on through training (conditional Concatenate) and inference (GetMlRepeatCount lookup). FF=off path remains the production default and is implicitly covered by every other ML test (which uses the pinned ml-model.bin fixture, trained without the feature, that this PR keeps loadable since ML.NET ignores unused TInput fields).
  • Post-deploy: verify with EXPLAIN ANALYZE on a sample GetTextRepeatCount query that idx_event_msg_text_md5 is used.
  • After enabling the flag in bot_setting and triggering a retrain: the new model's metrics row in ml_trained_model.created_at updates, and the bot logs show training succeeded with the new feature column count (visible in the metrics block posted to ALL_LOGS_CHANNEL_ID).

🤖 Generated with Claude Code

Repeated spam ("campaigns") is the dominant failure mode of the
30-day-window SDCA model: 54% of top-50 campaign hits have a max
dormant gap >30 days, so the bot forgets a campaign between bursts
and re-learns it on each retrain. Adding text_repeat_count as an
input feature lets the model condition on "how often have we seen
this exact text" — a precision-improving prior that survives
dormant-gap cycles even at the current training window.

The feature is gated by a new bot_setting ML_REPEAT_COUNT_ENABLED
(default off). When off, the trained pipeline does not concatenate
repeatCountF and inference skips the DB lookup, so behavior is
identical to today. When on, training adds the column and inference
populates it via DB.GetTextRepeatCount, which uses the new partial
index idx_event_msg_text_md5 from V35 to keep the lookup cheap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant