Vahter: gated text-repeat ML feature behind ML_REPEAT_COUNT_ENABLED#19
Open
Szer wants to merge 1 commit into
Open
Vahter: gated text-repeat ML feature behind ML_REPEAT_COUNT_ENABLED#19Szer wants to merge 1 commit into
Szer wants to merge 1 commit into
Conversation
Repeated spam ("campaigns") is the dominant failure mode of the
30-day-window SDCA model: 54% of top-50 campaign hits have a max
dormant gap >30 days, so the bot forgets a campaign between bursts
and re-learns it on each retrain. Adding text_repeat_count as an
input feature lets the model condition on "how often have we seen
this exact text" — a precision-improving prior that survives
dormant-gap cycles even at the current training window.
The feature is gated by a new bot_setting ML_REPEAT_COUNT_ENABLED
(default off). When off, the trained pipeline does not concatenate
repeatCountF and inference skips the DB lookup, so behavior is
identical to today. When on, training adds the column and inference
populates it via DB.GetTextRepeatCount, which uses the new partial
index idx_event_msg_text_md5 from V35 to keep the lookup cheap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional
text_repeat_countML feature, gated by a newbot_settingkeyML_REPEAT_COUNT_ENABLED(defaultfalse). When enabled, the SDCA model conditions on how often the exact text has been seen in the training window — a precision-improving prior that survives campaign dormancy cycles.Motivated by experiments documented offline:
longWindow+repeatshad marginally higher precision thanlongWindowalone across every age bucket and the best LogLoss — i.e. fewer false-positive bans.The window-bump itself (
ML_TRAIN_INTERVAL_DAYS=30→120) is a separatebot_settingchange you can roll out independently of this PR.Mechanism
DB.MlDataalways returnstext_repeat_countvia a newtext_repeat_countsCTE keyed on the existingevent.msg_text_md5(V32).MlRepeatCountEnabled. Read once on startup, refreshable viaPOST /reload-settings.repeatCountF = log1p(text_repeat_count)in theConcatenatewhen the flag is on. log1p compresses the long tail (max ~200 → ~5.3) so SDCA doesn't over-weight a single very-repeated campaign.Bot.GetMlRepeatCountlooks up the count viaDB.GetTextRepeatCount(also gated by the FF). Skipped when the flag is off so production hot-path latency is unchanged.MlTrainInterval, so the inference distribution lines up with training.Toggling guidance
Behaviour after toggling the FF takes effect on the next retrain (daily, or via your usual retrain trigger). Until then the loaded model still has the old pipeline:
Migration
V35__event_msg_text_md5_index.sqladds a partial indexevent(msg_text_md5) WHERE event_type='MessageReceived' AND msg_text_md5 IS NOT NULL. Without it,DB.GetTextRepeatCountwould seq-scan the ~700k-row event log on every spam check.Test plan
dotnet test tests/VahterBanBot.Tests -c Release. The existingMLTrainingPipelineTestsfixture now seedsML_REPEAT_COUNT_ENABLED=true, so the smoke test exercises FF=on through training (conditionalConcatenate) and inference (GetMlRepeatCountlookup). FF=off path remains the production default and is implicitly covered by every other ML test (which uses the pinnedml-model.binfixture, trained without the feature, that this PR keeps loadable since ML.NET ignores unused TInput fields).EXPLAIN ANALYZEon a sampleGetTextRepeatCountquery thatidx_event_msg_text_md5is used.bot_settingand triggering a retrain: the new model's metrics row inml_trained_model.created_atupdates, and the bot logs show training succeeded with the new feature column count (visible in the metrics block posted toALL_LOGS_CHANNEL_ID).🤖 Generated with Claude Code