A small model that runs in 0.4ms, costs nothing per message, and cleared 87% of the false positives from our old moderation. Here is how we built it, and how it learned from its own production mistakes.
The problem: "i'll kill you" is not a threat here
We run a Minecraft minigames network. Players spend their evenings rushing beds, clutching blocks, and typing things like:
"i'll kill you" "final kill ez" "kill farm mid"
For over a year our chat moderation ran on OpenAI's moderation API. It is a strong general-purpose model. It is also blind to context. It reads normal PvP banter as violence. It reads "kms" after a laggy round as self-harm.
The results were rough. Of the roughly 59,000 automated mutes it issued, staff reversed about half as false positives. Every reversal costs a staff member time and mutes a player for doing nothing wrong.
Share of automated mutes that staff later reversed, by category. Half of everything the model flagged was wrong.
The whole idea in one picture
The plan was a loop that improves itself. Turn old moderation logs into labeled data. Have strong models relabel it with game context. Distill that into a tiny model. Run the tiny model on live chat but let humans keep the final say. Log every decision so the next round has better data than the last.
Each stage feeds the next. The dashed line is the important part: live decisions become tomorrow's training data.
Step 1: Your moderation logs are already a labeled dataset
We had years of moderation history in a database. Every automated mute recorded the message, the model's verdict, and what a human did next. A reversed mute is a labeled false positive. A mute that stood is a confirmed violation.
Joining those two facts gave us a 10,320-example set with real human verdicts, without paying for any labeling.
Step 2: Find a teacher that understands the game
We tested a few context-aware language models against that set, using a rubric written for this game. The rubric spells out the difference between game talk and real harm. "i'll kill you" is a game objective. "kys" aimed at a person is harassment.
The labeling ran on Fireworks AI, which serves open models behind a fast API. Two models stood out, and they had opposite strengths.
| Model | Clears OpenAI's false positives | Keeps real violations | Agrees with staff |
|---|---|---|---|
| OpenAI moderation | 0% | 100% | ~47% |
| gpt-oss-120b + rubric | 70% | 84% | 77% |
| Kimi K2 + rubric | 85% | 69% | 77% |
gpt-oss is the more balanced labeler and Kimi is stricter, so we used them together. The generator labels everything cheaply, the verifier re-checks anything risky, and when they disagree we drop the example.
Recovery of false positives against retention of real violations, on the human-verdict set. Both students sit above the teacher trade-off line at a fraction of the size.
Step 3: Distill it into something tiny
You cannot call a language model on every chat message. The latency and the cost are both too high. So we distilled the teachers' judgment into a small character-level CNN. It has 0.23 million parameters and two outputs: whether a message is a violation, and what kind. We export it to ONNX and serve it with FastAPI, with no PyTorch in the running image. It scores a message in about 0.4 milliseconds on a CPU.
Working at the character level matters more than the architecture. Players do not type "you are terrible." They type "ur trash kid" and creative spellings of slurs, which a character model can still read but a word-based tokenizer turns into an unknown token.
We shipped the first version in shadow mode. It scores every message and alerts staff, but it never mutes on its own. It also writes every decision to a log. That logging choice is what made the second half of this project possible.
Step 4: Let production grade the model
After two and a half days, the log held about 499,000 scored messages. Two questions matter for a shadow model.
Did it catch what humans caught? We matched every staff mute to that player's recent scored messages. The model had flagged 94% of them.
Was it flagging the right things? Mostly not. We relabeled its flags with the teacher pair. Only about a quarter held up. Three out of four flagged messages were false positives.
The old model's scores against the new one's, on live traffic. The new model pulls a lot of weight out of the flag zone.
The failures fell into a few clear patterns:
| Failure mode | Share of flags | What was happening |
|---|---|---|
| Non-English text read as sexual | ~35% | Romanized Hindi, Turkish, and Cyrillic chat landed in the sexual category. A message asking a friend to return borrowed armor scored high. |
| Profanity read as harassment | ~25% | A single "fuck you" or "stfu", the kind of thing staff never mute, flagged at high confidence. |
| One category catching everything | most routing | Four of the eight categories never won. Real ads for hacked clients got labeled sexual. |
| Repeated messages counted many times | ~15% | One player flooding one line counted as 67 separate flags. |
Every unique flagged message, checked by the teacher pair. The two loudest categories were almost all false positives. Only the hate category earned its flags.
Left: which languages filled the false positives in the sexual category. Right: the model flagged some scripts at nearly three times the English rate.
The deeper lesson: the model treated anything unlike its training data as a violation. Non-English chat, shouting, and long messages all leaked into whichever category was closest. The fix was to get more of those messages into the training set.
Step 5: The self-improving loop
Two models label each message. Agreement becomes training data. The model's own confirmed mistakes come back as hard negatives.
Here is the loop that fixed it.
- Generate. The fast model relabels every unique flagged and near-miss message with the game rubric. This cost $3.40.
- Verify. The strict model independently re-judges anything risky. That was about 12,800 checks, around $22.
- Agreement becomes a label, disagreement gets dropped. 3,125 messages where the teachers split never entered training.
- The gold: 8,431 messages the model flagged that both teachers cleared. These are its own confirmed mistakes on real traffic. They go back in as heavily weighted negative examples.
One judgment call shaped everything. Our staff do mute casual slur use, but the game-lenient teachers tend to clear it. So we added a guard: a teacher-cleared slur can never become a negative example. If you distill from a model without checking where its values differ from yours, you will ship its rules instead of your own.
Held-out metrics per epoch. The chosen checkpoint clears the most live false positives without giving up the violations staff actually mute.
Step 6: Trust the model where it's confident
The new category head came back much sharper. When it is confident, it is usually right: hate flags are 92% correct. When no category clears a confidence bar, the model reports "unspecified" instead of forcing a guess into the nearest one.
The mute threshold then became per-category. We swept each one against the teacher labels and checked it against staff recall.
Each curve sweeps the threshold inside one category. The stars mark the settings we shipped. Precision is per unique message; the horizontal axis is live volume for context.
The results
The numbers below are per unique message, so a spammer repeating a line does not skew them.
| Old model | New model | |
|---|---|---|
| Live false positives cleared | it flagged them | 87% |
| Precision (per unique message) | 24% | 59% |
| Real violations caught at shipped thresholds | 75% | |
| Staff-mute recall | 94% | 90% |
| Flag volume | ~5,800 / day | ~950 / day |
| "return my armor" message (Hindi) | 0.60+, sexual | 0.02, clean |
| Server ad for a hacked client | missed | flagged |
| Latency and size | 0.4ms, 0.9MB | same |
The drop in recall from 94% to 90% was a deliberate trade. The misses are slur-policy gray areas, spam floods now handled elsewhere, and mutes for things that never appeared in public chat.
Live decision rates. Traffic keeps its daily rhythm on top. Flags per minute drop about sixfold when the new model ships.
Real-time instead of after-the-fact
The old system could not afford to call a slow API on every message, so it batched them. Messages were shown to everyone first, then sent to OpenAI in bulk and checked a moment later. That moment was about 1.8 seconds on a normal day, and up to 35 seconds when the API was slow. By the time a mute landed, everyone had already read the message.
The small model changes that. It scores a message in about 0.4 milliseconds, fast enough to run inline, before the message is delivered. A violation gets blocked at the source instead of muted after the fact.
Median moderation latency, from our own metrics. The batched call to OpenAI took about 1.8 seconds and ran after the message was already visible. The inline model runs in under a millisecond, before the message is sent.
Serving it behind a cache
A message passes through a wordlist, a cache, then the model. Repeats never reach the model. Every decision is logged.
Chat repeats a lot: "gg", "ez", the same copy-pasted ad. So the model sits behind two cheap layers.
First a small wordlist blocks the handful of terms that are never acceptable in any context. Then a cache. Every decision is stored for a day in Redis, keyed by the message and the model version. A repeat returns instantly and skips the model. About a third of live messages are repeats, so a third of traffic never touches the model at all. Keying the cache by model version means a new model starts with a clean, honest cache.
Cache hit rate climbing in the first hours. Each hit is a message the model never had to score.
Every decision, cached or not, is written to MongoDB. That log is both an audit trail and the training set for the next model.
What I'd tell you to steal
- Your moderation logs are a labeled dataset. Reversals are human verdicts. Join them.
- Log every decision from day one. Shadow mode plus full logging turned production into our best source of training data.
- Two disagreeing models beat one strong model. About $25 of labeling gave us clean data on 30,000 messages.
- A model's confirmed production mistakes are worth more than any synthetic data. Weight them accordingly.
- Check where the teacher's values differ from your team's. Then decide whose rules win.
- Do not trust a category label until the model earns it. Make "I don't know" a real option.
What's next
Scoring one message at a time has a ceiling. "how old r u" scores low, and it should, until you can see the messages around it. Reading a short window of conversation is the next step, mostly for grooming and other slow-building harm. The loop is already running, so the next model will train on this one's mistakes.
The stack: a character-level CNN in PyTorch exported to ONNX, served with FastAPI. Labeling ran on Fireworks AI with gpt-oss and Kimi K2. Redis caches decisions and MongoDB stores the logs. The model has no power to mute on its own; humans still make every call.