What rspamd's neural module is actually learning

If you have switched on rspamd’s neural module, watched it tick along for a quarter, and noticed that the network is contributing almost nothing useful — NEURAL_HAM firing on most of the inbound stream, never moving the score in any direction that matters — the explanation is usually not a bug. The module is doing what the config tells it to. The mental model in your head doesn’t match.

The thing to internalise first is what the network actually reads. The body of the message never reaches it. There is no tokeniser, no embedding, no representation of “what spam looks like” learned from bytes on the wire. The inputs are the firing pattern of every other enabled rspamd symbol for the message being scored: BAYES_SPAM, DKIM_REJECT, your noisy RBL_FOO, the MULTIMAP-derived greylist hint, the spamtrap signal, whatever else you have wired in. That floating-point vector goes through a small feedforward network — one hidden layer in stock, two if somebody tuned it, tanh activation, small enough to evaluate in microseconds on the same core that did the regex pass — and the network emits one scalar that becomes the contribution of one more symbol (NEURAL_SPAM or NEURAL_HAM) to the final score.

The library underneath is KANN , by Heng Li. The whole thing is a few thousand lines of C. If you have not read neural-network source before, it is an unusually approachable place to start.

Where the labels come from

The part that catches people: auto-train does not get its labels from an oracle. It uses the current rspamd score. Anything above spam_score (default 8.0) becomes a spam training example, anything below ham_score (default -2.0) becomes ham, the middle is dropped. The classifier is learning to predict the decisions of the engine it is supposed to be improving.

Once that clicks the operational behaviour stops being mysterious. Over a few thousand samples, the network converges to a learned weighted sum of whatever symbols are already moving the score. On most installations that means it converges, fairly quickly, to a smoothed approximation of BAYES_SPAM, because bayes is the loudest classifier in the chain. On one busy site I bothered to measure recently the trained network’s output correlated at 0.94 with BAYES_SPAM alone after a quarter of uninterrupted auto-training. At that point you could replace the whole module with NEURAL_SPAM = 0.4 * BAYES_SPAM and nobody would notice.

That is a feedback loop, not a coincidence. Borderline messages are scored slightly more confidently because neural is amplifying whichever symbol was already pushing them, those scores then qualify as training labels for the next cycle, the loop tightens, the network drifts further toward its existing dominant input. It is not pathology; it is what you get when the loss function is “predict your own outputs.”

None of which is a reason to turn it off. It is a reason to know what you have switched on, and to recognise the failure shape when you see it. The diagnostic for bayes mimicry is one config change and a test pass: set BAYES_SPAM weight to zero in local.d/groups.conf, rerun a representative corpus through the controller, look at what happens to neural’s discrimination. If it collapses with bayes, you have a learned bayes restatement. If it holds up, neural is finding signal somewhere else and earning its place.

There is a quieter failure that’s worth recognising. A message arrives during a partial config reload, or in the few seconds after ann_expire has kicked in and before the next training cycle has finished, with most of its symbols still unfired. Undefined symbols feed into the network as zero. Most trained networks have a small ham contribution on the all-zeros input, because the average ham message fires fewer symbols than the average spam one. The visible result is a population of messages with anomalously consistent NEURAL_HAM scores at roughly the same magnitude. It looks like a model bug. It’s a population artefact, and the cure is to skip training (and skip acting on neural’s score) for messages with fewer than some threshold of fired symbols. Ten works on the configs I have looked at.

Two more shapes worth knowing. Sharing one rule across inbound and outbound mail trains a network that is bad at both, because the distributions over symbols are too different — inbound never carries the authenticated-user signals, outbound never fires the inbound RBLs. One rule per direction, with distinct symbol_spam / symbol_ham names so they show up separately in /symbols. And a site behind aggressive upstream filtering (typical of a secondary MX behind something like Microsoft Defender) sees almost no real spam reach rspamd. The spam side of the auto-train buffer fills slowly, with low-confidence examples that barely cross the threshold, and the network sits there contributing nothing useful for months. The signature is NEURAL_SPAM rarely firing with meaningful weight, sustained over weeks. The fix is either feeding the trainer manually or admitting that neural is not earning its CPU on this site and disabling it.

Configuration, briefly

Trained networks live in Redis under keys prefixed rn_$rulename. A network’s identity is (rule name, symbol-set hash); the hash is computed at training time from the enabled symbols. The practical consequence is that any change to your symbol set silently invalidates the existing model and starts a retrain from zero on the next training cycle. If you deploy a new module on Monday morning, expect scoring to be a little noisy until the Tuesday training run completes. Worth knowing before someone files a ticket about it.

A minimum local.d/neural.conf:

enabled = true;
rules {
  default {
    train {
      max_trains     = 1000;
      max_usages     = 20;
      spam_score     = 8;
      ham_score      = -2;
      learning_rate  = 0.01;
      max_iterations = 25;
      train_prob     = 1.0;
    }
    symbol_spam = "NEURAL_SPAM";
    symbol_ham  = "NEURAL_HAM";
    ann_expire  = 86400;
  }
}

The two settings that matter most on auto-train are the labelling thresholds and train.train_prob. Default 8 / -2 is sensible; tightening to 12 / -4 reduces label noise at the cost of slower convergence. On high-traffic systems, leaving train_prob at 1.0 means your rolling window stops being a representative sample of recent traffic surprisingly fast — drop it to 0.1 or 0.05 there and the buffer reflects a longer slice.

Reading what it does

The fastest way to see what neural is contributing to a given message is the controller’s /symbols endpoint:

curl -s -X POST --data-binary @suspect.eml \
    -H "Password: $RSPAMD_PASSWORD" \
    -H "Content-Type: message/rfc822" \
    http://127.0.0.1:11334/symbols \
  | jq '.[] | {symbol, score, options}'

The useful question is whether neural is agreeing with bayes or disagreeing. Agreement adds nothing — if BAYES_SPAM is at +4 and NEURAL_SPAM is at +3 on the same message, neural is restating bayes with a learned multiplier. Disagreement is where the module either earns its place or wastes it. Those are the messages worth opening by hand.

You can quantify how often that happens. Hold out a week of messages, run them through with neural’s weights zeroed in groups.conf, run them again with weights restored, count the messages whose total crosses a classification boundary. The healthiest installation I have measured recently came out at a flip rate of 1.7%. The least healthy was 0.2%, of which roughly two thirds were flips in the wrong direction; after disabling neural, that site’s false-positive complaint count fell by about a tenth over the following quarter.

Training it manually

There isn’t a clean, version-stable CLI for “train this network from these mbox files.” What survives across versions is to stand up a separate rspamd instance, write a neural.conf with train_prob = 1.0 and the labelling thresholds tightened so the labels are unambiguous, and pipe a curated mailbox through it with rspamc scan. Once the buffer fills and the training cycle runs, the resulting network can be exported out of that Redis and imported into production, or both instances can write to a shared Redis if your topology allows. It’s not elegant. It’s what works.

The corpus doesn’t have to be enormous. A few thousand examples each side, labelled by somebody who has actually read the messages, will outperform months of auto-train on a busy production node. Spam shape drifts, so the corpus needs refreshing — quarterly is enough on the sites I have run this on; an annual refresh is enough to be visibly stale by the second half of the year.

When not to bother

If your symbol set is small (a handful of RBLs, bayes, not much else), the network has too little to combine and will mirror bayes. If your upstream filtering removes the unambiguous spam before rspamd sees it, the trainer never sees enough spam to learn from. And if nobody on the ops side is going to look at /symbols and check whether the module is doing useful work, you are paying CPU and Redis for a number nobody reads.

On the installations where it does earn its place — broad symbol set, real spam reaching the filter, somebody who watches what it does — it is a worthwhile addition. Just don’t enable it because the word “ML” came up in a meeting.

Further reading: the upstream docs at rspamd.com/doc/modules/neural.html are terse and mostly accurate. The KANN source is short and pleasant if you want to know what tanh actually compiles to. And src/plugins/lua/neural.lua in the rspamd tree is the integration layer; it is the canonical reference for the version you are actually running.