OpenSMTPd, jails and amavisd-new for virus scanning and DKIM signing and spam classification using SpamAssassin and DSPAM

Installation

Required ports

WIP mail/spamd Throttling
mail/opensmtpd OpenSMTPd MTA
mail/opensmtpd-extras OpenSMTPd extra filters
security/amavisd-new “A Mail and Virus Scanner”
mail/spamassassin Apache SpamAssassin e-mail classifier
~~mail/dspam Bayesian e-mail classifier~~
mail/rspamd rspamd e-mail classifier
security/clamav Open source virus scanner
DSPAM !SpamAssassin plug-in
One of MariaDB/MySQL, PostreSQL, SQLite or Hash database

Jails

mail (opensmtpd, dovecot)
scan (amavis, spamassassin, dspam, clamav)

Housekeeping

Jail	What	When	Why
scan	sa-update	daily	Keep SpamAssassin rules up-to-date
scan	freshclam	hourly	Keep ClamAV signatures up-to-date
scan\|db	dspam-purge	daily	Remove old messages and signatures from dspam database
scan	train dspam	hourly	Process false-positives/-negatives

Sub-component notes

Base OS

Disable sendmail

sendmail_enable="NONE"

spamd

Listens on 25 and proxies the inbound connections using black-/grey-/white-listing, tarpitting, etc.

Generates and maintains a hash-database of black-/grey-/whitelisted senders.

OpenSMTPd

Intended flow of mail from authenticated users to be relayed

Receive on port 25/465/587 (smtp/smtps/submission) with authentication
Classified as local when user is authenticated
DKIM-sign message
Relay

Intended flow for mail for any of my domains

Receive on port 25 (smtp)
Domain of receiver is one of allowed domains
Forward to AMaViS service
1. Bayesian classification using DSPAM b. Rule-based classification using SpamAssassin c. Virus scan using ClamAV
Receive from AMaViS service (tag Scanned)
Deliver to Dovecot using aliases db

All other mail must be bounced.

DKIM signing

Generate a 4096-bit DKIM signing key using LibreSSL in the location where the DKIM-signer filter expects it

$ /usr/local/bin/openssl genrsa -out /etc/ssl/private/rsa.private 4096

Once added to your amavisd config you can generate relevant DNS TXT records using

$ amavisd showkeys
default._domainkey.example.org.    3600 TXT (
  "v=DKIM1; p="
  "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3bcAwPnDM89FdJqWo8Om"
  "C4qfsfPIWIjqCnqp8H0QT03jAYwwzTwrKNb2w2dWjf9fp8D/Jm21XAT4GDhUWEQA"
  "XBxA2Ds5cHIPiSHRCstj7+kww1S40mQcS9WcUJjjO0jaGzH+Lfm02ZXC/E0i7YJY"
  "m9GKwPBz8M/8gMNCIbTuf9exVCmFW1gjnHMfQHOX0x8oBVLiby67vFl9UBwrIBSe"
  "+6T73t10fIWyTtkuFkJKNiqDnat8p0KMzpYwviBU8Aec1wTUFo9rK9bUgFJYdXUF"
  "q3gRN3glLlj4ho7yjZdqlofYO1CbdZ5FjyYs559jRd+dOUZh84Tmazg5PNx8Jeus"
  "MwIDAQAB")

Testing via any gmail account shows you if your DKIM setup is correct, look for the following in the smtp headers of a message that passed by gmail's service

Authentication-Results: mx.google.com;
    spf=neutral (google.com: 139.254.214.162 is neither permitted nor denied by best guess record for domain of brnrd@example.org) smtp.mail=brnrd@example.org;
    dkim=pass header.i=@example.org

Lessons learnt

OpenSMTPd foot-shooting notes:

accept/reject rules work in order
accept/reject rules make sure you have a from and for check
Ip-addresses of the host itself are seen as 'local'

Dovecot

Using Dovecot's LDA in OpenSMTPd provides the following benefits:

Mailbox indexing during mail delivery, providing faster mailbox access later
Quota enforcing (plugin)
Sieve language support (plugin)

This setup adds mail/dovecot2-antispam-plugin to the configuration for easy (re-)classification of mails. This triggers copying of a message when copied/moved into or out of the Spam folder. The NotSpam and IsSpam have to be trained with DSPAM.

AMaViSd

Listens on 10024 to scan mails

Scanning

ClamAV
~~DSPAM~~
rspamd
SpamAssassin

Viruses can be rejected without spam classification/scanning.
First pass through DSPAM for bayesian classification (I've found it to be far superior to SpamAssassin's bayesian classifier) and then through SpamAssassin to allow SpamAssassin to take the X-DSPAM headers generated by dspam into account when scoring

All this results in the following headers:

X-DSPAM-Processed: Sun Jun  7 11:03:51 2015
X-DSPAM-Confidence: 0.9899
X-DSPAM-Probability: 0.0000
X-Virus-Scanned: amavisd-new at brnrd.eu
X-Spam-Flag: NO
X-Spam-Score: -2.618
X-Spam-Level:
X-Spam-Status: No, score=-2.618 tagged_above=-999 required=6.2
        tests=[DSPAM.Innocent=-1.000, DSPAM_HAM_99=-3.23,
        MSGID_FROM_MTA_HEADER=0.001, RDNS_DYNAMIC=0.363, TO_MALFORMED=1.247,
        UNPARSEABLE_RELAY=0.001] autolearn=disabled
X-DSPAM-Result: Innocent
X-DSPAM-Signature: 447408f7488584401825356

DSPAM

NOTE: Documentation seems to be gone from the sourceforge.net project page NOTE2: I'm no longer using DSPAM, I've moved to rspamd (with spamassassin rules added)

This setup creates a 'site-wide' configuration, all mail is classified and trained under the vscan user.

To use dspamc from amavisd you must make /usr/local/etc/dspam.conf readable for the vscan user so dspamc can read its ClientHost configuration.

Training

Get a corpus of spam and ham to train dspam (in separate folders) as the vscan user.
dspam_train vscan /path/to/spamdir /path/to/hamdir

False positives/negatives

The Dovecot antispam-plugin must be configured to put mis-classified messages in /path/to/NotHam and /path/to/NotSpam folders. You'll need to create a script to feed these to dspam to update the database tokens

rspamd

Yet to be documented

SpamAssassin

NOTE: Now integrated into my rspamd config

Download the dspam SpamAssassin module (dspam.pm) and ruleset (dspam.cf) and drop them in /usr/local/etc/mail/spamassassin
Add to your init.pre

loadplugin Mail::SpamAssassin::Plugin::dspam /usr/local/etc/mail/spamassassin/dspam.pm`

and to your local.cf

include dspam.cf

The Bayesian tests and auto-learning can be disabled as they are replaced with DSPAM.

Validate your configuration with spamassassin --lint, when there's problems, adding -D shows you debug output that helps find the root of the problem.

Example configurations

NOTE: These are not complete configurations but show the relevant parts that require modification!

etc/mail/smtpd.conf

# generate db using makemap
table aliases db:/usr/local/etc/mail/aliases.db
table domains { example.org, example.net, example.com }
table secrets db:/usr/local/etc/mail/secrets.db
 
filter dkim-sign      dkim-signer "-Dexample.org"
filter dnsbl-sorbs    dnsbl       "-h dnsbl.sorbs.net"
filter dnsbl-spamcop  dnsbl       "-h bl.spamcop.net"
filter dnsbl-spamhaus dnsbl       "-h zen.spamhaus.org"
filter dnsbl-all      chain       dnsbl-sorbs dnsbl-spamcop dnsbl-spamhaus
 
lan_addr = "192.0.2.2"
scan_addr = "192.0.2.5"
 
# Define keys and certs (PEM encoded)
pki example.org certificate "/etc/ssl/certs/example.org.cer"
pki example.org key         "/etc/ssl/priv/example.org.key"
 
filter dkim-sign dkim-signer "-d example.org -p /usr/local/etc/mail/example.org.dkim.key"
 
# Inbound mail smtp, smtps, deliver
listen on $lan_addr port  25 filter dnsbl-all tls \
       pki example.org hostname example.org auth-optional
listen on $lan_addr port 465 smtps \
       pki example.org hostname example.org auth
listen on $lan_addr port 587 dkim-sign tls-require \
       pki example.org hostname example.org auth \
       filter dkim-sign
# Receive scanned mails from amavisd-new
listen on $lan_addr port 10025 tag Scanned
 
# Deliver locally messages coming back in from scanner
accept tagged Scanned from source $scan_addr \
       for domain <domains> alias <aliases> \
       deliver to lmtp "/var/run/dovecot/lmtp"
reject tagged Scanned
 
# Forward all mail received for local domains to amavis
accept from any \
       for domain <domains> \
       relay via "smtp://192.0.2.5:10024"
 
# Relay anything that came in from authenticated users
accept from local \
       for any \
       relay
# If you need to use a 'smarthost' use
#      relay via tls+auth://user@mta.example.net:587 <secrets>

Line 33: Use dovecot's LDA so it updates the indexes for the Maildir as well (otherwise indexing takes place when the user 'opens' the mailbox)
Line 39: There's no macro expansion in quoted strings, so you must use the IP-address or a valid hostname.

etc/amavisd.conf

$MYHOME = '/var/amavis';
$pid_file = "/var/run/amavisd/amavisd.pid";
 
@inet_acl = qw(192.0.2.1/24);
 
$inet_socket_port = [10024];
 
$notify_method  = 'smtp:[192.0.2.2]:10025';
$forward_method   = 'smtp:[192.0.2.2]:10025';
 
@spam_scanners = (
  ['DSPAM', 'Amavis::SpamControl::ExtProg', '/usr/local/bin/dspamc',
  [ qw(--mode=teft --deliver=stdout --user vscan) ],
    mail_body_size_limit => 256000, score_factor => 1,
  ],
  ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin' ],
);
 
@av_scanners = (
   ['ClamAV-clamd',
     \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.sock"],
     qr/\bOK$/m, qr/\bFOUND$/m,
     qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
);

dspam

Preference "signatureLocation=headers"

And make sure you have all storage backend params in order

SpamAssassin

use_bayes 0
 
loadplugin Mail::SpamAssassin::Plugin::dspam dspam.pm

Disable bayes (you can remove all lines referring to bayes as well)

Testing

amavisd

telnet or netcat/nc work fine for me to test amavisd. Prepare a complete mail-transmission to paste

EHLO localhost
MAIL FROM: <someone@example.org>
RCPT TO: <noone@example.net>
DATA
<the original mail content including headers>
.
QUIT

Remember that amavisd will deliver the mail to OpenSMTPd on port 10025 tagged Scanned

dspam

You can test dspam using the dspamc client program:
dspamc --mode=notrain --deliver=stdout --user vscan < test-ham.mail
Simulation works best when you switch user to vscan before executing the command (tests if dspamc can read the client config)

Scripts

dspam-learn

#!/bin/sh
 
notHamPath=/path/to/notHam
notSpamPath=/path/to/not/Spam
spamCorpusPath=/path/to/SpamCorpus
 
IFS="
"
 
extractHeader () {
headers=`grep -Eim3 '^(to|from|subject):' $1`
 
for header in $headers ; do
   case $header in
      [Ff][Rr][Oo][Mm]:* ) from="${header#[Ff][Rr][Oo][Mm]:}" ;;
      [Tt][Oo]:* ) to="${header#[Tt][Oo]:}" ;;
      [Ss][Uu][Bb][Jj]???:* ) subject="${header#[Ss][Uu][Bb][Jj][Ee][Cc][Tt]:}" ;;
   esac
done
echo -e "From:    ${from}\nTo:      ${to}\nSubject: ${subject}\n"
}
 
trainIsSpam () {
   cd "$notHamPath"
   [ "$(ls -A .)" ] || return
 
   # Mails that have been mis-classified
   for file in `grep -E -l "^X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
      extractHeader "${file}"
      /usr/local/bin/dspamc --user vscan --class=spam --source=error < "${file}"
      mv "${file}" "${spamCorpusPath}"
   done
   [ "$(ls -A .)" ] || return
 
   # Mails that have already been classified as Spam
   for file in `grep -E -l "X-DSPAM-Result: Spam" *` ; do
      rm "${file}"
   done
   [ "$(ls -A .)" ] || return
 
   # Mails that were not previously classified
   for file in * ; do
      extractHeader "${file}"
      /usr/local/bin/dspamc --user vscan --class=spam --source=inoculation < "${file}"
      mv "${file}" "${spamCorpusPath}"
   done
}
 
trainIsHam () {
   cd "$notSpamPath"
   [ "$(ls -A .)" ] || return
 
   # Mails that have been mis-classified
   for file in `grep -E -l "^X-DSPAM-Result: Spam" *` ; do
      extractHeader "${file}"
      /usr/local/bin/dspamc --user vscan --class=innocent --source=error < "${file}"
      rm "${file}"
   done
   [ "$(ls -A .)" ] || return
 
   # Mails that have already been classified as Ham
   for file in `grep -E -l "X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
      rm "${file}"
   done
   [ "$(ls -A .)" ] || return
 
   # Mails that were not classified
   for file in * ; do
      extractHeader "${file}"
      /usr/local/bin/dspamc --user vscan --class=innocent --source=inoculation < "${file}"
      rm "${file}"
   done
}
 
trainIsSpam
trainIsHam

sa-update

#!/bin/sh
 
PATH=$PATH:/usr/local/bin ; export PATH
 
result=`sa-update -v 2>&1`
RC=$?
 
# Output of sa-update is "Update finished, no fresh updates were available"
# when no new update is available, additionally RC=1
[ "$result" == "Update finished, no fresh updates were available" ] && exit 0
 
# If we reach this, we have a new ruleset
echo New ruleset: RC=$RC
 
# Generate some some text for your periodic/cron output so it sends you a mail
echo "$result" | grep "Update available for channel"
 
result=`sa-compile 2>&1`
 
# AMaViDS needs to be restarted to pick up the new rules
service amavisd restart

dspam-purge

Run daily

#!/bin/sh
mysql -u <dspamuser> -p <dspampass> -h db.host <dspamdb> < /usr/local/share/examples/dspam/mysql/purge.sql

freshclam.sh

Run every hour, main purpose is to silence cron mails

#!/bin/sh
 
result=`/usr/local/bin/freshclam`
RC=$?
 
# All databases are up to date, exit
[ `echo "$result" | grep -c "is up to date"` -eq 3 ] && exit 0
 
# DB Updated, Clamd notified, no errors, exit
[ "$result" == "${result%Database updated (*) from *}" ] && DBupdated=yes
[ "$result" == "${result%Clamd successfully notified about the update.}" ] && clamdNotified=yes
[ -n "$DBupdated" -a -n "$clamdNotified" -a $RC -eq 0 ] && exit 0
 
# If we reach this, we have a new ruleset and something non-default happened
echo ClamAV updates: RC=$RC
 
# Generate some some text for your periodic/cron output so it sends you a mail
echo "$result"

https://paste.xinu.at/ypY62/

HowTo

Table of Contents

Installation

Housekeeping

Sub-component notes

Base OS

spamd

OpenSMTPd

DKIM signing

Lessons learnt

Dovecot

AMaViSd

Scanning

DSPAM

Training

False positives/negatives

rspamd

SpamAssassin

Example configurations

etc/mail/smtpd.conf

etc/amavisd.conf

dspam

SpamAssassin

Testing

amavisd

dspam

Scripts

dspam-learn

sa-update

dspam-purge

freshclam.sh