This is an old revision of the document!


OpenSMTPd, jails and amavisd-new for virus scanning and DKIM signing and spam classification using SpamAssassin and DSPAM

Installation

Required ports

Jails

  • mail (opensmtpd, dovecot)
  • scan (amavis, spamassassin, dspam, clamav)

Housekeeping

  1. Keep SpamAssassin rules up-to-date using sa-update (daily)
  2. Keep ClamAV signatures up-to-date using freshclam (hourly)
  3. Keep DSPAM database clean (daily)
  4. Process false-positives/-negatives (hourly)

Sub-component notes

Intended flow of mail from authenticated users to be relayed

  1. Receive on port 25/465/587 (smtp/smtps/submission) with authentication
  2. Classified as local when user is authenticated
  3. Forward to DKIM-signing service
  4. Receive from DKIM-signing service (tag DKIM-signed)
  5. Relay

Intended flow for mail for any of my domains

  1. Receive on port 25 (smtp)
  2. Domain of receiver is one of allowed domains
  3. Forward to AMaViS service
  4. Receive from AMaViS service (tag Scanned)
  5. Deliver to Dovecot using aliases db

All other mail must be bounced.

OpenSMTPd foot-shooting notes:

  1. accept/reject rules work in order
  2. accept/reject rules make sure you have a from and for check
  3. Ip-addresses of the host itself are seen as 'local'

Using Dovecot's LDA in OpenSMTPd provides the following benefits:

  1. Mailbox indexing during mail delivery, providing faster mailbox access later
  2. Quota enforcing (plugin)
  3. Sieve language support (plugin)

This setup adds mail/dovecot2-antispam-plugin to the configuration for easy (re-)classification of mails. This triggers copying of a message when copied/moved into or out of the Spam folder. The NotSpam and IsSpam have to be trained with DSPAM.

Listens on 10024 to scan mails
Listens on 10026 to DKIM-sign mails

  1. ClamAV
  2. DSPAM

Viruses can be rejected without spam classification/scanning.
First pass through DSPAM for bayesian classification (I've found it to be far superior to SpamAssassin's bayesian classifier) and then through SpamAssassin to allow SpamAssassin to take the X-DSPAM headers generated by dspam into account when scoring

All this results in the following headers:

X-DSPAM-Processed: Sun Jun  7 11:03:51 2015
X-DSPAM-Confidence: 0.9899
X-DSPAM-Probability: 0.0000
X-Virus-Scanned: amavisd-new at brnrd.eu
X-Spam-Flag: NO
X-Spam-Score: -2.618
X-Spam-Level:
X-Spam-Status: No, score=-2.618 tagged_above=-999 required=6.2
        tests=[DSPAM.Innocent=-1.000, DSPAM_HAM_99=-3.23,
        MSGID_FROM_MTA_HEADER=0.001, RDNS_DYNAMIC=0.363, TO_MALFORMED=1.247,
        UNPARSEABLE_RELAY=0.001] autolearn=disabled
X-DSPAM-Result: Innocent
X-DSPAM-Signature: 447408f7488584401825356

Generate a DKIM signing key using LibreSSL (defaults to 2048 bit)

$ /usr/local/bin/openssl genrsa /etc/ssl/priv/my.dkim.key

Once added to your amavisd config you can generate relevant DNS TXT records using

$ amavisd showkeys
default._domainkey.example.org.    3600 TXT (
  "v=DKIM1; p="
  "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3bcAwPnDM89FdJqWo8Om"
  "C4qfsfPIWIjqCnqp8H0QT03jAYwwzTwrKNb2w2dWjf9fp8D/Jm21XAT4GDhUWEQA"
  "XBxA2Ds5cHIPiSHRCstj7+kww1S40mQcS9WcUJjjO0jaGzH+Lfm02ZXC/E0i7YJY"
  "m9GKwPBz8M/8gMNCIbTuf9exVCmFW1gjnHMfQHOX0x8oBVLiby67vFl9UBwrIBSe"
  "+6T73t10fIWyTtkuFkJKNiqDnat8p0KMzpYwviBU8Aec1wTUFo9rK9bUgFJYdXUF"
  "q3gRN3glLlj4ho7yjZdqlofYO1CbdZ5FjyYs559jRd+dOUZh84Tmazg5PNx8Jeus"
  "MwIDAQAB")

Testing via any gmail account shows you if your DKIM setup is correct, look for the following in the smtp headers of a message that passed by gmail's service

Authentication-Results: mx.google.com;
    spf=neutral (google.com: 139.254.214.162 is neither permitted nor denied by best guess record for domain of brnrd@example.org) smtp.mail=brnrd@example.org;
    dkim=pass header.i=@example.org

NOTE: Documentation seems to be gone from the sourceforge.net project page

This setup creates a 'site-wide' configuration, all mail is classified and trained under the vscan user.

To use dspamc from amavisd you must make /usr/local/etc/dspam.conf readable for the vscan user so dspamc can read its ClientHost configuration.

Get a corpus of spam and ham to train dspam (in separate folders) as the vscan user.
dspam_train vscan /path/to/spamdir /path/to/hamdir

The Dovecot antispam-plugin must be configured to put mis-classified messages in /path/to/NotHam and /path/to/NotSpam folders. You'll need to create a script to feed these to dspam to update the database tokens

Download the dspam SpamAssassin module (dspam.pm) and ruleset (dspam.cf) and drop them in /usr/local/etc/mail/spamassassin
Add to your init.pre

loadplugin Mail::SpamAssassin::Plugin::dspam /usr/local/etc/mail/spamassassin/dspam.pm`

and to your local.cf

include dspam.cf

The Bayesian tests and auto-learning can be disabled as they are replaced with DSPAM.

Validate your configuration with spamassassin --lint, when there's problems, adding -D shows you debug output that helps find the root of the problem.

Example configurations

NOTE: These are not complete configurations but show the relevant parts that require modification!

# generate db using makemap
table aliases db:/usr/local/etc/mail/aliases.db
table domains { example.org, example.net, example.com }
table secrets db:/usr/local/etc/mail/secrets.db
 
lan_addr = "192.0.2.2"
scan_addr = "192.0.2.5"
 
# Define keys and certs (PEM encoded)
pki example.org certificate "/etc/ssl/certs/example.org.cer"
pki example.org key         "/etc/ssl/priv/example.org.key"
 
# Inbound mail smtp, smtps, deliver
listen on $lan_addr port  25 tls \
       pki example.org hostname example.org auth-optional
listen on $lan_addr port 465 smtps \
       pki example.org hostname example.org auth mask-source
listen on $lan_addr port 587 tls-require \
       pki example.org hostname example.org auth mask-source
# Receive scanned mails from amavisd-new
listen on $lan_addr port 10025 tag Scanned
# Receive DKIM signed mails from amavisd
listen on $lan_addr port 10027 tag DKIM-Signed
 
# Deliver locally messages coming back in from scanner
accept tagged Scanned from source $scan_addr \
       for domain <domains> alias <aliases> \
       deliver to lmtp "/var/run/dovecot/lmtp"
reject tagged Scanned
 
# Forward all mail received for local domains to amavis
accept from any \
       for domain <domains> \
       relay via "smtp://192.0.2.5:10024"
 
# Relay messages that are already signed
accept tagged DKIM-Signed from source $scan_addr \
       for any \
       relay
 
# Forward local deliver mails to DKIM signing service
accept from local \
       for any \
       relay via "smtp://192.0.2.5:10026"

Line 28: Use dovecot's LDA so it updates the indexes for the Maildir as well (otherwise indexing takes place when the user opens the mailbox) Line 44: There's no macro expansion in quoted strings, so you must use the IP-address or a valid hostname.

$MYHOME = '/var/amavis';
$pid_file = "/var/run/amavisd/amavisd.pid";
 
@inet_acl = qw(192.0.2.1/24);
 
$inet_socket_port = [10024,10026];
 
$notify_method  = 'smtp:[192.0.2.2]:10025';
$forward_method   = 'smtp:[192.0.2.2]:10025';
 
$enable_dkim_signing = 1; 
dkim_key('example.org', 'default', '/usr/local/etc/mail/example.org.dkim.key');
 
$interface_policy{'10026'} = 'ORIGINATING';
$policy_bank{'ORIGINATING'} = {
  forward_method => 'smtp:[192.0.2.2]:10027',
};
 
@spam_scanners = (
  ['DSPAM', 'Amavis::SpamControl::ExtProg', '/usr/local/bin/dspamc',
  [ qw(--mode=teft --deliver=stdout --user vscan) ],
    mail_body_size_limit => 256000, score_factor => 1,
  ],
  ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin' ],
);
 
@av_scanners = (
   ['ClamAV-clamd',
     \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.sock"],
     qr/\bOK$/m, qr/\bFOUND$/m,
     qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
);
Preference "signatureLocation=headers"

And make sure you have all storage backend params in order

use_bayes 0
 
loadplugin Mail::SpamAssassin::Plugin::dspam dspam.pm

Disable bayes (you can remove all lines referring to bayes as well)

Testing

telnet or netcat/nc work fine for me to test amavisd. Prepare a complete mail-transmission to paste

EHLO localhost
MAIL FROM: <someone@example.org>
RCPT TO: <noone@example.net>
DATA
<the original mail content including headers>
.
QUIT

Remember that amavisd will deliver the mail to OpenSMTPd on port 10025 tagged Scanned

You can test dspam using the dspamc client program:
dspamc --mode=notrain --deliver=stdout --user vscan < test-ham.mail
Simulation works best when you switch user to vscan before executing the command (tests if dspamc can read the client config)

Scripts

#!/bin/sh
 
isEmptyDir () { 
   # RC = 0 when current dir is empty
   for file in ./* ; do [ "$file" = './*' ] && return 0 || return 1 ; done 
}
 
trainIsSpam () {
   cd /path/to/notHam
   isEmptyDir && return 0
 
   # Mails that have been mis-classified
   for file in `grep -E -l "^X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
      /usr/local/bin/dspamc --user vscan --class=spam --source=error < "${file}"
      mv "${file}" /path/to/spamCorpus
   done
   isEmptyDir && return 0
 
   # Mails that have already been classified as Spam
   for file in `grep -E -l "X-DSPAM-Result: Spam" *` ; do
      rm "${file}"
   done
   isEmptyDir && return 0
 
   # Mails that were not previously classified
   for file in * ; do
      /usr/local/bin/dspamc --user vscan --class=spam --source=inoculation < "${file}"
      mv "${file}" /path/to/spamCorpus
   done
}
 
trainIsHam () {
   cd /path/to/notSpam
 
   isEmptyDir && return 0
 
   # Mails that have been mis-classified
   for file in `grep -E -l "^X-DSPAM-Result: Spam" *` ; do
      /usr/local/bin/dspamc --user vscan --class=innocent --source=error < "${file}"
      rm "${file}"
   done
   isEmptyDir && return 0
 
   # Mails that have already been classified as Spam
   for file in `grep -E -l "X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
      rm "${file}"
   done
   isEmptyDir && return 0
 
   # Mails that were not classified
   for file in * ; do
      /usr/local/bin/dspamc --user vscan --class=innocent --source=inoculation < "${file}"
      rm "${file}"
   done
}
 
trainIsSpam
trainIsHam
#!/bin/sh
 
PATH=$PATH:/usr/local/bin ; export PATH
 
result=`sa-update -v 2>&1`
RC=$?
 
# Output of sa-update is "Update finished, no fresh updates were available"
# when no new update is available, additionally RC=1
[ "$result" == "Update finished, no fresh updates were available" ] && exit 0
 
# If we reach this, we have a new ruleset
echo New ruleset: RC=$RC
 
# Generate some some text for your periodic/cron output so it sends you a mail
echo "$result" | grep "Update available for channel"
 
result=`sa-compile 2>&1`
 
# AMaViDS needs to be restarted to pick up the new rules
service amavisd restart
  • bernardspil/mailserver.1435501793.txt.gz
  • Last modified: 2015/06/28 14:29
  • by BernardSpil