%%OpenSMTPd%%, jails and amavisd-new for virus scanning and DKIM signing and spam classification using %%SpamAssassin%% and DSPAM

====== Installation ======

Required ports
  * **WIP** [[http://freshports.org/mail/spamd|mail/spamd]] Throttling
  * [[http://freshports.org/mail/opensmtpd|mail/opensmtpd]] %%OpenSMTPd%% MTA
  * [[https://github.com/Sp1l/ports/tree/master/mail/opensmtpd-extras|mail/opensmtpd-extras]] %%OpenSMTPd%% extra filters
  * [[http://freshports.org/security/amavisd-new|security/amavisd-new]] "A Mail and Virus Scanner"
  * [[http://freshports.org/mail/spamassassin|mail/spamassassin]] Apache %%SpamAssassin%% e-mail classifier
  * <del>[[http://freshports.org/mail/dspam|mail/dspam]] Bayesian e-mail classifier</del>
  * [[http://freshports.org/mail/rspamd|mail/rspamd]] rspamd e-mail classifier
  * [[http://freshports.org/security/clamav|security/clamav]] Open source virus scanner
  * [[http://eric.lubow.org/projects/dspam-spamassassin-plugin/|DSPAM !SpamAssassin]] plug-in
  * One of [[http://freshports.org/databases/mariadb100-client|MariaDB/MySQL]], [[http://freshports.org/databases/postgresql94-client|PostreSQL]], [[http://www.freshports.org/databases/sqlite3/|SQLite]] or Hash database 

Jails
  * mail (opensmtpd, dovecot)
  * scan (amavis, spamassassin, dspam, clamav)

====== Housekeeping ======

| Jail | What | When | Why |
| scan | [[#sa-update|sa-update]] | daily | Keep %%SpamAssassin%% rules up-to-date |
| scan | [[#freshclam.sh|freshclam]] | hourly | Keep %%ClamAV%% signatures up-to-date |
| scan%%|%%db | [[#dspam-purge|dspam-purge]] | daily | Remove old messages and signatures from dspam database |
| scan | [[#dspam-learn|train dspam]] | hourly | Process false-positives/-negatives |

====== Sub-component notes ======

===== Base OS =====

Disable sendmail

<code>
sendmail_enable="NONE"
</code>

===== spamd =====

Listens on 25 and proxies the inbound connections using black-/grey-/white-listing, tarpitting, etc.

Generates and maintains a hash-database of black-/grey-/whitelisted senders.

===== OpenSMTPd =====

Intended flow of mail from authenticated users to be relayed
  - Receive on port 25/465/587 (smtp/smtps/submission) with authentication
  - Classified as ''%%local%%'' when user is authenticated
  - DKIM-sign message
  - Relay

Intended flow for mail for any of my domains
  - Receive on port 25 (smtp)
  - Domain of receiver is one of allowed domains
  - Forward to %%AMaViS%% service
    - Bayesian classification using DSPAM b. Rule-based classification using %%SpamAssassin%% c. Virus scan using %%ClamAV%%
  - Receive from %%AMaViS%% service (tag Scanned)
  - Deliver to Dovecot using aliases db

All other mail must be bounced.

==== DKIM signing ====
Generate a 4096-bit DKIM signing key using [[/LibreSSL]] in the
location where the DKIM-signer filter expects it

<code>
$ /usr/local/bin/openssl genrsa -out /etc/ssl/private/rsa.private 4096
</code>

Once added to your amavisd config you can generate relevant DNS TXT records using

<code>
$ amavisd showkeys
default._domainkey.example.org.    3600 TXT (
  "v=DKIM1; p="
  "MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA3bcAwPnDM89FdJqWo8Om"
  "C4qfsfPIWIjqCnqp8H0QT03jAYwwzTwrKNb2w2dWjf9fp8D/Jm21XAT4GDhUWEQA"
  "XBxA2Ds5cHIPiSHRCstj7+kww1S40mQcS9WcUJjjO0jaGzH+Lfm02ZXC/E0i7YJY"
  "m9GKwPBz8M/8gMNCIbTuf9exVCmFW1gjnHMfQHOX0x8oBVLiby67vFl9UBwrIBSe"
  "+6T73t10fIWyTtkuFkJKNiqDnat8p0KMzpYwviBU8Aec1wTUFo9rK9bUgFJYdXUF"
  "q3gRN3glLlj4ho7yjZdqlofYO1CbdZ5FjyYs559jRd+dOUZh84Tmazg5PNx8Jeus"
  "MwIDAQAB")
</code>

Testing via any gmail account shows you if your DKIM setup is correct, look for the following in the smtp headers of a message that passed by gmail's service

<code>
Authentication-Results: mx.google.com;
    spf=neutral (google.com: 139.254.214.162 is neither permitted nor denied by best guess record for domain of brnrd@example.org) smtp.mail=brnrd@example.org;
    dkim=pass header.i=@example.org
</code>

==== Lessons learnt ====

%%OpenSMTPd%% foot-shooting notes:

  - ''%%accept%%''/''%%reject%%'' rules work **in order**
  - ''%%accept%%''/''%%reject%%'' rules make sure you have a ''%%from%%'' //and// ''%%for%%'' check
  - Ip-addresses of the host itself are seen as 'local'

===== Dovecot =====

Using Dovecot's LDA in %%OpenSMTPd%% provides the following benefits:
  - Mailbox indexing during mail delivery, providing faster mailbox access later
  - Quota enforcing (plugin)
  - Sieve language support (plugin)

This setup adds [[http://www.freshports.org/mail/dovecot2-antispam-plugin/|mail/dovecot2-antispam-plugin]] to the configuration for easy (re-)classification of mails. This triggers copying of a message when copied/moved into or out of the Spam folder. The %%NotSpam%% and %%IsSpam%% have to be trained with DSPAM.

===== AMaViSd =====
Listens on 10024 to scan mails \\ 

==== Scanning ====
  - %%ClamAV%%
  - <del>DSPAM</del>
  - rspamd
  - %%SpamAssassin%%

Viruses can be rejected without spam classification/scanning. \\ 
First pass through DSPAM for bayesian classification (I've found it to be **//far//** superior to %%SpamAssassin%%'s bayesian classifier) and then through %%SpamAssassin%% to allow %%SpamAssassin%% to take the ''%%X-DSPAM%%'' headers generated by dspam into account when scoring

All this results in the following headers:

<code>
X-DSPAM-Processed: Sun Jun  7 11:03:51 2015
X-DSPAM-Confidence: 0.9899
X-DSPAM-Probability: 0.0000
X-Virus-Scanned: amavisd-new at brnrd.eu
X-Spam-Flag: NO
X-Spam-Score: -2.618
X-Spam-Level:
X-Spam-Status: No, score=-2.618 tagged_above=-999 required=6.2
        tests=[DSPAM.Innocent=-1.000, DSPAM_HAM_99=-3.23,
        MSGID_FROM_MTA_HEADER=0.001, RDNS_DYNAMIC=0.363, TO_MALFORMED=1.247,
        UNPARSEABLE_RELAY=0.001] autolearn=disabled
X-DSPAM-Result: Innocent
X-DSPAM-Signature: 447408f7488584401825356
</code>

===== DSPAM =====

**NOTE:** Documentation seems to be gone from the sourceforge.net project page
**NOTE2:** I'm no longer using DSPAM, I've moved to rspamd (with spamassassin rules added)

This setup creates a 'site-wide' configuration, all mail is classified and trained under the ''%%vscan%%'' user.

To use ''%%dspamc%%'' from ''%%amavisd%%'' you must make ''%%/usr/local/etc/dspam.conf%%'' readable for the ''%%vscan%%'' user so ''%%dspamc%%'' can read its ''%%ClientHost%%'' configuration.

==== Training ====

Get a corpus of spam and ham to train dspam (in separate folders) as the ''%%vscan%%'' user.\\ 
''%%dspam_train vscan /path/to/spamdir /path/to/hamdir%%''

==== False positives/negatives ====

The Dovecot antispam-plugin must be configured to put mis-classified messages in /path/to/%%NotHam%% and /path/to/%%NotSpam%% folders.
You'll need to create a script to feed these to dspam to update the database tokens

===== rspamd =====

Yet to be documented

===== SpamAssassin =====

**NOTE:** Now integrated into my rspamd config

Download the [[https://github.com/elubow/dspam-spamassassin|dspam]] %%SpamAssassin%% module (''%%dspam.pm%%'') and ruleset (''%%dspam.cf%%'') and drop them in ''%%/usr/local/etc/mail/spamassassin%%''\\ 
Add to your init.pre

<code>
loadplugin Mail::SpamAssassin::Plugin::dspam /usr/local/etc/mail/spamassassin/dspam.pm`
</code>

and to your local.cf

<code>
include dspam.cf
</code>

The Bayesian tests and auto-learning can be disabled as they are replaced with DSPAM.

Validate your configuration with ''%%spamassassin --lint%%'', when there's problems, adding ''%%-D%%'' shows you debug output that helps find the root of the problem.

====== Example configurations ======

**NOTE:** These are not complete configurations but show the relevant parts that require modification!

===== etc/mail/smtpd.conf =====

<code aconf>
# generate db using makemap
table aliases db:/usr/local/etc/mail/aliases.db
table domains { example.org, example.net, example.com }
table secrets db:/usr/local/etc/mail/secrets.db

filter dkim-sign      dkim-signer "-Dexample.org"
filter dnsbl-sorbs    dnsbl       "-h dnsbl.sorbs.net"
filter dnsbl-spamcop  dnsbl       "-h bl.spamcop.net"
filter dnsbl-spamhaus dnsbl       "-h zen.spamhaus.org"
filter dnsbl-all      chain       dnsbl-sorbs dnsbl-spamcop dnsbl-spamhaus

lan_addr = "192.0.2.2"
scan_addr = "192.0.2.5"

# Define keys and certs (PEM encoded)
pki example.org certificate "/etc/ssl/certs/example.org.cer"
pki example.org key         "/etc/ssl/priv/example.org.key"

filter dkim-sign dkim-signer "-d example.org -p /usr/local/etc/mail/example.org.dkim.key"

# Inbound mail smtp, smtps, deliver
listen on $lan_addr port  25 filter dnsbl-all tls \
       pki example.org hostname example.org auth-optional
listen on $lan_addr port 465 smtps \
       pki example.org hostname example.org auth
listen on $lan_addr port 587 dkim-sign tls-require \
       pki example.org hostname example.org auth \
       filter dkim-sign
# Receive scanned mails from amavisd-new
listen on $lan_addr port 10025 tag Scanned

# Deliver locally messages coming back in from scanner
accept tagged Scanned from source $scan_addr \
       for domain <domains> alias <aliases> \
       deliver to lmtp "/var/run/dovecot/lmtp"
reject tagged Scanned

# Forward all mail received for local domains to amavis
accept from any \
       for domain <domains> \
       relay via "smtp://192.0.2.5:10024"

# Relay anything that came in from authenticated users
accept from local \
       for any \
       relay
# If you need to use a 'smarthost' use
#      relay via tls+auth://user@mta.example.net:587 <secrets>
</code>

Line 33: Use dovecot's LDA so it updates the indexes for the Maildir as well (otherwise indexing takes place when the user 'opens' the mailbox)\\ 
Line 39: There's no macro expansion in quoted strings, so you must use the IP-address or a valid hostname.

===== etc/amavisd.conf =====

<code aconf>
$MYHOME = '/var/amavis';
$pid_file = "/var/run/amavisd/amavisd.pid";

@inet_acl = qw(192.0.2.1/24);

$inet_socket_port = [10024];

$notify_method  = 'smtp:[192.0.2.2]:10025';
$forward_method   = 'smtp:[192.0.2.2]:10025';

@spam_scanners = (
  ['DSPAM', 'Amavis::SpamControl::ExtProg', '/usr/local/bin/dspamc',
  [ qw(--mode=teft --deliver=stdout --user vscan) ],
    mail_body_size_limit => 256000, score_factor => 1,
  ],
  ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin' ],
);

@av_scanners = (
   ['ClamAV-clamd',
     \&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.sock"],
     qr/\bOK$/m, qr/\bFOUND$/m,
     qr/^.*?: (?!Infected Archive)(.*) FOUND$/m ],
);
</code>

===== dspam =====

<code perl>
Preference "signatureLocation=headers"
</code>

And make sure you have all storage backend params in order

===== SpamAssassin =====

<code perl>
use_bayes 0

loadplugin Mail::SpamAssassin::Plugin::dspam dspam.pm
</code>

Disable bayes (you can remove all lines referring to bayes as well)

====== Testing ======

===== amavisd =====
''%%telnet%%'' or ''%%netcat%%''/''%%nc%%'' work fine for me to test amavisd. Prepare a complete mail-transmission to paste

<code>
EHLO localhost
MAIL FROM: <someone@example.org>
RCPT TO: <noone@example.net>
DATA
<the original mail content including headers>
.
QUIT
</code>

Remember that amavisd will deliver the mail to %%OpenSMTPd%% on port 10025 tagged Scanned

===== dspam =====
You can test dspam using the ''%%dspamc%%'' client program:\\ 
''%%dspamc --mode=notrain --deliver=stdout --user vscan < test-ham.mail%%''\\ 
Simulation works best when you switch user to ''%%vscan%%'' before executing the command (tests if ''%%dspamc%%'' can read the client config)

====== Scripts ======

===== dspam-learn =====

<code bash>
#!/bin/sh

notHamPath=/path/to/notHam
notSpamPath=/path/to/not/Spam
spamCorpusPath=/path/to/SpamCorpus

IFS="
"

extractHeader () {
headers=`grep -Eim3 '^(to|from|subject):' $1`

for header in $headers ; do
   case $header in
      [Ff][Rr][Oo][Mm]:* ) from="${header#[Ff][Rr][Oo][Mm]:}" ;;
      [Tt][Oo]:* ) to="${header#[Tt][Oo]:}" ;;
      [Ss][Uu][Bb][Jj]???:* ) subject="${header#[Ss][Uu][Bb][Jj][Ee][Cc][Tt]:}" ;;
   esac
done
echo -e "From:    ${from}\nTo:      ${to}\nSubject: ${subject}\n"
}

trainIsSpam () {
   cd "$notHamPath"
   [ "$(ls -A .)" ] || return

   # Mails that have been mis-classified
   for file in `grep -E -l "^X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
      extractHeader "${file}"
      /usr/local/bin/dspamc --user vscan --class=spam --source=error < "${file}"
      mv "${file}" "${spamCorpusPath}"
   done
   [ "$(ls -A .)" ] || return

   # Mails that have already been classified as Spam
   for file in `grep -E -l "X-DSPAM-Result: Spam" *` ; do
      rm "${file}"
   done
   [ "$(ls -A .)" ] || return

   # Mails that were not previously classified
   for file in * ; do
      extractHeader "${file}"
      /usr/local/bin/dspamc --user vscan --class=spam --source=inoculation < "${file}"
      mv "${file}" "${spamCorpusPath}"
   done
}

trainIsHam () {
   cd "$notSpamPath"
   [ "$(ls -A .)" ] || return

   # Mails that have been mis-classified
   for file in `grep -E -l "^X-DSPAM-Result: Spam" *` ; do
      extractHeader "${file}"
      /usr/local/bin/dspamc --user vscan --class=innocent --source=error < "${file}"
      rm "${file}"
   done
   [ "$(ls -A .)" ] || return

   # Mails that have already been classified as Ham
   for file in `grep -E -l "X-DSPAM-Result: (Whitelisted|Innocent)" *` ; do
      rm "${file}"
   done
   [ "$(ls -A .)" ] || return

   # Mails that were not classified
   for file in * ; do
      extractHeader "${file}"
      /usr/local/bin/dspamc --user vscan --class=innocent --source=inoculation < "${file}"
      rm "${file}"
   done
}

trainIsSpam
trainIsHam
</code>

===== sa-update =====

<code bash>
#!/bin/sh

PATH=$PATH:/usr/local/bin ; export PATH

result=`sa-update -v 2>&1`
RC=$?

# Output of sa-update is "Update finished, no fresh updates were available"
# when no new update is available, additionally RC=1
[ "$result" == "Update finished, no fresh updates were available" ] && exit 0

# If we reach this, we have a new ruleset
echo New ruleset: RC=$RC

# Generate some some text for your periodic/cron output so it sends you a mail
echo "$result" | grep "Update available for channel"

result=`sa-compile 2>&1`

# AMaViDS needs to be restarted to pick up the new rules
service amavisd restart
</code>

===== dspam-purge =====
Run daily

<code bash>
#!/bin/sh
mysql -u <dspamuser> -p <dspampass> -h db.host <dspamdb> < /usr/local/share/examples/dspam/mysql/purge.sql
</code>

===== freshclam.sh =====
Run every hour, main purpose is to silence cron mails

<code bash>
#!/bin/sh

result=`/usr/local/bin/freshclam`
RC=$?

# All databases are up to date, exit
[ `echo "$result" | grep -c "is up to date"` -eq 3 ] && exit 0

# DB Updated, Clamd notified, no errors, exit
[ "$result" == "${result%Database updated (*) from *}" ] && DBupdated=yes
[ "$result" == "${result%Clamd successfully notified about the update.}" ] && clamdNotified=yes
[ -n "$DBupdated" -a -n "$clamdNotified" -a $RC -eq 0 ] && exit 0

# If we reach this, we have a new ruleset and something non-default happened
echo ClamAV updates: RC=$RC

# Generate some some text for your periodic/cron output so it sends you a mail
echo "$result"
</code>

https://paste.xinu.at/ypY62/

----

{{tag>HowTo}}