Zarafa on Debian how to part 2: SASL and autolearning for spamassassin

This is part two of a two part article describing how to set up Zarafa on Debian using Postfix. The first part of the article covered the essentials of spam and virus scanning for incoming mail via spamassassin and clamassassin. This part covers secure user authentication for sending mail (SASL) and training spamassassin to be better at recognising spam. Both methods rely on unencrypted IMAP, so you need to make sure that you've enabled IMAP access in zarafa-gateway. Since unencrypted IMAP is insecure I suggest blocking it at the firewall level and only exposing IMAP-SSL to your external users.

Setting up SASL

Let's start with SASL. While it's not needed for anyone who's accessing Zarafa via ActiveSync, the web frontend, or Zarafa's Outlook compatibility layer, it still has its place. It's essential to anyone who wants to use another mail client such as Thunderbird, Opera, Evolution, SeaMonkey, etc, or a non-ActiveSync phone. Many phones support email but not ActiveSync, and for some unknown reason Microsoft apparently designed ActiveSync so that it's impossible for one phone to use it for more than one account, so even people with ActiveSync capable phones might not be able to use it to access Zarafa if they have another ActiveSync account already active. While there is a mobile web interface for Zarafa, it is fairly basic, and users may well find IMAP and SASL easier to use.

Configure Cyrus SASL

Anyway, back to the setup guide... To give credit where it's due, the original instructions for configuring SASL to work with Zarafa appeared on the Zarafa forums thanks to users unixsmeden and godber. Before beginning you'll need to install the packages sasl2-bin and libsasl2-modules, which contain the necessary program files for Cyrus SASL. You then need to create the file /etc/postfix/sasl/smtpd.conf as follows: pwcheck_method: saslauthd mech_list: plain login Next, add postfix to the sasl system group by running the command gpasswd -a postfix sasl as root. Then Cyrus SASL must be configured via /etc/default/saslauthd. Here are the lines I've got in my configuration (with comments removed): START=yes DESC="SASL Authentication Daemon" NAME="saslauthd" MECHANISMS="rimap" MECH_OPTIONS="127.0.0.1" THREADS=0 OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd -r"

Configure Postfix

The final step is to configure Postfix to use Cyrus SASL to provide authentication. To do this you need to add the following options to /etc/postfix/main.cf:
smtpd_tls_auth_only = yes
smtpd_tls_security_level = may
smtpd_tls_key_file = /etc/ssl/private/filename.key
smtpd_tls_cert_file = /etc/ssl/certs/filename.crt
smtpd_tls_session_cache_database = sdbm:/var/lib/postfix/smtpd_scache
smtpd_tls_received_header = yes
smtpd_tls_session_cache_timeout = 3600s
tls_random_source = dev:/dev/urandom

smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination
smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated

smtpd_sasl_auth_enable = yes
broken_sasl_auth_clients = yes
The first of these options means that Postfix only offers authentication over TLS connections. By default SMTP connections are unencrypted, meaning that login details are sent in the clear and could potentially be intercepted. By only offering SMTP AUTH on TLS-encrypted connections this risk is avoided. The next few lines ensure that TLS is available - note that you will need an encryption key and certificate for the smtpd_tls_key_file and smtpd_tls_cert_file options. These can be self-signed but mail sending programs will probably give warnings about them, so to reassure your users it may be simpler to get an SSL certificate. Since there are many guides on creating and installing SSL certificates already online I won't go over it again here. The following two lines tell Postfix to allow users who have authenticated via SASL to send mail. Normally Postfix will only allow mail to be sent from trusted hosts, which are specified in the mynetworks option. The final two lines allow SASL authentication to be used and advertises it in such a way as to work around bugs in some mail clients (older versions of Microsoft Outlook are the most common, although there may well be others). Note that these instructions will only enable SASL on port 25, i.e. the default mail port. If you also want to offer SASL on port 465 (SMTP-SSL) and 587 (SMTP submission), which are used by some clients, you'll need to add the following sections to /etc/postfix/master.cf:
submission inet n       -       -       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o broken_sasl_auth_clients=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject_unauth_destination
  -o smtpd_sender_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       -       -       -       smtpd
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o broken_sasl_auth_clients=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject_unauth_destination
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject_unauth_destination
  -o smtpd_sender_restrictions=permit_sasl_authenticated,reject_unauth_destination
  -o milter_macro_daemon_name=ORIGINATING
Once the changes have been made, reload the Postfix configuration with /etc/init.d/postfix reload and try sending mail using your favourite client.

Setting up automatic spamassassin training

Now that SASL is working, we can go on to setting up automated training of spamassassin. To do this we'll use a Perl script by David Zendzian. You'll need to create a Zarafa user that the script can log in with, which should not be used for any other purpose (actually, it can be used as normal, but if you change the account password then you'll also need to change the script). Create a public folder called Spam, another one called "Non-spam", and grant all users, including the spamassassin training user, read/write access to both. Once that's done you're ready to set up the automated learning process. You'll need to have the packages perl and libmail-imapclient-perl installed. Here's my version of David Zendzian's script:
#!/usr/bin/perl
#
# Process mail from imap server shared folder 'Public folders/Spam' & 'Public folders/Non-spam'
# through spamassassin sa-learn
# Original by dmz [at] dmzs.com - March 19, 2004
# Modified by Pete Donnell, Kitson Consulting Ltd, March 5th 2010 - processed Non-spam messages are not deleted any more
# http://www.dmzs.com/tools/files/spam.phtml
# http://www.dmzs.com/tools/files/spam/DMZS-sa-learn.pl [modified for SMEServer]
# LGPL

use Mail::IMAPClient;

my $debug=0;
my $salearn;

# # # # # # # # # # EDIT USER AND PASSWORD  # # # # # # # # # #

my $imap = Mail::IMAPClient->new( Server=> '127.0.0.1:143',
                                  User => 'username',
                                  Password => 'password',
                                  Debug => $debug);

if (!defined($imap)) { die "IMAP Login Failed"; }

# If debugging, print out the total counts for each mailbox
if ($debug) {
 my $spamcount = $imap->message_count('Public folders/Spam');
 print $spamcount, " Spam to process\n";

 my $nonspamcount = $imap->message_count('Public folders/Non-spam');
 print $nonspamcount, " Non-spam to process\n" if $debug;
}

# Process the spam mailbox
$imap->select('Public folders/Spam');
my @msgs = $imap->search("ALL");
for (my $i=0;$i <= $#msgs; $i++)
{
 # I put it into a file for processing, doing it into a perl var & piping through sa-learn just didn't seem to work
 $imap->message_to_file("/tmp/salearn",$msgs[$i]);

 # execute sa-learn w/data
 if ($debug) { $salearn = `/usr/bin/sa-learn -D --no-sync  --spam /tmp/salearn`; } 
 else { $salearn = `/usr/bin/sa-learn --no-sync  --spam /tmp/salearn`; }
 print "-------\nSpam: ",$salearn,"\n-------\n" if $debug;

 # delete processed message
 $imap->delete_message($msgs[$i]);
 unlink("/tmp/salearn");
}
$imap->expunge();
$imap->close();

# Process the not-spam mailbox
$imap->select('Public folders/Non-spam');
my @msgs = $imap->search("ALL");
for (my $i=0;$i <= $#msgs; $i++)
{
 $imap->message_to_file("/tmp/salearn",$msgs[$i]);
 # execute sa-learn w/data
 if ($debug) { $salearn = `/usr/bin/sa-learn -D --no-sync  --ham /tmp/salearn`; }
 else { $salearn = `/usr/bin/sa-learn --no-sync  --ham /tmp/salearn`; }
 print "-------\nNotSpam: ",$salearn,"\n-------\n" if $debug; 

 unlink("/tmp/salearn");
}
$imap->close();

$imap->logout();

# integrate learned stuff
my $sarebuild = `/usr/bin/sa-learn --sync`;
print "-------\nRebuild: ",$sarebuild,"\n-------\n" if $debug;
You can test it by changing the script so that debug=1 and the username and password match the account that you set up earlier. Then make the script executable (chmod +x <filename>), run it from the command line and check the output. If it works without errors then you're all good to set it to run regularly as a cron job. To do this, copy the script to a suitable location (not forgetting to set debug back to 0 before you do!), e.g. /usr/local/bin/spamassassin_imap_learn.pl and create a file called /etc/cron.daily/train-spamassassin containing the following: #!/bin/sh nice /usr/local/bin/spamassassin_imap_learn.pl > /dev/null 2>&1 This file should also be executable. Obviously, if you'd like to run it hourly or weekly instead of daily then put it in /etc/cron.hourly or /etc/cron.weekly as you prefer. That's all there is to it. You now have secure SASL authentication enabled, and spamassassin can learn from its mistakes. You'll need to instruct your users to put any spam messages they receive in the public Spam folder, and any genuine messages that are mistakenly classed as spam in the Non-spam folder. Note that unlike David Zendzian's original version of the script, my version does not delete messages from the Non-spam folder, so users can retrieve them once the learning has taken place. There is obviously a privacy issue there, but for small setups it hopefully won't be a problem. As I said at the beginning of the first part of this article, I don't recommend this guide for large setups! As always, feedback and other comments are appreciated. Due to the amount of spam that this site receives comments are moderated by hand, so there will be a delay between submission of comments and their appearance on the site. We're very sorry about this!

Comments

Hi guys,

Good stuff here, I looked for something like that, since I am having a hard time setting up zarafa, postfix and spamassassin + avira mailgate. I got them working partly thanks to the first part of this article. Thumbs up!

But now with SASL I get the following warnings in /var/log/mail.log:

Oct 8 13:39:38 zarafa postfix/smtpd[6689]: warning: xsasl_cyrus_server_get_mechanism_list: no applicable SASL mechanisms
Oct 8 13:39:38 zarafa postfix/smtpd[6689]: fatal: no SASL authentication mechanisms
Oct 8 13:39:39 zarafa postfix/master[4054]: warning: process /usr/lib/postfix/smtpd pid 6689 exit status 1
Oct 8 13:39:39 zarafa postfix/master[4054]: warning: /usr/lib/postfix/smtpd: bad command startup -- throttling

What can I do about that? Is a complete cyrus (imapd) installation required?

I tried

apt-get install cyrus-sasl-*

since I couldn't find cyrus-sasl-plain and/or cyrus-sasl-sql they don't seem to be available in debian(?)

but I got the following error:
dpkg: unrecoverable fatal error, aborting:
syntax error: unknown user `cyrus' in statoverride file

so there is a user "cyrus" missing. Any hints on the mentioned problems? What about the requirements for SASL and spamassassin autolearning?

Thanks for your help in advance.

cheers,
Marcus

Hi Marcus,

It took me a while to figure out how to get SASL working too, it's quite tricky. You don't need to install another imapd, Zarafa provides IMAP and POP via the zarafa-gateway package. The first thing to double check is that imap is enabled in /etc/zarafa/gateway.cfg - look for the lines imap_enable = true and imap_port = 143. Cyrus saslauthd doesn't support secure IMAP when using the rimap method, hence the note in the article about blocking unsecured IMAP at the firewall level. If you don't, then users might inadvertently be sending their logins across the internet in plain text.

On one of test boxes I have the packages libsasl2-2, libsasl2-modules and sasl2-bin. Settings are exactly as described above and I've since also successfully adapted them to run on CentOS, so I can only suggest double checking that you haven't missed a step along the way. I think that by default saslauthd doesn't start, so you may need to run /etc/init.d/saslauthd start once you've changed the configuration settings as per the article.

Is there any further information in the other mail logs, e.g. mail.info, mail.warn or mail.err? They do seem to mostly duplicate the same information, but there might be more info in there.

As for spamassasin, there shouldn't be any further dependencies. Have you tried the debug method suggested, and if so what happened, did you get any error messages?

Hope that helps...

Hello Pete,

thanks a lot for your reply! Auto-learning works so far, but I'd like to get more privacy for our users. Therefore public folders cannot be used, since everybody can look into them. So I setup to new accounts (spam/non-spam) and wanted to use them with your perl script. I set them to be seen and writeable for everyone, yet nobody (except the admin) can look into the contents of the folders. Now I'd like to know how I access the folders with the user spam, I mean "his" own folder and the folder of the user nonspam. Of course I allow "spam" to access the folders of "non-spam" in zarafa.

Loggin in wasn't a problem, but I get the first error while reading
"Read: 2 NO STATUS error finding folder" (all debug output at the end of this post)

I changed
my $spamcount = $imap->message_count('Public Folders/Spam');
to
my $spamcount = $imap->message_count('spam');

and
my $nonspamcount = $imap->message_count('Public Folders/Non-Spam');
to
my $nonspamcount = $imap->message_count('non-spam');

and
$imap->select('Public Folders/Spam');
to
$imap->select('spam');

and
$imap->select('Public Folders/Non-Spam');
to
$imap->select('non-spam');

Do you have any idea about what else I have to change to get this working with non-public IMAP accounts.
Thanks for your help!

This is the debug output of spamassassin_imap_learn.pl:
Started at Tue Oct 26 17:47:29 2010
Using Mail::IMAPClient version 3.08 on perl 5.010000
Connecting to 127.0.0.1:143 port 143
Connected to 127.0.0.1:143
Read: * OK Zarafa IMAP gateway ready
Sending: 1 LOGIN "spam" spam
Sent 21 bytes
Read: 1 OK LOGIN completed
Sending: 2 STATUS spam (MESSAGES)
Sent 26 bytes
Read: 2 NO STATUS error finding folder
ERROR: 2 NO STATUS error finding folder
Spam to process
Sending: 3 STATUS non-spam (MESSAGES)
Sent 30 bytes
Read: 3 NO STATUS error finding folder
ERROR: 3 NO STATUS error finding folder
Non-Spam to process
Sending: 4 SELECT spam
Sent 15 bytes
Read: 4 NO SELECT error getting mails in folder
ERROR: 4 NO SELECT error getting mails in folder
Sending: 5 UID SEARCH ALL
Sent 18 bytes
Read: * SEARCH

5 OK UID SEARCH completed
Search successfull but found no matching messages
Sending: 6 EXPUNGE
Sent 11 bytes
Read: 6 NO EXPUNGE error opening folder
ERROR: 6 NO EXPUNGE error opening folder
Sending: 7 SELECT
Sent 11 bytes
Read: 7 BAD SELECT must have 1 argument
ERROR: 7 BAD SELECT must have 1 argument
Sending: 8 CLOSE
Sent 9 bytes
Read: 8 NO CLOSE error opening folder
ERROR: 8 NO CLOSE error opening folder
Sending: 9 SELECT non-spam
Sent 19 bytes
Read: 9 NO SELECT error getting mails in folder
ERROR: 9 NO SELECT error getting mails in folder
Sending: 10 UID SEARCH ALL
Sent 19 bytes
Read: * SEARCH

10 OK UID SEARCH completed
Search successfull but found no matching messages
Sending: 11 CLOSE
Sent 10 bytes
Read: 11 NO CLOSE error opening folder
ERROR: 11 NO CLOSE error opening folder
Sending: 12 LOGOUT
Sent 11 bytes
Read: * BYE Zarafa server logging out

12 OK LOGOUT completed

Hi again Marcus,

Your plan is a good idea for increasing privacy, a similar idea was discussed in the guide at http://wiki.contribs.org/Zarafa_Bayesian_Learning (which I should have referenced before, my bad). I didn't include it in my article because it requires an extra user account, although I guess that's not a big problem for most people.

Incidentally, you only need one extra user account, which can have both the spam and non-spam folders. There is a bit of a complication in that by default Zarafa (and probably Outlook) will move messages to the folders rather than copying them. This is fine for spam, of course, because the user almost certainly doesn't want to see it again, but for non-spam there is a problem - they can move the mail into the folder, but they can't get it back out again.

If you haven't got all that many users then one possible solution would be to make a separate subfolder for each, e.g. "user1-nonspam", "user2-nonspam" etc, and set the subfolder to be shared read/write with that user so they can retrieve messages once they've been learnt. If you have a lot of users this isn't going to be practical, of course, in which case the only option is to train users to copy the mail rather than move it (and I must confess I haven't figured out how to do this myself).

With regard to the error you're seeing, try debugging the connection from the command line and seeing what folders are visible, which might help you track down why you're getting the "error finding folder". Use the following commands, substituting things in angle brackets as appropriate:

telnet <server_ip> 143
. login "<username>" "<password>"
. list "" "*"
. logout

You'll get output between each command, after the list command you should get a list of folders which will hopefully help you work out what to put in the perl script. For more information look up "IMAP telnet" in your preferred search engine.

Best of luck, let us know how you get on...

Great post, easy read, worked great.

Could you please provide a link on how to integrate purchased SSL certificates. Im an idiot and cant seem to get my ssl cert to work!

Great work mate!

Thanks for the kind words.

Which components of the system are you having trouble with? Apache, Postfix and Zarafa can all use SSL certificates. There are plenty of guides for configuring Apache to use SSL out on the web, so if you've got that working you're most of the way there. After that it's just a question of setting the permissions on the key and cert files so that Postfix and Zarafa can read them (while ensuring that non-privileged users cannot read the key, of course!) and then specifying the paths to them in the relevant config file: look for smtp(d)_tls_[key|cert]_file in /etc/postfix/main.cf and ssl_[private_key|certificate]_file in /etc/zarafa/gateway.cfg. Good luck!

Add new comment

(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.

Filtered HTML

  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.

Plain text

  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.