Dr. Christopher Kunz

I have a little story from an administrator’s life for you today.

Today at around 1pm CEST, our mail cluster started behaving very odd. Although SMTP ports were open, establishing a connection to the initial server greeting took around 30 seconds on each of the 8 nodes.

Long delays normally indicate DNS problems, so I checked if the resolvers were b0rked. Both were working fine and DNS lookups on the command line on each of the mail servers were blazing fast. Ho-humm... stracing the qmail-smtpd processes showed that my assumption was correct - the delays were definitely caused by DNS lookups to our resolverss being slow. Just to be on the safe side and to rule out firewall or routing issues between our networks, I used the OpenDNS resolvers on one node - nothing changed.

My next guess was: Maybe a RBL shut down?! We are only filtering against XBL and the iX blacklist and both were up - I tried anyway. Still, SMTP was slow on every mail server.

We are using qmail with rblsmtpd, so I could quickly disable the rblsmtpd service altogether. Suddenly, the connection speed was up to past standards again. So it had to do with rblsmtpd, but with none of the RBLs it is filtering against. Weird.

I went to the resolvers and checked which queries were actually performed during a smtp connection initiation - and to my astonishment, I saw that queries to resolve 1.2.3.4.rbl.maps.vix.com were used during each connection initiation. I googled a bit and read that the long defunct MAPS RBL was the fallback blacklist that was used by rblsmtpd when no other RBLs were specified in the configuration.
However, I had two blacklists in the config - yet rblsmtpd tried to look up at the MAPS RBL, too.

It seems someone disabled the nameservers for the vix.com zone, so I took the liberty of telling my resolvers to serve it to all clients - empty, of course. Now, everything is back to normal. It cost me about two hours to solve that problem.

(This is mostly for me to remember for the other boxes, but maybe it helps someone)

When I boldly updated one of our SMTP/POP3/IMAP boxes (running qmail/vpopmail for MTA and POP3(s) and courier for imap) to Etch the other day, I ran into a number of issues. First, tcpserver and a lot of other DJB binaries errored out on me after the upgrade, downing the MTA:

This is due to the glibc having been updated to 2.3.x with Etch - qmail 1.03 without patches and all associated tools are used to older glibc. The errno.h header is pretty much the only incompatible change and requires patch + recompile for all qmail stuff. You can get errno.h patches here:

http://djbware.csi.hu/patches/qmail-1.03.errno.patch (qmail 1.03)

http://djbware.csi.hu/patches/daemontools-0.76.errno.patch (daemontools)

http://djbware.csi.hu/patches/ucspi-tcp-0.88.errno.patch (ucspi-tcp)

Apply with patch -p 0 < patchfile.patch, make && make check install and triple-check the locations! On my box, all binaries were in three (!) different locations.

After that, at least the services started again. Kinda. All spam filtering (done with a maildrop script on our machines) failed, though - “Error 0x06”. Googling yields a number of possible solutions, I (think I) fixed it by downloading the latest version and recompiling maildrop as well as setuid’ing the binary.

The last thing that bugged me was the fact that the stunnel-based pop3s service didn’t work anymore (and there was no error message to be found in log/pop3s/current). This is due to the fact that stunnel now expects a DH exchange parameter in each certificate PEM file. Thanks for pointing that out, infodrom! The remedy goes like this:

   dd if=/dev/urandom count=2 | openssl dhparam -rand - 512

This creates 3 lines similar to

Prepend those to your certificate PEM file, yielding this:

-----BEGIN DH PARAMETERS-----
[some blah blah]
-----END DH PARAMETERS-----
-----BEGIN RSA PRIVATE KEY-----
[a lot of blah blah blah]
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
[an even bigger lot of blah blah blah]
-----END CERTIFICATE-----

Also, you might wanna use the run file from my extended entry instead, since I think stunnel has been updated to version 3 (from 2) in Etch.
That’s it, after this ordeal your qmail/vpopmail machine should run again.