Configuring TLS for Sendmail (STARTTLS)

Getting this to work took quite some time – mostly because the articles I have found on the subject are wrong or contain errors making them useless. Is that deliberate? It almost felt that way!

The basic structure isn’t difficult, but it is critical to get the right files in the right places. These instructions have been ripped from a number of different web sites, but have been fixed so they actually work!

The system in question is RedHat 5.6 with Sendmail 8.13.8. I also did this on CentOS 6.2 x64 with Sendmail 8.14.4 in testing. I am assuming a functioning Sendmail system.

1. Install openssl-perl:
yum -y install openssl-perl

2. You need to edit the CA.pl file. This was a common area of errors since the instructions for the edit were WRONG. Rather than screwing around with diff files, you just need to make the following changes (copying the existing /etc/pki/tls/misc/CA.pl to /etc/pki/tls/misc/CA1.pl)

Now edit CA1.pl and find:
system ("$REQ -new -x509 -keyout newkey.pem -out newcert.pem $DAYS");
and make it
system ("$REQ -new -x509 -nodes -keyout newkey.pem -out newcert.pem $DAYS");

then find
system ("$REQ -new -keyout newkey.pem -out newreq.pem $DAYS");
and make it
system ("$REQ -new -nodes -keyout newkey.pem -out newreq.pem $DAYS");

I also changed the $CADAYS variable to ‘3650’ (10 years). YMMV

3. Create the Certificate Authority (CA)
./CA1.pl -newca
DO NOT ENTER A FILE NAME! Answer the questions as appropriate, but don’t use any ‘extra’ information.

4. Now create your request
./CA1.pl -newreq
Answer the questions as appropriate, and again, don’t give any ‘extra’ info.

5. Now ‘sign’ the Certificate Request with the CA you created earlier
./CA1.pl -sign

For some reason, no matter what I enter for the $DAYS variable, the cert is only ever good for one year. I have tried many variations and commands, but have not been able to get by this. The CA is still good fro 10 years.

6. Create a certificates directory
mkdir /etc/mail/certs

7. Copy the relevant files to the certs directory (This is another place I have found errors in other documentation)
cp /etc/pki/CA/cacert.pem /etc/mail/certs/CAcert.pem
cp /etc/pki/tls/misc/newkey.pem /etc/mail/certs/MYkey.pem
cp /etc/pki/tls/misc/newcert.pem /etc/mail/certs/MYcert.pem

8. Sendmail is very picky about permissions, so set them on the certs folder and files
chmod -R 600 /etc/mail/certs

9. Edit your sendmail.mc file to include the following:
define(`confCACERT_PATH', `/etc/mail/certs')dnl
define(`confCACERT', `/etc/mail/certs/CAcert.pem')dnl
define(`confSERVER_CERT', `/etc/mail/certs/MYcert.pem')dnl
define(`confSERVER_KEY', `/etc/mail/certs/MYkey.pem')dnl
define(`confCLIENT_CERT', `/etc/mail/certs/MYcert.pem')dnl
define(`confCLIENT_KEY', `/etc/mail/certs/MYkey.pem')dnl

Then recompile your sendmail.cf

10. If you are using the packages this shouldn’t be a problem, but you may also check to make sure STARTTLS is compiled into Sendmail. Run this:
sendmail -bt -d0.8 < /dev/null

And look for this:

Compiled with: DNSMAP HESIOD HES_GETMAILHOST LDAPMAP LOG MAP_REGEX
MATCHGECOS MILTER MIME7TO8 MIME8TO7 NAMED_BIND NETINET NETINET6
NETUNIX NEWDB NIS PIPELINING SASLv2 SCANF SOCKETMAP STARTTLS
TCPWRAPPERS USERDB USE_LDAP_INIT

11. Send a HUP to Sendmail:
kill -HUP `head -1 /var/run/sendmail.pid`

If everything is correct, you may not see anything in the logs. But if you:
telnet localhost 25

and issue a proper ‘EHLO’, you should see this:

250-STARTTLS

The purpose of this document is to fix syntax errors is other documents I have found. The resulting errors are just to vague to get any real help. Hopefully this will help others avoid the hours of searching I had to go through!