qmail + vpopmail + Dovecot | Roberto's qmail notes

Hi

I'm "managing" more than 50 virtual server with multiple domains in each one, it's something hard to have all up to date :D
Every day I find something that can be get better, and some times I found thinks hard to do :D
In the last month I get some reports from customers that can't sent mails to some addresses and the common things is that all the MXs rejecting mails are requesting STARTTLS !
I really don't know if this is something that will become "normal", but I'm looking the way to enable TLS on outgoing mail (qmail-remote / qmail-send) and I'm lost !
May be I have all configured an running ok, and the other server is working bad, but, as I can't find a way to debug qmail-remote, I don't know wath to do :(

I received the next message trying to send mails to remote host

Connected_to_xxx.xxx.xx.xxx_but_my_name_was_rejected./Remote_host_said:_530_#5.7.0_Must_issue_a_STARTTLS_command_first/

So ... I'm lost !
My be I missing some control file ?

Hi Pablo,

qmail-remote should use TLS. Can you check that qmail-remote is linked to libssl and libcrypto?

# ldd /var/qmail/bin/qmail-remote 
       linux-vdso.so.1 (0x00007fffdfed2000) 
       libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f055595f000) 
       libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f055567a000)

Then verify that the connection is done with tls in this way

# /var/qmail/bin/qmail-remote host sender@yourdomain recipient
from: sender@yourdomain
to: recipient
subject: test 

test 

. 
r K74.125.206.27 accepted message. 
Remote host said: 250 2.0.0 OK  1720157566 ffacd0b85a97d-36791d6e378si3182501f8f.541 - gsmtp

you can use the IP instead of host in square brackets [IP]

If you do the same with strace

strace -F -o strace.log -s 1024 command

you can see if it's using TLS

or you can test with your own server and enable recordio to look at the smtpd conversation

I tested qmail-remote as explained above sending a message from a test virtual server to my smtp.sagredo.eu main server, where I enabled recordio, and I can see that qmail-remote is issuing the STARTTLS verb correctly:

024-07-05 13:19:39.994198500 3799 > 220 smtp.sagredo.eu ESMTP 
024-07-05 13:19:39.994318500 3799 < EHLO smtp.other.qmail.server
024-07-05 13:19:39.995738500 3799 > 250-smtp.sagredo.eu 
024-07-05 13:19:39.995738500 3799 > 250-STARTTLS 
024-07-05 13:19:39.995739500 3799 > 250-PIPELINING 
024-07-05 13:19:39.995739500 3799 > 250-8BITMIME 
024-07-05 13:19:39.995748500 3799 > 250 SIZE 10000000 
024-07-05 13:19:39.997908500 3799 < STARTTLS 
024-07-05 13:19:39.999948500 3799 > 220 ready for tls

Hi

Thanks for the data !
Finally works !
I missed the clientcert.pem and some permissions !

Hi
I'm looking for the way to "control" the email traffic by size not by quantity
Knows someone if there is some patch for this ?

By the way, I want to say "Thanks !!!", one more time, because I'm updating servers and I was looking for what was changed in certbot, and I found it here again !

Thanks again for all the share knowledge

Hi Pablo, thanks for the appreciation.

Do you mean a filter based on a per mail size or a filter which sums the size of msg in given period of time? Regarding the second option I don't know 

Hi !

the 2nd option !
I just find the way
Qmailanalog !
The program process de qmail-send log and make a nre file with the information arranged in better way, and has a lot scripts to generate different reports

Great! It is a program that I wanted to play with soon or later

Here is a script I made to process the qmail-send logs

#!/usr/local/bin/bash
#
# 
# This script is for use on the log for the day before (yesterday :D)
# 
# I use multilog, so I have to check if the log 
# of the day is in more than one file
#

send_log_dir = "/var/log/qmail/send"

archivos=`find -s ${send_log_dir}/* -type f -mtime 1 ! -size 0c`
cat $archivos sin_procesar > send_log

# Fecha desde
desde=`date -v-1d -v0H -v0M -v0S +%s`

# For test only, using a day you want to test
# just remove # and change the date
#
#desde=`date -j -f "%Y%m%d%H%M%S" "20230902000000" +%s`

# Fecha hasta
hasta=`date -v0H -v0M -v0S -v-1S +%s`

#
# For test only, using a day you want to test
# just remove # and change the date
# 
#hasta=`date -j -f "%Y%m%d%H%M%S" "20230902235959" +%s` 

# The multilog use tai64 for date/time, and qmailanalog use seconds since 1970
# so I have to parse the log using tai64nfrac and filtering the result using awk
cat send_log | tai64nfrac | awk -v desde=$desde -v hasta=$hasta '{ if ($1 >= desde && $1 <= hasta) print $0 }' > log_$desde

# Here is where qmailanalog do "the magic" :D
cat log_$desde | /var/qmail/qmailanalog/bin/matchup >procesado_$desde 5>sin_procesar

#
# Then you can use the script in BIN dir, the ones starting with Z or X
cat procesado_$desde | /var/qmail/qmailanalog/bin/zoverall

Thanks for sharing 

Hi all. Has anyone of you sucessfully installed on an arm64? I am planning to install this on a Raspberry PI 4.

Please share your experience.

Thank you

Hi, thanks for your post. I've never played with arm but I'm curious. Do you run slackware on it? Let us know if you do it!

I just wanted to drop a quick THANK YOU for the great work you do with keeping all the valueable qmail information online.

I have studied the qmail source code back in '99 and learned a lot about C programming back then. IMHO, qmail is still one of the best MTA when it comes to send out tons of mail quickly.

 Your website has helped me tremendously through the past years, so again – thanks! I know it takes a lot of effort to keep up such a project.

Hello Roberto,

my qmail server has been working perfectly for the past year and I'm now planning to patch and upgrade O.S. and the entire qmail stack; I usually keep just the roundcube part constantly updated.

We don't have a upgrade complete set of instructions, right ? Any particular issues to take care of, besides:

- [backup tasks]

- qmailctl stop

- configure && make && make install

- [same per dovecot]

- qmailctl start

Thank you !

Maurizio

Ciao Maurizio,

generally the upgrade depends on your installed version, but sometime you can find tips of what you have to do when upgrading.

Concerning qmail, it depends on your current version of the big patch. If it's not too old just patch, stop, compile, start the server.

Concerning dovecot, if you are upgrading from 2.3.x to the latest version you can do the same. If moving from 2.2 to 2.3 version, take a look at the "Moving to 2.3 branch" paragraph.

Concerning spamassassin you have to recompile and restart. There are new configurations of razor, pyzor, spamcop that you may want to upgrade as well.

Roundcube is very easy to upgrade from every version. Look at the top of the page to find out how.

Clamav: recompile and install on top of the previous one

simscan: patch recompile and install on top of the previous version.

Thank you Roberto !

Looks reasonable to me. My qmail patch is about "one year old", I will have to double check everything before proceeding.

Do we have a simple way of backupping the current qmail installation (I'm referring to binary files), besides saving a copy of /usr/bin/ - qmail/control - etc. ?

(I'm snapshotting my virtual server just in case....)

I think that these simple steps you are reporting could become a standalone page, they are very useful and I think a lot of other sysadmins have to deal with upgrading issues, especially in the present unsecure world :-)

Thank you, ciao and Happy Christmas !

Maurizio

If your installed patch is one year old you can just recompile and restart.

I don't have suggestions to quickly backup the server. I'm used to daily rsync all configs and messages and also clone my virtual server before migrating or doing very big updates

Thank you Roberto, same procedure for me :-)

Has anyone set up any load balances for Qmail SMTP?

I am currently using HAProxy, but I am having difficulty maintaining Remote IP on connections when sending to the Qmail server.

Hi,

Anyone here has a working setup with Qmail and rspamd? (is this possible at all?)

Hi,

It is certainly possible, either through running rspamc in the desired .qmail files or through hackery :)

Personally I've hacked together an ugly patch that will run rspamc through simscan, which works and lets me enable it for entire domains, but it's an ugly patch at the moment ("works for me") that could use quite a bit of polish.

That said, if you want to try it without warranty and on your own risk, you can grab it here: http://notes.benv.junerules.com/wp-content/uploads/2019/04/simscan-1.4.0.rspamc.patch

Improvements welcome ;)

Regards,

Wouter

Hello Wouter,

Sounds promising, might give it a try quick response also :)

cdr

Great write up, I've used your notes a number of times and it works great.

Hi,

I had built a new toaster with the latest patch. It was about 2 days old. Everything was working fine and today it happened that every email sent, there a a bounced message. The bounced message  complaints that it tried to deliver the message to log@serverhost.

The intended recipient still maanged to get the email.

Bounced mail below :-
Hi. This is the qmail-send program at domain.com.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.
<log@domain.com>:
Sorry. Although I'm listed as a best-preference MX or A for that host, it isn't in my control/locals file, so I don't treat it as local. (#5.4.6)

I decided to create log@domain.com and tried to see what is being recieved, it happened to be the exact email sent out.

Any idea what went wrong?

thanks

nic

The mails sent to log@domain.com are needed just to improve the qmail-send log; read the qmail-extra patch on the purpose.

But domain.com must have a valid MX and has to be included in you locals file. Try to see what

qmail-showctl |grep local 

says

Thanks Roberto.

Since it is needed, i will just keep the account log@domain.com and put up a crontab to delete mails nightly.

no, this is not how it is programmed to work. There's no actual email to that account. Check the contents of the file /var/qmail/alias/.qmail-log and compare with what I reported in the qmail-extra patch documentation

Hi Roberto

I have the same problem as Nic. My clients always receive bounced emails saying that they cannot send email to mslog@domain.com. Because I installed qmail from complied binary, I do not have an access to extra.h. How should I stop this logging? In my qmail log, it contains many "starting delivery xxxxx: msg xxxx to remote msglog@domain.com".

Thank you

Sam.

So your qmail has nothing to do with my patch and the present guide is not suitable for your installation. For this reason if you follow my instructions you will get an unpredictable behaviour

Hi.

Sorry for the late reply.

My .qmail-log as follows

| awk '/^$/ { exit } /^[mM][eE][sS][sS][aA][gG][eE]/ { print } /^[rR][eE][cC][eE][iI][vV][eE][dD]:/ { print; } /^[fF][rR][oO][mM]:/ { print } /^[tT][oO]:/ { print } /^[sS][uU][bB][jJ][eE][cC][tT]:/ { print } /^[xX]-[mM][aA][iI][lL][eE][rR]/ { print } /[hH][eE][lL][oO]/{ print } /^[rR][eE][pP][lL][yY]-[tT][oO]/{ print } /^[rR][eE][tT][uU][rR][nN]-[pP][aA][tT][hH]/{ print } /^[cC][cC]:/{ print } /^[dD][eE][lL][iI][vV][eE][rR][eE][dD]-[tT][oO]/{ print } /^[dD][aA][tT][eE]:/{ print } / by /{ print } / id /{ print } /<.*>/{ print }'
| echo "---"

thanks
nic

it seems correct... double check everything

its like this:

if /var/qmail/control/locals - does NOT include hostname.yourdomainname.com

then qmail is trying to send an e-mail to remote log@hostname.yourdomainname.com

if u put hostname.yourdomainname.com in /var/qmail/control/locals

then all goes ok.

Thank you for pointing out this. I thought that having hostname.yourdomainname.com in /var/qmail/control/locals is a good practice regardless of the fact that you have that patch installed. This is the reason why I didn't even mentioned that.

I've just added a note about this in the "Configuring qmail" page

I don't know if this might be useful, but since I'm using a local account named "log@domain.com", and I cannot rename it, in order to avoid the duplicate email problem I had to revert this part of the patch:

-#define QUEUE_EXTRA ""
-#define QUEUE_EXTRALEN 0
+#define QUEUE_EXTRA "Tlog\0"
+#define QUEUE_EXTRALEN 5

So my extra.h looks like this:

#ifndef EXTRA_H
#define EXTRA_H

#define QUEUE_EXTRA ""
#define QUEUE_EXTRALEN 0

#endif

Regards,

miz

FIrst of all, 

congratulations for your work and tutorial.

we follow and mounted a new qmail install.

we repair that the new server trys to send a "copy" of every message to log@domain.com

can you tell us where you put this conf, cause we don't found any reference to this.

Best regards and congratulations again.

Matheus

I added a clarification about this functionality at the bottom of the configuration page

Hi,

I didn't write a related page yet, just a note here http://notes.sagredo.eu/node/82#queue-extra. Actually it is a way to improve the qmail-send log.

Is the functionality working fine for you?

Hi

This documentation is the best for qmail, I'm working with qmail a year without problems, but it feels like I have to advance a bit.

Is there any possibility to setup a mirror qmail server? i mean have two servers working and when one is down the other keep working with all the services. Imap, Pop3, SMTP, etc.

Thanks in advance.

uh... I really don't know how exactly, but I guess that a lot of work should be done to dinamically change the IP in the name server in the event that a program like ping (?) returns a certain response. And also to sync back and restore everything automatically.

Thanks for the qmail info! Helped a ton when I had to update a old server recently and didn't want to switch away from qmail.

Saved me tons of time!

Roberto, you did a great job! thanks a lot!

to be noted:

- your vpopmail guide is not --large-site=y, so your vpopmail mysql table only had 1 single 'vpopmail' table. i cannot find dovecot support --large-site=y configuration for sql auth, i am backward to use vpopmail auth instead to make it work atm.

regards,

GW

I think you mean --enable-many-domains (http://www.inter7.com/vpopmail/FAQ.txt FAQ #19). It can be a choice, of course, in case of many domains.

So, if I understand well, this option is not dovecot compliant? If you use the SQL backend it should be sufficient to adjust the sql query accordingly. I can't get the vpopmail driver working anymore...

YES indeed, dovecot compliences issue, and another issue to be added, there is corrupted dovecot.index.chache each time new message arrived! Actually i can get running dovecot 2.2.5 with vpopmail 5.4.33 with --enable-many-domains option using vpopmail driver BUT another issues came up:

- LDA permission to user socket connection denied

- Dovecot Indexing causes dovecot.index.chache corrupted

- LDA issue causes Managesieve/Sieve did not work.

I've never tried this configuration, but if I were in you I would try to switch to the mysql driver.

Let us know...

i can do migration from vpopmail to sql properly but imap-auth scheme support is only LOGIN and PLAIN, it is failed when i 'm using DIGEST-MD5 and CRAM-MD5. i will try to solve this later but the LDS Indexing issue still remind eventhough dovecot always fix it after checking it.

You can get rid of DIGEST-MD5 and CRAM-MD5 enabling imaps and pop3s...

Indeed, but LDA Indexing corrupted still annoying while courier works properly...stuck with corrupted dovecot.index.cache...

please note the URL has been moved to

www.fehcom.de/qmail/qmail.html

 Thank you for putting these notes together.

David.

Roberto,

We migrated to our new mail server yesterday (built from the ground up using your notes as our guide).  It’s working great!  SpamAssassin is working!  ClamAV is working! SPF is working! DKIM is working!  RoundCube rocks!  In short—everything works! 

Up til now, we had been limping along with a partially broken, spam-laden mail server.  Worse yet, I didn’t know what steps to take to fix it—seemed like any fixes I implemented ended up breaking something else.

Thanks so much for notes.sagredo.eu—it’s been a Godsend!

-Dave Martin

Hello,

Congratulations for this how to, i think this is one of the most complete on web. Updated tools and perfect to Slackware, my favorite distro.

Hello,

I just convert the mailboxes configuration with myself/scripts/bash etc.. nvm..  It's done.  this issue is close.
2 days ago, I switch my company domain, to the new mail server, and it works great! Good spam, virus protecion. Nice looking webmail, and fresh packages.  Bling-bling ;))

I'am very verrrryy glad that You help and assist me at time of the configuration of your toaster. Thank you! good job, nice howto ;-)