Expunging expired Junk and Trash emails with dovecot

December 4, 2020 by Roberto Puzzanghera 18 comments

Of course we want to delete old Junk and Trash emails to save hard disk space.

If you want to expunge your Junk and Trash folder after 60 days you can set 15-mailboxes.conf as follows:

namespace {
  mailbox Junk {
    special_use = \Junk
    auto = subscribe
    autoexpunge = 60d
  }

  mailbox Trash {
    special_use = \Trash
    auto = subscribe
    autoexpunge = 60d
  }
}

Be aware that the messages saved to Inbox 60 days ago and moved to Junk today will not be deleted.

In case you need to act more precisely on your mailboxes you may want to take a look at this script by Tony Fung.


I leave intact these following notes for those who are not satisfied with the previous method and prefer to set up an expunge script to be run via cronjob. Those who set up the autoexpunge as explained above can jump to the next page.

This command

doveadm expunge -A mailbox Junk savedbefore 60d

will do a connection to the userdb, sql/MySQL in our case, and iterate in all (-A option) user's mailbox looking for expired emails, moved to the Junk folder more than 60 days ago. 

To achieve the purpose it will be sufficient to create a shell script like this

> nano /usr/local/dovecot/etc/dovecot_expunge.sh

#!/bin/bash
#
DOVEADM="/usr/local/dovecot/bin/doveadm";

$DOVEADM expunge -A mailbox Trash savedbefore 90d
$DOVEADM expunge -A mailbox Junk  savedbefore 60d

assign the +x priviledge:

chmod +x /usr/local/dovecot/etc/dovecot_expunge.sh

and run the script once a month or whatever as a cronjob

> crontab -e

# dovecot delete spam & trash
#minute hour mday month wday command
40 3 12 * * /usr/local/dovecot/etc/dovecot_expunge.sh

What to do when using the vpopmail auth driver

Unfortunately, the vpopmail's APIs seem not to provide an iteration feature. The good news is that Costel Balta has a brilliant solution. You can download his script here.

#!/bin/bash
#
# Author: Costel Balta
# Slightly modified by Roberto Puzzanghera
#

# Config and executables path  
VPOPMAIL_MYSQL_CONFIG="/home/vpopmail/etc/vpopmail.mysql"  

# mysql bin
MYSQL=$(which mysql)

# Extract mysql params  
HOST=$(sed -n "/#/! s/^\(.*\)|.*|.*|.*|.*/\1/p" $VPOPMAIL_MYSQL_CONFIG)  
PORT=$(sed -n "/#/! s/^.*|\(.*\)|.*|.*|.*/\1/p" $VPOPMAIL_MYSQL_CONFIG)  
USER=$(sed -n "/#/! s/^.*|.*|\(.*\)|.*|.*/\1/p" $VPOPMAIL_MYSQL_CONFIG)  
 PWD=$(sed -n "/#/! s/^.*|.*|.*|\(.*\)|.*/\1/p" $VPOPMAIL_MYSQL_CONFIG)  
  DB=$(sed -n "/#/! s/^.*|.*|.*|.*|\(.*\)/\1/p" $VPOPMAIL_MYSQL_CONFIG) 

# dovecot details
DOVEADM="/usr/local/dovecot/bin/doveadm";

# Output sql to a file that we want to run
echo "USE vpopmail; select concat(pw_name,'@',pw_domain) as username from vpopmail;" > /tmp/query.sql;

# Run the query and get the results (adjust the path to mysql)
results=`$MYSQL -h $HOST -u $USER -p$PWD -N < /tmp/query.sql`;

# Loop through each row
for row in $results
        do
        echo "Purging $row Trash and Junk mailbox..."
        # Purge expired Trash
        $DOVEADM -v expunge mailbox Trash -u "$row" savedbefore 90d
        # Purge expired Junk
        $DOVEADM -v expunge mailbox Junk  -u "$row" savedbefore 60d
done

This script does a mysql query selecting all users from the vpopmail's database, stores the results in a variable and iterates through each user's mailbox deleting old emails from Trash and Junk folders. Since this script stores the mysql access it must be run by root and must have root's read priviledges.

If you want to avoid email notifications about this task, because you have tons of users, simply comment the echo inside the script.

cd /usr/local/dovecot/etc
wget https://notes.sagredo.eu/files/qmail/dovecot_expunge/dovecot_expunge-cb
mv dovecot_expire dovecot_expire.sh
chown root:root dovecot_expire.sh
chmod 0700 /usr/local/dovecot/etc/dovecot_expire.sh

Run the script once a month or whatever as a cronjob

> crontab -e

# dovecot delete spam & trash
#minute hour mday month wday command
40 3 12 * * /usr/local/dovecot/etc/dovecot_expire.sh

Comments

mysql: [Warning]

Hi Roberto,

when i execute /usr/local/dovecot/etc/dovecot expire.sh manually in the output there is below warning:

mysql: [Warning] Using a password on the command line interface can be insecure.

I created small patch for it:

cat > /usr/local/src/dovecot_expire.patch <<__EOF__
--- /usr/local/dovecot/etc/dovecot_expire.sh.old 2024-10-04 11:48:15.436321335 +0000
+++ /usr/local/dovecot/etc/dovecot_expire.sh 2024-10-04 11:49:36.847198893 +0000
@@ -24,7 +24,7 @@
echo "USE vpopmail; select concat(pw_name,'@',pw_domain) as username from vpopmail;" > /tmp/query.sql;

# Run the query and get the results (adjust the path to mysql)
-results=`$MYSQL -h $HOST -u $USER -p$PWD -N < /tmp/query.sql`;
+results=`export MYSQL_PWD=$PWD; $MYSQL -h $HOST -u $USER -N < /tmp/query.sql`;

# Loop through each row
for row in $results
__EOF__


For all with already enabled dovecot_expire.sh cron can apply the patch:

patch /usr/local/dovecot/etc/dovecot_expire.sh < /usr/local/src/dovecot_expire.patch

Reply |

mysql: [Warning]

I don't think that the script is unsecure. It is not readable and it doesn't hold any pwd... that warning can be safely ignored

Reply |

script

wget https://notes.sagredo.eu/files/qmail/dovecot_expunge/dovecot_expunge-cb - ERROR 404

downloading dovecot_expunge-cb then moving dovecot_expire ??? - anyway it's not there

# Config and executables path  
VPOPMAIL_MYSQL_CONFIG="/home/vpopmail/etc/vpopmail.mysql"

ADD MYSQL=$(which mysql) - else script fails

Reply |

script

Thank you. Corrected

Reply |

Also may need additional settings

https://doc.dovecot.org/configuration_manual/namespace/#mailbox-settings says about autoexpunge:

mailbox_list_index = yes is highly recommended when using this setting, as it avoids actually opening the mailbox to see if anything needs to be expunged.

mail_always_cache_fields = date.save is also recommended when using this setting with sdbox or Maildir, as it avoids using stat() to find out the mail’s saved-timestamp. With mdbox and obox formats this isn’t necessary, since the saved-timestamp is always available.

Reply |

Also may need additional settings

I see that mail_always_cache_fields has been added in v2.3.17 but I can't find it in any file of the example-config folder. I think putting it in 10-mail.conf will be fine

Reply |

Dovecot Expunging

Hi Roberto,

I created a script to expunge mailboxes to remove emails in before specified days per domain/user/folder which defined in a conf file.  Schedule to run the script daily and it shall clean up the defined mailboxes.  I run it on CentOS 8 almost a year and save me a lot of time to house keep some mailboxes in use of notification.  Also, it could be applied as mail retention control for all or defined mailboxes.  Hope this share is helpful.

Here is the script (/usr/local/dovecot/bin/dovecot-expunge):
Change the variables (DOVEADM, CONF, LOG) in the script to desired values for your environment if necessary.

#!/bin/sh

DOVEADM="/usr/local/dovecot/bin/doveadm"
CONF="/usr/local/dovecot/etc/dovecot/dovecot-expunge.conf"
LOG="/var/log/dovecot/dovecot-expunge.log"

if [ ! -e "$CONF" ]; then
   echo $(date +"%Y%m%d %T") \
   "aborted with $CONF not found" >> $LOG
   exit
fi

while read line; do
         # Skip commented and blank lines:
         if [[ ${line:0:1} = "#" ]] || [ -z "$line" ]; then
            continue
         fi

         # Put fields to variables per line:
         DOMAIN=$(echo $line|cut -d"|" -f1)
         USER=$(echo $line|cut -d"|" -f2)
         DAY=$(echo $line|cut -d"|" -f3)
         FOLDER=$(echo $line|cut -d"|" -f4)

         # List mailbox folders per user:
         $DOVEADM mailbox list -u "$USER@$DOMAIN" | \
         while read list; do
                  if [[ "$USER" = "*" ]]; then
                     u=$(echo $list|cut -d" " -f1)
                     f=$(echo $list|cut -d" " -f2)
                  else
                     u="$USER@$DOMAIN"
                     f=$list
                  fi

                  if [[ "$FOLDER" = "*" ]]; then
                     $DOVEADM expunge -u $u mailbox $f savedbefore "$DAY""d"
                     echo $(date +"%Y%m%d %T") \
                     "expunged $u:$f:$DAY days before" >> $LOG
                  else
                     if [ -z "${FOLDER##*$f*}" ]; then
                        $DOVEADM expunge -u $u mailbox $f savedbefore "$DAY""d"
                        echo $(date +"%Y%m%d %T") \
                        "expunged $u:$f:$DAY days before" >> $LOG
                     fi
                  fi
         done
done < $CONF

exit

Here is the conf file and info (/usr/local/dovecot/etc/dovecot/dovecot-expunge.conf):

# Entry format: (email_domain)|(email_user)|(days_before)|(mail_folders)
#
# (email_domain) is the domain name of email address, can be "*" for all.
# (email_user) is the user name of email address, can be "*" for all.
# (days_before) is the number of days to remove in before of.
# (mail_folders) is the folders to expunge, can be "*" for all,
# or combination in any of "Drafts,Junk,Trash,Sent,INBOX".
#
# All blank and commented (begin with "#") lines are skipped.
#
# =============================================
#
# Example: example.com|postmaster|30|*
#
# This example will expunge postmaster@example.com,
# with all mail folders in before of 30 days.
#
# ---------------------------------------------
#
# Example: example.com|*|15|Trash
#
# This example will expunge all users of example.com,
# with mail folder Trash only in before of 15 days.
#
# ---------------------------------------------
#
# Example: example.com|*|60|Drafts,Trash
#
# This example will expunge all users of example.com,
# with mail folders Drafts and Trash in before of 60 days.
#
# ---------------------------------------------
#
# Example: *|*|90|Trash
#
# This example will expunge all users per domain,
# with mail folder Trash in before of 90 days.
#
# Using "*" for domain is advised to be general purpose,
# Such that the days before should be the largest in compare with other settings.
#
# =============================================

Here is an example of cron job:

# Daily dovecot expunge:
1 0 * * * root /usr/local/dovecot/bin/dovecot-expunge > /dev/null 2>&1

Reply |

Dovecot Expunging

Hi, very usefull, indeed. How do I set 2 or 3 or more users but not one specific or all, as using space as username's separation doesn't seems to work? Thanks

Reply |

Dovecot Expunging script

Here's a patch to your script to allow spaces in the names of mailboxes and a fix to script type (it should be a bash script, otherwise, it  won't work on some systems):

--- /orig/dovecot-expunge 2021-01-17 11:23:55.845365502 +0200
+++ new/dovecot-expunge 2021-01-17 11:25:21.060576680 +0200
@@ -1,8 +1,9 @@
-#!/bin/sh
+#!/bin/bash

@@ -27,19 +28,19 @@
while read list; do
if [[ "$USER" = "*" ]]; then
u=$(echo $list|cut -d" " -f1)
- f=$(echo $list|cut -d" " -f2)
+ f=$(echo $list|sed 's/[^ ]* //')
else
u="$USER@$DOMAIN"
f=$list
fi

if [[ "$FOLDER" = "*" ]]; then
- $DOVEADM expunge -u $u mailbox $f savedbefore "$DAY""d"
+ $DOVEADM expunge -u $u mailbox "$f" savedbefore "$DAY""d"
echo $(date +"%Y%m%d %T") \
"expunged $u:$f:$DAY days before" >> $LOG
else
if [ -z "${FOLDER##*$f*}" ]; then
- $DOVEADM expunge -u $u mailbox $f savedbefore "$DAY""d"
+ $DOVEADM expunge -u $u mailbox "$f" savedbefore "$DAY""d"
echo $(date +"%Y%m%d %T") \
"expunged $u:$f:$DAY days before" >> $LOG
fi

Reply |

Dovecot Expunging

It seems the <...> are removed, the followings are replaced by (...).  Hope this time can display correctly.

# Entry format: (email_domain)|(email_user)|(days_before)|(mail_folders)
#
# (email_domain) is the domain name of email address, can be "*" for all.
# (email_user) is the user name of email address, can be "*" for all.
# (days_before) is the number of days to remove in before of.
# (mail_folders) is the folders to expunge, can be "*" for all,
# or combination in any of "Drafts,Junk,Trash,Sent,INBOX".

Reply |

Dovecot Expunging

Thank you, Tony. Very appreciated.

I think this can be useful expecially to clean mailboxes like postmaster which are not accessed very often. I'm going to link this post later

Reply |

Other folders

Hi, what about other folders?

When you mark a message as deleted in INBOX folder?

I think you should go through all user folders :)

Best,

Rafal.

Reply |

Rafal, what do you mean by

Rafal, what do you mean by "mark a message as deleted in INBOX folder"?

Reply |

Hi Roberto,I wasn't clear

Hi Roberto,

I wasn't clear before, I am sorry. Please look at this example:

1) A new message (Seen Flag)

root@mail J:0 S:1 Maildir# tree {cur,new,tmp}
cur
??? 1417989074.M205847P32064.XXX,S=2582,W=2640:2,S
new
tmp

0 directories, 1 file

2) The message is still in INBOX folder but has got a "Trashed" flag

root@mail J:0 S:1 Maildir# tree {cur,new,tmp}
cur
??? 1417989074.M205847P32064.XXX,S=2582,W=2640:2,ST
new
tmp

0 directories, 1 file

3) I tried to expunge deleted messages from Trash folder

root@mail J:0 S:1 # doveadm expunge -u XXX mailbox Trash savedbefore 1s deleted

4) The message is still in INBOX folder (still uses storage)

root@mail J:0 S:1 Maildir# tree {cur,new,tmp}
cur
??? 1417989074.M205847P32064.XXX,S=2582,W=2640:2,ST
new
tmp

0 directories, 1 file

5) I deleted all messages with "Trashed" flag from INBOX folder and..

root@mail J:0 S:1 Maildir# doveadm expunge -u XXX mailbox INBOX savedbefore 1s deleted

6) The message has been removed from the hard-drive

root@mail J:0 S:1 Maildir# tree {cur,new,tmp}
cur
new
tmp

0 directories, 0 files

If a mail reader doesn't support "Expunge" every folder on exit, you will have a lot of deleted emails that use your storage and they are still in dovecot's index file which affects performance...

Your script is good but you should consider to "clean" users folders from deleted messages which have not been moved to Trash or Junk folders (like savedbefore 60d).

I would also change this line:

$DOVEADM -v expunge mailbox Trash -u $row savedbefore 90d

to:

$DOVEADM -v expunge mailbox Trash -u "$row" savedbefore 90d

or

$DOVEADM -v expunge mailbox Trash -u "${row}" savedbefore 90d

If somehow in user's name is a spacebar that would "crash" the script... and it is vulnerable to inject user's code (eg. read other users emails).

I hope you won't consider my message as something bad or negative. These are just my conclusions. You can always delete my comment :)

Best,

Rafal.

Reply |

how do you get that tag?

how do you get that Trashed tag? I expect that as soon as you delete the message by your client it is moved to .Trash

PS: I've never missed to reply/publish a comment, when it contains interesting things to learn. And I think this is the case, so thanks to you :)

Reply |

Hi Roberto,I have been busy

Hi Roberto,

I have been busy recently...

Try for example in Thunderbird to delete message by pressing Shift + Delete when a message is highlighted. You can do the same through telnet and other mail clients.

I hope it helps.

Best,

Rafal.

Reply |

Ok, I see. But I can't find a

Ok, I see. But I can't find a way to iterate over all mailboxes...

any help?

Reply |

There is a simple solution

There is a simple solution when using vpopmail:

40 4 12 * * for user in $(/home/vpopmail/bin/vpopbull -nV); do /usr/local/bin/doveadm -v expunge -u $user \( mailbox Trash OR mailbox Junk \) SAVEDBEFORE 8w ; done

Reply |

Recent comments
See also...
Recent posts

RSS feeds