Solr
is a Lucene indexing server. Dovecot
communicates to it using HTTP/XML
queries. With this indexing server, you can do text searches in your emails.
- Download latest version: v. 9.1.0
- Solr documentation page
- Solr FTS Engine documentation page for Dovecot
Upgrading to version 9.1.0
- More info here
Dovecot
has recently released the schema files updated to version 9, alternative to the ones that you can find here
Before starting check that your java
is at least at version 11.
Download version 9.1.0:
wget https://www.apache.org/dyn/closer.lua/solr/solr/9.1.0/solr-9.1.0.tgz?action=download -O solr-9.1.0.tgz
Then stop your Solr
server and run the upgrade with the -f
(upgrade) and -n
(do not start the server when finished) options:
tar xzf solr-9.1.0.tgz solr-9.1.0/bin/install_solr_service.sh --strip-components=2 sudo bash ./install_solr_service.sh solr-9.1.0.tgz -f -n
Slackware
users will have to do:
wget https://notes.sagredo.eu/files/qmail/solr/install_solr_slackware.sh chmod +x install_solr_slackware.sh ./install_solr_slackware.sh solr-9.1.0.tgz -f -n
Now download and install the new schema and configuration files for Dovecot
cd /var/solr/data/dovecot/conf rm -f schema.xml managed-schema solrconfig.xml wget https://notes.sagredo.eu/files/qmail/solr/schema-9.1.0.xml -O schema.xml wget https://notes.sagredo.eu/files/qmail/solr/solrconfig-9.1.0.xml -O solrconfig.xml chown solr:solr solrconfig.xml schema.xml
The new configuration file replaces LRUCache
with CaffeineCache
and changes the location of the .jar
libraries (diff here).
Configure your /etc/default/solr.in.sh file, as many options are changed. Then restart the Solr
server.
Finally upgrade the indexes (edit the downloaded script to insert your Dovecot
password)
wget https://notes.sagredo.eu/files/qmail/solr/solr_rescan_index.sh chmod +x solr_rescan_index.sh chown root:root solr_rescan_index.sh chmod o-wrx solr_rescan_index.sh ./solr_rescan_index.sh Stopping Dovecot . <?xml version="1.0" encoding="UTF-8"?> <response> <lst name="responseHeader"> <int name="status">0</int> <int name="QTime">20</int> </lst> </response> Starting Dovecot.
If the script does not return errors (status=0
) you are ok. If you get errors, double check the Authorization and the Solr
's dovecot user credentials.
Installing
Solr
is a java
servlet which requires openjdk
v. 11 or later.
# java -version openjdk version "11.0.17" 2022-10-18 OpenJDK Runtime Environment (build 11.0.17+8) OpenJDK 64-Bit Server VM (build 11.0.17+8, mixed mode)
Be sure that you have the java
binary in your path and that you have defined the variable JAVA_HOME
, for example
PATH=$PATH:/usr/lib64/java/bin/ JAVA_HOME=/usr/lib64/java/
Download the binary version of Solr
and install
cd /usr/local/src wget https://www.apache.org/dyn/closer.lua/solr/solr/9.1.0/solr-9.1.0.tgz?action=download -O solr-9.1.0.tgz
Extract the installer from the archive and run it. The installer will work for most Linux
distributions based on systemd
.
tar xzf solr-9.1.0.tgz solr-9.1.0/bin/install_solr_service.sh --strip-components=2 sudo bash ./install_solr_service.sh solr-9.1.0.tgz
The server will be launched by systemd
at boot time.
Installing on Slackware
If you are a Slackware
user like me, the installer will not work. Use instead my modified install_solr_slackware.sh script to do the installation:
wget https://notes.sagredo.eu/files/qmail/solr/install_solr_slackware.sh ./install_solr_slackware.sh solr-9.1.0.tgz
Configuration
The location of the server is /opt/solr, while the logs are in /var/solr/logs.
The servers' configuration file is stored in /opt/solr/bin/solr.in.sh.orig file, which is conveniently copied to /etc/defaults/solr.in.sh so that we can keep it after a future upgrade. Be aware that the double quotes are important in this file.
SOLR_TIMEZONE="Europe/Rome" SOLR_IP_ALLOWLIST="127.0.0.1, 10.0.0.0/24" SOLR_JETTY_HOST="0.0.0.0" SOLR_SECURITY_MANAGER_ENABLED=true SOLR_OPTS="$SOLR_OPTS -Dsolr.allowUrls=http://solr.yourdomain.tld:8983"
SOLR_IP_ALLOWLIST
allows connections from my DMZ
. Be aware that SOLR_IP_ALLOWLIST
was SOLR_IP_WHITELIST
before version 9.
Setting SOLR_JETTY_HOST="0.0.0.0"
was a little difficult to guess for me, because I have Solr
in a different virtual server than qmail
and apache
. It allows Solr
to accept connections from the outnet. If you have all the servers in the same host, you can leave SOLR_JETTY_HOST=127.0.0.1
commented out.
Setting the limits
Solr
will run as the user solr:solr
, which is created for you during the installation process. This user will ask to raise the open file limit up to 65000. In my Slackware
the default limit is 1024. To increase the limit for the solr
user edit your /etc/security/limits.conf or add a solr.conf file in your limits.d folder with the following instructions:
solr soft nofile 65536 solr soft nproc 65536 solr hard nofile 65536 solr hard nproc 65536
In my case solr
lives in an LXC
unprivileged container, so the above limit had to be set not only for the solr
user inside the container, but also in the host for the user who runs the container itself. In addition, the container must have this option in its config file:
lxc.prlimit.nofile = 65536
Running
We are ready to run the server or restart if it's already running:
sudo systemctl stop solr sudo systemctl start solr sudo systemctl status solr
Slackware
users will do as follows (remember to add the start command in your rc.local
)
/etc/init.d/solr start sleep 5 /etc/init.d/solr status
Control panel setup
General settings are available also via web panel. You have to setup a virtual domain where Solr
will be listening. This is how to configure an apache
virtual host to serve the control panel. I have set a proxy to connect via regular port 443.
<VirtualHost *:443> SSL stuff here ServerName solr.mydomain.tld ErrorLog ${LOGDIR}/solr_error.log LogLevel warn CustomLog ${LOGDIR}/solr_access.f2b.log combined SSLProxyEngine On ProxyRequests Off ProxyPass / http://solr.mydomain.tld:8983/ ProxyPassReverse / http://solr.mydomain.tld:8983/ </VirtualHost>
When you browse to https://solr.mydomain.tld you will see that the control panel does not require any login and that it complains of the lack of any security policy.
Securing
Look to where your solr.home
environment variable is set navigating to the dashboard:
Change to that directory and install a security.json file which will define your plugins for authentication, authorization and auditlogging:
cat > /var/solr/data/security.json << __EOF__ { "authentication":{ "blockUnknown": true, "class":"solr.BasicAuthPlugin", "credentials":{"solr":"IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}, "realm":"My Solr users", "forwardCredentials": false }, "authorization":{ "class":"solr.RuleBasedAuthorizationPlugin", "permissions":[{"name":"security-edit","role":"admin"}], "user-role":{"solr":"admin"} }, "auditlogging":{ "class": "solr.SolrLogAuditLoggerPlugin", "async": true, "blockAsync" : false, "numThreads" : 2, "queueSize" : 4096, "eventTypes": ["REJECTED", "ANONYMOUS_REJECTED", "UNAUTHORIZED", "COMPLETED", "ERROR"] } } __EOF__ chown solr:solr /var/solr/data/security.json chmod o-r /var/solr/data/security.json
Now restart the server and you should be able to login. The login user is 'solr', with a temporary password 'SolrRocks'. Change it after your first login.
The security section suggests to fix some permission issues. In addition, you have to create a "dovecot" user and grant it read and update permissions in order to perform its tasks. Do not use special characters like "@" in your password, because you can have parsing failures in the dovecot
connection.
This is my set up:
I have TLS
disabled since the connection between the apache
proxy and the Solr
server is via http
.
From now on, you can manage users, roles and permissions acting on the control panel rather than editing the above security.json file, which will be modified for you.
Dovecot
core setup
To use Solr
with Dovecot
, it needs to be configured specifically (Solr
must be running):
sudo -u solr /opt/solr/bin/solr create -c dovecot
The data and the configuration files for Dovecot
are under /var/solr/data/dovecot.
The official Dovecot
documentation recommends to change a couple of configuration files:
cd /var/solr/data/dovecot/conf rm -f schema.xml managed-schema solrconfig.xml wget https://notes.sagredo.eu/files/qmail/solr/9.1/schema-9.1.0.xml -O schema.xml wget https://notes.sagredo.eu/files/qmail/solr/9.1/solrconfig-9.1.0.xml -O solrconfig.xml chown solr:solr solrconfig.xml schema.xml
The managed-schema file is generated based on schema.xml.
We already compiled Dovecot
with the Solr
support (--with-solr
configuration). Add the plugin in the 10-mail.conf file:
mail_plugins = $mail_plugins fts fts_solr
and add the configuration in the plugin{...}
block inside the 90-plugin.conf file:
plugin { fts = solr fts_solr = url=http://dovecot:password@solr.mydomain.tld:8983/solr/dovecot/ # eventually add debug to the previous line ... }
of course dovecot:password
are the credentials for the Solr
's user dovecot, that you created before, and solr.mydomain.tld is the domain where Solr
will be reachable.
Testing
# telnet 0 143 Trying 0.0.0.0... Connected to 0. Escape character is '^]'. * OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ STARTTLS AUTH=PLAIN AUTH=LOGIN] Dovecot ready. a login user@mydomain.tld password a OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NA MESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LI TERAL+ NOTIFY SPECIAL-USE QUOTA] Logged in a select Inbox * FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $MDNSent Junk $label3 $Forwarded) * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $MDNSent Junk $label3 $Forwarded \*)] Flags permitted. * 308 EXISTS * 0 RECENT * OK [UIDVALIDITY 1285590712] UIDs valid * OK [UIDNEXT 19895] Predicted next UID * OK [HIGHESTMODSEQ 31675] Highest a OK [READ-WRITE] Select completed (0.001 + 0.000 secs). a SEARCH text "Dovecot" * SEARCH 35 50 51 55 56 57 58 62 63 74 75 76 77 121 129 130 146 150 151 158 163 164 165 173 196 201 202 203 213 225 226 227 228 230 231 232 249 250 262 263 264 289 a OK Search completed (0.309 + 0.001 + 0.043 secs). a logout
Open your dovecot
log and be sure that there are no errors. Then look at the Solr
's logs. This is a clean /var/solr/logs/solr.log:
2023-01-04 13:42:44.423 INFO (qtp1141500277-21) [ dovecot] o.a.s.c.S.Request webapp=/solr path=/select params={q={!lucene+q.op%3DAND}(hdr:Dovecot+OR+body:Dovecot)&fl=uid,score&sort=uid+a sc&fq=%2Bbox:8af5d82b9ae4c94cbe610000364df272+%2Buser:user@mydomain.tld&rows=21842&wt=xml} hits=44 status=0 QTime=15 2023-01-04 13:42:44.424 INFO (audit-31-thread-1) [ ] o.a.s.s.SolrLogAuditLoggerPlugin type="COMPLETED" message="Completed" method="GET" status="200" requestType="SEARCH" username="dovecot" resource="/select" queryString="wt=xml&fl=uid,score&rows=21842&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Dovecot+OR+body:Dovecot)&fq=%2Bbox:8af5d82b9ae4c94cbe610000364df272+%2Buser:user@mydomain.tld" collections=[]
and this is the request on the log file /var/solr/logs/YYYY_MM_DD.request.log. You can see that it was served correctly with a final 200 code:
10.0.0.4 - - [04/Jan/2023:13:42:44 +0000] "GET /solr/dovecot/select?wt=xml&fl=uid,score&rows=21842&sort=uid+asc&q=%7b!lucene+q.op%3dAND%7d(hdr:Dovecot+OR+body:Dovecot)&fq=%2Bbox:8af5d82b9ae 4c94cbe610000364df272+%2Buser:user@mydomain.tld HTTP/1.1" 200 4550
If you see a 451 error code you have to double check the authorizations.