Notmuch E-Mail with Emacs, mbsync and friends

17.04.2025 Permalink

Introduction

Over the course of a workday I receive plenty of mails, many are notifications regarding tickets, commits and the like. Some are calls for help that require a response on the same day.

For more than ten years I used Thunderbird on Linux for mail and scheduling, and it worked ok for me. Nevertheless there was room for improvement:

Each incoming mail is a potential interruption of whatever I'm busy with. I don't like to be bothered by tens of desktop notifications each day, let alone audible signals. On the other hand I'd like to see if there are unread messages without having to switch into a mail client. I tried Birdtray to place some reduced, non-intrusive information onto my desktop bar, but it didn't satisfy me.
I work for several clients and projects and it is important for me to have a distinct mail folder for each. Thunderbird offers message filters that automatically sort messages into folders, which had great benefits for me. However, a somewhat simpler and more flexible approach I prefer is automated tagging.
For me, searching for mails in Thunderbird feels cumbersome and slow.
I make extensive use of Emacs Org mode for task management and I wanted a seamless connection between todo items resulting from a mail with the mail itself. I consider a mail as read as soon as at least one todo item points to it. This leads to Inbox Zero which seems to me like a perfect addition to a GTD style of productivity management.
The Org agenda can display diary items, but my appointments are stored in systems that provide scheduling data in iCalendar format, so a conversion would be very welcomed.
I like the idea of having mails as plain text files, which enables automated processing with simple scripts.

Meet the team

After some research I got a list of some components that promised to satisfy my requirements:

three new folders in my home: ~/Mail, ~/Calendar and ~/Contacts,
the mail indexer notmuch and its Emacs package as a frontend,
mbsync and mpop to fetch mail from IMAP and POP3 servers,
msmtp to send mails via SMTP,
vdirsyncer to synchronize calendar and contact items with iCal servers,
pass for password management and provision, and
systemd for scheduling.

You can see that there are quite some pieces needed to form an open E-Mail system that can be used with Emacs. Compare this with a one-stop-go solution like Thunderbird, which, btw, I still need for scheduling appointments with colleagues. Nevertheless, it was worth the effort to now have my mails in Emacs, because I can operate on them very quickly using only the keyboard.

And I like the Linux/Unix approach of having composable tools, each having a single purpose and being good at that. Each piece of the puzzle can be understood and tested separately using the command line.

What follows is a description of how to configure these tools so that they work together nicely. We'll start with the folder organization and the overall service management and then look at each piece in more detail.

Folder organization

Mail

My ~/Mail folder contains a main folder for each account, plus some special folders:

Mail
├── .notmuch
├── archives
│   └── private
│       ├── This
│       ├── That
│       └── Anything_Else
├── drafts
├── templates
├── falko.riemenschneider@arcor.de
│   ├── Archives
│   ├── Drafts
│   ├── INBOX
│   ├── Junk
│   ├── Sent
│   ├── Spam
│   └── Trash
├── falko.riemenschneider@gmail.com
│   ├── Drafts
│   ├── INBOX
│   ├── Sent
│   └── Trash
└── info@falkoriemenschneider.de
   ├── Drafts
   ├── INBOX
   ├── Sent
   └── Trash

The .notmuch folder contains three shell scripts, so called hooks in notmuch parlance. I cover them in the notmuch section below.

My mail signatures are simple .txt files which I also keep in ~/Mail.

Calendar & Contacts

The ~/Calendar and ~/Contacts folder look similar to each other. For each account there is one main folder.

Calendar
├── private
├── dt
└── family
    └── calendar

Contacts
├── private
└── family
    └── addresses

The Emacs diary format allows for includes, so I put a diary file into ~/Calendar to ensure that all event items show up in my Org agenda. It looks like this:

#include "~/Calendar/private.diary"
#include "~/Calendar/dt.diary"
#include "~/Calendar/family.diary"

Scheduling synchronization

The synchronization of mails, events and contacts must be done on a regular basis. For me every five minutes is good enough. systemd is a reasonable choice for managing user related services and scheduling tasks. You can control it via unit files stored in ~/.config/systemd/user. For my needs I needed three units:

notify.service is a unit used by the other two and serves only to produce a desktop notification in case the synchronization service failed.

[Unit]
Description=Send Systemd Notifications to Desktop

[Service]
Type=oneshot
ExecStart=/usr/bin/notify-send -a "Systemd" %i

[Install]
WantedBy=default.target

notmuch.service is repeatedly started every five minutes, which just executes a simple command notmuch new:

[Unit]
Description=Sync and index mail with mbsync and notmuch
OnFailure=notify.service

[Service]
ExecStart=/usr/bin/notmuch new
Restart=always
RestartSec=300
RuntimeMaxSec=300

[Install]
WantedBy=default.target

diary.service serves the syncronization of contacts and calendar items. Its complexity is hidden in a cal--sync.sh shell script which I describe in the tool section below:

[Unit]
Description=Sync icalendar entries and overwrite Emacs diary files
OnFailure=notify.service

[Service]
ExecStart=/home/riemenschneider/bin/cal--sync.sh
Restart=always
RestartSec=300
RuntimeMaxSec=300

[Install]
WantedBy=default.target

The files are placed in ~/.config/systemd/user and can be enabled using the systemctl command.

Tool configuration

pass

pass is an automation-friendly, terminal-based tool for password management on Linux. Each password (and accompanying info) is stored in its own plain text file somewhere in ~/.password-store, encrypted via a gpg public key. It is well documented, has clients for Chrome and Firefox and grants access based on gpg-agent and a single master password which is essentially the passphrase to a private gpg key. The only piece of configuration I had to change was the default-cache-ttl value in ~/.gnupg/gpg-agent.conf. Since its primary interface is the command line it can easily be used by other tools whenever a password is required.

mbsync

mbsync is used for synchronization of IMAP mail folders. Its configuration lives in ~/.mbsyncrc. For each account you need to specify the local folders, the remote folders and some rules for synchronization between them. As an example here's my configuration for my Gmail account:

IMAPStore falko.riemenschneider@gmail.com-remote
Host imap.googlemail.com
Port 993
User falko.riemenschneider
# contains an app password, see https://myaccount.google.com/apppasswords
PassCmd "pass private/google-mbsync"
SSLType IMAPS
# not needed, so turned of
# AuthMechs XOAUTH2
CertificateFile /etc/ssl/certs/ca-certificates.crt

# Local mailbox falko.riemenschneider@gmail.com
MaildirStore falko.riemenschneider@gmail.com-local
Subfolders Verbatim
Path ~/Mail/falko.riemenschneider@gmail.com/
INBOX ~/Mail/falko.riemenschneider@gmail.com/INBOX
Flatten .

# Syncronization falko.riemenschneider@gmail.com
Channel falko.riemenschneider@gmail.com
Far :falko.riemenschneider@gmail.com-remote:
Near :falko.riemenschneider@gmail.com-local:
Patterns INBOX Drafts Sent Trash
SyncState *
Create Both
Expunge Both
MaxMessages 0

Since Google usually requires a two-factor authentication (2FA) you'll need to setup an app password, which you must store (without any spaces) in a file in ~/.password-store/.

To test if syncronization works you can use a command similar to mbsync falko.riemenschneider@gmail.com.

mpop

mpop is used for synchronization of POP3 mail accounts. Its configuration file is ~/.mpoprc. Here's an example for one account:

account info@falkoriemenschneider.de
delivery maildir ~/Mail/info@falkoriemenschneider.de/INBOX
host pop3.netcologne.de
timeout 10
user info@falkoriemenschneider.de
tls on
tls_starttls on
auth plain
passwordeval pass show private/netcologne-mail-info@falkoriemenschneider.de
keep off

To test if the synchronization works you can use the command mpop --all-accounts.

msmtp

msmtp is a tool for actually sending mail via SMTP protocol. The configuration is stored in ~/.msmtprc and looks for example like this:

defaults
syslog on

account falko.riemenschneider@arcor.de
from falko.riemenschneider@arcor.de
host mail.arcor.de
port 587
auth plain
tls on
tls_starttls on
user falko.riemenschneider@arcor.de
passwordeval pass private/vodafone

account falko.riemenschneider@gmail.com
from falko.riemenschneider@gmail.com
host smtp.googlemail.com
port 465
auth plain
tls on
tls_starttls off
user falko.riemenschneider@gmail.com
passwordeval pass private/google-mbsync

vdirsyncer

vdirsyncer is used to synchronize contacts and calendar items via CardDAV and CalDAV protocols. Its configuration and working files are stored within a folder ~/.vdirsyncer. As an example here is what I need in ~/.vdirsyncer/config for synchronizing family related data with my ~/Contacts and ~/Calendar folders:

[general]
status_path = "~/.vdirsyncer/status/"

#
# family Calendar
#

# CALDAV
[pair family_calendar]
a = "family_calendar_local"
b = "family_calendar_remote"
collections = ["calendar"]
metadata = ["displayname", "color"]

# To resolve a conflict the following values are possible:
#   `null` - abort when collisions occur (default)
#   `"a wins"` - assume a's items to be more up-to-date
#   `"b wins"` - assume b's items to be more up-to-date
conflict_resolution = "b wins"

[storage family_calendar_local]
type = "filesystem"
path = "~/Calendar/family/"
fileext = ".ics"

[storage family_calendar_remote]
type = "caldav"
url = "http://internalservername/radicale/family/calendar/"
username = "family"
password.fetch = ["command", "pass", "private/family-vdirsyncer"]

# CARDDAV
[pair family_contacts]
a = "family_contacts_local"
b = "family_contacts_remote"
collections = ["addresses"]
metadata = ["displayname"]

[storage family_contacts_local]
type = "filesystem"
path = "~/Contacts/family/"
fileext = ".vcf"

[storage family_contacts_remote]
type = "carddav"
url = "http://internalservername/radicale/family/addresses/"
username = "family"
password.fetch = ["command", "pass", "private/family-vdirsyncer"]

To test if synchronization works you can use a command like vdirsyncer sync family_calendar/calendar.

notmuch

notmuch is a super-fast mail file indexer and a search facility. To retrieve new mail the command notmuch new executes two custom shell scripts (called hooks) that I keep in ~/Mail/.notmuch:

pre-new executes the mbsync and mpop commands for all my accounts. In addition it moves mails with the deleted tag to a Trash folder or deletes them permanently if they are older than 21 days. It recovers mails in Trash without the deleted tag by moving them to the INBOX/cur folder for the correspondent account.
post-new executes an array of notmuch tag commands that attach all kinds of tags to mails according to From or To addresses. This makes searching for mails mind-bogglingly fast.

In addition there is post-insert which allows you to apply tags to mails that you have created and sent.

The main configuration is placed in ~/.notmuch-config and looks like this:

[database]
path=/home/riemenschneider/Mail
hook_dir=/home/riemenschneider/Mail/.notmuch

[user]
name=Falko Riemenschneider
primary_email=riemenschneider@doctronic.de
other_email=falko.riemenschneider@arcor.de;admin@falkoriemenschneider.de;falko.riemenschneider@gmail.com

[new]
ignore=.mbsyncstate;.mbsyncstate.lock;.uidvalidity;signature-dt.txt;signature-private.txt;signature-falkoriemenschneider.txt;templates

[search]
exclude_tags=trash;deleted

[maildir]

[query]
attention=tag:unread and -tag:spam and -tag:delivery and -tag:notifications and -tag:calendar

Once you have the mail synchronization based on mbsync and/or mpop in place and working, you'll want to include their execution commands in pre-new. Then the main command for fetching new mail is notmuch new. After you have some mails in your new mail folders please take some time to make yourself familiar with notmuch tagging and searching. You'll quickly see that you can now work with your mail on a different level.

i3 block for output of mail count on the desktop

I use i3wm as a window manager and i3blocks to display some information in the desktop status bar. To retrieve the number of unread mails the following script suffices:

#!/bin/bash
notmuch search --output=files --duplicate=1 tag:unread | wc -l

With this in place I can always see if there is something new by looking at the top of my screen. No notification and no interruption whatsoever. I decide when to turn my attention to it.

Emacs

There is a nice major mode for reading mail in Emacs in the package notmuch. Please see some screenshots from their website.

The notmuch-hello screen is basically a starting point for triggering queries. Once you have sensible tags defined and applied them using post-new you'll pre-define queries for your notmuch-hello screen. You can tab between the queries or write an individual one. When you hit the Enter key you see the query result in an instant. You can add or remove tags to listed messages via + and - keys. Now, you can copy a link to a mail (for me with C-x C-l) and insert it into an Org mode todo item (for me with C-c C-l). When you need to open the mail again as part of dealing with the todo item, just hit Enter on the link within the item (if you have configured Org mode like this). This kind of handling mail feels totally different compared to a bloated reading app that has no connection to my Org.

My Notmuch related Emacs configuration is on GitHub. To support multiple mail accounts for writing and sending new mail I had to extend the default a bit.

From .ics to Emacs .diary

Syncing calendar and contact data can be done via a command similar to vdirsyncer sync family_calendar/calendar. But Emacs Org agenda needs event data in .diary format. To convert the synced .ics contents to Emacs .diary format I use Emacs in a batch mode like this to execute the Elisp function icalendar-import-file:

emacs -batch -kill -eval '(progn (require '\''calendar) (setq calendar-date-style '\''european) (icalendar-import-file "~/Calendar/family-all.ics" "~/Calendar/family.diary"))'

The whole script cal--sync.sh as referred to by the systemd user service diary.service then looks like this:

#!/bin/bash
#
# Sync iCalendar collections and create diary files from ics collections
#

case "$TERM" in
    xterm*|rxvt*)
        NORMAL="$(tput sgr0)"
        RED="$(tput setaf 1)"
        GREEN="$(tput setaf 2)"
        ;;
    *)
        ;;
esac

reachable=$(ping -c 1 internalservername 2>/dev/null 1>&2; echo $?)
if [[ 0 -eq $reachable ]]; then
    echo "${GREEN}internalservername is reachable${NORMAL}"
    vdirsyncer sync family_calendar/calendar
    cat ~/Calendar/family/calendar/*.ics > ~/Calendar/family-all.ics
    cp /dev/null ~/Calendar/family.diary
    emacs -batch -kill -eval '(progn (require '\''calendar) (setq calendar-date-style '\''european) (icalendar-import-file "~/Calendar/family-all.ics" "~/Calendar/family.diary"))'
    if [[ $? -ne 0 ]]; then
        notify-send --category=email "Error syncing with family calendar"
    fi
else
    echo "${RED}family calendar is UNreachable${NORMAL}"
fi

This script produces one or more .diary files that are included in ~/Calendar/diary. My Emacs configuration for the calendar package sets the diary-file to this location. Now Org agenda can fetch the event data using this pointer.

Summary

It's clear from the description above that this is already a little project in its own right. If you're not into crafting with linux tools you should not embark on this trip.

I have been using it for more than a year now and it has also been through a re-installation of my Linux machine. Since it consists only of files at specific places it's fairly robust and I did not experience any major problems so far.

Reading and managing mail has become much quicker for me. With the easy link from todo items to mails, Inbox Zero has become a reality. Writing plain mail in Emacs is fine, even with attachments, but I admit it somehow still feels a bit unusual.

For the time being Thunderbird remains a daily companion but only for scheduling appointments. I didn't open its messages tab for months now.