Syncing Gmail with mbsync

2019-05-14

I've decided that I want to try to reduce my dependence on Google's services. A large part of my Google footprint is Gmail, which I have used for about 15 years.

I've not entirely planned through my migration yet, but I think I want to use email in a more transactional manner, as was intended; I want to retrive emails from a server, store and back them up locally on hardware which I control, and then send via SMTP. I really don't find having access to my email via a browser that convenient as it's rare I don't have a laptop or a phone from which I can SSH into my own computers. Push notifications for email are an annoyance; I'd rather interact with my email on my own schedule in batches.

No matter what the overall plan, the first step is to being to sync a considerable volume of email from Google's servers to my own computer.

For this, I'm using isync, which can reliably sync email from GMail's IMAP servers to a local Maildir, in one (or both) directions. Note that while the package is called isync, the binary I'm using is called mbsync, and the configuration file is .mbsyncrc.

That isync can sync mail in both directions is important as I want to initially have my local setup mirror my current GMail setup, with access via either method, and to later transition to only downloading from GMail.

It's also necessary to setup an 'App Password' in GMail, which will allow IMAP access to your mailbox outside of the usual Google Account authentication flow. Once that password has been created, I suggest you encrypt it locally and extract it via gpg (as the configuration below demonstrates).

My configuration file (~/.mbsyncrc) is as follows:

IMAPAccount personal
Host imap.gmail.com
User jon@jonatkinson.co.uk
PassCmd "<run gpg here>"

SSLType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt

IMAPStore personal-remote
Account personal

MaildirStore personal-local
Subfolders Verbatim
Path ~/mail/personal/
Inbox ~/mail/personal/inbox

Channel personal-default
Master :personal-remote:
Slave :personal-local:
Patterns * ![Gmail]* 
Create Both
SyncState *
Sync All

Channel personal-sent
Master :personal-remote:"[Gmail]/Sent Mail"
Slave :personal-local:sent
Create Slave
Sync Pull

Channel personal-trash
Master :personal-remote:"[Gmail]/Trash"
Slave :personal-local:trash
Create Slave
Sync Pull

# Get all the channels together into a group.
Group personal
Channel personal-default
Channel personal-sent
Channel personal-trash

The isync configuration files are in an odd, stanza-based format. The concepts are based around groups and channels. A channel is a bidirectional mapping between a two mail stores (in this case the remote IMAP server and the local Maildir, but this could just as easily be two remote IMAP services). From top to bottom, that configuration file does the following:

The first sync will take a long time (for my decade and a half of email, to my home server over a 75mb link it was around a day, but that also accounts for some gentle throttling on the remote IMAP service). I ran this in a screen session:

$ mbsync -Dmn personal

Once the initial sync has completed, subsequent activity will be much quicker. I use the following systemd timer and service to sync every five minutes:

$ cat /etc/systemd/user/mbsync.timer 
[Unit]
Description=Mailbox synchronization timer

[Timer]
OnBootSec=2m
OnUnitActiveSec=5m
Unit=mbsync.service

[Install]
WantedBy=timers.target

$ cat ~/.config/systemd/user/mbsync.service 
[Unit]
Description=Mailbox synchronization service

[Service]
Type=oneshot
ExecStart=/usr/bin/mbsync -Va