programming (and other) musings
11 Mar 2021

notmuch queries via consult

This package provides notmuch queries in emacs using consult. It offers interactive functions to launch search processes using the notmuch executable and present their results in a completion minibuffer and, after selection of a candidate, single message and tree views.

The package's elisp code is tangled from this literate program.

Dependencies and installation

We depend on the emacs packages consult and notmuch:

(require 'consult)
(require 'notmuch)

Both, as well as this package, are available in MELPA, so that's the easiest way to install consult-notmuch. You can also simply tangle this document and put the resulting consult-notmuch.el file in your load path (provided you have its dependencies in your load path too), or obtain it from its git repository.

Usage

Consult interface to notmuch search

Our objective is to use consult to perform notmuch queries, and show their results in the completions minibuffer and, upon selection, on a dedicated notmuch buffer. Notmuch knows how to show either single messages or full threads (or trees), so our public interface is going to consist of two autoloaded commands:

;;;###autoload
(defun consult-notmuch (&optional initial)
  "Search for your email in notmuch, showing single messages.
If given, use INITIAL as the starting point of the query."
  (interactive)
  (consult-notmuch--show (consult-notmuch--search initial)))

;;;###autoload
(defun consult-notmuch-tree (&optional initial)
  "Search for your email in notmuch, showing full candidate tree.
If given, use INITIAL as the starting point of the query."
  (interactive)
  (consult-notmuch--tree (consult-notmuch--search initial)))

They're implemented in terms of a common search driver, with the only difference being how we show the final result of the auto-completion call.

When using them you'll notice how we automatically inject consult's split character (# by default) as the first one in the query string, so that our search term can be divided into a string passed to notmuch and a second half (after inserting a second # in our query) that is used by emacs to filter the results of the former. You'll also see that there's a preview available when traversing the list of candidates in the minibuffer.

It is also worth remembering that the input is a generic notmuch query, so one can, for instance, use the initial contents to define specific query commands. For example, i have a set of mailboxes under a subdirectory called feeds (where mails retrieved by rss2email end up after a dovecot sieve), so i could define this command in my init files:

(defun jao-consult-notmuch-feeds (&optional tree)
  (interactive "P")
  (let ((init "folder:/feeds/ "))
    (if tree (consult-notmuch-tree init) (consult-notmuch init))))

or be more generic and read from a completing prompt the subfolder of my ~/var/mail directory i want to use:

(defun jao-consult-notmuch-folder (&optional tree folder)
  (interactive "P")
  (let* ((root "~/var/mail/")
         (folder (if folder
                     (file-name-as-directory folder)
                     (thread-first (read-directory-name "Mailbox: " root)
                       (file-relative-name root))))
         (folder (replace-regexp-in-string "/\\(.\\)" "\\\\/\\1" folder))
         (init (format "folder:/%s " folder)))
    (if tree (consult-notmuch-tree init) (consult-notmuch init))))

(defun jao-consult-notmuch-feeds (&optional tree)
  (interactive "P")
  (jao-consult-folder tree "feeds"))

An appetiser: address search

As a simple example of how the simplest asynchronous consult query looks, let's write a function that allows us to select a mail address using a notmuch address query, which in the CLI looks like

notmuch address --format=text <query>

A brute force approach is to issue the command above synchronously, split the result, and use completing-read:

(defun read-notmuch-address (input)
  (let* ((cmd (format "notmuch address --format=text '%s'" input))
         (res (shell-command-to-string cmd)))
    (completing-read "Notmuch address: " (split-string res "\n"))))

If your notmuch database is not that big, this approach actually works reasonably well, but it doesn't scale with its size. That's where an asynchronous command can help: we call the shell command as the input varies, collecting and displaying the results as they're available. Here's how that looks using consult asynchronous command infrastructure:

(defun consult-notmuch--address-command (input)
  "Spec for an async command querying a notmuch address with INPUT."
  `("notmuch" "address" "--format=text" ,input))

(defun consult-notmuch-address-compose (address)
  "Compose an email to a given ADDRESS."
  (let ((other-headers (and notmuch-always-prompt-for-sender
                            `((From . ,(notmuch-mua-prompt-for-sender))))))
    (notmuch-mua-mail address
                      nil
                      other-headers
                      nil
                      (notmuch-mua-get-switch-function))))

(defun consult-notmuch--address-prompt ()
  (consult--read (consult--async-command #'consult-notmuch--address-command)
                 :prompt "Notmuch addresses: "
                 :sort nil
                 :category 'notmuch-address))

;;;###autoload
(defun consult-notmuch-address (&optional multi-select-p initial-addr)
  "Search the notmuch db for an email address and compose mail to it.
With a prefix argument, prompt multiple times until there
is an empty input."
  (interactive "P")
  (if multi-select-p
      (cl-loop for addr = (consult-notmuch--address-prompt)
               until (eql (length addr) 0)
               collect addr into addrs
               finally (consult-notmuch-address-compose
                        (mapconcat #'identity
                                   (if initial-addr
                                       (cons initial-addr addrs)
                                     addrs)
                                   ", ")))
    (consult-notmuch-address-compose (consult-notmuch--address-prompt))))

Buffer narrowing

If you have many buffers, you may want a convenient way to switch specifically among notmuch buffers. The consult-notmuch-buffer-source source can be used for this purpose:

(defun consult-notmuch--interesting-buffers ()
  "Return a list of names of buffers with interesting notmuch data."
  (consult--buffer-query
   :as (lambda (buf)
         (when (notmuch-interesting-buffer buf)
           (buffer-name buf)))))

;;;###autoload
(defvar consult-notmuch-buffer-source
  '(:name "Notmuch Buffer"
    :narrow (?n . "Notmuch")
    :hidden t
    :category buffer
    :face consult-buffer
    :history buffer-name-history
    :state consult--buffer-state
    :items consult-notmuch--interesting-buffers)
  "Notmuch buffer candidate source for `consult-buffer'.")

This source can be used with consult-buffer by adding it to consult-buffer-sources:

(add-to-list 'consult-buffer-sources 'consult-notmuch-buffer-source)

With the above configuration, you can initiate consult-buffer and then type n followed by a space to narrow the set of buffers to just notmuch buffers.

Customization

As customary, we're going to use a customization group, as a subgroup of notmuch's one:

(defgroup consult-notmuch nil
  "Options for `consult-notmuch'."
  :group 'consult)

and our first user option will tell us whether we display single messages in the matches list (extracted via notmuch-show) or thread groups (a la notmuch-search):

(defcustom consult-notmuch-show-single-message t
  "Show only the matching message or the whole thread in listings."
  :type 'boolean)

When displaying search results in the minibuffer, we'll want to extract the authors, date and subject and thread count for each message and give them a format defined by the custom variable:

(defcustom consult-notmuch-result-format
  '(("date" . "%12s  ")
    ("count" . "%-7s ")
    ("authors" . "%-20s")
    ("subject" . "  %-54s")
    ("tags" . " (%s)"))
  "Format for matching candidates in minibuffer.
Supported fields are: date, authors, subject, count and tags."
  :type '(alist :key-type string :value-type string))

which has the same semantics as notmuch-search-result-format.

Implementation

Consult search function

The core of our implementation should a call to consult--read with a closure to obtain completion candidates based on a call to notmuch search or notmuch show as an asynchronous process. For that, we'll use consult's helper consult--async-command. This function takes as first argument a string representing the command to be called to obtain completion candidates, followed by any transformations we want to apply to them before being displayed. Thus, our candidates generator will look like:

(defun consult-notmuch--command (input)
  "Construct a search command for emails containing INPUT."
  (if consult-notmuch-show-single-message
      `("notmuch" "show" "--body=false" "--unthreaded" ,input)
    `("notmuch" "search" ,input)))

(defun consult-notmuch--search (&optional initial)
  "Perform an asynchronous notmuch search via `consult--read'.
If given, use INITIAL as the starting point of the query."
  (setq consult-notmuch--partial-parse nil)
  (consult--read (consult--async-command
                     #'consult-notmuch--command
                   (consult--async-filter #'identity)
                   (consult--async-map #'consult-notmuch--transformer))
                 :prompt "Notmuch search: "
                 :require-match t
                 :initial (consult--async-split-initial initial)
                 :history '(:input consult-notmuch-history)
                 :state #'consult-notmuch--preview
                 :lookup #'consult--lookup-member
                 :category 'notmuch-result
                 :sort nil))

In the code above we're also using a preview function (described below), and a history variable:

(defvar consult-notmuch-history nil
  "History for `consult-notmuch'.")

and the candidates transformer will depend on whether we're displaying threads or single messages:

(defun consult-notmuch--transformer (str)
  "Transform STR to notmuch display style."
  (if consult-notmuch-show-single-message
      (consult-notmuch--show-transformer str)
    (consult-notmuch--search-transformer str)))

Formatting search results

Using consult-notmuch-result-format, we are going to return a string representation from a plist describing the current message, reusing notmuch's facility notmuch-tree-format-field, with the added trick of storing the current message or thread id in a text property, so that it can latter be used for displaying the message preview:

(defun consult-notmuch--format-field (spec msg)
  "Return a string for SPEC given the MSG metadata."
  (let ((field (car spec)))
    (cond ((equal field "count")
           (when-let (cnt (plist-get msg :count))
             (format (cdr spec) cnt)))
          ((equal field "tags")
           (when (plist-get msg :tags)
             (notmuch-tree-format-field "tags" (cdr spec) msg)))
          (t (notmuch-tree-format-field field (cdr spec) msg)))))

(defun consult-notmuch--format-candidate (msg)
  "Format the result (MSG) of parsing a notmuch show information unit."
  (when-let (id (plist-get msg :id))
    (let ((result-string))
      (dolist (spec consult-notmuch-result-format)
        (when-let (field (consult-notmuch--format-field spec msg))
          (setq result-string (concat result-string field))))
      (propertize result-string 'id id 'tags (plist-get msg :tags)))))

(defun consult-notmuch--candidate-id (candidate)
  "Recover the thread id for the given CANDIDATE string."
  (when candidate (get-text-property 0 'id candidate)))

(defun consult-notmuch--candidate-tags (candidate)
  "Recover the message tags for the given CANDIDATE string."
  (when candidate (get-text-property 0 'tags candidate)))

Parsing notmuch show results

When consult-notmuch-show-single-message is set to nil, we're showing single messages as completion candidates, and, therefore, we are going to need to parse the output of that command, which looks like:

     message{ id:emacs-circe/circe/issues/401@github.com depth:0 ...
     header{
      <Sender (tags)>
      Subject: <subject>
      From: <from>
      To: <to>
      ...
      Date: Fri, 03 Sep 2021 12:46:53 -0700
     header}
      message}

Now, all we need is to parse the output of notmuch show and fill in the message metadata plist:

(defvar consult-notmuch--partial-parse nil
  "Internal variable for parsing status.")
(defvar consult-notmuch--partial-headers nil
  "Internal variable for parsing status.")
(defvar consult-notmuch--info nil
  "Internal variable for parsing status.")

(defun consult-notmuch--set (k v)
  "Set the value V for property K in the message we're currently parsing."
  (setq consult-notmuch--partial-parse
        (plist-put consult-notmuch--partial-parse k v)))

(defun consult-notmuch--show-transformer (str)
  "Parse output STR of notmuch show, extracting its components."
  (if (string-prefix-p "message}" str)
      (prog1
          (consult-notmuch--format-candidate
           (consult-notmuch--set :headers consult-notmuch--partial-headers))
        (setq consult-notmuch--partial-parse nil
              consult-notmuch--partial-headers nil
              consult-notmuch--info nil))
    (cond ((string-match "message{ \\(id:[^ ]+\\) .+" str)
           (consult-notmuch--set :id (match-string 1 str))
           (consult-notmuch--set :match t))
          ((string-prefix-p "header{" str)
           (setq consult-notmuch--info t))
          ((and str consult-notmuch--info)
           (when (string-match "\\(.+\\) (\\([^)]+\\)) (\\([^)]*\\))$" str)
             (consult-notmuch--set :Subject (match-string 1 str))
             (consult-notmuch--set :date_relative (match-string 2 str))
             (consult-notmuch--set :tags (split-string (match-string 3 str))))
           (setq consult-notmuch--info nil))
          ((string-match "\\(Subject\\|From\\|To\\|Cc\\|Date\\): \\(.+\\)?" str)
           (let ((k (intern (format ":%s" (match-string 1 str))))
                 (v (or (match-string 2 str) "")))
             (setq consult-notmuch--partial-headers
                   (plist-put consult-notmuch--partial-headers k v)))))
    nil))

Parsing notmuch search results

When consult-notmuch-show-single-message is set, our candidates generator uses the following transformer to format the raw results returned by the notmuch search command. Here, every line contains already all elements we need:

(defun consult-notmuch--search-transformer (str)
  "Transform STR from notmuch search to notmuch display style."
  (when (string-match "thread:" str)
    (let* ((id (car (split-string str "\\ +")))
           (date (substring str 24 37))
           (mid (substring str 24))
           (c0 (string-match "[[]" mid))
           (c1 (string-match "[]]" mid))
           (count (substring mid c0 (1+ c1)))
           (auths (string-trim (nth 1 (split-string mid "[];]"))))
           (subject (string-trim (nth 1 (split-string mid "[;]"))))
           (headers (list :Subject subject :From auths))
           (t0 (string-match "([^)]*)\\s-*$" mid))
           (tags (split-string (substring mid (1+  t0) -1)))
           (msg (list :id id
                      :match t
                      :headers headers
                      :count count
                      :date_relative date
                      :tags tags)))
      (consult-notmuch--format-candidate msg))))

Displaying candidates

consult-notmuch--search is going to return a candidate, and we'll want to display it either as a single message or a tree. notmuch.el already provides functions for that, so our display functions are really simple. Let's start with the one showing previews.

Previews

We're going to use always the same buffer for previews, closing it when we're done, and use notmuch-show to show a candidate. Remember that we've stashed the message or thread id needed by that function as a property of of our candidate string, and provided an accessor for it, so we have all the ingredients:

(defvar consult-notmuch--buffer-name "*consult-notmuch*"
  "Name of preview and result buffers.")

(defun consult-notmuch--show-id (id buffer)
  "Show message or thread id in the requested buffer"
  (let ((notmuch-show-only-matching-messages
         consult-notmuch-show-single-message))
    (notmuch-unthreaded id nil nil buffer)))

(defun consult-notmuch--preview (action candidate)
  "Preview CANDIDATE when ACTION is 'preview."
  (cond ((eq action 'preview)
         (when-let ((id (consult-notmuch--candidate-id candidate)))
           (when (get-buffer consult-notmuch--buffer-name)
             (kill-buffer consult-notmuch--buffer-name))
           (consult-notmuch--show-id id consult-notmuch--buffer-name)))
        ((eq action 'exit)
         (when (get-buffer consult-notmuch--buffer-name)
           (kill-buffer consult-notmuch--buffer-name)))))

Messages and trees

Displaying a message is practically identical to previewing it, we just change the buffer's name to include the query:

(defun consult-notmuch--show (candidate)
  "Open resulting CANDIDATE in ‘notmuch-show’ view."
  (when-let ((id (consult-notmuch--candidate-id candidate)))
    (let* ((subject (car (last (split-string candidate "\t"))))
           (title (concat consult-notmuch--buffer-name " " subject)))
      (consult-notmuch--show-id id title))))

and for a tree we just use notmuch-tree instead:

(defun consult-notmuch--tree (candidate)
  "Open resulting CANDIDATE in ‘notmuch-tree’."
  (when-let ((thread-id (consult-notmuch--candidate-id candidate)))
    (notmuch-tree thread-id nil nil)))

Integration with Embark

Embark actions

We can integrate consult-notmuch with Embark by defining a keymap with actions on notmuch messages and associating it with the completion category of notmuch-result. In this keymap we associate + and - (like in notmuch buffers) to a function that tags a message:

(defvar consult-notmuch-map
  (let ((map (make-sparse-keymap)))
    (define-key map (kbd "+") 'consult-notmuch-tag)
    (define-key map (kbd "-") 'consult-notmuch-tag)
    map)
  "Keymap for actions on Notmuch entries.")

(set-keymap-parent consult-notmuch-map embark-general-map)
(add-to-list 'embark-keymap-alist '(notmuch-result . consult-notmuch-map))

Additionally, we should integrate our address selection functions as well, so that you can act on the addresses.

(defun consult-notmuch--address-to-multi-select (address)
  "Select more email addresses, in addition to the current selection"
  (consult-notmuch-address t address))

(defvar consult-notmuch-address-map
  (let ((map (make-sparse-keymap)))
    (define-key map (kbd "c") #'consult-notmuch-address-compose)
    (define-key map (kbd "m") #'consult-notmuch--address-to-multi-select)
    map))

(set-keymap-parent consult-notmuch-address-map embark-general-map)
(add-to-list 'embark-keymap-alist
             '(notmuch-address . consult-notmuch-address-map))

consult-notmuch-tag should take as argument the search result as a propertized message string. Because Embark feeds it this string, this function does not need to be interactive:

(defun consult-notmuch-tag (msg)
  (when-let* ((id (consult-notmuch--candidate-id msg))
              (tags (consult-notmuch--candidate-tags msg))
              (tag-changes (notmuch-read-tag-changes tags "Tags: " "+")))
    (notmuch-tag (concat "(" id ")") tag-changes)))

We can also create bespoke functions to automatically tag a message with certain tags using Embark. For example, here is a function that returns a tagger:

(defun consult-notmuch-make-tagger (tags)
  "Make a function to tag a message with TAGS."
  (lambda (msg)
    "Tag a notmuch message using Embark."
    (when-let ((id (consult-notmuch--candidate-id msg)))
      (notmuch-tag (concat "(" id ")") (split-string tags)))))

We use this to map Embark actions that trash, archive or flag messages to d, a and f respectively:

(define-key consult-notmuch-map (kbd "d") (consult-notmuch-make-tagger "+trash -inbox"))
(define-key consult-notmuch-map (kbd "a") (consult-notmuch-make-tagger "-inbox"))
(define-key consult-notmuch-map (kbd "f") (consult-notmuch-make-tagger "+flagged"))

Embark export

To export search results to a notmuch search buffer with Embark, we can define a configurable exporter:

(defvar consult-notmuch-export-function #'notmuch-search
  "Function used to ask notmuch to display a list of found ids.
Typical options are notmuch-search and notmuch-tree.")

(defun consult-notmuch-export (msgs)
  "Create a notmuch search buffer listing messages."
  (funcall consult-notmuch-export-function
   (concat "(" (mapconcat #'consult-notmuch--candidate-id msgs " ") ")")))

Associating this exporter with consult-notmuch is a matter of adding to embark-exporters-alist:

(add-to-list 'embark-exporters-alist
             '(notmuch-result . consult-notmuch-export))

Package boilerplate

consult-notmuch.el

The file consult-notmuch.el is automatically generated from this org document, and has the typical breakdown in sections of an emacs package:

;;; consult-notmuch.el --- Notmuch search using consult  -*- lexical-binding: t; -*-

<<package-boilerplate>>

;;; Code:

<<dependencies>>

<<customization>>

<<private-functions>>

;; Embark Integration:
(with-eval-after-load 'embark
  <<embark-actions>>)

<<public-functions>>

(provide 'consult-notmuch)
;;; consult-notmuch.el ends here

ELPA headers

The standard header boilerplate will make it publishable as a regular ELPA package

;; Author: Jose A Ortega Ruiz <jao@gnu.org>
;; Maintainer: Jose A Ortega Ruiz
;; Keywords: mail
;; License: GPL-3.0-or-later
;; Version: 0.7
;; Package-Requires: ((emacs "26.1") (consult "0.9") (notmuch "0.31"))
;; Homepage: https://codeberg.org/jao/consult-notmuch

License (GPL 3+)

;; Copyright (C) 2021, 2022  Jose A Ortega Ruiz

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <https://www.gnu.org/licenses/>.

Commentary blurb

;;; Commentary:

;; This package provides two commands using consult to query notmuch
;; emails and present results either as single emails
;; (`consult-notmuch') or full trees (`consult-notmuch-tree').
;;
;; The package also defines a narrowing source for `consult-buffer',
;; which can be activated with
;;
;;   (add-to-list 'consult-buffer-sources 'consult-notmuch-buffer-source)

;; This elisp file is automatically generated from its literate
;; counterpart at
;; https://codeberg.org/jao/consult-notmuch/src/branch/main/readme.org

Acknowledgements

The initial implementation of consult-notmuch was heavily inspired by Alexander Fu Xi's counsel-notmuch.

This package also contains code contributions from Karthik Chikmagalur and Miciah Masters, and has also benefited from their ideas for new functionaliy.

S.M Mukarram Nainar suggested the idea and a working implementation for consult-notmuch-address.

Tags: emacs
Creative Commons License
jao.io by jao is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.