programming (and other) musings
11 Mar 2021

notmuch queries via consult

This package provides notmuch queries in emacs using consult. It offers interactive functions to launch search processes using the notmuch executable and present their results in a completion minibuffer and, after selection of a candidate, single message and tree views.

The package's elisp code is tangled from this literate program.

Dependencies and installation

We depend on the emacs packages consult and notmuch:

(require 'consult)
(require 'notmuch)

Both, as well as this package, are available in MELPA, so that's the easiest way to install consult-notmuch. You can also simply tangle this document and put the resulting consult-notmuch.el file in your load path (provided you have its dependencies in your load path too), or obtain it from its git repository.


Consult interface to notmuch search

Our objective is to use consult to perform notmuch queries, and show their results in the completions minibuffer and, upon selection, on a dedicated notmuch buffer. Notmuch knows how to show either single messages or full threads (or trees), so our public interface is going to consist of two autoloaded commands:

(defun consult-notmuch (&optional initial)
  "Search for your email in notmuch, showing single messages.
If given, use INITIAL as the starting point of the query."
  (consult-notmuch--show (consult-notmuch--search initial)))

(defun consult-notmuch-tree (&optional initial)
  "Search for your email in notmuch, showing full candidate tree.
If given, use INITIAL as the starting point of the query."
  (consult-notmuch--tree (consult-notmuch--search initial)))

They're implemented in terms of a common search driver, with the only difference being how we show the final result of the auto-completion call.

When using them you'll notice how we automatically inject consult's split character (# by default) as the first one in the query string, so that our search term can be divided into a string passed to notmuch and a second half (after inserting a second # in our query) that is used by emacs to filter the results of the former. You'll also see that there's a preview available when traversing the list of candidates in the minibuffer.

It is also worth remembering that the input is a generic notmuch query, so one can, for instance, use the initial contents to define specific query commands. For example, i have a set of mailboxes under a subdirectory called feeds (where mails retrieved by rss2email end up after a dovecot sieve), so i could define this command in my init files:

(defun jao-consult-notmuch-feeds (&optional tree)
  (interactive "P")
  (let ((init "folder:/feeds/ "))
    (if tree (consult-notmuch-tree init) (consult-notmuch init))))

or be more generic and read from a completing prompt the subfolder of my ~/var/mail directory i want to use:

(defun jao-consult-notmuch-folder (&optional tree folder)
  (interactive "P")
  (let* ((root "~/var/mail/")
         (folder (if folder
                     (file-name-as-directory folder)
                     (thread-first (read-directory-name "Mailbox: " root)
                       (file-relative-name root))))
         (folder (replace-regexp-in-string "/\\(.\\)" "\\\\/\\1" folder))
         (init (format "folder:/%s " folder)))
    (if tree (consult-notmuch-tree init) (consult-notmuch init))))

(defun jao-consult-notmuch-feeds (&optional tree)
  (interactive "P")
  (jao-consult-folder tree "feeds"))

Buffer narrowing

If you have many buffers, you may want a convenient way to switch specifically among notmuch buffers. The consult-notmuch-buffer-source source can be used for this purpose:

(defun consult-notmuch--interesting-buffers ()
  "Return a list of names of buffers with interesting notmuch data."
   :as (lambda (buf)
         (when (notmuch-interesting-buffer buf)
           (buffer-name buf)))))

(defvar consult-notmuch-buffer-source
  '(:name "Notmuch Buffer"
    :narrow (?n . "Notmuch")
    :hidden t
    :category buffer
    :face consult-buffer
    :history buffer-name-history
    :state consult--buffer-state
    :items consult-notmuch--interesting-buffers)
  "Notmuch buffer candidate source for `consult-buffer'.")

This source can be used with consult-buffer by adding it to consult-buffer-sources:

(add-to-list 'consult-buffer-sources 'consult-notmuch-buffer-source)

With the above configuration, you can initiate consult-buffer and then type n followed by a space to narrow the set of buffers to just notmuch buffers.


As customary, we're going to use a customization group, as a subgroup of notmuch's one:

(defgroup consult-notmuch nil
  "Options for `consult-notmuch'."
  :group 'consult)

and our first user option will tell us whether we display single messages in the matches list (extracted via notmuch-show) or thread groups (a la notmuch-search):

(defcustom consult-notmuch-show-single-message t
  "Show only the matching message or the whole thread in listings."
  :type 'boolean)

When displaying search results in the minibuffer, we'll want to extract the authors, date and subject and thread count for each message and give them a format defined by the custom variable:

(defcustom consult-notmuch-result-format
  '(("date" . "%12s  ")
    ("count" . "%-7s ")
    ("authors" . "%-20s")
    ("subject" . "  %-54s")
    ("tags" . " (%s)"))
  "Format for matching candidates in minibuffer.
Supported fields are: date, authors, subject, count and tags."
  :type '(alist :key-type string :value-type string))

which has the same semantics as notmuch-search-result-format.


Consult search function

The core of our implementation should a call to consult--read with a closure to obtain completion candidates based on a call to notmuch search or notmuch show as an asynchronous process. For that, we'll use consult's helper consult--async-command. This function takes as first argument a string representing the command to be called to obtain completion candidates, followed by any transformations we want to apply to them before being displayed. Thus, our candidates generator will look like:

(defun consult-notmuch--command (input)
  "Construct a search command for emails containing INPUT."
  (if consult-notmuch-show-single-message
      `("notmuch" "show" "--body=false" ,input)
    `("notmuch" "search" ,input)))

(defun consult-notmuch--search (&optional initial)
  "Perform an asynchronous notmuch search via `consult--read'.
If given, use INITIAL as the starting point of the query."
  (setq consult-notmuch--partial-parse nil)
  (consult--read (consult--async-command
                   (consult--async-filter #'identity)
                   (consult--async-map #'consult-notmuch--transformer))
                 :prompt "Notmuch search: "
                 :require-match t
                 :initial (consult--async-split-initial initial)
                 :history '(:input consult-notmuch-history)
                 :state #'consult-notmuch--preview
                 :lookup #'consult--lookup-member
                 :category 'notmuch-result
                 :sort nil))

In the code above we're also using a preview function (described below), and a history variable:

(defvar consult-notmuch-history nil
  "History for `consult-notmuch'.")

and the candidates transformer will depend on whether we're displaying threads or single messages:

(defun consult-notmuch--transformer (str)
  "Transform STR to notmuch display style."
  (if consult-notmuch-show-single-message
      (consult-notmuch--show-transformer str)
    (consult-notmuch--search-transformer str)))

Formatting search results

Using consult-notmuch-result-format, we are going to return a string representation from a plist describing the current message, reusing notmuch's facility notmuch-tree-format-field, with the added trick of storing the current message or thread id in a text property, so that it can latter be used for displaying the message preview:

(defun consult-notmuch--format-field (spec msg)
  "Return a string for SPEC given the MSG metadata."
  (let ((field (car spec)))
    (cond ((equal field "count")
           (when-let (cnt (plist-get msg :count))
             (format (cdr spec) cnt)))
          ((equal field "tags")
           (when (plist-get msg :tags)
             (notmuch-tree-format-field "tags" (cdr spec) msg)))
          (t (notmuch-tree-format-field field (cdr spec) msg)))))

(defun consult-notmuch--format-candidate (msg)
  "Format the result (MSG) of parsing a notmuch show information unit."
  (when-let (id (plist-get msg :id))
    (let ((result-string))
      (dolist (spec consult-notmuch-result-format)
        (when-let (field (consult-notmuch--format-field spec msg))
          (setq result-string (concat result-string field))))
      (propertize result-string 'thread-id id))))

(defun consult-notmuch--thread-id (candidate)
  "Recover the thread id for the given CANDIDATE string."
  (when candidate (get-text-property 0 'thread-id candidate)))

Parsing notmuch show results

When consult-notmuch-show-single-message is set to nil, we're showing single messages as completion candidates, and, therefore, we are going to need to parse the output of that command, which looks like:

     message{ id:emacs-circe/circe/issues/ depth:0 ...
      <Sender (tags)>
      Subject: <subject>
      From: <from>
      To: <to>
      Date: Fri, 03 Sep 2021 12:46:53 -0700

Now, all we need is to parse the output of notmuch show and fill in the message metadata plist:

(defvar consult-notmuch--partial-parse nil
  "Internal variable for parsing status.")
(defvar consult-notmuch--partial-headers nil
  "Internal variable for parsing status.")
(defvar consult-notmuch--info nil
  "Internal variable for parsing status.")

(defun consult-notmuch--set (k v)
  "Set the value V for property K in the message we're currently parsing."
  (setq consult-notmuch--partial-parse
        (plist-put consult-notmuch--partial-parse k v)))

(defun consult-notmuch--show-transformer (str)
  "Parse output STR of notmuch show, extracting its components."
  (if (string-prefix-p "message}" str)
           (consult-notmuch--set :headers consult-notmuch--partial-headers))
        (setq consult-notmuch--partial-parse nil
              consult-notmuch--partial-headers nil
              consult-notmuch--info nil))
    (cond ((string-match "message{ \\(id:[^ ]+\\) .+" str)
           (consult-notmuch--set :id (match-string 1 str))
           (consult-notmuch--set :match t))
          ((string-prefix-p "header{" str)
           (setq consult-notmuch--info t))
          ((and str consult-notmuch--info)
           (when (string-match "\\(.+\\) (\\([^)]+\\)) (\\([^)]+\\))$" str)
             (consult-notmuch--set :Subject (match-string 1 str))
             (consult-notmuch--set :date_relative (match-string 2 str))
             (consult-notmuch--set :tags (split-string (match-string 3 str))))
           (setq consult-notmuch--info nil))
          ((string-match "\\(Subject\\|From\\|To\\|Cc\\|Date\\): \\(.+\\)?" str)
           (let ((k (intern (format ":%s" (match-string 1 str))))
                 (v (or (match-string 2 str) "")))
             (setq consult-notmuch--partial-headers
                   (plist-put consult-notmuch--partial-headers k v)))))

Parsing notmuch search results

When consult-notmuch-show-single-message is set, our candidates generator uses the following transformer to format the raw results returned by the notmuch search command. Here, every line contains already all elements we need:

(defun consult-notmuch--search-transformer (str)
  "Transform STR from notmuch search to notmuch display style."
  (when (string-match "thread:" str)
    (let* ((thread-id (car (split-string str "\\ +")))
           (date (substring str 24 37))
           (mid (substring str 24))
           (c0 (string-match "[[]" mid))
           (c1 (string-match "[]]" mid))
           (count (substring mid c0 (1+ c1)))
           (auths (string-trim (nth 1 (split-string mid "[];]"))))
           (subject (string-trim (nth 1 (split-string mid "[;]"))))
           (headers (list :Subject subject :From auths))
           (msg (list :id thread-id
                      :match t
                      :headers headers
                      :count count
                      :date_relative date)))
      (consult-notmuch--format-candidate msg))))

Displaying candidates

consult-notmuch--search is going to return a candidate, and we'll want to display it either as a single message or a tree. notmuch.el already provides functions for that, so our display functions are really simple. Let's start with the one showing previews.


We're going to use always the same buffer for previews, and close it when we're done:

(defvar consult-notmuch--buffer-name "*consult-notmuch*"
  "Name of preview and result buffers.")

(defun consult-notmuch--close-preview ()
  "Close the message preview, by killing its buffer."
  (when (get-buffer consult-notmuch--buffer-name)
    (kill-buffer consult-notmuch--buffer-name)))

and use notmuch-show to show a candidate.

Remember that we've stashed the message or thread id needed by that function as a property of of our candidate string, and provided an accessor for it:

(defun consult-notmuch--preview (candidate _restore)
  "Open resulting CANDIDATE in ‘notmuch-show’ view, in a preview buffer."
  (when-let ((thread-id (consult-notmuch--thread-id candidate)))
    (notmuch-show thread-id nil nil nil consult-notmuch--buffer-name)))

The additional _restore argument it's used by consult when we install the function above via consult--read's :state keyword.

Messages and trees

Displaying a message is practically identical to previewing it, we just change the buffer's name to include the query:

(defun consult-notmuch--show (candidate)
  "Open resulting CANDIDATE in ‘notmuch-show’ view."
  (when-let ((thread-id (consult-notmuch--thread-id candidate)))
    (let* ((notmuch-show-only-matching-messages
           (subject (car (last (split-string candidate "\t"))))
           (title (concat consult-notmuch--buffer-name " " subject)))
      (notmuch-show thread-id nil nil nil title))))

and for a tree we just use notmuch-tree instead:

(defun consult-notmuch--tree (candidate)
  "Open resulting CANDIDATE in ‘notmuch-tree’."
  (when-let ((thread-id (consult-notmuch--thread-id candidate)))
    (notmuch-tree thread-id nil nil)))


This implementation was heavily inspired by Alexander Fu Xi's counsel-notmuch.

Tags: emacs
Creative Commons License by jao is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.