in no particular order xv

Interesting bits elsewhere:

• Yes, I did drop 90GB of my work to Public Domain
• Functional Algorithm Design, Part 0, Part 1, Part 2
• Using Emacs for College Assignments - DEV Community
• Gleam: Lean BEAM typing machine - Louis Pilfold
• 25 Great Book Reviews From the Past 125 Years - The New York Times

consulting spotify in a better way

After my latest adventures writing a small spotify library and learning in the process a bit more about consult, its author, Daniel Mendler, was kind enough to comment on how i had implemented the asynchronous search using consult's API, showing me better ways.

You can read the full discussion in this issue over at codeberg (where you can also find the latest version of the library), but the gist of it, regarding consult, is as follows.

To define a new asynchronous consult command, one wants to use consult--read, passing to it a function that generates our dynamic list of completion candidates. To create that function, one can use a pipeline of closures that successively create and massage those candidates. In the case of espotify that layering might look like this¹:

(thread-first (consult--async-sink)
  (consult--async-refresh-immediate)
  (espotify--async-search type filter)
  (consult--async-throttle)
  (consult--async-split))

where we only have to implement espotify--asynch-search to construct the generator of completion candidates (more about it in a moment). The rest are helpers already provided by consult:

• consult--async-split: splits the input string, one part for async, one part for filtering
• consult--async-throttle: throttles the user input
• consult--async-refresh-immediate: refreshes when candidates are pushed
• consult--async-sink: collects the candidates and refreshes

Consult offers also a few more closure generators that we haven't used (yet):

• consult--async-map: transform candidates
• consult--async-refresh-timer: refreshes, when candidates are pushed, throttles with a timer
• consult--async-filter: filter candidates
• consult--async-process, a source generator handy when your candidates come from the output of executing a local process

Back to our candidates generator. It must be a function that takes a continuation closure (the async after you in the pipeline) and returns an action dispatcher, that is, a function takiing that action as its single argument (possibly passing its results, or simply delegating, to the next handler in the pipeline). So our dispatcher generator is going to look something like this template, where we display all possible actions to be dispatched:

(defun espotify--async-search (next-async ...)
  ;; return a dispatcher for new actions
  (lambda (action)
    (pcase action
      ((pred stringp) ...) ;; if the action is a string, it's the user input
      ((pred listp) ...)   ;; if a list, candidates to be appended
      ('setup ...)
      ('destroy ...)
      ('flush ..)
      ('get ...))))

For each action, we must decide whether to handle it ourselves or simply pass it to next-async, or maybe both. Or we could ask next-async to perform new actions for us. In our case, we only care about generating a list of tracks when given a query string that ends on a marker character², and making sure it reaches the top level. Thus, our async has only work to do when it receives a string, simplifying my original implemetation to:

(defun espotify--async-search (next type filter)
  (lambda (action)
    (pcase action
      ((pred stringp)
       (when (string-suffix-p "=" action)
         (espotify-search-all
          (lambda (items)  ;; search results callback
            (funcall next 'flush)
            (funcall next (mapcar #'spotify--format-item items)))
          (substring action 0 (- (length action)
                                 (length espotify-search-suffix)))
          type
          filter)))
      (_ (funcall next action)))))

As you can see, when we receive a search string, we launch an asynchronous search and, upon receiving its results, we flush the layer above us (so that it discards previous candidates) and pass the new candidate list to it. It is ultimately the closure returned by consult--async-sink the one keeping track of those candidates, and making them accessible to consult--read. The latter is expecting candidates to be strings (possibly with properties), while our search callback is receiving, via its items parameter, a list of alists: that's why we need to map over them with espotify--format-item. If we prefer, we can make that transformation explicit by simply returning items in that callback (via (funcall next items)) and inserting consult--async-map in our pipeline, which would now look like:

(thread-first (consult--async-sink)
  (consult--async-refresh-immediate)
  (consult--async-map #'espotify--format-item)
  (espotify--async-search type filter)
  (consult--async-throttle)
  (consult--async-split))

With all that, our code looks tidier and easier to understand (i at least understand much better its workings) ³. You can always check its latest version, in literate version, here (C-c C-v t in the org buffer will generate espotify.el for you).

Thanks Daniel!

an emacs packages hat trick

i've found these last days a handful of really useful little emacs packages:

• smartscan, a simple way of navigating programming modes

  (use-package smartscan
    :ensure t
    :commands smartscan-mode
    :init (add-hook 'prog-mode-hook #'smartscan-mode)
    :diminish)
  

• git-link, because i almost never look at files using the browser, but some people don't have my repos checked out. Specially nice the bit about being able to use git config variables.

  (use-package git-link
    :ensure t
    :custom ((git-link-default-remote "bigml")))
  

• visible-mode, an Emacs built-in, especially useful in org buffers, where i hide markup and sometimes it's a chore to ascertain whether point is before or after a it. With a binding to this minor mode, it's a breeze to toggle them:

  (use-package visible-mode
    :bind (("s-v" . visible-mode)))
  

Tip of the hat to Daniel Mai, for a couple of the above. You might find some other interesting tidbits in his config.

an even better video wharf

A couple of days ago, i was writing about embark and my first experiment defining a new embarking to play remote video streams. Omar Antolín Camarena, embark's author, has been kind enough to not only read it, but comment on a couple of significant improvements that i think well deserve this follow-up.

First, you'll remember that we were defining a function to detect a video URL:

(defun jao-video-finder ()
  "Check whether we're looking at a video URL.
Return (video-url . <URL>) if so."
  (when-let ((url (thing-at-point-url-at-point)))
    (when (string-match-p jao-video-url-rx url)
      (cons 'video-url url))))

Once we've got a non-null url value, even if it's not a video URL, it's still certainly a URL, and embark has a url category, so we could save a new parsing by the default URL finder by saying:

(when-let ((url (thing-at-point-url-at-point)))
  (cons (if (string-match-p jao-video-url-rx url) 'video-url 'url) url))

This has the potential drawback that we're overriding embark's finder, embark-target-url-at-point, and we might prefer to keep the latter.

Turns out that we can do that thanks to embark's target transformers. One can add to embark-transformers-alist an arbitrary function to be applied to a target of any given category, and embark will apply its actions to the transformed value. Omar calls this process, very aptly, a refinement of the target; here's how we would do it:

(defun jao-refine-url-type (url)
  "Refine type of URL in case it is a video."
  (cons (if (string-match-p jao-video-url-rx url) 'video-url 'url) url))

(add-to-list 'embark-transformer-alist '(url . jao-refine-url-type))

With this strategy, we don't need jao-video-finder at all, and it also makes lots of sense, conceptually, to have our video-url defined as a refinement rather than a new target¹. Omar's second suggestion is also in line with this concept: surely we want all actions available for url also for our video-url, don't we? Well, that's exactly the reason why the embark-define-keymap macro we used to define our actions can inherit all the actions already defined in another keymap, using the :parent keyword²:

(embark-define-keymap jao-video-url-map
  "Actions on URLs pointing to remote video streams."
  :parent embark-url-map
  ("p" jao-play-video-url))

(add-to-list 'embark-keymap-alist '(video-url . jao-video-url-map))

It is worth noting that this ability to inherit a keymap is not really an embark add-on: vanilla Emacs keymaps already have it, via the standard function set-keymap-parent. You could actually define jao-video-url-map without using embark-define-keymap at all, and it'd work exactly the same.

So, our code has become shorter and more featureful: thanks, Omar!

(when-let ((url (or (w3m-anchor) (thing-at-point-url-at-point))))
  (cons (if (string-match-p jao-video-url-rx url) 'video-url 'url) url))

(defun jao-w3m-url-finder ()
  (when-let ((url (w3m-anchor)))
    (cons 'url url)))

(add-to-list 'embark-target-finders #'jao-w3m-url-finder)

embarking videos

Inspired by Prot's musings on completion, i've, ahem, embarked in a reconsideration of my completions setup (as you might have intuited from my recent experiments with the spotify API and consult). As it happens, i'm starting to feel quite at home with a combination of selectrum, prescient and consult, and the ideas to augment what i have with contextual actions using embark seem really natural to me.

The main premise of embark is pretty simple: one has categories of targets that can be acted upon (URLs, regions, files, buffers…), and, for each category, a collection of commands that take an instance of the target and do something with it (that is, a collection of actions on target instances). As you may imagine, user-defined categories and actions are a thing in embark.

A little application of those ideas immediately popped to my mind when i started reading about embark: streaming videos from video hosting platforms. For many reasons, i try to avoid switching my context to graphical browsers and prefer to play those kind of videos by passing their URLs to mpv, which in turn will use youtube-dl to stream the contents.

Now, it's very easy to recognise when a URL is a video URL, we just need to match it against an appropriate regular expression. For example:

(defvar jao-video-url-rx
  (format "^https?://\\(?:www\\.\\)?%s/.+"
          (regexp-opt '("youtu.be"
                        "youtube.com"
                        "blip.tv"
                        "vimeo.com"
                        "infoq.com")
                      t))
  "A regular expression matching URLs that point to video streams")

Given that regular expression, we can write a "video-url target" finder function, using for instance the Emacs standard library thing-at:

(defun jao-video-finder ()
  "Check whether we're looking at a video URL.
Return (video-url . <URL>) if so."
  (when-let ((url (thing-at-point-url-at-point)))
    (when (string-match-p jao-video-url-rx url)
      (cons 'video-url url))))

Once we have a URL in our hands, playing it with mpv is very easy ¹:

(defun jao-play-video-url (&optional url)
  (interactive "sURL: ")
  (let ((cmd (format "mpv %s" (shell-quote-argument url))))
    (start-process-shell-command "mpv" nil cmd)))

Now, we can put both things together with embark and tell it, first, that we have a new kind of target (video-url), and, second, what actions are associated to it (jao-play-video-url, among others).

The new target category definition is accomplished by adding its finder function to embark-target-finders alist:

(add-to-list 'embark-target-finders #'jao-video-finder)

Similarly, registering the possible actions for a target is done by adding a keymap for it to embark-keymap-alist. Embark provides the utility macro embark-define-keymap for easily defining new keymaps, so, putting both things together we have:

(embark-define-keymap jao-video-url-map
  "Actions on URLs pointing to remote video streams."
  ("p" jao-play-video-url)
  ("b" browse-url)
  ("f" browse-url-firefox))

(add-to-list 'embark-keymap-alist '(video-url . jao-video-url-map))

Besides our default, jao-play-video-url, i've thrown in for good measure a couple other actions. Note that they don't need to be specially defined, they're just stock Emacs commands taking one argument; and also that jao-video-url-map is just a run-of-the-mill Emacs keymap. As such, it can also be used and modified via the standard Emacs utilities: embark will just use the final result ².

With that in place, we're done³: one can invoke M-x embark-act (most probably via a keyboard shortcut) when point is around (or right before) a video stream URL, and then press, say, p, to call mpv on it.

You might be thinking that writing a single Emacs command doing the same thing for us and binding it to a key is very easy too, so why bother? Well, as a matter of fact, "writing a standard Emacs command and binding it to a key" is exactly what we did, so, for starters, making it all part of the embark system is not giving us much additional trouble, if any (a couple of calls to add-to-list, all in all). In exchange, we reap the benefits of making our new action interplay with the generic utilities and integrations provided by embark (for instance, you'll see that a host of other generic actions are available and readily accessible for free for our new video URL target). This is, again, a good illustration of (and a homage to) the modularity and orthogonality of a system composed of pieces such as embark and friends.

consulting spotify

We have two kinds of interaction with Spotify: via its HTTP API to perform operations such as search, and via our local DBUS to talk to client players running in our computer, such as the official client, spotifyd or mopidy-spotify. Our goal is to obtain via the former a track or album identifier that we can send then to the latter to play, with emacs completion mechanisms (consult and friends in this case) providing the glue between both worlds.

Note: you can access this post as an org file in my jao/elibs repository, and transform it to an emacs-lisp file with M-x org-babel-tangle.

Let's start with an umbrella customization group:

;;; espotify.el - spotify search and play -  -*- lexical-binding: t; -*-

(defgroup espotify nil
  "Access to Spotify API and clients"
  :group 'multimedia)

(defvar espotify-spotify-api-url "https://api.spotify.com/v1")
(defvar espotify-spotify-api-authentication-url
  "https://accounts.spotify.com/api/token")

(defvar espotify-client-id nil "Spotify application client ID.")
(defvar espotify-client-secret nil "Spotify application client secret.")

(defun espotify--basic-auth-credentials ()
  (let ((credential (concat espotify-client-id ":" espotify-client-secret)))
    (concat "Basic " (base64-encode-string credential t))))

(defun espotify--with-auth-token (callback)
  (let ((url-request-method "POST")
        (url-request-data "&grant_type=client_credentials")
        (url-request-extra-headers
         `(("Content-Type" . "application/x-www-form-urlencoded")
           ("Authorization" . ,(espotify--basic-auth-credentials)))))
     (url-retrieve espotify-spotify-api-authentication-url
                   (lambda (_status)
                     (goto-char url-http-end-of-headers)
                     (funcall callback
                              (alist-get 'access_token (json-read)))))))

(espotify--with-auth-token
 (lambda (token) (message "Your token is: %s" token)))

(defun espotify--make-search-url (term types &optional filter)
  (when (null types)
    (error "Must supply a non-empty list of types to search for"))
  (let ((term (url-encode-url term)))
    (format "%s/search?q=%s&type=%s&limit=50"
            espotify-spotify-api-url
            (if filter (format "%s:%s" filter term) term)
            (mapconcat #'symbol-name types ","))))

(espotify--make-search-url "dream blue turtles" '(album))

(defun espotify--with-query-results (token url callback)
  (let ((url-request-extra-headers
         `(("Authorization" . ,(concat "Bearer " token)))))
    (url-retrieve url
                  (lambda (_status)
                    (goto-char url-http-end-of-headers)
                    (funcall callback
                             (let ((json-array-type 'list))
                               (json-read)))))))

(defun espotify-search (callback term types &optional filter)
  (let ((query-url (espotify--make-search-url term types filter)))
    (espotify--with-auth-token
     (lambda (token)
       (espotify--with-query-results token query-url callback)))))

(defvar espotify-query-result nil)
(espotify-search (lambda (res) (setq espotify-query-result res))
                 "dream blue turtles"
                 '(album artist))
(sit-for 0)

(mapcar 'car espotify-query-result)

(defun espotify--type-items (res type)
  (alist-get 'items (alist-get (intern (format "%ss" type)) res)))

(defun espotify-search* (callback term types &optional filter)
  (let* ((types (if (listp types) types (list types)))
         (cb (lambda (res)
               (let ((its (mapcar (lambda (tp)
                                    (espotify--type-items res tp))
                                  types)))
                 (apply callback its)))))
    (espotify-search cb term types filter)))

(defvar espotify-query-result nil)
(espotify-search* (lambda (al ar)
                    (message "Found %s albums, %s artists"
                             (length al) (length ar))
                    (setq espotify-query-result (cons al ar)))
                 "blue turtles"
                 '(album artist))
(sit-for 0)
(list (mapcar 'car (car (car espotify-query-result)))
      (mapcar 'car (car (cdr espotify-query-result))))

(defcustom espotify-service-name "mopidy"
  "Name of the DBUS service used by the client we talk to.

The official Spotify client uses `spotify', but one can also use
alternative clients such as mopidy or spotifyd."
  :type 'string)

(defcustom espotify-use-system-bus-p t
  "Whether to access the spotify client using the system DBUS.")

(defun espotify-call-spotify-via-dbus (method &rest args)
  "Tell Spotify to execute METHOD with ARGS through DBUS."
  (apply #'dbus-call-method `(,(if espotify-use-system-bus-p :system :session)
                              ,(format "org.mpris.MediaPlayer2.%s"
                                       espotify-service-name)
                              "/org/mpris/MediaPlayer2"
                              "org.mpris.MediaPlayer2.Player"
                              ,method
                              ,@args)))

(defun espotify-play-uri (uri)
  (espotify-call-spotify-via-dbus "OpenUri" uri))

(defun consult-spotify-by (type &optional filter)
  (consult--read (format "Search %ss: " type)
                 (espotify--search-generator type filter)
                 :lookup 'espotify--consult-lookup
                 :category 'spotify-query-result
                 :initial "#"
                 :require-match t))

(defun espotify--search-generator (type filter)
  (thread-first (consult--async-sink)
    (consult--async-refresh-timer)
    (espotify--async-search type filter)
    (consult--async-throttle)
    (consult--async-split)))

(defun espotify--async-search (next type filter)
  (let (candidates)
    (lambda (action)
      (pcase action
        ((or 'setup 'refresh) (funcall next action))
        ('destroy (setq candidates nil) (funcall next 'destroy))
        ((pred stringp)
         (espotify-search*
          (lambda (x)
            (setq candidates (mapcar 'espotify--format-item x))
            (funcall next candidates))
          action ;; when we receive a string as the action,
                 ;; it's the user input
          type
          filter))
        ('get candidates)
        (_ (funcall next action))))))

(defun espotify--format-item (x)
  (propertize (format "%s" (alist-get 'name x)) 'espotify-item x))

(defun espotify--item (cand)
  (get-text-property 0 'espotify-item cand))

(defun espotify--uri (cand)
  (alist-get 'uri (espotify--item cand)))

(require 'seq)
(defun espotify--consult-lookup (_input cands cand)
  (seq-find (lambda (x) (string= cand x)) cands))

(defun espotify--maybe-play (x)
  (when-let (uri (espotify--uri x))
    (espotify-play-uri uri)))

(defun consult-spotify-album (&optional filter)
  (interactive)
  (espotify--maybe-play (consult-spotify-by 'album filter)))

(defun espotify-marginalia-annotate (cand)
  (when-let (x (espotify--item cand))
    (marginalia--fields
     ((alist-get 'type x "") :face marginalia-mode :width 10)
     ((or (alist-get 'name (car (alist-get 'artists x))) "")
      :face 'marginalia-file-name :width 50)
     ((if-let (d (alist-get 'total_tracks x)) (format "%s tracks" d) "")
      :face 'marginalia-size :width 12)
     ((if-let (d (alist-get 'release_date x)) (format "%s" d) "")
      :face 'maginalia-modified :width 10))))

(add-to-list 'marginalia-annotators-heavy
             (cons 'spotify-query-result 'espotify-marginalia-annotate))

(defun consult-spotify-playlist (&optional filter)
  (interactive)
  (espotify--maybe-play (consult-spotify-by 'playlist filter)))

(provide 'espotify)

what's not to like

I've just discovered Codeberg, a code hosting site that, finally, has let me create a user with plain emacs-w3m, shows me content reasonably well there, with a refreshingly uncluttered layout, handles graciously org files (why, it's even generating a table of contents for me), has a good privacy policy, it's not under the wings of any corporation and had my preferred username free for grabs.

So i'm definitely testing the waters for it to become my default site for (newish) published code: here's my jao/elibs: Emacs libraries and little utilities repository for starters (and, if you look closely, you'll find also my emacs config nearby). If and when that happens, they're going to to get my humble contribution too, i reckon.

appius and virginia

I recently learnt, at The Neglected Books Page (aptly subtitled Where forgotten books are remembered), about G.E. Trevelyan, an author all but forgotten nowadays too. More concretely, I read about her first novel, Appius and Virginia, and was very intrigued and happy to discover in this video feature that it was going to be re-printed by Abandoned Books Press. My copy arrived yesterday, and it's easily been the best book i've read this year. It's daring, original and exceptionally well-written. It reminds me of another of my favourites of all times, Aldous Huxley, or Woolf if she had tried to try her stream-of-consciousness experiments to the mind of an ape, or maybe Mary Shelley, had she imagined Frankenstein's creature as a puzzled orangutan.

G.E. Trevelyan died tragically due to a Nazi bomb, and had only time for 7 more novels. I'm looking forward to reading them!

two decades of gnu mdk

I've just published GNU MDK 1.3.0, its 28th release, which finally migrates MDK's graphical user interface to GTK+ 3, to keep up with the, ahem, not-so-modern times and see to it that MDK keeps alive for at least another decade or two.

Twenty years ago on this day, the first version of MDK 0.1 was released. Back then, it didn't have an Emacs or graphical interface¹, support for internationalization or any integration with Guile, and the debugger was really bare bones (it still is, but not that much). It wasn't yet a GNU package. But all those things, and then more, came in rapid succession, as i used the project to discover the Free Software world, both at a technical and a human level.

As a physicist out of grad school, i wasn't young, but i was just starting in the world of software and everything was fresh and new and exciting. I remember the thrill of receiving emails from total strangers willing to help and offering insightful advice (hi there, Philip King, wherever you are these days), and of thinking that someone else, out there, was finding my little program fun and useful². Or of exchanging messages with RMS, that guy i had read about in Hackers and admired so much; and then see in the process MDK entering the GNU umbrella. And of seeing it shortly afterwards on Knuth's own page on TAOCP, up there in the MIXware list.

MDK was my first project developed entirely in Emacs running on Debian (twenty years ago, that felt bolder than it sounds now), two constant companions up to ths day. Seeing it as a maintained Debian package was another big satisfaction and milestone for memory lane.

I also have vivid recollections of the mind-bending experience of discovering Scheme, because i learnt that Guile was the extension language of choice for GNU, and my baby steps on writing a lexer or an interpreter, and how humbled i felt when i took a proper compilers course a few years later and looked back at my clumsily rediscovered wheels³.

MDK was also my gateway to publishing a book for the first time, thanks to the nice guys of the GNU Press, with a polished version of its manual. Okay, it's more of a booklet, and out of press by now, but it was an enriching experience nonetheless; for instance i got a chance of seeing an editor in action.

After two decades, having grown older and bitter, all those things look small and of little importance, but as a freshman i truly had the best of times. I think that maybe the most important thing i learned was to collaborate with other people, and i've since been always very fond of the kindness of strangers, and how much one can learn from them.

Anyway, here's to the next twenty years. Happy hacking!

in no particular order xiv

Interesting bits elsewhere, emacs edition:

• EmacsConf - 2020
• Why I Chose Emacs as My New Text Editor - Take on Rules
• Making medieval manuscript transcription less painful with GNU Emacs - Joseph R. Johnson
• Emacs in Emacs: A triumph for EXWM - Tech.ToryAnderson.com