Writing an automated test to try to find an Emacs bug

BUG: Emacs is losing the default font information from init.el when adding a68-mode or alda-mode. It’s difficult to figure out what exactly is happening. It seems related to having a large number of packages installed, but perhaps its just the right combination of combination of packages.

DISCLAIMER: I strongly suspect there are better ways to do some of this, but what I have seems to be working.

I’m trying to write a test that starts with default font information specifying a font. It probably doesn’t matter what font, as the end result of the bug is this getting replaced with

(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(default ((t (:background nil)))))

So to check whether the bug has happened, I can either

• search for the pattern above, OR
• search for the specific font I’ve specified

I will search for the specific font, as I’d like to detect any changes, not just the change above.

The bug seems to happen when installing a68-mode or alda-mode just after starting Emacs. So, this test will be a loop of many Emacs invocations.

• Start Emacs
• Install a68-mode
• Check whether the default font information has been destroyed
  • If so, note the last package installed.
  • If not
    • delete (uninstall) a68-mode
    • install a random package
    • write the package name just installed to a file
    • exit Emacs
    • Run the test again

init.el

init.el contains the font information. I also initially added a “blank” custom-set-variables call, but over time I discovered that sometimes restart-emacs would see a process talking to ELPA and ask if I wanted to kill the process. No, I don’t want to be asked that. I found confirm-kill-processes was the variable I could set to nil to avoid getting asked. So into the set of of custom variables it went.

;;; -*- lexical-binding: t -*-
(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(confirm-kill-processes nil)
 )
(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(default ((t (:family "Fira Code" :foundry "CTDB" :slant normal :weight medium :height 120 :width normal)))))

start-test.sh

#!/bin/sh

TESTDIR=$HOME/bug-emacs.d
cd $TESTDIR
emacs --init-directory=$TESTDIR --script $TESTDIR/test.el

Test

;;;
;;; Copy this script into an empty user-emacs-directory along with
;;; the accompanying init.el.
;;;
;;; Run this script with
;;;
;;;     emacs --init-directory=<directory-path> --load=<directory-path>/test.el
;;;

Load up packages

List packages and set buffer to Packages

(progn
  (list-packages)
  (set-buffer "*Packages*")
  )

Install a68-mode

(message "%s" "Installing a68-mode")
(sit-for 0.5)
(package-install 'a68-mode)

Check that font information is still in init.el

(find-file (locate-user-emacs-file "init.el"))
(if (not (search-forward "(custom-set-faces
 ;; custom-set-faces was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(default ((t (:family \"Fira Code\" :foundry \"CTDB\" :slant normal :weight medium :height 120 :width normal)))))"
			 nil t))
    ;; String not found. Read last package installed from file and print it out. Then exit Emacs.
    ;;
    ;; Read string from file and print
    (progn
      (find-file "last-installed.txt")
      (display-message-or-buffer "Install of %s caused subsequent install of a68-mode to lose font information" (buffer-string))
      (sit-for 0.5)
      ;; Exit Emacs
      (kill-emacs) 
     )
    )

Uninstall/delete a68-mode

(message "%s" "Removing a68-mode")
(sit-for 0.5)
(package-delete (package-get-descriptor 'a68-mode))

Install random package

Install random package and write its name to last-installed file

(let ((pkg (car (seq-random-elt package-archive-contents))))
  (save-current-buffer
    (let ((buffer (find-file "last-installed.txt")))
      (set-buffer buffer)
      (kill-region (point-min) (point-max))
      (print pkg buffer)
      (save-buffer)
      )
    )
  ;; Now install pkg
  (message "Installing %s" pkg)
  (sit-for 1)
  (ignore-errors (package-install pkg))
  )

Restart emacs

I was wondering if I’d have to have a more complicated shell script to keep starting Emacs, but Emacs has restart-emacs that does the obvious.

(message "%s" "Restarting emacs")
(sit-for 0.5)
(restart-emacs)