emacs/admin/cl-packages.org

# -*- mode: org; eval: (auto-fill-mode 1); org-indent-mode: 1; -*-
#+STARTUP: show3levels

* Common Lisp packages for Emacs

This is an experimental implementation of CL packages for Emacs.
The question of the experiment is if it is possible to add CL packages
to Emacs with reasonable effort and reasonable compatibility.

Note that this branch is only known to build and run under macOS.  I
don't have access to other systems, so it might not compile or work on
other systems.  Patches welcome.

Please see a book like Common Lisp the Language (CLtL2) for a
description of the CL package systen.  The book is freely available
from CMU.

** Status
This builds and runs with unchanged Magit, Lsp-mode, and other
packages for me, so it seems to be pretty backwards-compatible.  I
can't gurantee anything, of course.  If you find a problem, please let
me know.

** User-visible functionality
There are three pre-defined packages.

The keyword package, named "keyword" or "" contains keywords.

The Emacs package, with name "emacs" contains all other symbols.  All
code is currently loaded in this package, for compatibility.  All
symbols in the package are currently exported.

The "emacs-user" package is intended for user-code, for example in
*scratch*, and uses the Emacs package, so that everything in Emacs can
be used.

These variables are defined:

"*package*" holds the current package like in CL.  It's buffer-local,
and you can't set it to a non-package value, to prevent havoc.

"*emacs-package*", "*keyword-package*", "*emacs-user-package*" hold
the package objects.  This is mainly for easier debugging and testing.
The variables may go at some point.  Or not.

"*package-registry* is a hash-table of registered packages.  The
variable may go at some point.  Or not.

A file-local variable "symbol-packages" can be used to enable or
disable reading or printing of symbols with package prefixes.  If
symbol-packages is nil, a symbol "a:b" will be read as a symbol with
name "a:b" in the Emacs package.  If symbol-packages is t, "a:b" will
be read as symbol "b" in package "a".  Default is nil.

Note that there is a small incompatibility here: In "normal" Emacs,
the symbol 'a:b' is printed as 'a:b'.  In this version of Emacs, it
prints as 'a\:b".  Both Emacs versions can read the latter form.  This
can be a problem with Transient which writes symbols containing a
colon to .emacs.d/transient/level.el.  If an old levels.el gets read
in this version of Emacs, this can lead to an error "Unknown package
'transient'".  Solution: add a backslash in front of the colons.

Various functions related to packages are defined.  Depending on the
time when you read this, this may be in some state of incompleteness,
and it probably has bugs.  Fixes welcome.

** Implementation notes
*** Where is it?
The C part is mainly in src/pkg.c.  I chose that name because
package.c resulted in conflicts in the tests (conflicts with
package.el tests).

The Lisp part is mainly in lisp/emacs-lisp/pkg.el.  I've done as much
of this in Lisp as possible because that's much easier and faster.  If
packages are used in files loaded in loadup, changes might be
necessary to make this possible.  I consider this out of scope, ATM.

*** No pure space support
The branch contains a patch by Stefan Monnier that makes it no longer
use pure space.  I didn't want to deal with pure space.  Note that a
small fix in init_vectors is needed for making Stefan's patch work.
There is nothing preventing the use of pure space though, in
principle.

*** Shorthands
Are currently not supported.

*** Lisp_Package
There is a new Lisp data type Lisp_Package defined in lisp.h.

***  Lisp_Symbol
Struct Lisp_Symbol has lost its interned flag and its next pointer.
Both were an implementation detail of obarrays, which are gone.

All symbols now have a package.  Uninterned symbols have a nil
package.

Keywords have the keyword package. Note that keyword symbol names do
not contain the colon.  The function symbol-name still returns a
string with a leading colon.  I found this was necessary to achieve
backwards-compatibility.  At least at this point.  The function
cl-symbol-name returns the real name of a keyword, without the colon.

Other symbols have the Emacs package.

*** Obarrays
Obarrays have been removed.  Backwards-compatibility is achieved by
the following

- The variable 'obarray' still exists.  Its value is now the Emacs
  package.
- intern, intern-soft, unintern, mapatoms still accept vectors (former
  obarrays).  When called with a vector, they secretly create and use
  packages.  This is done because legacy code uses make-vector instead
  of obarray-make to create obarrays.

*** Reader
The variable 'symbol-packages' determines if the reader will
interpret colons in a symbol name as part of a package name or not.
Default is nil.

*** Printer
The printer prints package prefixes if necessary, as in CL.

*** Completions
The completion functions accept packages as collections.

** Problems and how they are approached (currently)
*** Keywords
In CL, keywords are symbols in the keyword package.  The leading colon
of a keyword is not part of its symbol name, but a package prefix.
The keyword package has a nickname that is an empty string.

In Emacs, keywords are just symbols whose names start with a colon,
and that is expected in a ton of places both implicity and explicitly
and in various forms.

Current approach:

- Internally, keyword names don't contain the colon, which is TRT.
- symbol-name returns a name with colon for keywords.
- cl-symbol-name returns the symbol name as-is.
- intern and intern-soft when called with a name starting with a colon
  interpret that as wanting a keyword.

That's not at all pretty, but in an experiment with symbol-name
behaving like in CL showed serious problems that I couldn't solve so
far without modifying the code.

But see under Ideas and Todos.

*** Fake package qualification
Existing code contains symbols like GUI:xyz which look like GUI is a
package qualification.  That's the reason for the variable
symbol-packages which means to interpret the : as part of the symbol
name.

** Ideas / Todo
*** Completions
It might be useful to complete over all symbols in all packages.
I haven't added that.

*** Existing package extensions
There are some language extensions available in CL implementations
that might be nice to have

- Hierarchical packages
- Package locks
- Local nicknames

None of these are implemented.

*** Changing symbol names
A trap that I always fall into, constantly, in Emacs, is to use CL
functions without the cl- prefix.  It would be nice to have something
that makes these symbols available without the cl-.

Just ideas:

- (shadow-alias multiple-value-bind cl-multiple-value-bind) or maybe
  with regexs. Or something.
- (import sym as another-sym)

*** symbol-packages in functions
I'm wondering if it would be an idea to record the value of
symbol-packages at the time and in the buffer where functions are
compiled or eval'd.

We could then

- Bind symbol-packages around the execution of the function to that
  value.

- Return a name with leading colon from symbol-value if
  symbol-packages is nil, which means the function was compiled or
  eval'd in a "traditional" setting.  It would return the keyword name
  without the leading colon if symbol-packages is t.

- Make intern treat colons differently depending on the value of
  symbol-packages.  There are some places like transient.el which
  intern names with a leading colon which are a pain in the neck.

- Maybe calls to read could also behave differently.

For subrs (native-compiled and C code), there is plenty of room for 1
bit.  For byte-compiled functions, see make-bytecode + make-closure.

This should be doable from that perspective.  One probably just has to
try it out.

*** Modeline
A mode-line indicator showing the current package and symbol-packages
would be helpful.  Can be done with (:eval ...) in global-mode-string
now.  Or maybe in a header-line.

*** Tests
Should be much improved.

*** Documentation
Doesn't exist :-).

*** Other
- Add (declare (ignore ...)) and (declare (ignorable ...) goddam :-).