Next, we have fboundp.  This is another function for querying the state of the Lisp environment, asking whether a particular name is bound to a function or macro.  This function does not determine whether the name is interned as variable name.  Recall that, unlike C++, the Lisp language syntax unambiguously distinguishes between variable names and function names.  Consequently, these two types of symbols are not at risk of namespace collisions.

Now, you might think that this makes fboundp very simple.  The temptation is to say that if fboundp on a symbol returns nil, then calling that symbol in a function context will always fail, but things are a bit more complicated than that.  The fact is that it is possible to create named functions and macros which are not inserted into the global/persistent Lisp function namespace, but instead have a limited scope, following which time the functions effectively disappear.  The features labels, flet, and macrolet do this.  If, in your reading, you’ve seen a reference to “lexical functions”, the labels and flet features are what create them, and fboundp does not see lexical functions or macros.

And from this starting point, we can gain some helpful insight into the way Lisp works…

When one of labels, flet, or macrolet is used, it does not insert a new entry into the function symbol table, but it does shadow any existing functions with that name in the same scope.  If labels did insert an entry into the function symbol table, that would interfere with the view seen by other threads of execution.

This distinction between names in the function symbol table and names constructed by labels, flet, or macrolet manifests itself in one of the less obvious syntax requirements.  There are many contexts where the programmer can pass a function by referring to its symbol.  If the programmer has decided to use labels to build a function that happens to collide with an existing name in the function symbol table, how can he tell the program which one to use?  When passing a name that is not in the function symbol table, then rather than using a single-quote to quote the name, one must use a number sign followed by the single quote.  This is demonstrated here:
 

CL-USER> (defun my-adder (x)
           (+ x 2))
MY-ADDER
CL-USER> (labels
             ((my-adder (x) (+ x 3)))
           (mapcar 'my-adder '(1 2 3)))
(3 4 5)
CL-USER> (labels
             ((my-adder (x) (+ x 3)))
           (mapcar #'my-adder '(1 2 3)))
(4 5 6)

I begin by creating a function called my-adder in the function symbol table, one that adds 2 to the numbers passed to it.  Then, I use labels to shadow my-adder with a new definition, one that adds 3 to the numbers passed.  Note, though, that when I use the symbol ‘my-adder in mapcar, the one that is used is the one in the defun, not the one that I supposedly used to shadow it.  In order to use my new definition of my-adder, I have to use the #’my-adder syntax.  It is important to understand the difference between these two cases, and why the different forms are necessary.

It helps to understand what these notations are doing.  What, exactly, is the difference between mapcar acting on ‘my-adder and acting on #’my-adder?  In the first case, the mapcar function is called with a symbol as a parameter.  Inside mapcar, the symbol is resolved in the function symbol table to obtain the function that is to be used for the operation.  That is, mapcar is told “use the function whose symbol is designated by ‘my-adder“.  The second case, #’my-adder, is entirely different.  The # prefix is a reader macro that converts the text at load time.  The sequence #’my-adder is replaced by the text sequence (function my-adder) before the Lisp instance even sees it.  The function special operator resolves the name in the current lexical environment (before the mapcar function call), and returns a function object, not a symbol.  The mapcar function, rather than receiving a symbol and being told to look it up, receives a bare function itself, and uses that, without further resolution.

Perhaps now, there is a realization dawning.  You know that familiar construct for passing anonymous functions, lambda.  We start with a quoted list, the car of which is the keyword lambda.  Then, by prefixing the #, we convert this list into a function in the current lexical environment, and that function is what is passed downwards.  In effect, when we pass a function as a parameter, we can either choose to pass it by name, by sending a symbol down, or by value, by resolving the symbol into an actual function object and passing that instead.  And that is the difference between these two syntaxes, and the # prefix is a read-time shorthand for the second case.  Edit #3: 2014-02-27.  This phrasing was awkward, and now that we’ve covered read-macros under gensym, and have a separate article for lambda, this struck-out text should be ignored.  Instead, please refer to the article about lambda for more details.  The final point is this: many Lisp features which accept functions as arguments can be passed either a symbol from which the function is to be retrieved, or simply the bare function itself.  For functions that are not bound to a symbol, like those created with labels, flet, or lambda, it it necessary that the function object itself be passed.  The function object is retrieved within the scope where it is visible with the function special operator, for which #’ is a read-macro shortcut.  Finally, Common Lisp defines a lambda macro that expands to include the invocation of function, so the #’ is technically optional on lambda forms.

Edit #1:  2014-02-01

Some criticism has been raised on other sites about the terms and descriptions above.  It has been noted that referring to something like the “function symbol table” may confuse more than illuminate, as that is not a term commonly used.  I apologize for this, and it’s a good point to bring up.  My tendency in this series has been to try to describe logical parallels directed at the C++ programmer.  Not an excuse, merely an explanation.  But I certainly appreciate that the language I use above is strange or off-putting to a veteran Lisp programmer, a category in which I emphatically do not include myself.

I sincerely hope that when I reach the intern function, that the description I provide then will be more familiar and provide a better view of what is really going on.

For now, if you’ve read the original text above, and seen me talking about a “function symbol table”, don’t attach too much to that.  The underlying concept, I hope, is helpful.  There exists an mechanism in the Lisp image that allows a function to be referenced by its symbol.  The labels form, while superficially similar to defun, is, in fact, notably different in that it does not influence this aforementioned mechanism, and so does not interfere with the resolution of that symbol in other forms.

I also invite others to post copies of their comments and criticisms here, if they so desire.  I don’t want confusing or misleading postings to sit uncorrected on this site, while helpful criticism sits on other web sites that the casual visitor might not have come across.  I’ll point out that my motivation for writing this series of postings is to improve my understanding of Lisp, and deliberately researching every feature that I haven’t had the opportunity to use.  These posts are primarily an educational tool for myself, but I hope they help others.  If I say something wrong or confusing, please let me know.

Edit #2:  2014-02-02

I’ve posted an out of sequence article about interned symbols and packages here.  I hope that this provides more clarity, and invite the reader to go over that material carefully in order to avoid being mislead by some of the less precise terms I’ve been using in this series of posts.  Once again, I invite comments if the more experienced readers feel I’m failing to give helpful explanations.

Next in our list of Lisp features not necessarily encountered in a brief introduction is export.  This is related to the package system of Lisp.  You may find it useful to review the earlier article on delete-package, to understand what a package is in Lisp, and how it differs from C++ classes and namespaces.

Once again, the use of this function ties back to a fundamental difference in the creation of Lisp programs, as distinct from C++ programs.  In C++, the author edits one or more disc files, compiles them into a single executable unit, and then runs that executable.  If changes are to be made, the programmer edits the disc files once more, recompiles, and then restarts the C++ binary.

Lisp programs, on the other hand, are assembled by inserting code into a Lisp image.  Rather than creating a new stand-alone binary, you should think of this in terms of adding things to a blank Lisp image.  Functions, classes, structure, packages, and more, are added, serially, to a Lisp image in order to achieve a desired program state.  Some of the things that C++ would do with keywords and compiler or linker directives are, in Lisp, done with functions that act at the time of invocation, not at the time of compilation.

So, we mentioned packages earlier, and how they are a bit like namespaces, but have the private/public symbols the C++ programmer might associated with private/public methods in classes.  A symbol in a Lisp package is not exported by default.  To the beginner Lisp programmer, this looks like a small difference, calling those symbols requires using a double-colon after the package name, rather than a single colon.  In the context of package inheritance, however, an exported symbol is visible in derived packages that inherit from it, and exported symbols raise the possibility of namespace collisions in that context.

So, what does export do?  Well, unsurprisingly, it causes a particular symbol to be exported from the package.  Normally, the programmer lists the exported symbols in the package definition, as, for instance, in this earlier article, with the :DL-LIST package.  The implementation of the defpackage macro, however, generally calls export or something of equivalent functionality.  The programmer may have reason to choose to export certain symbols at runtime, for instance if optional packages that would supply those symbols are not loaded.  In most circumstances, this function will be only rarely used.

Next in our review of less-commonly seen Lisp features is eval-when.  This is a bit of a subtle one, that has to do with executing certain forms in certain circumstances.

There are three arguments you might use to eval-when.  :compile-toplevel, :load-toplevel, and :execute.  The first two apply only in situations that involve compiling Lisp code.  This does not include entering a defun at the command line, or even loading a .lisp file from disc.  It’s important to note that :load-toplevel applies only to the loading of compiled Lisp files, not to the loading of the source code in a .lisp disc file.

To understand things a bit, let’s talk about what happens when you load a .lisp file.  You may look at the file and think it looks like you’re telling it to load a bunch of functions and macros, but you should think of it more like running a shell script.  A set of commands that are immediately executed, in sequence.  Of course, when a form begins with defun or defmacro, it doesn’t do anything obvious, merely sets up a function or macro.  But if a toplevel form were something like a format output command, that would be executed, and the output appear on your screen as you load the .lisp file.  Loading a .lisp file looks like typing those commands into the Lisp prompt, in order.  So, as I said, think shell script, acting in a functioning Lisp environment.

Compiling a file, however, is different.  The compiler executes in a specially modified Lisp environment, scanning the input file and compiling functions, but deliberately not executing toplevel forms.  Once the file has been compiled, loading the resulting object file looks a lot like loading a .lisp file, toplevel forms are executed.  So, you can think of compile-file as compiling a script, and the following load operation as running the compiled script.

Sometimes, however, the programmer has some forms that he or she wants to have execute in the compilation environment, perhaps to modify the behaviour of the compiler.  This might be used for optimization settings, or maybe to record when a file was compiled.  Another interesting use is in modifying the language, and adjusting the compiler to handle it.

I’ve talked before about Lisp language features that I miss when I’m writing C++ code.  Well, there’s a feature of C/C++ that I miss in Lisp.  That is, preprocessor concatenation of string literals.  When I’m writing a long string literal, and don’t want the text to wrap, I can type in a sequence of string literals, enclosed in double-quotes and separated by blanks, knowing the the preprocessor will assemble them into a single long string.  This feature isn’t present in the Lisp reader.  Here, though, I’ll show how to implement it at compile time, so that the compiler can operate on files containing this modified Lisp syntax.
 

(format t "Hello there, this is a toplevel executing form.~%")

(eval-when (:compile-toplevel)

  (defparameter *rt-copy* (copy-readtable))

  (defun bar-reader (stream char)
    (declare (ignore char))
    (let ((stringlist (read stream t nil t)))
      (apply 'concatenate 'string stringlist)))

  (set-macro-character #\_ #'bar-reader)

  (multiple-value-bind (sec min hour day month year)
      (decode-universal-time (get-universal-time))
    (with-open-file (s "e-w-demo-compile-time.lisp"
                       :direction :output 
                       :if-exists :supersede)

      (format s "~S~%"
              `(unintern '+compile-time+))
      (format s "~S~%"
              `(defconstant +compile-time+
                (format nil 
                 "~2,'0D:~2,'0D:~2,'0D ~0,4D-~2,'0D-~2,'0D"
                 ,hour ,min ,sec ,year ,month ,day))))))

(eval-when (:compile-toplevel :load-toplevel)
  (load "e-w-demo-compile-time.lisp" :if-does-not-exist nil))

(defun demonstrate ()
  (cond
    ((boundp '+compile-time+)
     (format t "File compiled at: ~A~%" +compile-time+))
    (t
     (format t "No compile time information available.~%")))
  (format t _("This is a long string that I don't "
              "want to see linewrap when formatting "
              "for the blog post~%")))

(eval-when (:compile-toplevel)
  (copy-readtable *rt-copy* *readtable*)
  (unintern '*rt-copy*))

CL-USER> (compile-file "eval-when")
; compiling file "/home/neufeld/programming/lisp/blogging/eval-when.lisp" (written 17 JAN 2014 19:59:43 AM):
; compiling (FORMAT T ...)
; compiling (LOAD "e-w-demo-compile-time.lisp" ...)
; compiling (DEFUN DEMONSTRATE ...)

; /home/neufeld/programming/lisp/blogging/eval-when.fasl written
; compilation finished in 0:00:00.018
#P"/home/neufeld/programming/lisp/blogging/eval-when.fasl"
NIL
NIL
CL-USER> (load "eval-when")
Hello there, this is a toplevel executing form.
T
CL-USER> (demonstrate)
File compiled at: 20:05:42 2014-01-17
This is a long string that I don't want to see linewrap when formatting for the blog post
NIL

Note that the format statement was not printed out when compiling the file, but was printed when loading it.  The compile-time modification to the reader that I made asks it, when presented with the ‘_’ (underline) character, to read the next object, which must be a list of strings, and return a string which is the concatenation of those in the list.  When this new syntax is used in the demonstrate function, it is recognized and understood.  Without the use of eval-when, this would not be possible, as the compiler deliberately avoids executing forms such as the one to modify the read syntax, unless specifically told otherwise.

What, exactly, is going on in this file?  Well, at compile time, a copy is made of the read table, so that we can restore it after we’ve finished using our modified read table.  That is to avoid polluting the Lisp image on which we’re performing the compilation, it’s bad form for the compile operation to modify the behaviour of the image after the compilation has completed.  Next, we set up our modified read table to concatenate strings, and we write out the current time to a file on disc.  This disc file allows us to record new forms generated at compile time but absent at load time.  The file is compiled, and then at the end of the compilation we return the read table to its former state.

I haven’t mentioned the final eval-when option.  The third option to eval-when, :execute, is the “everything else” category.  These forms are evaluated when neither in the compiler, nor loading a compiled file.

One thing the newcomer to Lisp from C++ might wonder about is why there are so many ways to compare for “equality”.  The short list includes eq, eql, equal, equalp, and =.  Newcomers writing code tend to just toss one of those in and then change it around if the code doesn’t behave as desired, but this can be dangerous as the resulting code may only be producing the desired results accidentally, on a particular platform, and not be correct, portable code.  Some introductions to Lisp do a good job of explaining the differences between these equality comparison functions, while others do not.  Let’s go over them, with an eye to how the C++ programmer might view them.

In C++, it is common for the programmer to define an operator== to compare two objects of the same or different classes for “equality”.  This is helpful to the programmer, but can be a problem for the maintainer, as the programmer generally codes the operator to be “equal for my purposes, in the code I plan to write”.  When another programmer comes along, they will likely have to review all definitions of operator== to make sure that they understand what definition of equality is being tested here.

In Lisp, a few different definitions of equality are available with these functions.  The programmer is always free, of course, to define other named functions to test yet other definitions of equality, preferably with function names that help to clarify to the reader of the code what, exactly, is being checked.

To begin, please review the discussion on object allocation in this earlier article on define-modify-macro.  As outlined there, entities in Lisp tend to fall into three allocation categories.  There are objects and structures, which must always be explicitly allocated, and are always passed by reference.  There are arrays and lists, which form their own category of in-memory objects, but which can have multiple entry points giving different views.  Finally, there are numbers, characters, keywords, and symbols, which can be passed by value, and are not explicitly constructed.

So, we start with eq.

• Two references to objects or structures are only equal by the definition of eq when they are pointers to the same in-memory object.  If a modification is made to one of these data structures through one reference, the change is reflected in the other reference, and the references continue to compare as eq after this modification.
• The same rule applies for arrays and lists.  Two references to a list are equal under eq only if they point to the same in-memory list, and both point to the same starting node.  You saw the consequences of this in the discussion of defconstant earlier in this series.
• The third category of numbers, characters, keywords, or symbols.  Numbers and characters should not be compared with eq, as there is no guarantee of behaviour.  Keywords and symbols will compare eq if they are the same symbol (generally, if they have the same printable representation).

Next, we go to eql.

• On objects and structures, eql behaves the same as eq.
• The eql function is not intended for arrays and lists.  It may not behave as expected.  Use equal or equalp.
• The eql function also compares for equality on numbers and characters.  It should not be used on keywords and symbols.  Also, it is not guaranteed to behave as expected on floating-point numbers.  Use = for numeric comparison.

On to equal.

• Do not use equal to compare objects or structures.
• For arrays and lists.  Two references to arrays are equal if they are references to the same in-memory object.  Strings and bit-vectors, which are specialized arrays, are treated differently, and are compared element-by-element.  Two lists are equal if they have the same structure, and all of their elements are separately equal.
• The equal function on numbers and characters behaves exactly like eql.

Next, equalp.

• This function can operate on structures, but not usefully on objects.  Two structures are equalp if they have the same class (i.e. are incarnations of the same structure), and all slots in the two structures are separately equalp.
• On arrays and lists, two references are equalp if they have the same layout (number of dimensions for arrays, cons structure for lists), and all the elements are separately equalp.  In effect, if they would print out the same, they are equalp.
• Two numbers are equalp if they have the same numeric value.  For keywords and symbols, use only eq.  For characters, equalp is the same as equal.

Finally, =.

• This function cannot be used on structures or objects.
• This function cannot be used on arrays and lists.
• This function behaves exactly like equalp on numeric values.  It cannot operate on non-numeric arguments.

The first four functions form a progression: eq eql equal equalp.  It will never be the case that eq is true, while one of the functions to its right is false, and that rule applies down the chain.  Two objects that are equal will always be equalp.

You’ll note that equalp behaves differently for instances of classes and for structures.  This is unlike C++, where classes and structures are semantically identical save for the default access qualifier (in the absence of public/protected/private keywords).

That’s the overview of the functions.  Now, looking at it from the other direction, how do you compare categories of objects:

• Objects (instances of classes) are compared with eq, which tells  you whether two references are to the same in-memory object.  Any other comparison must be made with user-written functions.
• Structures are compared with eq, eql, equal or equalp.  If eq, eql, or equal, they are the same in-memory structure.  If equalp, they may or may not be the same structure, but they are instances of the same structure definition, with equal slot values.
• Arrays (but not strings or bit-vectors) are equal if they point to the same array, and equalp if they have the same dimensions and the same contents in each element.
• Strings and bit-vectors are equal if they have the same contents.
• Lists are eq if they point to the same entry point of the same in-memory object.  They are equalp if they have the same layout with equalp contents.
• Symbols and keywords are eq if they evaluate the same, generally if they have the same printable representation.
• Characters are eql if they print the same.  Testing under eq is not guaranteed to produce the expected results.
• Numbers are = if they are mathematically equal, regardless of type.

Next, we show these comparison in action:
 

(defclass example-class ()
  ((slot-A      :accessor get-A
                :initform :A)
   (slot-B      :accessor get-B
                :initform :B)))

(defun compare-instances ()
  (let ((instance-1 (make-instance 'example-class))
        (instance-2 (make-instance 'example-class)))
    (format t "Two different instances of example-class, with same contents~%")
    (format t "Under EQ, they compare to: ~A~%"
            (eq instance-1 instance-2))
    (format t "Under EQUALP, they compare to: ~A~%"
            (equalp instance-1 instance-2))))

(defstruct (example-struct)
  (slot-A       :A)
  (slot-B       :B))

(defun compare-structs ()
  (let ((struct-1 (make-example-struct))
        (struct-2 (make-example-struct)))
    (format t "Two different instances of example-struct, with same contents~%")
    (format t "Under EQ, they compare to: ~A~%" 
            (eq struct-1 struct-2))
    (format t "Under EQUALP, they compare to: ~A~%"
            (equalp struct-1 struct-2))))

(defun compare-arrays ()
  (let ((array-1 (make-array '(2 2) 
                             :initial-contents '((:A :B)
                                                 (:C :D))))
        (array-2 (make-array '(2 2) 
                             :initial-contents '((:A :B)
                                                 (:C :D)))))
    (format t "Two different arrays, with same dimensions and contents~%")
    (format t "Under EQUAL, they compare to: ~A~%" 
            (equal array-1 array-2))
    (format t "Under EQUALP, they compare to: ~A~%"
            (equalp array-1 array-2))))

(defun compare-strings ()
  (let ((string-1 "abc")
        (string-2 "abc"))
    (format t "Two different strings, with same contents~%")
    (format t "Under EQ, they compare to: ~A~%" 
            (eq string-1 string-2))
    (format t "Under EQUAL, they compare to: ~A~%" 
            (equal string-1 string-2))))

(defun compare-lists-1 ()
  (let* ((struct-1 (make-example-struct))
         (struct-2 (make-example-struct))
         (list-1 (list struct-1))
         (list-2 (list struct-2)))
    (format t "Two different lists, each with its own example-struct, ~%")
    (format t "those two structs being EQUALP~%")
    (format t "Under EQUAL, they compare to: ~A~%" 
            (equal list-1 list-2))
    (format t "Under EQUALP, they compare to: ~A~%" 
            (equalp list-1 list-2))))

(defun compare-lists-2 ()
  (let* ((struct-1 (make-example-struct))
         (list-1 (list struct-1))
         (list-2 (list struct-1)))
    (format t "Two different lists, each containing the same example-struct~%")
    (format t "Under EQ, they compare to: ~A~%" 
            (eq list-1 list-2))
    (format t "Under EQUAL, they compare to: ~A~%" 
            (equal list-1 list-2))))

(defun compare-numbers ()
  (let ((val-1 (/ 1 2))
        (val-2 0.5d0)
        (val-3 (complex 0.5 0.0)))
    (format t "The value 1/2, expressed as rational, real, and complex:~%")
    (format t "Comparing rational to real under =  ~A~%" 
            (= val-1 val-2))
    (format t "Comparing rational to complex under =  ~A~%" 
            (= val-1 val-3))
    (format t "Comparing real to complex under =  ~A~%" 
            (= val-2 val-3))))

CL-USER> (compare-instances)
Two different instances of example-class, with same contents
Under EQ, they compare to: NIL
Under EQUALP, they compare to: NIL
NIL
CL-USER> (compare-structs)
Two different instances of example-struct, with same contents
Under EQ, they compare to: NIL
Under EQUALP, they compare to: T
NIL
CL-USER> (compare-arrays)
Two different arrays, with same dimensions and contents
Under EQUAL, they compare to: NIL
Under EQUALP, they compare to: T
NIL
CL-USER> (compare-strings)
Two different strings, with same contents
Under EQ, they compare to: NIL
Under EQUAL, they compare to: T
NIL
CL-USER> (compare-lists-1)
Two different lists, each with its own example-struct, 
those two structs being EQUALP
Under EQUAL, they compare to: NIL
Under EQUALP, they compare to: T
NIL
CL-USER> (compare-lists-2)
Two different lists, each containing the same example-struct
Under EQ, they compare to: NIL
Under EQUAL, they compare to: T
NIL
CL-USER> (compare-numbers)
The value 1/2, expressed as rational, real, and complex:
Comparing rational to real under =  T
Comparing rational to complex under =  T
Comparing real to complex under =  T
NIL

Another rarely used Lisp function is ensure-generic-function.  This is used to create a dynamic function when one does not exist, or verify the compatibility of an existing dynamic function’s lambda list (think “argument list” in C++) with one supplied in the arguments.

To see an example of this in use, see the earlier article on add-method.