Tag Archives: compiling

The less-familiar parts of Lisp for beginners — dynamic-extent

Another obscure Lisp feature that the newcomer might not have encountered is dynamic-extent.  This is used as part of a declare operation.  It is purely a compiler hint, the Lisp implementation is free to ignore dynamic-extent entirely.

What a dynamic-extent declaration does is to indicate to the compiler that one or more symbols are entirely local to the form.  Their values do not return to the caller, they are not inserted into objects or lists, they live and die completely in the form where the dynamic-extent declaration appears.  They are local storage.  When this is known, the compiler has the option of stack-allocating the objects referenced by the symbols, rather than throwing them into the garbage-collected heap.  This can reduce computational overhead in the maintenance of the heap, and so speed up execution.

Building with dependencies in Lisp

A simple project, whether in C++ or in Lisp, is easy enough to implement.  In C++, you’d write one .h file and one .cpp file, and when you wanted to test changes, you would compile with something like this

g++ -ggdb example.cpp -o example

However, as the project becomes more complicated, it typically gets broken up into separate files.  In C++, you might reserve a pair of .h/.cpp files for one class or one hierarchy of descended classes.  Then, as changes continue, it becomes important to figure out what files need to be recompiled when a header file has changed.  This is where GNU Make, and its relatives like cmake and qmake, come in.

What about Lisp?  Well, in a similar fashion, you tend to break up your project into separate files.  It isn’t common to break out your file into declarations and definitions, the way you try to do in C++, but it is possible to do a bit of that, if desired.  However, you still have cases where one .lisp file depends on definitions created in another .lisp file.  In that case, it’s important to load the dependent file ahead of the depending one, to avoid runtime errors at load time.  How is this managed?

In Lisp, the commonly used dependence manager is called ASDF. A particular project is built with an .asd file declaring what files are needed for the project, and what files depend on what other files.  Here’s an example of an .asd file from one of my old projects, a sudoku puzzle generator:
  

(defpackage puzzle-system
  (:use :common-lisp :asdf
#+clisp :screen
#-clisp :cl-ncurses
))

(in-package :puzzle-system)

(defsystem "puzzle"
    :description "puzzle:  An interactive Sudoku game."
    :version "0.2"
    :author "Christopher Neufeld <EMAIL-REDACTED>"
    :licence "GPL"
    :components ((:file "generate")
                 (:file "printed-puzzles" 
                        :depends-on ("generate" "eval-difficulty"))
                 (:file "interactive" 
                        :depends-on ("generate" "eval-difficulty"))
                 (:file "eval-difficulty" 
                        :depends-on ("generate")))
    :depends-on (
                 #-clisp :cl-ncurses
                         ))

This declares that we’re creating a system called ‘puzzle-system’.  If compiled under clisp, it depends on the :screen feature, otherwise it depends on :cl-ncurses (the cl-ncurses package is now deprecated, using the older UFFI where now we would use CFFI and cl-charms).  There are four input files, some of which depend on others.  By creating this .asd file, the programmer can avoid having to figure out what files need to be recompiled and loaded after a change has been made.  One simply invokes this command:

(asdf:oos ‘asdf:load-op ‘<SYSTEM-NAME>)

The asdf system then compiles the appropriate files and loads them.  You should ensure that asdf is loaded when your Lisp starts up.  For SBCL, I use this ~/.sbclrc file:
  

(require :asdf)
(setf asdf:*central-registry*
      (list* '*default-pathname-defaults*
             #p"/home/neufeld/.sbcl/systems/"
             asdf:*central-registry*))

This tells SBCL that it should load the :asdf system at startup.  It also tells it where to look for additional systems.  In the directory /home/neufeld/.sbcl/systems, I have placed symbolic links to the .asd files of the different Lisp packages I have downloaded.  For instance, there are links to cl-ncurses.asd, cl-ppcre.asd, cffi.asd, and so on.  This way, I can build a project that depends on one of those packages, and the ASDF make system will locate the appropriate system wherever it might be, and include it as appropriate.

ASDF is capable of much more than this, I’ve presented little more than the basic mode of operation.  As your projects become more sophisticated, you may find yourself learning the more advanced features of ASDF for proper compiler and loader control.  This, though, is a beginner’s guide to dependency tracking and compiling using ASDF in Lisp.

Installing in non-standard places

Uncategorized

I mentioned earlier the possibility of choosing an install prefix like /usr/local/samba, which installs the Samba libraries in a directory that may not commonly exist on distribution-managed machines. One possible effect of this is that you may turn up bugs in configuration and compilation scripts of other packages.

A configure script for another package may accept arguments related to the location of Samba libraries and header files, but compiling the package with these options set might not work. This isn’t very surprising, it’s a compilation option that is probably rarely used, so bit rot has a tendency to set in. A change somewhere that accidentally breaks the compilation when Samba is installed in an unusual place might not be noticed for some time. By putting Samba in its own directory, you are setting yourself up to test a valid, but rarely exercised option. You may find yourself submitting bug reports and patches to the package maintainers.

As I’ve said before, maintaining your box without a package manager and distribution is not easy. It’s quite a bit more work, but it does force you to understand more about how the system is set up and what it’s doing. For people who like the extra control and understanding this provides, this is a useful technique.

Choosing an install prefix

Uncategorized

As noted in this posting, you generally will have to choose an install prefix for software that you are compiling yourself. Most packages you encounter will be configured to install under /usr/local, though some will be configured for /usr.

The first thing you’ll want to do is to see if you already have an older version of the software installed anywhere. If the software was previously installed under /usr/local, and you install the new package under /usr, not only will you needlessly consume disk space, but the version that is run will depend on the setting of your PATH environment variable. A user may report that he can’t use a certain feature in the new version, and it may take you a while to notice that his environment variable differs from yours, and that he’s still running the old software. So, find the name of an executable that you expect will be installed. For example, if you’re installing the binutils software, you will expect that the ld binary should be installed somewhere. Next, type the command:

which ld

to see where it is currently installed. If you see it in “/usr/bin/ld”, then you’ll probably want to use a prefix of “/usr”, so that your new versions install over top of the old ones. If, on the other hand, it’s in “/usr/local/bin/ld”, you’ll want a prefix of “/usr/local”.

Sometimes a package installs only one or a few binaries. You may decide to install this into its own directory. For example, I install firefox into the prefix /usr/local/firefox, SBCL into the prefix /usr/local/sbcl, and the apache httpd into /usr/local/apache2. These get their own directories because, while they may install a very small number of executables, they come with a large set of ancillary files. Rather than installing over top of the old directory, I move the old directory to a new location, say “/usr/local/sbcl.old”, and then install and test the new version. If the new version doesn’t work properly, I can revert to the old one by deleting the new install and renaming the “.old” directory. Alternatively, I can compare the two installations, the previously working one against the new one, and see if there are any obvious differences that could account for problems.

Of course, you probably won’t be able to type the command firefox and expect it to run if it’s installed in /usr/local/firefox/bin/. You will either want to add that directory to the PATH variable, or, more conveniently, put a symbolic link to the appropriate executable from a directory that is in your PATH. This command:

ln -s /usr/local/firefox/bin/firefox /usr/X11/bin/firefox

puts the firefox executable into your PATH, piggy-backing on the /usr/X11/bin entry that is probably there already. Note, however, that if you re-install X11 (we’ll get to that in another posting), you might destroy this symbolic link, and you’ll have to re-create it then.

So, you really have a couple of choices. Put the program into a standard place, like /usr or /usr/local (and if upgrading try to install over top of the old version by using the same prefix that was used then), or installing the software in its own dedicated directory, like /usr/local/firefox or /usr/local/sbcl.

Now, when you set the prefix in an autoconf configure script, it also sets a number of derived values which can be separately overridden. Configuration files are, by default, put in /etc, libraries in /lib, headers in /include, man pages in /share/man (sometimes omitting the ‘share’ component), log files in <prefix&gt/var/log, and so on. The configure program lets you override these defaults separately, so that you can put configuration files into, say, /etc/http with the option “–sysconfdir=/etc/http”, and so on. Think carefully about whether you want these additional directories to keep their defaults. You probably don’t want your X-server log to be in /usr/X11/var/log, nobody will know where to look for it.

Compiling and installing by hand

Uncategorized

If you’re not using a package manager, or if you are, but there is no package available for a piece of software you’d like to install, you’ll find yourself compiling the software by hand. Generally, you start by locating the official web page of the software, downloading an appropriate version of the source code, and extracting the tar file to a directory somewhere.

At this point in the process, you are not doing anything as the root user. You’ll become root much later in this process.

The next thing you’ll do is look in the top level of the extracted directory for promising looking files, like README, INSTALL, or Makefile. It is likely that you will see an executable script called “configure”. It’s always a good idea to start by looking at the README and INSTALL files, if present. They may be in the toplevel directory, or in a documentation directory, which will often have a name like “doc”, “docs”, or “documentation”, possibly with different capitalizations.

If The Package Came With A Makefile

If there’s a Makefile in the toplevel, that’s usually because the software package is fairly small. You will want to look over the Makefile to ensure that it is correct for your intended installation. The most important things to look for are the installation directory and any optional features that might have to be turned on by editing the Makefile. If you can’t find the installation directory, type the command:

make -n install

This will ask “make” to print out the sequence of commands that it will be using to install the package. Since you haven’t compiled anything yet, it will start with the sequence of commands required to compile your software, so look for the installation commands to occur near the end of the output generated by this command.

If your package came with a Makefile, you will now modify the Makefile if necessary, perhaps changing the installation directory of the product. You should do this before compiling it, because sometimes character strings holding the pathnames of configuration files are inserted into the compiled binary, so changing the installation target after compiling may result in an installation that doesn’t work correctly. Editing the Makefile will usually not force a recompilation of the objects under its control, that is the Makefile is not, by default, considered a dependency for the targets in the file.

After this, you will, still as your non-root user, compile the package. This is usually done by simply entering the command make. If errors are encountered during the compile, you’ll have to figure out what happened and how to fix it. The most common causes of errors are:

  • missing include files – you might have to add a “-I” to the CFLAGS, CXXFLAGS, or CPPFLAGS variables in your Makefile.
  • missing libraries – you might have to add a “-L” to the LDFLAGS variable in your Makefile.
  • bad version – the compilation may depend on a library you have on your machine, but the version you have may not be compatible with the software package. You might have to download a different version of that library and install it before you can continue with the software package.
  • apparent code errors – the compiler may generate errors related to missing variables, bad function declarations, or syntax errors. Resist the urge to correct these immediately, and try to understand why you are seeing these errors. Remember, this package probably compiled for somebody before they released it, why doesn’t it work for you? Is it that your compiler is a different version, and flags as errors things that used to be warnings? Is the Makefile configured for the wrong architecture or platform? Something else?

Once you get a clean compile, you’re almost ready for the install. I usually prefer to run the command

make -n install | less

once and read through the output, just to make sure that the install isn’t going to do something weird. Look for things like configuration files going into /usr/etc, which might not be what you expect, or binaries going into /bin (you should try to keep in that directory only those executables that are necessary to get the computer to boot through its startup scripts up to the point where the network starts up).

At this point, move down to the section of the text called “Installing The Software”.

You Have A “configure.am” Script, But No “configure” Script

If you have a “configure.am” script, but no “configure” script, you’ll have to generate the configure script. If there is an executable in this directory with a name like “autogen.sh”, run it. This should be sufficient to set up the configure script. If you don’t have an autogen script, you should run the commands automake then autoconf. This will often generate warnings, but unless the configure script you generate doesn’t run, you can ignore those. So, now you have a configure script, you continue to the next section.

You Have A “configure” Script

If you generated the configure script yourself, you know that it’s an autoconf configure script. Sometimes, though, software is produced that has a completely different script that happens to be called “configure”. This can be confusing if it doesn’t recognize the switch “–help”. Start by typing:

./configure --help | less

and look at the output. If it produces a list of options that are available to you, review them carefully and see if there are any optional behaviours that you would like to turn on, or unwanted options that you want to remove (possibly you don’t have library support for these, and don’t need them). If, instead, the configure script appears to run and do things, you don’t have an autoconf configure script, go back and look at the documentation again to see how to use their particular configuration script.

There are a few things to look at in the options you get from “configure”. One of them is the prefix location, and choosing that properly can require some care, which is discussed here. For now, let’s assume that you’ve chosen a set of options that look suitable. You re-run the configure script with those options, and without the “–help” option. It will do some things, it may take a considerable amount of time to run. Eventually, the script should exit, sometimes generating a list of all options and whether or not they are active. Examine this list if present, there might be an option that you want to enable that has been turned off because the configure script failed to find a particular library, in which case you’ll have figure out why that option was disabled and figure out how to get it working. When you’re satisfied with the compilation options, type “make”. If an error is encountered, see the possibilities mentioned in the earlier section referring to building from a Makefile. If you succeed in compiling the software package, go to next section, “Installing The Software”.

Installing The Software

Now, you can become the root user. Change directory to the location where you compiled the binary, and run

make install

If the thing you’re installing has any shared objects (libraries, usually with names that end in “.so”, possibly followed by more dots and numerals), you should type

ldconfig

to make sure that the dynamic linker knows where to find the libraries you’ve just installed.

Many packages these days produce a pkg-config file. This is usually a filename that ends in “.pc”, and is installed in a directory like “…/lib/pkgconfig/”. The pkg-config application often looks for these files when “configure” is being run, but it has a fairly definite idea of where to look. If your .pc file was installed into a directory where pkg-config doesn’t normally look, you’ll have to find some way to make this file visible to that program. There are three ways you can handle this:

  • Add the appropriate directory to the system-wide environment variable PKG_CONFIG_PATH. Usually this means editing /etc/profile. You likely want it set at least to “/usr/lib/pkgconfig:/usr/local/lib/pkgconfig:/usr/X11/lib/pkgconfig”, but you may want to add more search directories to it, if you expect many packages to be installed in the same prefix.
  • Copy the .pc file into a directory that pkg-config searches. This is unwise, you may install another version of the software some time later, and unless you remember this step your .pc file will still be the old one, causing much aggravation as “configure” insists you still have a version of the package that you know you just replaced.
  • Put a symbolic link to the file from a directory that is searched by pkg-config. Do this if you’ve got only one or two .pc files in this prefix, and don’t expect to put in more.

Test your newly-installed software. It’s best to find problems now, when you’ve just finished installing it and remember what you did, than two weeks from now and have to go through the whole thing again just to figure out how it’s set up.

Two more hints: “configure” writes its command line into a comment near the top of the file “config.log”. If you need to remember how you last ran “configure”, you will find the options you used there.

If you have a particularly detailed set of configure options, you might want to record them in a directory somewhere for future reference, both to see quickly what options you enabled when you compiled the software and to re-use the command the next time you recompile it after downloading a new version.