Lisp is for Stupid People

[yes, the title is clickbait.]

I’ve been programming in Lisp for many years. It has a reputation as language only suitable to forbiddingly smart people (and indeed, most Lisp programmers I know are MIT-trained or the equivalent). I never quite understood that though. In fact, I realized recently that the reason I like Lisp, why I am always drawn back to it despite its general lack of commercial marketability, is not because I am so smart, but because I’m so stupid.

Or, more precisely, it is because Lisp is a better tool for overcoming my own mental limitations. It’s not that I am particularly stupid – my limitations are largely shared by everybody else. My attention is limited, my processing capacity is limited, my short-term memory capacity is limited, simply by virtue being a human being.

Programming is thought to require being able to hold complicated structures in the head. Here’s a cartoon illustration, which is pretty accurate as far as it goes, and here’s some actual interesting research on programmer’s mental functions. It may be that the fabled 10x or 100x programmers are just a little bit better at dealing with complexity than normal. I’m pretty good at it, but not superhuman, especially now that I’m getting alarmingly old.

The alternative to requiring programmers to be amazing jugglers of objects and acrobats of attention is powerful abstraction and good design. Abstraction is the basic way in which programmers encapsulate a bit of complexity in a single object or function call. All programming languages let you do this but some make it easier and/or offer more ways to do it. Good design means organizing a system around a few fundamental principles so that things make sense and navigating through the code doesn’t require a lot of effort.

These qualities are linked. Clean design means choosing abstractions so that they work together in powerful ways, and better abstraction tools make it easier to do that. It is certainly possible to do good software design in a less-powerful language. Some of the most impressive pieces of software design I’ve seen recently has been the series of visualization libraries coming out of Jeff Heer’s group, including prefuse (in Java) and the newer d3.js (in Javascript). The designers of these libraries have carefully chosen their abstractions so they fit together in powerful ways.

But, I maintain, it is a lot easier to do this in a language that is designed for it, and Lisp is that language. In Lisp, macros and other features make it trivially easy to hide all the details away in suitable abstractions, with the result that the meat of the program can be compact, so that the programmer can focus on it. In Java, by contrast, the important part of a program generally tends to be hidden away amidst boring boilerplate code that is necessary to please the Java compiler but not germane to the problem at hand (this article refers to the novel software metric “beef-to-bun ratio”, which is low in Java but apparently improving).

Thus Lisp has always seemed to me a much better tool than any other for augmenting my own limited brain functions. I don’t know why the rest of the world, presumably equally suffering from the same limitations, doesn’t see it that way.

Lispitaph

For John McCarthy:

Here lies a Lisper
Uninterned from this mortal package
Yet not gc'd
While we retain pointers to his memory

Also, too.

Prolixity of RDF vs Lisp

I'm messing with OWL/RDF and other semantic web goodness. Here is how you define an enumerated lists of strings (that can be the value of some property:

<code>
<owl:oneof parsetype="Resource">
<rdf:rest parsetype="Resource">
  <rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">ACTIVATION</rdf:first>
  <rdf:rest parsetype="Resource">
     <rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">INHIBITION-ALLOSTERIC</rdf:first>
    <rdf:rest parsetype="Resource">
      <rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">INHIBITION-COMPETITIVE</rdf:first>
      <rdf:rest parsetype="Resource">
        <rdf:rest parsetype="Resource">
          <rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">INHIBITION-NONCOMPETITIVE</rdf:first>
          <rdf:rest parsetype="Resource">
            <rdf:rest parsetype="Resource">
              <rdf:rest parsetype="Resource">
                <rdf:rest parsetype="Resource">
                  <rdf:rest resource="http://www.w3.org/1999/02/22-rdf-syntax-ns#nil">
                  <rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">ACTIVATION-ALLOSTERIC</rdf:first>
                </rdf:rest>
                <rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">ACTIVATION-NONALLOSTERIC</rdf:first>
              </rdf:rest>
              <rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">INHIBITION-UNCOMPETITIVE</rdf:first>
            </rdf:rest>
            <rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">INHIBITION-OTHER</rdf:first>
          </rdf:rest>
        </rdf:rest>
        <rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">INHIBITION-IRREVERSIBLE</rdf:first>
      </rdf:rest>
    </rdf:rest>
  </rdf:rest>
</rdf:rest>
<rdf:first datatype="http://www.w3.org/2001/XMLSchema#string">INHIBITION</rdf:first>
</rdf:rest></owl:oneof>
</code>

That's 1853 characters, to do something that is obviously derived from Lisp and which in Lispy syntax can be done like this:

(oneof
 "ACTIVATION"
 "INHIBITION-ALLOSTERIC"
 "INHIBITION-COMPETITIVE"
 "INHIBITION-NONCOMPETITIVE"
 "ACTIVATION-ALLOSTERIC"
 "ACTIVATION-NONALLOSTERIC"
 "INHIBITION-UNCOMPETITIVE"
 "INHIBITION-OTHER"
 "INHIBITION-IRREVERSIBLE"
 "INHIBITION")

Or 244 characters. The OWL format is 7.6 times the size, and this is a relatively simple example.

Of course the RDF format is grossly inefficient in terms of space and bandwidth, but my real problem with it is that it is also vastly inferior in terms of human comprehensibility.

One of Lisp's real strength is in human interface -- its external representations are simple and direct representations of its internal structures. This is what makes hacking Lisp fun, and powerful. As some Lisp guru once said, "you can feel the bits between your toes", but it's not the bits, its the actual conceptual data structures that have an almost tangible existence in a Lispy environment.

Pretty much nothing since Lisp has retained this quality. Modern IDEs do a lot to make code more tangible, but are pretty primitive when it comes to data. And XML/RDF is only barely human-readable, and not at all human-typable at any scale.

invokedynamic

Sun has just now decided that adding support for dynamic language to the JVM might be useful. It's only been about nine years since I and many others have been making awkward implementations of Lisp and other dynamic languages on the JVM. I did my work in this area when I was at IBM research working on Java tools, and I was working with a project that was working on a JVM implementation, and I tried to get them interested in adding some extensions like this. No go back then, but better late than never.

Lisp infects another formerly-normal person

A Microsoft guy, heavily involved with .NET and Visual Studio, discovers Lisp and finds it to be a sin, but in a good way. Is he aware that God wrote in Lisp? The theological implications are staggering.

The discussion let me to L Sharp, a Lispy scripting language for .NET, which is something I've been needing. I wrote something like this myself long ago (for Java), and haven't had the heart to try and do it again for yet another platform, so I'm happy to have someone else on it.

Too much information

Cosma on the horrible information overload of real biology:

One reason papers like this gladden my heart is my basic intellectual cowardice: the sheer endless proliferating detail of biology overwhelms me, especially when something drives home the fact that we keep finding utterly new stuff everywhere we look. Here we are, looking at our own guts, and coming up with stuff like this: "Three sequences from two subjects ... appear to represent a novel lineage, deeply branching from the Cyanobacteria phylum and chloroplast sequences." See? There are organisms whose closest relatives are the stuff that turns ponds and leaves green living inside us, and until now we had no idea. And when we eventually look inside them, they're going to turn out to be weirdly complicated and uniquely strange, exactly like everything else. And of course the damn things will have histories, again exactly like everything else. Biology just doesn't stop, and at some point the details and special cases make me wish my head would explode.

Read the whole thing, it's funny. But I feel the same way, there's just too damn much detail. If there is a spectrum of detail-oriented vs. abstractifying in cognitive style, I am way way to the right. I can't remember stuff, which is why I am a software guy, and why I am emotionally tied to elegant languages and systems like Lisp (the more powerful the abstraction, the less you have to rememeber) and also have a deep interest in user interfaces (because a good UI will remove cognitive burdens via affordances).

This is what bioinformatics is all about, figuring out some way to abstract from all the detail into some form of useful knowledge. Statistics pulls a lot of weight, but since my mathematical skills have atrophied my efforts are more in the line of making environments that let scienteists link various types of knowledge together, perform computations over them, and visualize the results.

My real interest is broader; it's in figuring out just what it means to know something in a condition where you have a limited brain and senses surrounded by effectively infinite amounts of data. This is not a problem unique to bioinformatics. Basically it's everybody's problem, the problem of the googlectual. Google is a great tool but causes as many new problems as it solves.

Bruce Sterling's most recent book is promoting a vision of a world full of SPIME, which roughly means objects with a trackable identity (think RFID) and that have their entire history monitored, recorded, and available. This is going to save our environmental asses, somehow. Let's say he's right, how are we going to make use of all this data? I mean, it would be nice to be able to do a Google-local-physob search for that screwdriver I misplaced the other day, or have a dashboard widget that is tracking my laundry status and raises a red flag when clean sock levels are dangerously low. But presumably there are more interesting forms of knowledge to be extracted from a complete record of the interacting histories of objects.