Lambda the ultimate incantation

I endorse this:

Thus, I believe it is possible...to understand programming languages as the latest instance of a dream and set of technologies developed by mystics, alchemists, philosophers, mathematicians, scientists, and engineers. These languages do not just represent things, they also do things in the world. They are both symbolic and material in form. They are central to the disenchantment of the world and, simultaneously, the substrate for a "reenchantment of the world." They are, to sacrilegiously misappropriate the lexicon of the Catholic Church, "the word incarnate." Programming languages melt the boundaries between science and religion because they are an unholy union of the two.

-- Warren Sack, The Software Arts (forthcoming)

(for more on this project, see here).

Computation is the intersection of quite a few different things: science and religion, mathematics and language, engineering and psychology, and more. I don't know that these unions are exactly unholy, but let's just say they tend to have relationship problems.

2013 Blogyear in Review

This was an unusual year for blogging. Thanks to Venkatesh Rao I got to be a “blogger in residence” at Ribbonfarm, and that occupied most of my writing and thinking efforts. It was a chance to do something more thematic, and I returned to some of the same obsessions I had in graduate school and apparently have not finished with: agency, collective action, personae and social interaction, morality, empathy. Big hairy topics that I have no particular standing to pontificate about, if not for the fact that nobody else seems to understand them very well, including the professionals.

So the home blog only got about half as many posts as usual. However, a few of these managed to go mildly viral, which means they were seen by thousands rather than dozens. That is gratifying, for perfectly normal reasons, although it feels weird to me to actually care about that. Weirdly normal. There should be a word for that.

The Popular 3

The current three most popular posts of all time are from this year, and all actually manage to express a clear and coherent point, which is not always the case. And the points seem worthwhile, or in other words, I think they deserve to be popular; the ideas contained therein seem actually valuable and slightly original, and they ought to be in more people’s heads. All three seem like they could be usefully expanded on and may generate some longer writing in the future.

Hostile AI: You’re soaking in it! develops a theory of human-hostile artificially intelligent systems that already exist. Plays off the LessWrong “friendly AI” people, and advocates that they turn their considerable talents to solving real problems.

Lisp is for Stupid People (nobody noticed my recursive acronym, sigh) was a successful effort to get attention on Hacker News, but there is a serious point there, which is that software developers need to get over their self-image as rock stars of the intellect and acknowledge their limitations. Languages and other tools are designed, usually not very explicitly, to augment the intellect, but if we are more honest about our limitations, we could do much better.

“God” == God leverages Alan Moore to solve forever the tedious argument between atheists and theists, by pointing out that gods are not the kind of things that can be said to exist or not-exist, but are best understood as concepts with actual causal power, a metafiction in Moore’s terminology. I connected this to the different powers words have in written vs oral cultures, and to software, which is a relatively new way to link symbols and causality.

The Idiosyncratic 3

And here are three posts that did not rise in the ranks, and don’t make any stunningly clear points, but I have a fondness for so giving them a small boost:

Engineers of Human Souls I really do believe that social media is reshaping who we are, and I sure wish it wasn’t being done so clumsily. I'm hardly the only one to say this, but I keep feeling a need to say it.

The Opposite of Mathematics How often have you picked up a book on some abstruse intellectual subject (in this case, the relationship between mathematics and narrative) and found someone you know in the subject matter?

Proposed Extensions to the Booleans is a minor rebellion against the constraints of my day job, which has to do with what we in the business call "knowledge representation". Doing this involves taking a very crabbed view of what both knowledge and representation are, but since it’s slightly less crabbed than what usually goes on in computers, it is on the cutting edge and potentially useful. But man, do we have a long way to go.

Lisp is for Stupid People

[yes, the title is clickbait.]

I’ve been programming in Lisp for many years. It has a reputation as language only suitable to forbiddingly smart people (and indeed, most Lisp programmers I know are MIT-trained or the equivalent). I never quite understood that though. In fact, I realized recently that the reason I like Lisp, why I am always drawn back to it despite its general lack of commercial marketability, is not because I am so smart, but because I’m so stupid.

Or, more precisely, it is because Lisp is a better tool for overcoming my own mental limitations. It’s not that I am particularly stupid – my limitations are largely shared by everybody else. My attention is limited, my processing capacity is limited, my short-term memory capacity is limited, simply by virtue being a human being.

Programming is thought to require being able to hold complicated structures in the head. Here’s a cartoon illustration, which is pretty accurate as far as it goes, and here’s some actual interesting research on programmer’s mental functions. It may be that the fabled 10x or 100x programmers are just a little bit better at dealing with complexity than normal. I’m pretty good at it, but not superhuman, especially now that I’m getting alarmingly old.

The alternative to requiring programmers to be amazing jugglers of objects and acrobats of attention is powerful abstraction and good design. Abstraction is the basic way in which programmers encapsulate a bit of complexity in a single object or function call. All programming languages let you do this but some make it easier and/or offer more ways to do it. Good design means organizing a system around a few fundamental principles so that things make sense and navigating through the code doesn’t require a lot of effort.

These qualities are linked. Clean design means choosing abstractions so that they work together in powerful ways, and better abstraction tools make it easier to do that. It is certainly possible to do good software design in a less-powerful language. Some of the most impressive pieces of software design I’ve seen recently has been the series of visualization libraries coming out of Jeff Heer’s group, including prefuse (in Java) and the newer d3.js (in Javascript). The designers of these libraries have carefully chosen their abstractions so they fit together in powerful ways.

But, I maintain, it is a lot easier to do this in a language that is designed for it, and Lisp is that language. In Lisp, macros and other features make it trivially easy to hide all the details away in suitable abstractions, with the result that the meat of the program can be compact, so that the programmer can focus on it. In Java, by contrast, the important part of a program generally tends to be hidden away amidst boring boilerplate code that is necessary to please the Java compiler but not germane to the problem at hand (this article refers to the novel software metric “beef-to-bun ratio”, which is low in Java but apparently improving).

Thus Lisp has always seemed to me a much better tool than any other for augmenting my own limited brain functions. I don’t know why the rest of the world, presumably equally suffering from the same limitations, doesn’t see it that way.

Illiterate Programming

For some reason, the group I am working with on a pretty complex Rails application does not believe in comments. They are big time adherents to agile methods, TDD, and that sort of thing, and the anti-comment stance seems to be a side effect of that. The reasons, I'm told, is that code should be self-explanatory, and also since comments can't be validated with tests they are likely to become outdated and inaccurate.

Well, the first reason seems like nonsense. I admit to sometimes taking that position myself, but I'm used to working in languages that have more powerful abstraction capabilities than Ruby (and so it is easier to shape programs to match human thought). And even there, it is almost always useful to put in some comments as guides. The Rails/Agile community seems to put a lot of thought into matching up user-visible structure, program structure, data structure, and the natural language ways to describe them, but their tools are really pretty crude and don't work in exactly the places where you need guidance.

For example, I ran across a method called sanitize_structure. Obviously this was supposed to take some representation of a (molecule) structure and produce a new one that had certain features removed, but what were they? There was no easy way to tell. And that's because this operation is not self-evident, it depends on domain knowledge that is not readily captured in standard Rails apparatus.

The second reason also seems pretty nonsensical. Yes comments may become outdated, but eliminating them entirely seems like having your legs amputated to get rid of toe fungus.

Now, this group works much more collaboratively than most places I've been involved with. And that means that perhaps code doesn't need to be as self-explanatory as I'm used to – instead, you are encouraged to go talk to people and get them to explain it to you. Talking is good, although it seems both inefficient and risky as a way to transmit software design information.

But I've really been missing the presence of English language in the code. I've tried addressing this technically – I found an Emacs package that lets me insert annotations into the code that do not get saved in the file but instead live in a side-channel. This means at least I can make my own notes and read them later, although nobody else sees them. And commit messages from source control are also a valuable source of human-scale explanations and design rationales.

A lot of agile seems to involve this kind of over-reaction to very real problems of earlier models of software development. But I guess this kind of over-reaction is fundamental to how any new thing establishes itself.

[to go up a few meta levels – I realized recently that the whole point of most of my work, including the commercial, research, and philosophical aspects of it, might be summed up as exploring ways for reconciling computational technology with the way human minds actually work. And given this, I realized that the agile methodology people might have something to teach me in that regard – that is one reason I took this job. I was pretty certain I would find adapting to agile to be a struggle, and it has been, but I think it will be a productive one. ]

[and more meta – I have been thinking about starting a different blog for technical stuff like this – having a technical blog seems to be practically a job requirement these days – but so far can't be bothered. I'll use tags.]

Who you calling "coder", coder?

Although Coders at Work is a fine collection of interviews with a stellar cast of software people, I hate the title and the way "coder" has become the word of choice for the kind of stuff I do. But I still haven't found a good name for it.

"Programmer" is OK but drab, "Hacker" is good but it has an unfortunate criminal second meaning, "Maker" is both too new and too broad, "Software Engineer" always sounded vaguely pretentious to me. "Software Designer" is probably the most accurate, but it never caught on. "Software Architect" isn't bad, although it tends to get interpreted too narrowly most of the time. Real architects (of buildings) integrate engineering concerns, the knowledge of and need to support patterns of human activity, and artistic striving…it's a label I'm comfortable with. Or simply "Developer", that is open-ended enough while capturing the creative essence of the activity.

But "Coder" is the worst of the lot, it suggests a drone grinding away in some 19th century office with a complete lack of creativity or engagement – someone who simply mechanically encodes ideas that were dreamt up by someone else. Blah.

Maybe it's a function of social context. I would be embarassed to be a labeled a "coder" at some non-specialist social gathering, but among software types it functions something as a badge of honor; it means that you are still hands-on and not some kind of distant manager or philosopher/flamer. Maybe it functions like "queer" or the n-word…a derogation being reappropriated as a badge of pride.

[update: I forgot "Computer Scientist", which is my actual current job title. Of course it is also quite pretentious, and it always brings to mind the adage that any field with "science" in its name isn't one]

Real Labor Day; Technical Work; Open Source Economics

Today is the real labor day; the one in September was an effort to disassociate the more conservative parts of the labor movement from the radical factions. May Day was relabeled as "Loyalty Day" by Eisenhower.

I'll link to this post I did on the fake labor day a couple of years ago on the nature of programming work.

Very few people in the computer industry seem to care about the unequal distribution of monetary gains in the technosphere. That some people get to be zillionaires while others slave away in cubicles seems very normal. In the heart of Silicon Valley where I work, everyone thinks they are going to get rich, and a significant enough fraction does. The open-source movement, a great idea in many ways, has only intensified the concentration of financial gain, where a few people who manage to occupy a strategic location in the system end up profiting over the unpaid work of others. Almost nobody seems to be critiquing this, but one exception is Seth Finkelstein, who focuses on how the unpaid labor of thousands of Wikipedia writers and editors has not only enriched the world, but a few individuals who get to take credit for this vast network of volunteers.

Oh well, I'm just bitter because I have managed pretty well to avoid getting rich, my interests have always either been non-commerical and/or mistimed (I had a proposal for a www-like system in 1986, a few years before the actual web took off). Luck has a lot to do with who wins in a winner-take-all economy; so does having a particular personality type. We live in a culture that worships outrageous success and disdains those whose accomplishments are modest. The genius of the labor movement was in giving a voice to the ordinary, in glorifying the mundane. Only partly successful, of course -- the dynamics that lead to inequality of status are powerful and perhaps innate to human existence; the global market economy did not invent them, it just perfectly embodies them.

The labor movement was a response to the dislocations of the industrial revolution. We are in the midst of a postindustrial revolution; new economic forms are being invented as we speak (virtualized companies, open source projects, intellectual commons...) and who knows how that's going to shake out. Nobody seems to have a very good model for how the information economy (where goods are expensive to create and free to reproduce) should relate to the everyday economy of scarcity, of things like food and energy. Open-source is creating great value for the world while the people who create it have to beg and scrounge to support their efforts. This seems wrong and unsustainable in the long term.

Programming as Labor

On Labor Day, it's time to reflect about the nature of technical work. It can be enormous fun, or complete sucky. It can be enormously lucrative, or lead to abysmal unemployment. Technical workers can easily be exploited by management. Dilbert-like working conditions abound. Older workers can easily be shoved aside in favor of younger workers who can be paid less and work longer hours. The globalization of work and the relative ease of outsourcing work has the technical world in a race to the bottom. Open source is a great boon for the world but it drives down programmer salaries, and the net economic effects are that programmers around the world are subsidizing big companies with free labor. Software becomes a winner-take-all industry, which is great for the winners, but not so great for the also-rans, which is inevitably going to be most of us.

That's the downside, which very rarely gets talked about. The upside of course is that technology is in fact an extremely dynamic and productive industry, with a great variety of opportunities, etc. But technology cheerleading is so prevalent and tedious, that perhaps its time to take a look at the human downside of all those wonders. Today's a day for thinking about the 50-year old laid-off software developer who can't get hired, or those who have to pay California mortgages while competing with Indian salaries, or those with medical conditions that prevent them from working or getting insured, or other victims of life vicissitudes. There is so much lionization and hero worship in the technology industry, and so little attention paid to ordinary workers.

Stock options, the dream of starting your own company, and transitions to management all serve to keep the workforce from achieving the sort of class consciousness that would permit them to organize on their own behalf. Programmers think of themselves as independent-minded, and oftebn have been infected with libertarianism. They are probably the last occupation on earth that could be unionized. Like the children of Lake Woebegon, they all think they are above average and are going to come out on top in the winner-take-all competition, and don't have much patience with those who aren't winning the race.

Here are a few (very lame) efforts at getting programmers to organize on their own behalf:

Programmers-Union: a chat site that seems dead. Here are some typical reactions from libertarianish programmers.

Cyberlodge. Sponsored by a real union, the IAMAW. Tagline: "Fight Offshoring". They actually seem to have a tiny bit of a clue, but the site hasn't been updated since April.

Programmer's Guild: somewhat more alive, this is also function mainly as to lobby for increased protectionism of domestic jobs. "The Programmers Guild advances the interests of U.S. technical and professional workers in information technology (IT) fields, and opposes the transfer of U.S. jobs, technology, and infrastructure overseas." Here they are exposing a sleazy law firm giving advice to company HR departments on how to game H1-B regulations by place fake classified ads. That's a valuable sort of interest-group advocacy work, although a long way from collective bargaining.

Update: here's another one, The Washington Alliance of Technology Workers, Communications Workers of America, Local 37083, AFL-CIO. This is the realest-looking one so far. h/t Wall Street Journal!

Would a programmer's union be a good thing, even if it was capable of getting off the ground? Probably not, at least not one that operated in the classical model. The tech industry is too distributed, entrepeneurial, and irregular to make such a thing workable. But that doesn't mean that tech workers can't start finding some common interests to organize around. Geeks are very effective when their interests are threatened, as organizations like the EFF, GNU, and Creative Commons show. But none of these address bread-and-butter economic issues, perhaps because geek culture is too young to have had to worry much about them. That will change.

[photo h/t: Happening Here]
[update: actual discussion of unions and the modern world happening here. I liked this comment:

the oft heard argument that "[name of multinational] moved its factories to China and Mexico. Therefore it is necessary to abandon unions" makes as much sense as "I had a lousy the Big Mac the other day. It is time to do away with restaurants."

]

UML kills me

So I'm being forced to sit through a long and tedious course on UML, taught by a guy who probably hadn't started high school when I first began doing object-oriented programming. It's 80% tedium, 10% idiocy, and maybe 10% useful (if you accept that UML can actually be a useful communications tool, which I do).

So yeah, a standard visual notation for expressing software designs is a good thing. Nonetheless UML appears supremely braindamaged to me, for many reasons:

- the presentation of UML seems inextricably linked to a particular notion of software development process. Thus, you have diagram types that are sort of similar (concept and class diagrams) but whose main difference is that they are intended to be used at different phases of the process (analysis and design). Unnecessarily redundant, and they are close enough to each other to cause cognitive confusion.

- UML concepts are loose, vague, sloppy, and weirdly named. For instance, "attributes" and "associations". Both of these are (ultimately) links between an object and something else, but the first is used for primitive types and the latter for object-object relationships. Argh. But actually, when I quizzed the presenter about this, he said that associations turn into "association attributes" or maybe it was "Attribute associations". Double argh.

- UML is a visual language, but the tools used to create diagrams all suck like a black hole. OK, maybe they don't all suck, I haven't tried them all, but Visio and ArgoUML and Rational Rose do. They all have one or more of: lousy UIs, missing features, idiosyncratic notation (to pile on top of the built-in idiosyncracies of UML).

- Since the tools suck in different ways (for instance, ArgoUML is actually sort of OK but is missing some of the diagram types), it might be possible to compensate by using more than one and passing the UML between them. Ha. UML is a well-established industry standard, right? You'd think that as a standard, it might be possible to write out a UML diagram from one UML editor and read it in with another. You'd be wrong. There is NO established serialization format for UML. This I find amazing and hard to understand even taking UML on its own terms.

- Related, but getting more philosophical, UML constantly fudges the extent to which it is trying to be a formal language. On the one hand, we are just drawing pretty pictures so the marketing guys can see what we are doing. On the other hand, we introduce all sorts of details that suggest we are really writing code, like public/private indicators on attributes. And then everybody tries to make tools that can do "round trip engineering" and go back and forth from UML to code (I've never seen one of these work).

- There's some undefineable grating quality about all discourse surrounding UML. I think it's related to the above, its attempting to straddle the gap between actual code and something else. This something else might be "business" but if so it's expressed in some way I don't understand.

I guess I'm extra miffed because I actually like diagrams and visual programming, but it seems to be that we have a case of a poor standard sucking all the air out of the space of visual object oriented programming. Also, I completely agree with what I see as the main goal of UML, which is enhanced communication between engineers and customers/domain experts. Again, I just think this is an abysmal way to go about it. Object-oriented programming is supposed to do this in itself, without an extra layer of sloppily-defined cruft.

Maybe I'm just missing some sort of business-oriented DNA. I went through an experience some years ago, where I (and my cow-orkers) first encountered relational databases. Everybody there was from sheltered academic environments where we had never seen such a thing before, and we regarded mostly with hostility and didn't make very good use of it. Eventually I had an epiphany where I finally got the value of the relational model, and understood how real-world applications might actually be interested in data. Perhaps UML similarly has value that is just hidden from me due to my lack of appropriate background, although I doubt it.

Keyword: venting,
Google fodder: UML sucks

Non-review: Code Complete

People at work worship at the metaphorical feet of a book called Code Complete, which is apparently a "runaway bestseller", a "software engineering classic", according to its back cover, where it is also endorsed by a variety of software gurus like John Vlissides, Alan Cooper, and Ralph Johnson. It's very big and covers everything from planning methodologies to how to structure your if statements.

I already know how to program, so I don't really want to spend time reading this classic. How can I dismiss it without having to read through the whole thing? Ah, here we go, look up recursion in the index and get this:

Recursion isn't useful often...For most problems, it produces massively complicated solutions -- in those cases, simple iteration is usually more understandable. Use recursion selectively.

Plonk.