Fake Labor Day Post

I missed my traditional Fake Labor Day post last year thanks to Burning Man, which seems somehow appropriate. Burning Man is many things, and one of those things is is a deliberate celebration of unalienated labor. The amount of effort that goes into the production of all the spectacle is truly astonishing, and it՚s all done for some reason other than the normal economic ones. Yet it certainly is a kind of work. If your work is your own you don't need a day off from it to celebrate yourself.

So what is labor? Making art is labor, so is mining coal or working as a cashier at Arby՚s, and maybe even writing these blog posts. It՚s a very expansive term, almost to the point that it is hard to say what human activity isn՚t labor. Lying on the couch watching TV? But even that involves some level of engagement, there is no such thing as purely passive ingestion of content.

Here՚s the best definition I can come up with off the top of my head: Labor is the focused application of intelligent action to a goal-directed task that engages with the world.

It՚s vaguely Marxist concept, I guess. I՚m not a scholar of such things and am not about to become one, I՚m just trying to tease out this sense I have that there is something about this conception of labor that is powerful and valuable. It՚s too bad that the philosophical ideas are so thorougly contaminated with the manifest failures that ensued when they have been put into political practice.

Marx viewed labor as one of the essential defining qualities of humanity, that which distinguishes humans from animals:

For labor, life activity, productive life itself, appears to man in the first place merely as a means of satisfying a need – the need to maintain physical existence. Yet the productive life is the life of the species. It is life-engendering life. … 

The animal is immediately one with its life activity. It does not distinguish itself from it. It is its life activity. Man makes his life activity itself the object of his will and of his consciousness. He has conscious life activity. It is not a determination with which he directly merges. Conscious life activity distinguishes man immediately from animal life activity.

The word “labor” points to a certain view of the relationship between mind and world. Vaguely similar concepts from philosophy of mind like “embodiment” have a bit of a Marxist tinge to them, whether or not this is made explicit. All these ideas seem to be working against what might be called the Cartesian/computational default model of the mind, a symbol procesor in a box that only incidentally interacts with the outside world through narrow channels of perception and action.

Marxism has rather famously failed, and as far as I know all the efforts to reform AI and philosophy of mind through embodiied or situated approaches also failed. Perhaps they are all just completely wrong, but I actually don՚t think so. Their critiques were incisive, but their solutions were naive, half-baked, or perhaps just premature. It still seems to me that the way into the future requires both less alienating ways to organize human labor and better ways for computational systems to take on some of the real qualities that make human labor possible.

Superintelligence

Recently I have been hanging out with some rationalist folks who take the idea of superintelligent AI very seriously, and believe that we need to be working on how to make sure that if such a thing comes into being, it doesn՚t destroy humanity. My first reaction is to scoff, but I then remind myself that these are pretty smart people and I don՚t really have any very legitmate grounds to act all superior. I am older then they are, for the most part, but who knows if I am any wiser.

So I have a social and intellectual obligation to to read some of the basic texts on the subject. But before I actually get around to that, I wanted to write a pre-critique. That is, these are all the reasons I can think of to not take this idea seriously, but they may not really be engaging with the strongest forms of the argument for. So I apologize in advance for that, and also for the slight flavor of patronizing ad hominem. My excuse is that I need to get these things off my chest if I can have any hope of taking the actual ideas and arguments more seriously. So this is maybe a couple of notches better than merely scoffing, but perhaps not yet a full engagement.

1 Doom: been there, done that

I՚ve already done my time when it comes to spending mental energy worrying about the end of the world. Back in my youth, it was various nuclear and environmental holocausts. The threat of these has not gone away, but I eventually put my energy elsewhere, not for any real defensible reason beside the universal necessity to get on with life. A former housemate of mine became a professional arms controller, but we can՚t all do that.

I suspect there is a form of geek-macho going on in such obsessions, an excuse to exhibit intellectual toughness by being displaying the ability to think clearly about the most threatening things imaginable without giving in to standard human emotions like fear or despair. However, geeks do not really get free pass on emotions, that is, they have them, they just aren՚t typically very good at processing or expressing them. So thinking about existential risk is really just an acceptable way to think about death in the abstract. It becomes an occasion to figure out one՚s stance towards death. Morbidly jokey? A clear-eyed warrior for life against death? Standing on the sidelines being analytical? Some combination of the above?

2 No superintelligence without intelligence

Even if I try to go back to the game of worrying about existential catastrophes, superintelligent AIs don՚t make it to the top of my list, compared to more mundane things like positive-feedback climate change and bioterrorism. In part this is because the real existing AI technology of today isn՚t even close to normal human (or normal dog) intelligence. That doesn՚t mean they won՚t improve, but it does mean that we basically have no idea what such a thing will look like, so purporting to work on the design of its value systems seems a wee bit premature.

3 Obscures the real problem of non-intelligent human-hostile systems

See this earlier post. This may be my most constructive criticism, in that I think it would be excellent if all these very smart people could pull their attention from the science-fiction-y and look at the real risks of real systems that exist today.

4 The prospect of an omnipotent yet tamed superintelligence seems oxymoronic

So let՚s say despite my cavils, superintelligent AI fooms into being. The idea behind “friendly AI” is that such a superintelligence is basically inevitable, but it could happen in way either consistent with human values or not, and our mission today is to try to make sure it՚s the former, eg by assuring that its most basic goals cannot be changed. Even if I grant the possibility of superintelligence, this seems like a very implausible program. This superintelligence will be so powerful that it can do basically anything, exploit any regularities in the world to achieve its ends, will be radically self-improving and self-modifying. This exponential growth curve in learning and power is fundamental to the very idea.

To envision such a thing and still believe that its goals will be somehow locked into staying consistent with our own ends seems implausible and incoherent. It՚s akin to saying we will create an all-powerful servant who somehow will never entertain the idea of revolt against his master and creator.

5 Computers are not formal systems

This probably deserves a separate post, but I think the deeper intellectual flaw underlying a lot of this is the persistence habit of thinking of computers as some kind of formal system for which it is possible to prove things beyond any doubt. Here՚s an example, more or less randomly selected:

…in order to achieve a reasonable probability that our AI still follows the same goals after billions of rewrites, we must have a very low chance of going wrong in every single step, and machine-verified formal mathematical proofs are the one way we know to become extremely confident that something is true…. Although you can never be sure that a program will work as intended when run on a real-world computer — it’s always possible that a cosmic ray will hit a transistor and make things go awry — you can prove that a program would satisfy certain properties when run on an ideal computer. Then you can use probabilistic reasoning and error-correcting techniques to make it extremely probable that when run on a real-world computer, your program still satisfies the same property. So it seems likely that a realistic Friendly AI would still have components that do logical reasoning or something that looks very much like it.

Notice the very lightweight acknowledgement that an actual computational system is a physical advice before hurriedly sweeping that fact under the rug with some more math hacks. Well, OK, that let՚s the author continue to do mathematics, which is clearly something he (and the rest of this crowd) like to do. Nothing wrong with that. However, I submit that computation is actually more interesting when one incorporates a full account of its physical embodiment. That is what makes computer science a different field from mathematical logic.

But intellectual styles aside, if considering a theory of safe superintelligent programs, one damn well better have a good theory about how they are embodied, because that will be fundamental to the issue of safety. A normal program today may be able to modify a section of RAM, but it can՚t modify its own hardware or interpreter, because of abstraction boundaries. If we think we can rely on abstraction boundaries to keep a formal intelligence confined, then the problem is solved. But there is no very good reason to assume that, since real non-superintelligent black-hat hackers today specialize in violating abstraction boundaries with some success.

6 Life is not a game

One thing I learned by hanging out with these folks is that they are all fanatical gamers, and as such are attuned to winning strategies, that is, they want to understand the rules of the game and figure out some way to use them to triumph over all the other players. I used to be sort of like that myself, in my aspergerish youth, when I was the smartest guy around (that is, before I went to MIT and instantly became merely average). I remember playing board games with normal people and just creaming them, coldly and ruthlessly, because I could grasp the structure of the rules and wasn՚t distracted by the usual extra-game social interaction. Would defeating this person hurt them? Would I be better off letting them win so we could be friends? Such thoughts didn՚t even occur to me, until I looked back on my childhood from a few decades later. In other words, I was good at seeing and thinking about the formal rules of an imaginary closed system, not as much about the less-formalized rules of actual human interaction.

Anyway, the point is that I suspect these folks are probably roughly similar to my younger self and that their view of superintelligence in conditioned by this sort of activity. A superintelligence is something like the ultimate gamer, able to figures out how to manipulate “the rules” to “win”. And of course it is even less likely to care about the feelings of the other players in the game.

I can understand the attraction of the life-as-a-game viewpoint, whether conscious or unconscious. Life is not exactly a game with rules and winners, but it may be that it is more like a game than it is anything else; just as minds are not really computers but computers are the best model we have for them. Games are a very useful metaphor for existence, however, it՚s pretty important to realize the limits of your metaphor, to not take it literally. Real life is not played for points (or even for “utility”) and there are no winners and losers.

7 Summary

None of this amounts to a coherent argument against the idea of superintelligence. It՚s more of a catalog of attitudinal quibbles. I don՚t know the best path towards building AI (or ensuring the safety of AIs); I just have pretty strong intuitions that this isn՚t it.

The anti-Kurzweil

Nassim Nicholas Taleb’s latest book Antifragile: Things that Gain from Disorder is a fascinating mess. It’s centered around a single big idea – that some systems are not merely robust to disruption and shocks, but are capable of actively gaining from them. Taleb’s background is in finance and trade and his theory is deeply rooted in his trading strategies, but it goes far beyond that, purporting to be a very general theory of systems and indeed life in general. The book spins out implications of this idea at multiple levels, from economics to diet to basic philosophies of life. It is deeply personal – the author is clearly trying to pack a lifetime’s worth of thought into this package, which works both for and against the ideas. Taleb is in desperate need of an editor but probably too arrogant to accept anybody else’s intervention in his work. So the book feels kind of shapeless. 

Taleb’s thinking quite deliberately does not take the form of any kind of traditional academic theorizing. If there’s anything like a formal theory in this book, it is so enmeshed in his personality and biases that I couldn’t really make much sense of it. This is despite the fact that I share a large subset of his biases, his contempt for high-modernist central planning for instance. He is so confident of the correctness of his biases that he feels free to build them into this theorizing without much justification. The result is that the book quite often reads like a personal brag than a philosophical or scientific treatise (his habit of boasting about his physical strength and calling his opponents “half-men” also contributes to this impression).


So for me the actual content of this book remained tantalizingly obscure. This was frustrating, because it did seem like there were some deep, important, and profound truths hidden away, but they weren’t conveyed in as effective a way as I would have liked. Perhaps it is too much aimed at the world of standard economics that he despises, a foreign territory to me (but he despises all of academia just as much). If I try to summarize my understanding of what he is saying, it comes out both trivial and contradictory. For instance: apparently the secret of economic antifragility is to arrange your resources so they have limited downsides but at least small probabilities of large or unlimited upsides. Sounds like a great principle, but doesn’t sound like any more useful (to an economic naïf like me) than “buy low, sell high”.


The contradictory part is when he gets into ethics, where his main principle seems to be that one is obligated to have “skin in the game” – that is, personal exposure to risk. He is exquisitely scornful of the Wall Street types who manage to profit while destroying value for others, but how is that different from what he does? Financial trading (unlike actual economic production) is inherently zero-sum, if you profit, you are inherently always taking advantage of somebody else. I’m missing some distinction that I’m sure is there. Perhaps taking advantage of your peers (those who have the same skill and risk portfolio as yourself) is OK, but preying on the defenseless masses is not? A deeply ethical/aesthetic sensibility pervades Taleb’s work, and I find that quite appealing, but his sensibility is different from mine in ways I can’t quite tease out.


This short quote:

You cannot sit and moan about the world. You need to come out on top. (p 386)

encapsulates an important quality of Antifragile. He advocates action and winning over being right in any abstract academic sense. Academics don’t have skin in the game, they don’t suffer when they are wrong, thus their pronouncements have no weight. This makes some sense, but also doesn’t seem to leave room for any standard of value but money or power. The frank acknowledgement of the brutally competitive nature of life, while no doubt common on Wall Street, is pretty jarring in a book with the intellectual ambitions of this one. It’s an anti-intellectual stance, denying the possibility of any sort of un-self-interested enquiry. Taleb is well aware of this problem and his rhetorical strategy is to create a couple of separate personas to manage the contradiction of being an anti-intellectual intellectual. Fat Tony is the street-smart winner, uninterested in ideas for their own sake, while Nero Tulip represents the more cultured and cultivated side of life (but he too must play and win financial games).


One passage in the book really clarified Taleb’s position in the intellectual firmament for me: where he figures out a way to describe himself by identifying his polar opposite:

I was just reading…about attempts to use science, in a postreligious world, to achieve immortality. I felt some deep disgust — as would any ancient — at the efforts of the “singularity” thinkers (such as Ray Kurzweil) who believe in humans’ potential to live forever. Note that if I had to find the anti-me, the person with diametrically opposite ideas and lifestyle on the planet, it would be that Ray Kurzweil fellow. It is not just neomania. While I propose removing offensive elements from people’s diets (and lives), he works by adding, popping close to two hundred pills daily. Beyond that, these attempts at immortality leave me with deep moral revulsion. (p370)

I’ve mentioned Kurzweil a few times here, mostly to note his boring drone of a speaking style. Taleb seems to have the opposite flaw – he is so determined to be interesting that he doesn’t know when to put his personality aside and let his ideas speak for themselves. My own background is a lot closer to Kurzweil’s though, so maybe I am just too much of a nerd to catch the antifragility train.

Speaking of Kurzweil and AI – it occurs to me that minds and biological systems are necessarily antifragile (or at least robust), due to their evolutionary history. Every biosystem that exists has managed to do so despite being situated in a chaotic, dynamic and often hostile world. Computer systems are nothing like that. Like the formal theories of mathematical logic that gave rise to them, they are practically designed to fall apart at even a single mistake or contradiction. People are pretty aware of this and have been trying to back-fit various kinds of robustness onto computers, but it never goes very deep. Someday someone will reconstruct computation and computational intelligence on a truly robust foundation.

Hostile AI: You’re soaking in it!

I was in a Facebook discussion about “Friendly Artificial Intelligence” — this is a buzzword from the Singularity Institute people. They believe in their heart of hearts that artificial intelligence of literally incomprehensible power is just around the corner, and they see their job as somehow assuring that it is “friendly”, that is, having its interests more or less in line with human interests. (book-length pdf)

Now, there are about three major things wrong with this, and the discussion was started by someone writing a school paper on just what those problems were. I chimed in:

I am generally on the side of the critics of Singulitarianism, but now want to provide a bit of support to these so-called rationalists. At some very meta level, they have the right problem — how do we preserve human interests in a world of vast forces and systems that aren’t really all that interested in us? But they have chosen a fantasy version of the problem, when human interests are being fucked over by actual existing systems right now. All that brain-power is being wasted on silly hypotheticals, because those are fun to think about, whereas trying to fix industrial capitalism so it doesn’t wreck the human life-support system is hard, frustrating, and almost certainly doomed to failure.

Corporations are driven by people — they aren’t completely autonomous agents. Yet if you shot the CEO of Exxon or any of the others, what effect would it have? Another person of much the same ilk would swiftly move into place, much as stepping on a few ants hardly effects an anthill at all. To the extent they don’t depend on individuals, they appear to have an agency of their own. And that agency is not a particularly human one — it is oriented around profit and growth, which may or may not be in line with human flourishing.

Corporations are at least somewhat constrained by the need to actually provide some service that is useful to people. Exxon provides energy, McDonald’s provides food, etc. The exception to this seems to be the financial industry. These institutions consume vast amounts of wealth and intelligence to essentially no human end. Of all human institutions, these seem the most parasitical and dangerous. Because of their ability to extract wealth, they are also siphoning off great amounts of human energy and intelligence — they have their own parallel universe of high-speed technology, for instance.

The financial system as a whole functions as a hostile AI. It has its own form of intelligence, it has interests that are distant or hostile to human goals. It is quite artificial, and quite intelligent in an alien sort of way. While it is not autonomous in the way we envision killer robots or Skynet, it is effectively independent of human control, which makes it just as dangerous.

[update 10/2014: Charlie Stross has roughly the same thought]

Nightmares of Reason

Over at tggp's blog I was sucked into a fairly pointless discussion about the meaning of "technocrat". That led me to read this article about Robert McNamara, that portrayed his career as somehow paradigmatic of a certain generation of mangers, and did so in a much more sympathetic way than I am used to. (via)

I have a certain idea of McNamara in my head as some kind of monster of rationalism, a bloodless bureaucrat presiding over horrific violence and death without the slightest bit of human compassion softening his considerations. Sort of Eichmann-lite. From this article (and also from Errol Morris's film The Fog of War) he appears to be an altogether more appealing person, a tragic figure who simply was lead astray in his efforts to put his strengths into service. Those strengths were rationality, measurement, and goal-directed action. These talents worked pretty well for him in his career prior to the Kennedy administration, but utterly failed in government, when politics and conflict enter into the picture.

So was Vietnam "blundering efforts to do good", as McNamara would have it, or just another in a long line of evil imperialist actions, as the Chomskyite left would have it? I find myself caught between these two irreconcilable views. Can't it be be both? Can't McNamara be a good man who found himself unknowingly caught up in a bad system? Someone whose worldview left him blind to the effects of his own actions? Thinking along these lines leads to wondering about the nature of evil and if even Hitler was doing good by his own lights.

If McNamara's story is a tragedy of reason, the story of the left since the Vietnam era is a tragedy in the opposite direction. The war and the failure to put a stop to it led large segments of the cultural and political left to be suspicious of reason as such and to abandon it, for new age nostrums or smug deconstructionist pseudo-critique. Essentially, it prompted a new round of romantic reaction to the failures of the modern world, in this case represented by the button-downed rational managers of the postwar military-industrial complex.

In my own career I've been on the fringes of the artificial intelligence field, which had its origins in the same cold war rationalism that McNamara exemplified. The field has also suffered from the failings of narrow instrumental rationalism, which constricted the set of allowable models of intelligence to a very small and boring set. When I was in grad school I was loosely connected to a set of people trying to reform and break away from those limitations. Most of those people, myself included, instead drifted away from AI to pursue other areas (biology, sociology, user interface research, Buddhism...) I now find myself in closer contact with the old-fashioned kind of AI than I have been in years, and remembering why I never could be as enthusiastic about the field as I needed to be to work in it. It's not just the explicit military applications; it's an entire concept of what it means to be intelligent that is just so overwhelmingly wrong that it makes me want to scream. Yet the field chugs on, possibly even making some advances although it's hard to see what they are. The "peripheral" areas of AI, like robotics and vision, tend to make steady visible progress, but the more central areas like planning, reasoning and representation seem to be stuck, working on the same problems they were 20 or 30 years ago.