Reading Recommendations for Rationalists

Here are a couple of somewhat congruent interactions between the deeply idealistic rationalist community and myself in the role of cynical old bastard.

(1) I was at a LessWrong event the other night where Ethan Dickinson spoke on social interactions and the problems that can arise and various ways of dealing with them. Here՚s the blurb:

Imagine a world where people cannot truly know each other. Miscommunication abounds. Well-intentioned remarks offend, while obvious truths are hidden or denied. Certain actions are deemed admirable by some, yet annoying or abhorrent by others. Entire communities enter into downward spirals of evaporative cooling, unproductive bickering, and bitter tribalized feuding.

This is the dystopian mindscape we find ourselves in when we fall prey to the biases and heuristics surrounding mental modeling and communication. What are these flaws in our understanding of each other? Is there a realistic path of self-improvement that can lead us to become better predictors and communicators? Can individual-level skills be parlayed into community-wide improvement?

My very immediate reaction to the first sentence was, wait, what other kind of world is there? Maybe the failure of imagination is mine, but I cannot imagine a world where we “truly know each other”, or even truly know ourselves for that matter. One thing you can say for humans, they are complex, and even for the people we know very well, there are always untapped depths.

But let՚s assume that we don՚t take “truly” overly literally. In fact, delete that sentence and I can՚t find anything objectionable at all. Still, I am not sure the idea of accurate representation of other people is the right model for human communication, any more than accurate physical cosmology is the right model for religion. Religion is about participating in ritual community; the truth of the words of a prayer have very little to do with it. And ordinary human communication also has a ritual quality to it, it is about expressing emotions; satisfying needs; finding, signalling, and reinforcing political/tribal realities; passing moral judgement; or simply enacting social roles. People interacting are only incidentally building more accurate models of each other; there is usually something else going on, and that something else has to be recognized and acknowledged.

My reading recommendation: All of Erving Goffman՚s work on strategic social interaction. Start with “On Cooling the Mark Out” and for more, see books like Interaction Ritual and Stigma: Notes on the Management of Spoiled Identity.

(2) Here՚s Scott Alexander making a typically lengthy and tightly-written fret about the fact that people use all sorts of irrational techniques to reinforce their beliefs, that tribalism is both a cause and a consequence of all sorts of questionable kinds of reasoning, and that lot of what should be rational debate is actually just mindless cheerleading for your side or hatred of the other side.

I hate to argue with him because he is so accurate in his diagnoses and so high-minded in his solutions. But essentially he wants human nature to change, for everybody to put down their emotion-based alliances and think objectively, which in cases like these mean among other things being able to empathetically take the viewpoint of an opponent.

As in the first case, this seems like a great idea but in some respects it misses the point of political speech, which is not primarily about reasoned debate and more about forming coalitions of power.

My reading recommendation: All of Bruno Latour՚s work on the politics of knowledge, starting with Science in Action. Also his bit on Socrates and Callicles in the more recent Pandora՚s Hope.

(3) Both rationalists that I am riffing off of are disturbed by the negative effects of emotion and power in human affairs. Rationality is seen as a corrective, a way of thinking (and being) that is at least in part insulated from such destructive forces. As a goal, that seems hard to argue with. But it may be that emotion and power are too fundamental to human behavior to be papered over by the rather thin layer of rationality available to us.

One could make an argument that these authors are actually more in tune with a certain kind of rationalism than positing a more naive sort of default agreeability. After all, there is no a priori reason why one agent in a conversation or other social relation should have the same goals as any other agent. Presumably they are out for themselves, and if they can find common cause, that՚s wonderful, but it՚s an achievement that has to be accounted for. And in all likelihood not as stable an achievement as we might like. So a properly rich rationalism would treat human interaction as more goal-oriented than accuracy-oriented.

The authors I am recommending have something in common: they both, in quite different ways, try to deal with the reality that humans are power- and status-seeking creatures well before they are truth- and comity-seeking. And whatever success they have at the latter is built using the machinery developed for the former. Latour and Goffmann both have developed a rich set of methods for describing the relationship between power and knowledge: Goffman applies the vocabulary of drama to ordinary life, and Latour merrily dispenses with normal ontological distinctions so he can describe power alliances between people, ideas, machinery, and nature. Conflict is an essential part of their world-pictures: people and other things have their own interests, the world is a chaos of competing interests, not a well-behaved unity.

This point of view may seem superficially cynical. But the depth of these authors՚ intellectual humanism elevates their work above mere cynicism. It is clear that despite seeing the often dirty and ugly machinery that underlies cognition and society, they still retain fondness for humans and their complex processes. And my argument for reading them is not based on either cynicism or idealism, but realism. They add a layer of depth to our understanding of human social processes.

Emacs hack: yank URL from Chrome (on Mac)

Damn is this going to be useful:

(defun yank-chrome-url ()
 "Yank current URL from Chrome"
  (interactive)
  (require 'apples-mode)
  (apples-do-applescript "tell application \"Google Chrome\"
 get URL of active tab of first window
end tell"
    #'(lambda (url status script)
        ;; comes back with quotes which we strip off
        (insert (subseq url 1 (1- (length url)))))))

I suppose I should have separate blogs for stupid hacks and Deep Contemplations of the Nature of Things, but who has the time to manage multiple web presences?

Lambda the ultimate incantation

I endorse this:

Thus, I believe it is possible...to understand programming languages as the latest instance of a dream and set of technologies developed by mystics, alchemists, philosophers, mathematicians, scientists, and engineers. These languages do not just represent things, they also do things in the world. They are both symbolic and material in form. They are central to the disenchantment of the world and, simultaneously, the substrate for a "reenchantment of the world." They are, to sacrilegiously misappropriate the lexicon of the Catholic Church, "the word incarnate." Programming languages melt the boundaries between science and religion because they are an unholy union of the two.

-- Warren Sack, The Software Arts (forthcoming)

(for more on this project, see here).

Computation is the intersection of quite a few different things: science and religion, mathematics and language, engineering and psychology, and more. I don't know that these unions are exactly unholy, but let's just say they tend to have relationship problems.

The Sacred and the Rational

At Burning Man, the Temple of Grace was something like the sacral center to the whole chaotic affair, an oasis of silence and tranquility amidst the noise, a place of meditation, where people went to commemorate and mourn the dead.

I made quite a few visits there, but my first was in the company of some friends, including the one who diagnosed irony in my dancing. He՚s a big-time rationalist, and I had an urge to start an argument/discussion with him there, to grill him on what the rationalist idea of the sacred was, how these two things were supposed to co-exist in his worldview. I did not do this, because that very urge seemed to go against the mood of the place, it would have been an attempt to generate heat and words in a place of calm and quiet. He (I assume) makes his own peace between reason and sacrality, and why should I trouble that peace at that moment? Quieting mental chatter, internal and external, also seems to go hand-in-hand with the sacred.

But I couldn՚t help mull over the opposition in my own mind, and am apparaently still doing so. Reason wants to break down and analyze and measure everything, the sacred exists outside all of that. The sacred is definitionally beyond reason, beyond all forms of contaminating human inquisition. It is that which demands the highest respect, and in that sense is the same for all religions and rituals. You don՚t have to believe in any particular god or creed to recognize it, and if you aren՚t a jerk you respect other people՚s sense of the sacred even when you disagree with their beliefs. It is that which can՚t be argued with. My own tendency to argue with everything gets put on hold (partially and temporarily) in its presence.

It is almost Yom Kippur, the holiest day in the Jewish calendar, and I am going to go be part of that (with the usual doubt and reluctance that apparently doesn՚t matter all that much). Ritual tunes up your relationship with the sacred. It is apparently something I need to do; probably something most people need to do.

Given our multicultural society there is not much that is held universally sacred. The minimal sacred object is personhood, the inner core of every person that deserves respect no matter how wrong, assholic, or fucked up they happen to be. That may not be enough, there is something about the sacred that demands public ritual. But it՚s better than nothing.

Previous Yom Kippur-related posts also seem to touch on this notion of that which is beyond language and reason. Hm. I am getting almost tediously repetitive. Maybe I need to call it a ritual.

Trip Report

So I went to Burning Man for the first time, and it was quite as amazing and overwhelming as I had hoped. I don՚t think I՚m going to go into details here. But I have to note that the fundamental thing about this event, which everybody knows but is not often made explicit, is that the whole thing is conceived, designed, built, and populated by people who are tripping balls most of the time (with exceptions). Black Rock City thus mirrors one of the more interesting phenomena of drug use: that a mind (or city) can be completely fucked up and yet still manage to perform most of the necessary operations to sustain daily life. I can՚t quite explain how this works, but in both cases it seems to say something about the robustness of the system architecture, for lack of a better term. A normal computer, by contrast, can՚t tolerate even a single component going wrong.

So having BM in an extremely hostile desert environment is something of a feature. Normal routines of survival are broken down, creating the necessity and opportunity of inventing other ones, that are more improvisational and more distributed. people or things getting fucked up are an occasion to fix them. There are always paths forward, generally not the ones you intended five minutes ago.

Here՚s the calling card I designed for myself beforehand (which, like me, may be a little too intellectual-jokey for the scene):

In that vein, my favorite moment (not the most beautiful or awe-inspiring, but the one that seemed most attuned to my own personal idiosyncrasies) was when I was out riding around with some friends, and we came to a small public square with a large purple phallic sculpture, climbable of course. So I climbed up and poked my head out and started dancing around, yelling look at me, I՚m a sperm. My friend shouted at me (from 30՚ away at least) to “stop dancing ironically, dance for real”. I have no idea what that meant, yet I successfully complied.

On the other hand, the heat, the nonstop electro-rave-whatever music that made it impossible to sleep, and the undertones of hippie self-satisfaction would occasionally combine to make me hate the place. I was hanging out with some much younger people and trying to follow their party schedule did a number on my aging body that՚s going to take a while to recover from. I think it was worth it.

I had multiple camera failures so have no pictures of my own, but on the other hand there is the upside that I didn't spend the whole time viewing things through a lens. A couple of good photo sets here and here. But (and I apologize to my readers for saying this) none of that captures the actuality of being there.

Superintelligence

Recently I have been hanging out with some rationalist folks who take the idea of superintelligent AI very seriously, and believe that we need to be working on how to make sure that if such a thing comes into being, it doesn՚t destroy humanity. My first reaction is to scoff, but I then remind myself that these are pretty smart people and I don՚t really have any very legitmate grounds to act all superior. I am older then they are, for the most part, but who knows if I am any wiser.

So I have a social and intellectual obligation to to read some of the basic texts on the subject. But before I actually get around to that, I wanted to write a pre-critique. That is, these are all the reasons I can think of to not take this idea seriously, but they may not really be engaging with the strongest forms of the argument for. So I apologize in advance for that, and also for the slight flavor of patronizing ad hominem. My excuse is that I need to get these things off my chest if I can have any hope of taking the actual ideas and arguments more seriously. So this is maybe a couple of notches better than merely scoffing, but perhaps not yet a full engagement.

1 Doom: been there, done that

I՚ve already done my time when it comes to spending mental energy worrying about the end of the world. Back in my youth, it was various nuclear and environmental holocausts. The threat of these has not gone away, but I eventually put my energy elsewhere, not for any real defensible reason beside the universal necessity to get on with life. A former housemate of mine became a professional arms controller, but we can՚t all do that.

I suspect there is a form of geek-macho going on in such obsessions, an excuse to exhibit intellectual toughness by being displaying the ability to think clearly about the most threatening things imaginable without giving in to standard human emotions like fear or despair. However, geeks do not really get free pass on emotions, that is, they have them, they just aren՚t typically very good at processing or expressing them. So thinking about existential risk is really just an acceptable way to think about death in the abstract. It becomes an occasion to figure out one՚s stance towards death. Morbidly jokey? A clear-eyed warrior for life against death? Standing on the sidelines being analytical? Some combination of the above?

2 No superintelligence without intelligence

Even if I try to go back to the game of worrying about existential catastrophes, superintelligent AIs don՚t make it to the top of my list, compared to more mundane things like positive-feedback climate change and bioterrorism. In part this is because the real existing AI technology of today isn՚t even close to normal human (or normal dog) intelligence. That doesn՚t mean they won՚t improve, but it does mean that we basically have no idea what such a thing will look like, so purporting to work on the design of its value systems seems a wee bit premature.

3 Obscures the real problem of non-intelligent human-hostile systems

See this earlier post. This may be my most constructive criticism, in that I think it would be excellent if all these very smart people could pull their attention from the science-fiction-y and look at the real risks of real systems that exist today.

4 The prospect of an omnipotent yet tamed superintelligence seems oxymoronic

So let՚s say despite my cavils, superintelligent AI fooms into being. The idea behind “friendly AI” is that such a superintelligence is basically inevitable, but it could happen in way either consistent with human values or not, and our mission today is to try to make sure it՚s the former, eg by assuring that its most basic goals cannot be changed. Even if I grant the possibility of superintelligence, this seems like a very implausible program. This superintelligence will be so powerful that it can do basically anything, exploit any regularities in the world to achieve its ends, will be radically self-improving and self-modifying. This exponential growth curve in learning and power is fundamental to the very idea.

To envision such a thing and still believe that its goals will be somehow locked into staying consistent with our own ends seems implausible and incoherent. It՚s akin to saying we will create an all-powerful servant who somehow will never entertain the idea of revolt against his master and creator.

5 Computers are not formal systems

This probably deserves a separate post, but I think the deeper intellectual flaw underlying a lot of this is the persistence habit of thinking of computers as some kind of formal system for which it is possible to prove things beyond any doubt. Here՚s an example, more or less randomly selected:

…in order to achieve a reasonable probability that our AI still follows the same goals after billions of rewrites, we must have a very low chance of going wrong in every single step, and machine-verified formal mathematical proofs are the one way we know to become extremely confident that something is true…. Although you can never be sure that a program will work as intended when run on a real-world computer — it’s always possible that a cosmic ray will hit a transistor and make things go awry — you can prove that a program would satisfy certain properties when run on an ideal computer. Then you can use probabilistic reasoning and error-correcting techniques to make it extremely probable that when run on a real-world computer, your program still satisfies the same property. So it seems likely that a realistic Friendly AI would still have components that do logical reasoning or something that looks very much like it.

Notice the very lightweight acknowledgement that an actual computational system is a physical advice before hurriedly sweeping that fact under the rug with some more math hacks. Well, OK, that let՚s the author continue to do mathematics, which is clearly something he (and the rest of this crowd) like to do. Nothing wrong with that. However, I submit that computation is actually more interesting when one incorporates a full account of its physical embodiment. That is what makes computer science a different field from mathematical logic.

But intellectual styles aside, if considering a theory of safe superintelligent programs, one damn well better have a good theory about how they are embodied, because that will be fundamental to the issue of safety. A normal program today may be able to modify a section of RAM, but it can՚t modify its own hardware or interpreter, because of abstraction boundaries. If we think we can rely on abstraction boundaries to keep a formal intelligence confined, then the problem is solved. But there is no very good reason to assume that, since real non-superintelligent black-hat hackers today specialize in violating abstraction boundaries with some success.

6 Life is not a game

One thing I learned by hanging out with these folks is that they are all fanatical gamers, and as such are attuned to winning strategies, that is, they want to understand the rules of the game and figure out some way to use them to triumph over all the other players. I used to be sort of like that myself, in my aspergerish youth, when I was the smartest guy around (that is, before I went to MIT and instantly became merely average). I remember playing board games with normal people and just creaming them, coldly and ruthlessly, because I could grasp the structure of the rules and wasn՚t distracted by the usual extra-game social interaction. Would defeating this person hurt them? Would I be better off letting them win so we could be friends? Such thoughts didn՚t even occur to me, until I looked back on my childhood from a few decades later. In other words, I was good at seeing and thinking about the formal rules of an imaginary closed system, not as much about the less-formalized rules of actual human interaction.

Anyway, the point is that I suspect these folks are probably roughly similar to my younger self and that their view of superintelligence in conditioned by this sort of activity. A superintelligence is something like the ultimate gamer, able to figures out how to manipulate “the rules” to “win”. And of course it is even less likely to care about the feelings of the other players in the game.

I can understand the attraction of the life-as-a-game viewpoint, whether conscious or unconscious. Life is not exactly a game with rules and winners, but it may be that it is more like a game than it is anything else; just as minds are not really computers but computers are the best model we have for them. Games are a very useful metaphor for existence, however, it՚s pretty important to realize the limits of your metaphor, to not take it literally. Real life is not played for points (or even for “utility”) and there are no winners and losers.

7 Summary

None of this amounts to a coherent argument against the idea of superintelligence. It՚s more of a catalog of attitudinal quibbles. I don՚t know the best path towards building AI (or ensuring the safety of AIs); I just have pretty strong intuitions that this isn՚t it.

Refactoring War

I seem to be obligated to have an opinion of some sort on the current fighting in Israel and Gaza. I am, after all a politically engaged and intellectual sort of person, or claim to be. All sorts of people I know are weighing in on one side or the other of the conflict. Some are quick to assign blame, others make heroic efforts to construct a balanced view where moral faults are parceled out to both sides in accordance with a detailed and sensitive knowledge of the history of the region. (Here՚s the best of those efforts I՚ve found so far, from none other than Amos Oz). I have family and co-workers in Israel, so am pulled in that direction, yet I am temperamentally and politically drawn to support the underdog, and that is not Israel in this fight. So I can՚t easily choose a single side for condemnation or support. But being balanced requires putting more time than I am willing to invest into learning all the agonizing details.

I could just shut up, of course, and mostly I have, because the situation seems to be definitionally hopeless. And my meta-heuristics say to stay away from hopeless topics, no matter how much they seem to want to pull me in. I՚m starting to see some merits in the LessWrongian slogan “politics is the mind-killer” – war and politics are after all basically two variants of the same thing, and while politics may kill the mind, war kills actual people as well. Why join in? If I thought there was some actual good to be done by expressing an opinion, that would be one thing, but the only benefit seems to be the very minor satisfactions of moral posturing, and the downside would be losing friends.

But silence is not really a viable option for me, for a multitude of reasons, social, moral, whatever. Doesn՚t matter – as Trotsky said, you may not be interested in war, but war is interested in you. So that means having to have an opinion, and that largely means figuring out how to assign blame. Isn՚t that what people really want to know when they raise this subject? They want to know which side to root for, as if it were a football game or pro wrestling or something.

Consider this post an effort to assign blame while avoiding picking a side. And there will be blame, someone or something has to answer. But using our patented refactoring technology may help us find different culprits than usual. And actually being unable to settle on a stable good guys vs bad guys story helps me out, in that it helps me to reflect in as abstract a way as I can manage on the nature of conflict in general. Abstraction is sort of what I do for a living; if there՚s any useful contribution I can make, it has to lie in that direction.

I came up with a refactoring of conflict a while back, a kind of childish and obvious idea really, but I keep it in my intellectual toolbox. Instead of seeing a conflict as between the two ostensible sides, view it as a battle between those who profit from war on both sides and those who are victimized. So in Vietnam the war was not between the US and the communists, but between the warriors on both sides, the military industrial complex in the US and the corresponding war machines of Russia, China, and their allies – and on the other, people trying to live their lives. Sometimes people trying to live are forced to enlist in this battle; hence the anti-war movement. Again, this isn՚t a particularly new idea – during the Vietnam era this was known as “the war at home” – but I rarely see it made explicit, and I haven՚t thought of as a refactoring until just now.

So instead of focusing on the ostensible conflict, focus on the internal conflict between warmakers and peacemakers. The dynamics become pretty visible in something like the Palestinian conflict, where both sides at one time contained a mix of hardliners and more reasonable people, but it was a lot easier for the hardliners to escalate the conflict than for the peacemakers to de-escalate it. Such escalation raises the relative status of the hardliners within their own side, so they have an interest in keeping the conflict going. As a result the Israeli peaceniks like Oz have had their power and stature diminished. In this other war, Hamas is Netanyahu՚s best ally and vice versa.

I՚m sorry, I՚m trying to keep this on as abstract a plane as possible, trying to suss out the utilitarian algebra that generates conflict in general, not this conflict in particular. I shouldn՚t even mention the actual warriors, I՚ll just get myself in trouble, even though I՚m very carefully avoiding even momentarily taking one side or another.

I am very partial to stories about heroic mutinies, like this one about how German workers ended WWI. And related stories that reveal the fractures within aggressive coalitions, like this one about what MPs are really for. It supports my refactoring story, obviously, and makes it possible to see the noble and peace-loving people being manipulated into conflict by their status-seeking superiors. I don՚t know how well this mythology can be applied to the Middle East, though; the very real ethnic hatred seems to be pervasive, not merely a creation of the violence entrepreneurs. Of course Israel is self-selected for Jews who want to turn ethnicity into political/military power – those are the ones who were drawn there (my uncle went there fleeing Nazi Europe; my mother and father turned west and went to England and the US). Palestinians too are probably self-selecting for collective belligerence – the ones who were individualistic and capable emigrated rather than join in ethnic warfare. Part of what makes this fight intractable is that it isn՚t all that refactorable. But people have tried.

A further refactoring occurs to me. In both the normal and refactored framing, we still tend to think of individuals being on one side or another. Jew or Palestinians, hawk or peacenik, it is a question of membership. But a more enlightened and even more refactored view is that everybody has a version of the war-making machinery in them, and peace-making as well, although either may be well-hidden. Then war is seen not as some external conspiracy of a few people against the many but an expression of tendencies we all have. Sometimes the machinery behind those tendencies simply gets the upper hand.

This is also not a terribly original idea. It is, after all, one of the bases for the nonviolent techniques of Gandhi and King, the idea that all humans have a conscience which can be reached.

King’s notion of nonviolence had six key principles. First, one can resist evil without resorting to violence. Second, nonviolence seeks to win the ‘‘friendship and understanding’’ of the opponent, not to humiliate him. Third, evil itself, not the people committing evil acts, should be opposed….

I think back to my childhood, where I was a very junior participant in the movement against the Vietnam war. These sappy posters were everywhere:

The sixties anti-war movement soon moved on from such sweet thoughts into more aggressive forms of opposition. Partly due to increasing pushback by the government and assassination of its prophets of nonviolence, but also because peace is too wimpy a cause to rally around. The only ones who can make war on war without becoming as bad as the thing they aim to defeat seem to be backed by a religious faith which I don՚t share. I could never really see myself as a flower-bearing peacenik, I՚m too contentious by nature, no saint. And more importantly, an approach to politics based on sainthood doesn՚t seem like it is workable, that it could scale.

On the other hand saints do appear on occasion. Somehow we normals have to figure out what to do in the meantime.

It is interesting that religion seems to be the ultimate glue holding coalitions together, whether they are sides in an ethnic war or a movement against war.

Buddhists seem to have their own refactoring of conflict, at least, they talk a lot about aggressive qualities of mind as a distinct thing which can be noticed and worked on and eliminated (or at least tamed to the point where it is non-destructive). Personally I am reluctant to give up my anger, it seems too fundamental to my being, to how I think. The world is full of things that deserve anger, should I let them all slide just for my own peace of mind? I would hate myself if I could no longer hate appropriately.

Still there is something to be said for getting aggressiveness under control, for learning to wield it as a weapon against targets that matter, including itself.

But vain the Sword & vain the Bow
They never can work Wars overthrow
The Hermits Prayer & the Widows tear
Alone can free the World from fear 

For a Tear is an Intellectual Thing
And a Sigh is the Sword of an Angel King
And the bitter groan of the Martyrs woe
Is an Arrow from the Almighties Bow

The hand of Vengeance found the Bed
To which the Purple Tyrant fled
The iron hand crushed the Tyrants head
And became a Tyrant in his stead 

— from “The Grey Monk”, William Blake

[This post owes something to a recent and widely read post on Slate Star Codex (my favorite blog right now) about how narrow interest-seeking on a large scale makes the world shitty. I՚ve been trying to work up a response; this is not that response but some influence has crept in. ]