Now, there are about three major things wrong with this, and the discussion was started by someone writing a school paper on just what those problems were. I chimed in:
I am generally on the side of the critics of Singulitarianism, but now want to provide a bit of support to these so-called rationalists. At some very meta level, they have the right problem — how do we preserve human interests in a world of vast forces and systems that aren’t really all that interested in us? But they have chosen a fantasy version of the problem, when human interests are being fucked over by actual existing systems right now. All that brain-power is being wasted on silly hypotheticals, because those are fun to think about, whereas trying to fix industrial capitalism so it doesn’t wreck the human life-support system is hard, frustrating, and almost certainly doomed to failure.Corporations are driven by people — they aren’t completely autonomous agents. Yet if you shot the CEO of Exxon or any of the others, what effect would it have? Another person of much the same ilk would swiftly move into place, much as stepping on a few ants hardly effects an anthill at all. To the extent they don’t depend on individuals, they appear to have an agency of their own. And that agency is not a particularly human one — it is oriented around profit and growth, which may or may not be in line with human flourishing.
Corporations are at least somewhat constrained by the need to actually provide some service that is useful to people. Exxon provides energy, McDonald’s provides food, etc. The exception to this seems to be the financial industry. These institutions consume vast amounts of wealth and intelligence to essentially no human end. Of all human institutions, these seem the most parasitical and dangerous. Because of their ability to extract wealth, they are also siphoning off great amounts of human energy and intelligence — they have their own parallel universe of high-speed technology, for instance.
The financial system as a whole functions as a hostile AI. It has its own form of intelligence, it has interests that are distant or hostile to human goals. It is quite artificial, and quite intelligent in an alien sort of way. While it is not autonomous in the way we envision killer robots or Skynet, it is effectively independent of human control, which makes it just as dangerous.
[update 10/2014: Charlie Stross has roughly the same thought]