An (Intuitive?) Introduction to Entropy

There’s a concept in Probability Theory called “entropy,” which borrows a bit from the physical concept of the same name. Intuitively, it’s the degree of “surprise” of an observation, or of a probability distribution. Alternatively, it’s how much information you gain when you observe something, as opposed to something else.

Let’s be more specific.

Suppose I have a biased coin that gives heads 80% of the time and tails 20% of the time. If I toss it and see tails I’ll be more surprised than if I see heads. If I toss it 5 times and see 4 tails and 1 head I’ll be much more surprised than if I see 4 heads and 1 tail. So it seems that whatever definition of entropy I use must reflect that; it should show me a higher number for more surprising observations, i.e. ones with a lower probability. We conclude then that this number is decreasing in the probability of the thing: lower probability means higher entropy.

So, for now, let’s call the entropy of an observation h(\cdot). Then the above condition says that P(a) < P(b) \leftrightarrow h(a) > h(b).

Back to our example. Suppose I observe tails twice. Should I be twice as surprised as if I had observed it once? That sounds reasonable to me, so for now we have that h(a,b) = h(a)+h(b). Since P(a,b)=P(a)P(b) (when a and b are independent), a general form of h(x)=-\log_2P(x) seems to do the trick just fine; the negative sign is to make it an increasing function on the probability, and the base of the logarithm will be explained in a bit (this is a retroactive pun and was completely unintended at the time of writing).

Now that we know how to calculate the entropy of a single observation, what’s the entropy of a distribution? That is, if I toss that coin a bunch of times, how surprised should I expect to be, on average? We can just calculate the expectation of that function \mathbb E[h]=\sum\limits_{x\in X}P(x)h(x)=-\sum\limits_{x\in X} P(x) \log_2 P(x) = H(X) (where X is the random variable that can take the relevant values). This is the average entropy of observations drawn from a distribution, usually just called the entropy of that distribution. In the case of our biased coin, then, we have:

H(\text{coin})=-0.8 \log_20.8 - 0.2 \log_20.2=0.722\text{ bits}

(Bits are short for “binary digits” and are the unit of \log_2 numbers. They’re also the names of those little 0s and 1s in a computer, and this is relevant, too. Now’s when the pun becomes relevant. Har-de-har-har.)

Continue reading

Posted in Mathematics, Probability Theory, Uncategorized | Tagged , , , , , , , | Leave a comment

Mathematical Hells

[Warning: Memetic hazard and philosophical trip. Also, probably incorrect. Talks about death and torture and robots.]

The universe is probably infinite, flat, uniform, and ergodic. This means not only that there are an infinity of copies of the Earth, all of them identical, all of them containing “a you” that’s reading this post written by “a me.” In fact, all possible distributions of matter happen somewhere an infinity of times.

The Many-Worlds Interpretation of Quantum Mechanics is probably correct. That means everything that “can happen” will, in “some universe.” This means all possible distributions of matter happen everywhere.

There’s a reasonable chance the process that spawned our universe was some form of Eternal or Chaotic Inflation, in which there’s a huge “field” whose local fluctuations spawn universes. In fact, due to the above, it’s likely that our universe repeats an infinite number of times.

It’s fun to think about the possibility that some strong form of Mathematical Platonism is true, and that the only thing that exists is mathematics, and our universe is no more than a mathematical structure. The implication of a myriad ways all distributions of matter can happen is left as an exercise to the reader.

(Why, yes, these are exactly Tegmark’s multiverses. They’re fun to think about.)

I listed the above possibilities in what I think is a decreasing order of probability. But suppose any one of them is true.

How do you know which “you” you are? If all things happen, why did no one ever see anything unusual? Why doesn’t Santa Claus suddenly materialise in the middle of Times Square, why does everything follow such neat and predictable laws as if there was only one way things could be?

In Quantum Mechanics, there is a thing called an “amplitude,” which is, well, a number. And the frequency with which we observe a certain outcome is proportional to the square of the amplitude of that outcome. Why? Beats me.

Some people call this probability a “measure.” That’s because there’s a part of maths called “measure theory” which deals with, amongst other things, comparing the “relative size” of infinities. So, for example, if there are an infinite number of “yous” that are in Earths and an infinite number of “yous” that are being run in a computer, a measure is something that compares “how many” there are of each of them relative to the other, even if there are an infinity of both.

I call it “magical reality fluid” because I have no idea how it works and it’s misleading to call it something that looks Serious when you don’t understand it. You might start believing you do.

Regardless, this would mean that, somehow, there’s more magical reality fluid in the “yous” that are in Earths identical to yours than in the “yous” that are Boltzmann brains and only exist for a femtosecond.

However, you have no way, even in principle of telling whether “you” are the you in this Earth or that Earth, or whether you’re being run on a computer, or what. In fact, there’s not even a question of the matter; you are all identical, the same person, so asking “which of them” is the real you is like asking whether this 3 or this 3 is the real 3.

However, there are ways of ruling out certain “classes of you” that you’re not. For example, there is probably a “you” that chose something other than you did for breakfast, or didn’t eat breakfast at all, or actually ate it when you didn’t. That you is certainly not you. I mean, it’s not you right now. They might’ve been you until yesterday, but now they’re not.

And likewise, if there is some computer that’s (and therefore an infinity of computers that are) running a you that’s identical to you up until 4AM tomorrow, when it will suddenly turn you into a talking duck with all your memories, then you won’t know until tomorrow at 4AM whether you’re one of those.

However, suppose that all the “yous” suddenly died right now, except for the yous that will turn into a duck tomorrow. Then you will never notice. Subjectively, nothing will change for you. Except now there’s a 100% probability (Well.) that you’ll turn into a duck tomorrow. There’ll be no “yous” that will experience anything different.

We have reason to suppose that the vast majority of your magical reality fluid is in Earths that were formed naturally and follow the inexorable emergent determinism of physics. In those Earths, there is very little room for variation, and most of it is likely in the form of quantum fluctuations that may not affect all that much – or maybe they do, who knows, maybe brains are quantum computers, or maybe quantum fluctuations are enough to make the uncertainty about which sperm fertilises which ovum significant. But in any case, whatever happens to one you in such an Earth probably happens to the vast majority of them. And so when you die in one Earth, you’ll die in most of them.

Subjectively speaking, though, you’ll never notice it. Because there is some magical reality fluid in versions of you that aren’t on Earths, but are nonetheless identical to you. They are you, for all intents and purposes. Except they didn’t die.

I mean, some of them did die. But after they do, subjectively, the only yous that remain are the ones that didn’t – and who, inevitably, broke the laws of physics to do so.  Past that point… all bets are off. We don’t have any way, even in principle, to predict what’s going to happen.

But worse than that is that most of these computer simulations won’t even be someone consciously simulating you. You’ll probably just be a byproduct of some other computation. You might be the consequence of the calculation of some function, or something. There’s no reason to expect, a priori, that your future computation will be benign – quite the opposite, since there are many more ways for a human to suffer and be disfigured beyond recognition than there are ways for them to thrive and have a reasonably satisfactory life.

Before you died, that didn’t really matter; the measure – pardon, magical reality fluid – of “yous” in those awful mathematical hells was so absurdly tiny compared to the fluid in deterministic Earths that the probability you’d face them was effectively zero. After all the deterministic Earths have gotten rid of you, however, you’re left with the effective zero fluid as your total fluid, and subjectively you’re… well, who the hell knows where you’ll be? Some mathematical hell, probably.

A benign superintelligence wishing to offset this risk would probably spend a reasonable amount of resources simulating what it believed were reasonable approximations of people who have died in order to try and make the most post-death magical reality fluid of them not be in a mathematical hell, but who knows if this will work, or be enough?

I have a friend who doesn’t want to be cryo-preserved because he’s afraid that humanity, in the future, has a reasonable chance of becoming horrible people who will torture past humans for fun (I want to link a certain recent SMBC comic here about robots that resurrect humans in order to inflict them the maximal amount of pain they can feel but can’t find it), and his probability for that is high enough that the negative effect offsets any of the positive effect of eternal life. The above argument says we’re screwed anyway, but he doesn’t believe it, though I have no idea why, since he’s already entertaining the idea of future humans torturing us for fun.

Ah, well. I guess it’s far-fetched anyway.

It does give the search for immortality a more urgent tone, however.

Posted in Mathematics, Philosophy | Tagged , , , , , , , , , | 5 Comments

A non-absurdity heuristic, and why I’m not a Singularitarian

So er…


Yeah. This is no longer a thing.

I wrote a post exactly six months ago explaining why I was a Singularitarian. Or, well, so I thought. Except then I thought about it long and hard. And I finished reading Bostrom’s book. And, well…

My core argument there, that there are many, many ways of getting to an AGI, is sound, I think. The prediction “AGI will be a thing” is disjunctive, and it’s probably correct. However, of the many forms AGI can take, software AI seems to be the murkiest, least well understood one. And… it’s the only one that really promises a “singularity” in the strictest of senses.

The argument, basically, is that a smarter-than-human software AI with access to its own source code and to our knowledge on how it was built would get into a huge feedback loop where it’d constantly improve itself and soar. And that’s a very intuitive argument. Humans are very likely not the “smartest” possible minds, and just eliminating all the cognitive biases and giving us faster processing power would probably be a huge step in the right direction.

But the absurdity heuristic has a converse: if we dismiss things that sound intuitively absurd before looking at their own merits, we accept intuitively plausible ideas too readily before criticising them. And I don’t think this heuristic should have a name, because, well, it’s probably not a single thing, it’s the set of all biases and heuristics, it’s just intuition itself, but my point here is that the argument… has a hard time surviving probing. It’s intuitive, we accept it readily, and we don’t question it enough. Or at least, I didn’t.

We don’t have a well-defined notion of what agency and intelligence are, we have absolutely no idea how to even begin building a software agent, and even if we did there is very, very little exploration on actual theoretical hard limits on improvement. Complexity theory, information theory, computability theory, all of those are highly necessary for us to even begin having a grasp on what’s possible and what’s not.

Which is not to say superintelligence won’t happen! In 300, maybe 200, maybe 100 years, it might be here. I don’t know. I can’t predict that. But right now, the Singularity is Pascal’s Mugging, or some other kind of mugging where the situation is so completely out of any reference classes we’ve known that even giving it a probability would be a farce.

And this is also not to say that research into AI safety isn’t necessary. What MIRI is doing, right now, is foundational research, it’s trying to create the field of AI safety as an actual field, with actual people doing research on it. And yes, it will probably include complexity, computability, information, logic, all of that. They’re starting with logic, because logic can prove things for us that are true everywhere, they’re a place to start. They’re working on decision theory, they’re working on value alignment. Those things are good and necessary, and I’m not going to discuss here what priority I personally believe should be given to each of those approaches or how effective MIRI is.

But I no longer think this is an urgent problem. I no longer believe this is something that needs doing immediately. I’ve unconvinced myself that this is a high-impact high-importance project, right now. I’ve unconvinced myself that… I should work on it.

So what, now? I spent the past five years of my life geared towards that goal, I have built a fairly large repertoire of knowledge that would help me there, I have specialised. My foundation is no longer there.

So I guess I’m going to try to use that, my skills and interests and capacity, to make an impact, somehow.

Posted in Philosophy, Rationality | Tagged , , , , , , , | 3 Comments

Truth, Probability, and Unachievable Consistency

What is truth?

Many an author has written lengthy philosophical treatises that begin with exactly this question, but, however shaky my identification with the group may be, as a rationalist my first and foremost answer to that question – or my first and foremost interpretation of that question – is a practical one. And to help with it, let’s first ask rather how we identify truth.

Whatever metaphysical definition you might be going with, whether you’re a Tegmark-Level-IV mathematical realist or some form of solipsist, if you ask someone the colour of the sky on a clear cloudless sunny day, they’d probably answer it’s “blue” – unless, of course, you started with a big preamble about “what truth is” or made the conversation seem any more than just a question, in which case they might go off on a philosophical tangent. But for the purposes of this post, let’s assume that’s not what’s happening, and they’ll just answer you it’s blue. If that’s a problem, maybe ask them another question, one that isn’t so obviously tied up with philosophical conundrums, such as “Who is the current President of the United States?” (it’s Barack Obama) or “What’s Brazil’s official language?” (it’s Portuguese).

Regardless of anything else, to a broad first approximation and to all intents and purposes that come up in daily life, we can say that the above answers are true. It’s true that, today, Obama is President of the United States, we speak Portuguese in Brazil, and the sky is blue on a clear sunny day. It’s true that if you walk off a cliff you will fall to your death. So far so good, I am fairly certain there is nothing controversial above these claims.

In practice, we recognise truth in a somewhat positive way. Which is not to take the extreme position that only empirical claims are “cognitively meaningful”; I’m a moral non-realist yet I see meaning in sentences such as “it is (ceteris paribus) wrong to murder people” even if there is no clear or direct empirical verification of the “wrongness” predicate (I mostly see predicates like “wrong” as two-subject predicates, one being the action itself and another being a given moral theory).

But in any case, as I was saying, in practice we use the concept of truth in a positive way. We propose truth based on evidence, and we defend truth based on expectation. A hypothesis is true if, of all mutually exclusive hypotheses, it leads us to expect reality, if it gives us predictions that turn out to be true, if it gives the most probability to what actually happened and will actually happen, if believing it’s true causes you to be less surprised about what you see than otherwise.

This may not be an immediately intuitive definition of “truth.” It’s almost certainly not the first thing most people think of, when they think of truth. But I think it sounds like a reasonable description. If, of literally all possible hypotheses, you have a given one that predicts your observations best, then you probably use that.

Except… not quite? Let’s talk probability (of course).

Continue reading

Posted in Mathematics, Probability Theory, Rationality | Tagged , , , , , , | 2 Comments

What’s logical coherence for anyway?

Time for a writeup! Or something.

So I’ve written before about Logical Uncertainty in a very vague way. And a few weeks ago I wrote about a specific problem of Logical Uncertainty which was presented in the MIRI workshop. I’m gonna reference definitions and results from that post here, too, though I’ll redefine:

Definition 1 (Coherence). A distribution \mathbb P over a set of sentences is coherent if:

  • Normalisation: \mathbb P(\top) = 1
  • Additivity: \forall \varphi,\psi:\mathbb P(\varphi) = \mathbb P(\varphi\land\psi)+\mathbb P(\varphi\land\neg\psi)
  • Non-negativity: \forall\varphi:\mathbb P(\varphi)\geq 0
  • Consistency: \forall\varphi,\psi:\varphi\equiv\psi\rightarrow\mathbb P(\varphi) = \mathbb P(\psi)

The \top symbol stands for a tautology (e.g. 0 = 0), and if \bot stands for a contradiction (e.g. 0 = 1) then normalisation and additivity imply \mathbb P(\bot) = 0, and the whole definition implies \forall\varphi:0\leq \mathbb P(\varphi) \leq 1.

TL;DR: A distribution \mathbb P is coherent if it’s an actual probability distribution and obeys the basic laws of probability (to perform inference you’d define \mathbb P(\varphi|\psi) = \frac{\mathbb P(\varphi\land\psi)}{\mathbb P(\psi)}).

Continue reading

Posted in Computer Science, Logic, Mathematics, Philosophy, Probability Theory | Tagged , , , , , , , , | Leave a comment

The Gaifman Condition and the Π1-Π2 problem

So I’m at a MIRI workshop on Logical Uncertainty, and I’m gonna make a more complete post about it later, but I wanted to talk about a thing that has been on my mind.

So we’re trying to build a probability distribution \mathbb P over logical sentences. Let’s get a few definitions out of the way:

Definition 1 (Coherence). \mathbb P is a coherent distribution if \mathbb P(\phi) = 1 for all proven \phi and \forall \phi,\psi. \mathbb P(\phi) = \mathbb P(\phi \land \psi) + \mathbb P(\phi \land \neg \psi).

Definition 2 (Computable approximability). \mathbb P is said to be computably approximable if there exists some computable function f(x,n) such that \lim\limits_{n\rightarrow +\infty}f('\phi', n) = \mathbb P(\phi).

That is, I can compute successive approximations to the probability of \mathbb P(\phi). This is an interesting definition because any coherent probability distribution is necessarily uncomputable.

Definition 3 (Arithmetical hierarchy). A predicate \phi is \Pi_0 and \Sigma_0 if it contains no quantifiers or only bounded quantifiers. It is \Pi_n (resp. \Sigma_n) if it’s of the form \forall x_0, x_1, x_2, ... \psi(x_0, x_1, x_2, ...) (resp. \exists x_0, x_1, x_2, ... \psi(x_0, x_1, x_2, ...)) where \psi is \Sigma_{n-1} (resp. \Pi_{n-1}).

So basically you should think of the position in the arithmetical hierarchy of formulae as alternating between existential and universal quantifiers n times.

Definition 4 (Gaifman Condition). \mathbb P is said to be Gaifman if \mathbb P(\phi) = 1 for all true \Pi_1 sentences \phi.

This is an interesting condition for a logical probability distribution to obey, but unfortunately…

Theorem 1. No coherent, computably approximable, Gaifman distribution \mathbb P gives nonzero probability to all true \Pi_2 sentences.

Or, in other words, if \mathbb P has the above properties, then there exists some true \Pi_2 sentence \phi with \mathbb P(\phi) = 0. To prove this, we’ll use the following lemma:

Lemma. If \mathbb P is coherent and Gaifman, then all false \Pi_2 sentences have probability 0.

Proof. Suppose \phi is a false \Pi_2 sentence. Then its negation is true, and it’s of the form \exists x_1, x_2, x_3... \psi(x_1, x_2, x_3, ...) where \psi is a \Pi_1 predicate. Take c = (c_1, c_2, c_3, ...) to be the value that makes \psi true. Then \psi( c) is a true \Pi_1 sentence, and by the Gaifman Condition has probability 1. But if that’s so, then that’s a proof that \phi is false, and coherence implies that the probability of this sentence must be 0. □

So now we’re ready to prove the theorem.

Proof of the Theorem. Assume that \mathbb P(\phi) > 0 for all true \Pi_2 sentences \phi. The lemma implies then that for all \Pi_2 sentences, \phi \leftrightarrow \mathbb P(\phi) > 0. However, computable approximability says that \mathbb P(\phi) > 0 \leftrightarrow \lim\limits_{n\rightarrow +\infty} f('\phi', n) >0, and this implies that \exists b,n_0. \forall n > n_0. f('\phi', n) > \frac 1 {2^b}. In other words:

\phi \leftrightarrow \exists b,n_0. \forall n > n_0. f('\phi', n)>\frac 1 {2^b}

Now, \phi is \Pi_2 and the formula on the right-hand side is \Sigma_2, and for all n > 0, not all \Pi_n formulae are equivalent to a \Sigma_n formula, therefore we got to a contradiction.

A clearer/more convincing proof is by diagonalisation. Define:

\phi : \leftrightarrow \forall b, n_0. \exists n > n_0. f('\phi', n) < \frac 1 {2^b}

The right-hand side is a \Pi_2 sentence, and it implies that \mathbb P(\phi) = 0 by computable approximability. Therefore, the above is equivalent to \phi \leftrightarrow \mathbb P(\phi) = 0. □

So we’re kinda screwed here. But then I thought of a weaker form of computable approximability:

Definition 5 (Computable \varepsilon-Approximability). \mathbb P is said to be computably \varepsilon-approximable if there exists some computable function f(x,n) such that |\lim\limits_{n\rightarrow +\infty}f('\phi', n) - \mathbb P(\phi)| < \varepsilon.

This definition seems to me to escape the two proofs of the \Pi_1-\Pi_2 problem.



raginrayguns talked about a sentence similar to the following and Benja Fallenstein proved the theorem:

Theorem 2. Let \mathbb P be coherent, computably \varepsilon-approximable, and Gaifman. Then there exist true \Pi_2 sentences that have probability less than 2\varepsilon.

Proof. We’ll build one such sentence:

\phi : \leftrightarrow \forall n_0.\exists n > n_0. f('\phi', n) < \varepsilon

If \phi is false, then \mathbb P(\phi) \geq \varepsilon, which by the lemma implies \phi is true.

If \phi is true, then \forall n_0. \exists n > n_0. f('\phi', n) < \varepsilon, which implies that \mathbb P(\phi) < 2\varepsilon. □

So we didn’t escape the problem after all, just put it in the bound.

Posted in Logic, Mathematics, Probability Theory | Tagged , , , , , , , | 9 Comments

The Metaphor of the Hills

Imagine that one day you’re standing on a very foggy field of grassy hills. It’s so foggy you can’t see even an inch in front of your nose, and what’s worse, you have no GPS or 4G reception. You decide to try to climb up one of those hills; maybe if you’re higher up you’ll be able to see something, or at least get some GPS signal.

Now, you can’t really see anything, so to start out you have to sort of fumble in the fog. You take a tentative step forward, and that doesn’t change anything. You take another, and you actually get to ground that’s a bit lower than where you were, so you take a step back. You try going left, and there the ground’s higher. On you go, fumbling around, trying to find somewhere with some at least minimal GPS reception.

Eventually you reach the top of one hill. There’s some GPS reception there; it’s not great, it’s in fact pretty crappy, but at least it’s there, unlike down where you started out. Except… what if there’s a higher hill somewhere nearby? You only found the one hill, and the only way to find another would be going back down again, and in this fog, there are no guarantees that you’ll even find your way back up the same hill you just climbed; you might even get lost and end up at a shorter hill, and that’d be pretty awful.

So, in sum, you’re faced with two choices: you can either continue exploring the landscape, or you can try to exploit what you have found. Maybe Google Maps will load in less than an hour, who knows?

You might be thinking now that this is a metaphor for something, and of course you’d be right. I mean, what would be the point of me even mentioning imaginary hill fields? How did you even get there in the first place? And where’d the fog come from? And what earthly carrier are you even using that can’t give you reception in the middle of an open field? Also there’s the title of the post which should be a dead giveaway. So, yeah, metaphor.

Continue reading

Posted in Computer Science, Intuitive Mathematics, Mathematics, Philosophy, Rationality | Tagged , , | 2 Comments

An anti-conjunction fallacy, and why I’m a Singularitarian

When anyone talks about the possibility or probability of the creation/existence of an UFAI, there are many failure modes into which lots of people fall. One of them is the logical fallacy of generalisation from fictional evidence, where people think up instances of AI in fiction and use that as an argument. Another is how the harder a problem is, the faster someone solves it, without spending even five minutes thinking about it. The absurdity heuristic makes an appearance, too.

But someone who’s familiar with LW or the whole cognitive biases shizzaz might be a bit cleverer and argue that most futurists get it wrong and predicting the future is actually really hard (conjunction fallacy). Ozy wrote a post about donating to MIRI in which zie points this out, but in the end mentions talking to, well, yours truly about it, and I think overall there are three points where I disagree with zir.

First, I propose the existence of a fallacy related to the conjunction fallacy and the sophisticated arguer effect, something I’ll call the Anti-Conjunction Fallacy, or perhaps the Disjunction Fallacy, or something. Maybe this is not a direct countercounterargument to Ozy’s point, but it’s a more general countercounterargument to the counterargument that “predicting AIs typically invokes a highly complex narrative with a high Complexity Penalty.”

The Conjunction Fallacy is a fancy name to the idea that sometimes people judge P(A\land B) > P(A), which is to say that a more complex proposition with more details seems to us more probable than a simpler one due to appealing to our sense of narrative. This is a fallacy because it’s a theorem of probability that the exact negation of that sentence is true, no matter what A and B are; that is, it is always the case that P(A\land B) \leq P(A). But conversely, we have that P(A\lor B)\geq P(A), that is, a disjunctive story is more likely than any of its components.

My proposed fallacy is this: many people (particularly rationalists) who see a long tale have an instinct to cry complexity penalty without actually checking whether the logical connective between the elements of that tale is a conjunction or a disjunction, AND or OR, and thus fall into the trap of saying that a disjunctive story has a low probability due to this instinct. And in my experience, most AGI predictions seem to be heavily disjunctive, in that the people making them (such as Nick Bostrom in his book) suggest a myriad possible disjunctive ways a superintelligence could arise, each of which relatively probable given current trends (e.g. whole brain emulations are an active research area which has seen actual results), so the posterior probability of the enterprise as a whole is much higher than that of each of those paths. This is true of many parts of the superintelligence narrative, from its formation to its takeoff to its potential powers. I don’t need five minutes to think of five different ways a superintelligence could reasonably take over the world and I’m not superintelligent.

So the moral of this part here is that, when you see a long prediction about something, first see whether it’s disjunctive or conjunctive before looking for fallacies. Isaac Asimov may have been wrong about the exact picture the future would paint, but by golly a large number of his individual predictions did in fact come true!

My second point is not so much an objection as a sort of reminder about what MIRI is actually doing. I’m not sure what its original goals were, but it most certainly isn’t trying, by itself, to program a superintelligence, at least not right now. Ozy says:

So it seems possible the solution is not independent funding, but getting the entire AGI community on board with Friendliness as a project. At that point, I can assume that they will deal with it and I can return to thinking of technology funding as a black box from which iPhones and God-AIs come out.

The thing is, that is one of MIRI’s explicit goals, outreach about AI dangers. And they seem to be at least mildly successful, or at any rate something was, given that Google created an AI Ethics board when it bought DeepMind, and given the growing number of prominent intellectuals that have been talking about the dangers of AI lately, some of which directly mentioning MIRI.

My third and final objection is that I think zie misunderstood me when I talked about the predictive skill of people who actually build technologies. I didn’t mean that they have some magical insider information or predictive superpowers that allow them to know these things; I meant that when you’re the one building a thing, what you’re doing isn’t predicting as much as it is setting goals. Predicting what Google is going to do is one thing, being inside Google actually doing the things is a whole ‘nother, and when AGI researchers talk about AGI there is frequently an undertone of “even if no one else is gonna do it, I am.” Someone who works at MIRI isn’t concerned so much with the prediction that a superintelligence is possible as they are with their own ability to bring it about, or raise the odds of a good outcome if/when it does.

My last point is something Ozy touched upon and on which I want to elaborate. Zie mentioned AGI is fundamentally different than other “large-scale” projects from before in that, unlike, say, nukes, the way it’s done will severely impact its outcome. As it is, I’d argue that almost no conclusions at all can be drawn from the past funding and development of technological advances because… the sample space is tiny. We can’t judge whether individuals funding research is an effective method of getting that research done because this idea, and the means to do so effectively, are brand new. During the 20th century, most technological advances happened due to the military, but that’s perfectly understandable given the climate: two full wars and a cold one spanning large powers, constant change in political and economic climates…

But large tech companies are a new invention, and it is my impression that, since at least mid-nineties, most of the technological advancements have had at least a hand of the private sector, and this seems to increasingly be the case. I’m not sceptical at all of the ability of individually funded technologies, especially software technologies, to play a large part in the future, because that’s what they’re doing right now, in the present.

But at any rate, there are a number of ways AGI could come about, and MIRI is trying to do what it can. So far, other than that, the FHI, and mmmmaaaaybe Google, it seems no one else is.

Posted in Rationality | Tagged , , , , , | 7 Comments

Alieving Rationality

Almost six years ago, Scott wrote a post on LW about the apparent lack of strong correlation of real-world instrumental success and studying what he calls “x-rationality” – that is, OB/LW-style rationality, of the kind that’s above and beyond the regular skills you can get from being a generally intelligent, thoughtful, and scientifically-minded person.

I’d be quite interested in hearing what his opinion is six years into the future, but my current one is that this situation hasn’t changed much, in general. In fact, I was linked to his post by a recent reply Ozy sent someone on zir blog, while commenting that zie didn’t spread LW memes because zie didn’t feel they were very useful. I’m not alone in this, then. (Let’s remember that CFAR exists now, though.)

I’d like to share my thoughts on another potential factor contributing to this case, something that was alluded to by the post and many commenters of it (including himself and Anna Salamon), something I’ve noticed that… I do. A skill, maybe.

Aspiring x-rationalists are the people who look at the mountain of stuff on Ovecoming Bias, Less Wrong, and other such sources, and decide that it makes sense, that their lives would be improved by the application of these techniques, so they go on and learn everything about it. They memorise it, they absorb all these memes to the point of being able to recite by heart many of the more famous quotes. And yet there isn’t a strong correlation! We’re not producing superheroes every other Tuesday! What gives?

I’d say it’s that believing rationality and alieving rationality are really different things.

Continue reading

Posted in Basic Rationality, Rationality | Tagged , | Leave a comment

On Arrogance

having or revealing an exaggerated sense of one’s own importance or abilities.

A friend of mine once mentioned on a comment written in response to some post or another in a facebook debate group that he had knowledge of maths far above the Brazilian average. That is a simple factual sentence, a true statement (which isn’t exactly surprising given what the Brazilian average actually is). The next few comments called him arrogant.

(ETA: This is an even better example of what I’m talking about here.)

I wonder what goes on in people’s heads when they say something like that. And by “wonder” I mean “sigh exasperatedly at the silliness of rules of etiquette.”

It’s clear, if you look at society and people in general, that people do not like feeling inferior. Not only that, people dislike feeling inferior so much that it’s become a generalised heuristic not to show superiority in any aspect. It’s rude to be seen as better than anyone at anything. It will give you trouble in most social circles. That can probably be easily explained: if you’re superior at something, everyone feels jealous, and stops helping you socially, so you end up being worse off than if you were just average.

It’s okay to want to be better than yourself. But being better than other people? You have to be more humble! How can you possibly think you could actually be better than other people?? That’s incredibly arrogant of you!

Yudkowsky makes a distinction between humility and social modesty: the latter is the kind of social thing you have to show, the “don’t-stick-out” heuristic; the former is actual, real, rational humility, the kind that recognises exactly how sure they are about the outcome of a decision and what steps must be taken in order to minimise the possibility of disaster.

So people calling you arrogant is frequently, in fact, a motte-and-bailey argument. The definition I presented at the top, of a false belief in one’s superiority (or even just a belief in one’s “general superiority” as if that existed), that’s the motte. The bailey is expressing superior aptitude at anything at all without paying your due to social modesty; it’s acknowledging your skills when they’re actually good. How dare you claim you’re better than anyone else? You’re just as flawed and imperfect as all of us! Even if you’re not. You have to pretend you are, just to not commit social suicide.

What I usually say is this: it’s not arrogance if it’s true.

Continue reading

Posted in Basic Rationality, Rationality | Tagged , , , , | Leave a comment