simont | Thoughts on thoughts (IV)

Thoughts on thoughts (IV)

Gosh; it's been a couple of years since I last made a post in this irregular series, which makes it quite irregular indeed.

I had coffee with feanelwa a couple of weeks ago, and we had a conversation in which it occurred to me that some kinds of programming, perhaps particularly at the level where you're only just getting the hang of it, are a fundamentally introspective process. If you want to program a computer to be able to do some task your own mind already knows how to do, one way to start working out how is to do it or imagine yourself doing it; then you watch your mind's process of thinking about it, closely enough to break it down into smaller steps. Then you write each of those steps in your program, perhaps by applying the same technique again. In other words, you're reverse-engineering algorithms and procedures out of your own subconscious: converting procedural knowledge into declarative. It had never occurred to me to think of it in those terms before, but I'm glad I did, because I've been strongly introspective from a pretty early age and now I feel as if I have a better explanation for why it comes naturally to me.

One obvious thing my brain does, of course, is to think, in the sense of self-aware world-aware general-purpose sentience. It is a source of occasional mild frustration to me that this process doesn't seem to be easily susceptible to the above technique of reverse engineering. It feels (in principle, before you find out how hard a problem AI really is) as if being a competent programmer with the ability to look hard at a sentient mind from the inside really ought to be sufficient to allow one to acquire an understanding of how to replicate the same processes in a computer.

(Yes, I know I've said before – in the previous post in this series and elsewhere – that I think constructing true machine sentiences would generally be a bad idea for practical and/or moral reasons. I haven't recanted that, really. But I'm not immune to the lure of curiosity; I want to know how to do it even if I think it probably shouldn't be done, and if I did know how then I would probably feel a strong temptation to try it just to find out whether it worked. So, probably best all round that I remain ignorant.)

Anyway. So recently I was imagining myself sitting down and writing a program-that-thinks, just to see what would be the first point where my imagination realised it wasn't sure what came next. It occurred to me that one of the very first big problems you run up against is the imprecise definition of almost all concepts used in normal human thought (or at least those outside pure maths). For almost any commonly used concept – ‘person’, ‘home’, ‘game’, ‘thought’, ‘happiness’, and so on – it's fairly easy to find borderline cases where there is disagreement or uncertainty about whether something qualifies as an instance of it.

Naturally, the fuzziness of such concepts makes life difficult for software written in the traditional precise and algorithmic sense. You can't typically apply standard propositional logic to a world of fuzzy concepts and expect to get reliable answers back out, because every step of your reasoning has some wiggle room and with a long enough chain of steps you eventually end up finding too many of them have wiggled in the same direction and the final link of the chain is pointing in precisely the opposite direction from the first. People have worked on various approaches to developing systems of computer inference which either try to avoid getting into this situation (e.g. ‘fuzzy logic’, or tagging every statement with a probability and using Bayesian methods to determine the certainty level of every deduction you make), or which try to avoid getting too confused once they are (e.g. by fiddling with the rules of propositional logic to prevent the system being able to deduce that even false things are true once it has a single contradiction). To some extent these have helped, but it's not very surprising that the most successful attempts to make computer programs compete with intelligent people have been from fields fairly close to pure mathematics, such as chess, where this sort of thing isn't much of a problem in the first place.

But what occurred to me, when trying to imagine sitting down and writing a thinking program from scratch, is that that human minds aren't merely capable of dealing with fuzzily specified concepts. Before we deal with them, we must first construct them.

In fact, in some sense every one of us has constructed in their own brain every fuzzy concept we ever think about. Even if in a given case that construction didn't occur ab initio but as a result of someone explaining the concept to us in words, those words are themselves imprecise; so in order to acquire anything one might call an understanding of the new concept, we have had to think about what the explanation actually means and what sorts of things it might or might not cover, and then we construct the concept in our head in basically the same way as we might have done without the explanation. All an explanation really does is to show us where to look for a new concept; we have to build our own actual understanding of it once we can see it.

This was important to me while trying to imagine how a program would think like a human, because I suddenly realised that jumping straight to an attempt to devise modes of reasoning which can operate effectively given some fuzzy concepts is putting the cart before the horse. The very first thing a program would have to do would be to actually construct its fuzzy concepts in the first place; reasoning with them comes later.

So how, and why, do we construct fuzzy concepts in the first place? I think we do it by looking at the world and seeing regularities in it. Initially, those regularities are in the slew of raw data coming in from our perception organs; so we quickly form a lot of fuzzy concepts for kinds of thing we commonly see – initially concepts like ‘round thing’ and ‘straight thing’, later on ‘person’ or ‘house’ or ‘car’. Later still we apply the same process more introspectively, which I'll come back to in a moment.

So I saw the process of concept-formation in my imagination as basically a statistical exercise. Imagine a sheet of paper with a lot of dots drawn on it. Suppose the dots aren't evenly distributed, so there are irregularities of various kinds. A human would have no difficulty in pointing and saying ‘look, there's a particularly intense cluster of them’ or ‘there's a mostly empty region’ or ‘hey, there's a set of dots all in a line’. This, I suddenly felt, is basically analogous to the process of concept-forming: a human brain is given a large amount of perceptual or introspective input, and its primary job is to find statistical, approximate regularities and patterns in it. Then it reapplies those regularities to form expectations and opinions about things beyond its immediate perception.

The interesting part is that this process isn't only applied to the data coming in from perception. By watching the inside of our own brain as concepts move past each other, one might observe a general regularity in the patterns in which they recur, leading one to derive things such as rules of reasoning – deduction, inference, analogy – as fuzzy concepts in their own right. In other words, we don't have to be shown how to reason or have to have it hard-wired into the brain at source; we intuitively develop our own ideas of how to think, simply by noticing that certain things seem to work in many cases and forming a pattern, then later on arranging other concepts deliberately into the same pattern and seeing where the conclusion of the pattern leads us.

It's only later on that we go to school and learn things like the abstraction of those ideas into formal propositional logic. (Which, incidentally, was invented by people who had the introspective ability to look at those intuitive fuzzy reasoning rules in their own heads and reverse-engineer out the most important points, in the manner I described above referring to programming.) Also at school we tend to have to have it beaten into our heads that not all of these modes of reasoning are as reliable as each other, and that (for example) reasoning by analogy is only a rough guide to what might be true and usually needs to be supported by a more rigorous argument if you want to conclude that something you've thought up by analogy actually is true. But in the absence of school, we'd learn this stuff anyway by hard knocks if we had to; indeed, before schools were ever thought of, we did have to.

So this view of sentience is one in which the fundamental act of intelligence is this statistical and approximate pattern-recognition process which takes vague and imprecise irregularities in data and forms concepts describing them. If you can do that, and do it well, nearly everything else a human brain does follows naturally from it – and you don't need to invent fuzzy logic to deal with the concepts, because the concept-forming framework itself will invent its own forms of logic and self-correct when they aren't quite right.

Of course, I'm well aware that all of this is no more likely to be true than any other crackpot theory about AI dreamed up by someone out of nothing but introspection. And I'm sure that if I did sit down and try to write an intelligent program on this basis (which I couldn't even if I wanted to, because statistics isn't really my thing – I identify a need for it above, but doing it is not my forte), it would turn out to be a lot more complicated than I've just made it sound; there would be no end of weird emergent problems owing to the concept-former having some undesirable property, and as soon as you started tweaking the rules to eliminate those problems you'd find you had wallpaper-bubble syndrome with a vengeance. So I'm not under the impression that I've just revolutionised AI; but I do think that I've at the very least given myself an interesting new way to think about thinking and to perceive what happens in my own brain.

It's also a little depressing, in a way. Suppose it's all true: suppose that the fundamental function of sentience on which all else is based is this concept-forming process, and that the concepts thereby formed are always fuzzy and imprecise, and that it's vital to the whole thing working that they should be so. Then we deduce … that the only thing enabling human beings to function at all is that they are inaccurate. The only thing which enables us to ever get anything even vaguely right is our ability to get things wrong, or at best only vaguely right.

A lot of hard SF which talks about artificial intelligences tends to see them as either inherently, or at least capable of being made, better, faster and above all more precise than human minds. Suddenly I feel that this is a contradiction in terms: a more precise mind would probably fail to be a mind at all. It is our imprecision which enables us to function in the first place, and despite its perceptible downsides, one cannot construct an AI which doesn't suffer from them because its upsides are all-pervasive and indispensable. Intelligence is a huge 90% solution, a great con trick, a hideous hack, and it's a miracle it works at all let alone as well as it does.

Flat | Top-Level Comments Only

I don't know, I think that "the only thing which enables us to ever get anything even vaguely right is our ability to get things wrong" is very accurate and reassuring. I was thinking this morning* about something I read about middle class teenage girls' abject terror of failure and how it messes them up for the rest of their lives and their children's lives. And, perhaps that's another secret trick of the sexism meme, instilling excessive perfectionism into females with the otherwise most hopeful background, so that their natural state is to sit there unable to ever achieve something real. Maybe raising people to believe failure is the worst possible thing is a secret way of removing their intelligence before it has a chance to develop.

*If I could get up in the morning without thinking so much I would be late a lot less of the time

Introspecting myself, I think there's great scope for an AI made on this model to be able to think arbitrarily more precisely than I do, because I am limited by headspace constraints and this limits the precision of concepts that I can maintain. Precision happens by nesting concepts, and it doesn't follow from the outer concepts being fuzzy that the inner concepts are as fuzzy. I think if you had a mind built on hardware that expanded over time, rather than starting to contract again after twenty years, you might be able to start with this and end up somewhere more precise.

Possibly by modularising? It seems a great constraint that all one's specialist knowledge has to fit inside the same head. If your AI could over time get access to new sub-systems to populate with specialist knowledge it could maintain more attention on an overview. Internalise organisal structure?

Also, I think there's scope for:

* standard human brain, but make it run much faster
* standard human brain, but with connection to more traditional "computer" -- instant checking of facts, which ought to make it easier for the AI to spot where its fuzziness is causing problems
* make the AI less impatient; the human brain has evolved to take lots of shortcuts and not necessarily bother thinking things through, because for hard real-time tasks like "avoid getting eaten by lion" you can't afford to take the time to do that. You could make the AI more contemplative.

Yes. You could dispense with having to direct the majority of your thoughts towards keeping fed, housed and warm. Although then you might end up with motivation problems and it devoting all its energies to developing innovative AI porn.

If you make it less impatient it won't check the facts instantly :)

There's a lot of experience from practical AI (such as machine translation) that the more data you can bring to bear, the more accurate you can be. So I think Simon's conclusion in the last paragraph is wrong: AI can be better if it can be bigger than the brain, even if the basic implementation technique are the same.

I'm not convinced that it's actually possible for a human brain to introspect usefully about how it's sentient. It's like being sat inside a virtual machine -- you can look at how processes also inside the VM work, but you don't necessarily have any access to the real hardware that the VM sits on top of.

True, but in the analogous situation with real VMs you wouldn't actually have to. You'd be able to observe the behaviour implemented by the VM's instruction set, and then you'd know everything you needed in order to write a VM implementation of your own which was indistinguishable from the original one to processes running inside it.

The trouble with brains is that even working out what the VM's analogue of instructions is is a total pain, because it's all so continuous and vague.

eta: In fact it seems more analogous to the problem of determining the underlying rules of physics than to VMs. You can clearly see various kinds of macroscopic behaviour, but there are dark corners where it's less obvious what's going on, and working out the underlying rule set of which all of that behaviour is emergent consequences is generally a hard problem.

Edited 2008-04-13 11:55 (UTC)

You'd be able to observe the behaviour implemented by the VM's instruction set, and then you'd know everything you needed in order to write a VM implementation of your own

I don't think you can (necessarily) observe the implemented behaviour (in sufficient detail) as a program running *inside the VM*, for your average (real, computer) VM. Just to pick something at random, the VM might not have any facilities for letting you load new code into it. And even if you can determine the behaviour exactly, knowing that "executing instruction X causes effect Y" doesn't mean you know how to actually implement effect Y in your own VM.

(I'm reminded of Greg Egan's _Permutation City_, which includes some entities who think they're determining the underlying rules of physics but are actually dealing with the behaviour of the VM they live in.)

Oh, and suppose that what you were interested in was how to go from an inert pile of bytecode to a running thread of execution. If the VM just has some kind of thread.start primitive, then being inside the VM is really of little use for identifying how this might work. In a similar way, my guess is that the stuff which actually makes us conscious is more like a primitive provided by the underlying mess of neurons; from our perspective it just works and we have no insight into its workings.

I agree almost entirely.

I'll add though, that your idea of "concept-forming" is of course a concept, and hence is somewhat of an abstraction of what actually happens.

I think that thought's best modeled by understanding the idea of spreading activation and reinforcement in the brain; when the chemical receptors for a particular smell always get triggered at the same time as certain patterns of visual and taste input, and are then followed by a kinesthetic perception of feeling full, then if you repeat that pattern enough they all become linked, and seeing one is enough to spread activation to the other areas - hence the famous Pavlov experiment. I'd suggest it's this which implements your observed "concept-forming".

The other important component is an understanding of the various bootstraps which get the whole thing going and also curb those nasty emergent phenomena you hypothesised. Things like "if it could be a pattern or could not be a pattern, model it as a pattern and proceed according to that model", or particular receptivity to certain kinds of pattern learning, e.g. linguistic learning in the young.

So, rather than identifying "concept-forming" as the primary mechanism for thought, I think I'd identify "memory" - encoded as associations between various brain-parts which cause more sympathetic spreading activation between those parts. But, I think these are kinda the same thing. :)

I think the process of designing software according to "how would I solve this problem?" is very sound. That's the way we work with our character recognition software. Whenever a new problem comes up, we ask the guys who commonly key that kind of document, "hey, how do you know what to key in, or where the important data is?". Then, I try and codify that for our hardcore algorithm guys to chew on and implement. :)

Not sure if I'm following...

I'm envisaging "an AI" as a program that is being fed a sequence of inputs (visual inputs from the world around it, say, or text inputs from someone its interacting with), and comes up with a sequence of outputs (speech, actions, ... ).

Gathering the input/output sequence, it seeks patterns, perhaps particularly patterns with arbitrary time-delays (cluster X of inputs often leads to cluster Y of inputs with some delay).

How do you give it a (or bootstrap its) decision-making process? Why should it choose one output over another? Or rather, what kind of goals should it have?

You could simply feed it data, ask it to extract patterns, and then give it some short-term goal, I guess. I'm not sure how much freedom that gives it for exploring nicely.

Or I may be missing your point completely ... ?

(The beginning of this paper, for example, sounds a little like you're describing:
http://citeseer.ist.psu.edu/296596.html as far as learning concepts is concerned. (I haven't read the paper)).

I'm taking an AI course at the OU at the moment... and, yes, one of the things I've been learning about is neural networks that are trained to classify things that are presented to them. And there's two ways of training - one has the "answers" with the training questions (like a parent saying "no, dear, the plant is not your dinner") and the other way doesn't and just leaves the AI to find similarities.

These sorts of things turn out to be quite good at, for example, picking out the concept of "noun" or "animate thing" when presented with suitable training data.

What I find hard when working with computers (and apparently people working in AI find hard too) is representing visual input in a "fuzzy" way, indeed I find doing any visual work on a computer without using a GUI to manipulate it very difficult to get from in-my-head to in-the-computer (I've been programming a simple GUI without a GUI GUI-editor for work and it HURTS MY BRAIN); and computers are bad at "recognise a handwritten letter" - although they are getting better.

I think most of how humans learn these things is being told off for doing it wrong and rewarded for doing it right - either by other people or in more direct ways (like getting ill from eating the wrong things) so we make fuzzy categories that are very similar to those that our parents and friends have - and then we get confused when our fuzzy categories turn out to be very unlike other people's when we argue with people on the internet (for example).

I think the difficulty with AI is not to give it a big database but rather to learn how to write code that allows the AI to *learn* effectively from sets of training data (or from interacting with trainers). We're pretty rubbish at understanding how humans learn; and I'm not sure that knowledge would transfer directly to computers anyway (because the storage medium is very different), but learning definitely seems to be key.

Humans find it really hard dealing with fuzzy concepts. Just because we can make a firm decision based on partial information doesn't mean it's sound. See, for example, "framing" in political discussion, "priming", confidence tricks, psychology of gambling, etc.

The interesting thing to build in an AI is not reasoning but motivation.

This reminds me, I have an LJ entry in my head about "statistical morality" and why we can't cope with it; that is, dealing with actions that have a very slight negative effect on a very large number of people, or actions that very slightly increase risks.

I don't think it's fuzziness that your (fundamentally impossible) 'programmed intelligence' is missing: it's intentionality.

S.

Flat | Top-Level Comments Only

Thoughts on thoughts (IV)

Almost unrelated tangent, sorry

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject

no subject