While I'm musing about programming and programmers, there's another thing I've been wondering over the past few years, which is how the next generation of programmers are going to learn.

A modern computer system (let's say Windows, for the sake of argument) is a huge collection of layers of abstraction piled on top of one another in an untidy heap, going all the way from the top-level GUI shell to the machine code level (or even further down if you're a hardware person). In between the GUI and the bare metal, we take in GUI-embedded languages such as Visual Basic; more general-purpose things like Java and C#; virtual machine layers abstracting those away from the real CPU; the C/C++ level below that, and the machine code to which C and C++ compile. At the C/C++ level your interface to the operating system is mediated through multiple layers of libraries: MFC and its ilk, the ANSI C standard library functions, and below both of those the ‘real’ Windows API on which both of those are based and which you can access directly if you prefer to know exactly what you're doing. Underneath that, there's the understanding that your processes are running in virtual address spaces implemented partly by real RAM and partly by swapping to disk; there's the understanding that your processes are isolated from directly accessing real hardware but instead have to ask the OS nicely to do things for them; there's the OS kernel, hardware abstraction layers and device drivers within that, filesystems, and eventually you come up against real machine code which is talking directly to hardware I/O registers.

And as Joel Spolsky points out, all abstractions leak; so if you're working in the layer at the top, you're inevitably going to have to notice the tower of layers below you at some point, when it sways gently in the breeze and you lose your balance. I have friends who occasionally come to me for Perl advice, for example, and every so often Perl does something weird which I can't sensibly explain without talking about what's really happening at the Unix system-call level; this generally confuses them because they were hoping to be able to just learn Perl, but you can't get real work done in Perl without understanding what's below it. And that's a much shorter tower than the modern Windows one, which involves more levels of libraries and virtual machines.

So in order to use such a machine really competently, you need a decent understanding of every level of the machine, from the GUI down to the bare metal. I'm not asking for godlike systems-programming skills here including the ability to sit down and rewrite any given layer on demand; but I do think it's important to understand what all the layers are up to well enough to predict or explain their effects on whatever I might happen to be trying to use the machine for at any given moment. Things like, for example, the effects of virtual memory on certain types of data-processing application (namely the fact that they'll slow down unbearably as soon as the working set gets bigger than available real RAM), and the ability to predict reasonably accurately what sorts of application will be thus affected (which comes from knowing what those applications will be having to do at the C level, which might not have been completely obvious to someone writing such an application using things like pre-written image processing libraries which hide all the complexity from you, or high-level languages which let you write a string concatenation operator without bothering to tell you you're potentially moving 100Mb of data bodily around with that one punctuation symbol).

So how would somebody, faced with the staggering tower of leaky abstractions that is a modern (say) Windows box, go from being a total novice to being the kind of person who has that conceptual overview of the entire system plus a decent amount of detail at many of its levels? Because it occurred to me recently that neither I, nor any of the other people I know who do have that kind of understanding, acquired it by playing with a modern Windows box.

I got a lot of my early programming experience on much simpler computers like the Spectrum and the BBC, which simply didn't have the huge number of layers that a Windows machine has. On a BBC, you had a BASIC interpreter, and then you had raw machine code plus some ROM routines to help out with your interaction with the hardware. On that sort of platform, it was feasible to gain a comprehensive overview of the entire system in not very many steps: program in BASIC for a while, then take a few steps into the machine code world, then think for a bit about what sorts of operation are easier in BASIC and try to work out how you'd do them in machine code. After that you were pretty much there: you could work at the lowest level of the system if you needed to, and you had a clear idea of what useful things BASIC was adding and what it was taking away, and thus you could judge when it was appropriate to use that level of the system instead.

After I understood the BBC, it was only a short step to MS-DOS on the PC. DOS is not that much different from a BBC, really; you have a bunch of hardware which you can access directly with IN and OUT machine instructions if you want to, but it's usually easier to get the ROM routines (the BIOS) to do jobs like interrupt-driven timing for you. DOS adds a secondary layer of service routines to handle file systems and a few other administrative tasks, which is roughly equivalent to the BBC's ADFS or NFS ROM forming a layer on top of the basic MOS. Beyond that, you're on your own; you can write machine code directly, or you can appeal to a language compiler or interpreter to help you with the tedious bits. Not all that different at all, really; each level is a bit more involved and confusing (due partly to ‘it just grew’ design and partly to the machine being bigger) but the levels are much the same in nature.

The Amiga taught me about multitasking operating systems without going straight to virtual memory and inter-process security protections; the idea that every so often a timer interrupt went off and the operating system would swap out all the machine registers and go and execute a different process for a while was interesting and new, but simple enough to describe in one sentence, and I had a few years to explore the consequences of it. Then properly virtualised operating systems such as Unix and Win95 started to show up in my life, and once again there were only one or two new concepts to cope with at a time. (The biggest culture shock I had when first encountering Unix, actually, was that it was multitasking but not GUI-based. It hadn't previously occurred to me that the two could be separate, but as soon as I saw it I had one of those moments of revelation and went ‘oh yeah, they're actually completely independent aren't they’.) And a few years later people started to invent things like MFC, and then things like Java, and things like Visual Basic, and in each case I was able to judge the idea based on a clear understanding of how computers worked in its absence, and then decide what it added and what it took away and how it would interact with everything else and when it was or was not appropriate to use it for something.

So, in large part, I learned about all these layers by starting at the bottom and working up. This works well, because layers above you don't generally affect you, whereas all the layers below you do; so by starting at or near the bottom I was able to very quickly reach the point where I understood all the layers below me and wasn't getting any unpleasant surprises.

But it seems to me that someone sitting down at a modern Windows box and deciding to try to learn how to program it is inevitably going to have to start at the very top, which means that their initial experience of trying to understand the system they're working on is going to be chock full of unpleasant surprises caused by lower layers they haven't even begun to understand doing things they would never have been able to predict. Working down from Visual Basic to bare-metal systems programming is surely going to be a years-long process of repeatedly throwing away everything you thought you understood to make way for a completely different understanding of the world which you will eventually come to see that your previous understanding vaguely approximated one small corner of. Much like learning chemistry at school, as I recall, with its endless series of layers of Lies-To-Children unfolding as you progress through the mutually contradictory syllabuses.

Well, some people do persevere through the various chemistry syllabuses and emerge at the far end of a university course with something which must by that time surely at least resemble a proper understanding of chemistry. So I suppose, by analogy, there might be people able to start from the very top of a Windows system and end up with a top-to-bottom comprehension of its entire software architecture; but I cannot believe that it would be as easy, or nearly as much fun, to learn that way as it was for me to learn from the bottom up. Knowing that you understand what's going on, and really understand it rather than working with a novice's approximation, is a good and motivating feeling, and a top-down student must surely continue to lack that until they finally hit bottom, whereas I had it for most of the way up.

(Meanwhile, Windows has the additional fun property that things you don't know about yet can seriously bite you. If you start by operating at the Windows GUI level and don't really understand the difference between programs and data yet, how are you possibly to know which e-mail attachments contain viruses which will take over your computer, and which are harmless data which can at worst show you offensive images or advertise useless products at you?)

So if someone were to hand me the kind of intelligent five-year-old that I once was, and explain that this kid would be in my charge for twenty years and could I please make a serious programmer out of them, I currently think that I would not start by sitting them down at a Windows machine at all. I'd probably start by digging out some kind of older machine – something at most as complicated as MS-DOS, although I might even go back as far as the BBC – and have them learn from the bottom up, as I did. Not from right at the bottom, of course; going straight into machine code and direct I/O port access would probably be excessive. But I'd start them with some kind of lowish-level language, perhaps C or perhaps Borland Pascal (though I'd certainly take the opportunity not to pollute them with BASIC!), and then I'd extend their education down through DOS and BIOS support functions to I/O ports and machine code, until they had an understanding of everything from Pascal down to the bare metal; and then, just as happened to me, I'd start piling additional layers on the top. Because I know that approach worked for me and for (as far as I'm aware) all the serious programmers I know; and I currently have no evidence at all, other than a tenuous analogy with school chemistry lessons, to make me believe that the top-down approach is a sensibly feasible way to create serious programmers.

Unfortunately, in the modern world the bottom-up approach takes a long time before it gets you any useful results. Knowing about I/O ports is really not going to be helpful when no machine except your learning-exercises box lets you access them directly because there's an OS in the way! In my day, bottom-up learning meant I could get useful stuff done on all the kinds of computer I gradually moved through. Do it today and you'd be looking at ten years before you had any directly usable skills, so you'd have to be very sure you wanted to be a programmer to go through all that.

So, can anyone reassure me by arguing that the top-down approach can work sensibly? Or suggest other plausible ways to learn all this stuff? I'm getting old enough now that there might actually be programmers in the younger generation who didn't start in the Spectrum/BBC era and didn't have the opportunity to learn from the bottom up to the extent I did; if any of those people are reading, I'd be interested if they'd share their experiences.