Musings on programming (II)

[simont]

While I'm musing about programming and programmers, there's another thing I've been wondering over the past few years, which is how the next generation of programmers are going to learn.

A modern computer system (let's say Windows, for the sake of argument) is a huge collection of layers of abstraction piled on top of one another in an untidy heap, going all the way from the top-level GUI shell to the machine code level (or even further down if you're a hardware person). In between the GUI and the bare metal, we take in GUI-embedded languages such as Visual Basic; more general-purpose things like Java and C#; virtual machine layers abstracting those away from the real CPU; the C/C++ level below that, and the machine code to which C and C++ compile. At the C/C++ level your interface to the operating system is mediated through multiple layers of libraries: MFC and its ilk, the ANSI C standard library functions, and below both of those the ‘real’ Windows API on which both of those are based and which you can access directly if you prefer to know exactly what you're doing. Underneath that, there's the understanding that your processes are running in virtual address spaces implemented partly by real RAM and partly by swapping to disk; there's the understanding that your processes are isolated from directly accessing real hardware but instead have to ask the OS nicely to do things for them; there's the OS kernel, hardware abstraction layers and device drivers within that, filesystems, and eventually you come up against real machine code which is talking directly to hardware I/O registers.

And as Joel Spolsky points out, all abstractions leak; so if you're working in the layer at the top, you're inevitably going to have to notice the tower of layers below you at some point, when it sways gently in the breeze and you lose your balance. I have friends who occasionally come to me for Perl advice, for example, and every so often Perl does something weird which I can't sensibly explain without talking about what's really happening at the Unix system-call level; this generally confuses them because they were hoping to be able to just learn Perl, but you can't get real work done in Perl without understanding what's below it. And that's a much shorter tower than the modern Windows one, which involves more levels of libraries and virtual machines.

So in order to use such a machine really competently, you need a decent understanding of every level of the machine, from the GUI down to the bare metal. I'm not asking for godlike systems-programming skills here including the ability to sit down and rewrite any given layer on demand; but I do think it's important to understand what all the layers are up to well enough to predict or explain their effects on whatever I might happen to be trying to use the machine for at any given moment. Things like, for example, the effects of virtual memory on certain types of data-processing application (namely the fact that they'll slow down unbearably as soon as the working set gets bigger than available real RAM), and the ability to predict reasonably accurately what sorts of application will be thus affected (which comes from knowing what those applications will be having to do at the C level, which might not have been completely obvious to someone writing such an application using things like pre-written image processing libraries which hide all the complexity from you, or high-level languages which let you write a string concatenation operator without bothering to tell you you're potentially moving 100Mb of data bodily around with that one punctuation symbol).

So how would somebody, faced with the staggering tower of leaky abstractions that is a modern (say) Windows box, go from being a total novice to being the kind of person who has that conceptual overview of the entire system plus a decent amount of detail at many of its levels? Because it occurred to me recently that neither I, nor any of the other people I know who do have that kind of understanding, acquired it by playing with a modern Windows box.

I got a lot of my early programming experience on much simpler computers like the Spectrum and the BBC, which simply didn't have the huge number of layers that a Windows machine has. On a BBC, you had a BASIC interpreter, and then you had raw machine code plus some ROM routines to help out with your interaction with the hardware. On that sort of platform, it was feasible to gain a comprehensive overview of the entire system in not very many steps: program in BASIC for a while, then take a few steps into the machine code world, then think for a bit about what sorts of operation are easier in BASIC and try to work out how you'd do them in machine code. After that you were pretty much there: you could work at the lowest level of the system if you needed to, and you had a clear idea of what useful things BASIC was adding and what it was taking away, and thus you could judge when it was appropriate to use that level of the system instead.

After I understood the BBC, it was only a short step to MS-DOS on the PC. DOS is not that much different from a BBC, really; you have a bunch of hardware which you can access directly with IN and OUT machine instructions if you want to, but it's usually easier to get the ROM routines (the BIOS) to do jobs like interrupt-driven timing for you. DOS adds a secondary layer of service routines to handle file systems and a few other administrative tasks, which is roughly equivalent to the BBC's ADFS or NFS ROM forming a layer on top of the basic MOS. Beyond that, you're on your own; you can write machine code directly, or you can appeal to a language compiler or interpreter to help you with the tedious bits. Not all that different at all, really; each level is a bit more involved and confusing (due partly to ‘it just grew’ design and partly to the machine being bigger) but the levels are much the same in nature.

The Amiga taught me about multitasking operating systems without going straight to virtual memory and inter-process security protections; the idea that every so often a timer interrupt went off and the operating system would swap out all the machine registers and go and execute a different process for a while was interesting and new, but simple enough to describe in one sentence, and I had a few years to explore the consequences of it. Then properly virtualised operating systems such as Unix and Win95 started to show up in my life, and once again there were only one or two new concepts to cope with at a time. (The biggest culture shock I had when first encountering Unix, actually, was that it was multitasking but not GUI-based. It hadn't previously occurred to me that the two could be separate, but as soon as I saw it I had one of those moments of revelation and went ‘oh yeah, they're actually completely independent aren't they’.) And a few years later people started to invent things like MFC, and then things like Java, and things like Visual Basic, and in each case I was able to judge the idea based on a clear understanding of how computers worked in its absence, and then decide what it added and what it took away and how it would interact with everything else and when it was or was not appropriate to use it for something.

So, in large part, I learned about all these layers by starting at the bottom and working up. This works well, because layers above you don't generally affect you, whereas all the layers below you do; so by starting at or near the bottom I was able to very quickly reach the point where I understood all the layers below me and wasn't getting any unpleasant surprises.

But it seems to me that someone sitting down at a modern Windows box and deciding to try to learn how to program it is inevitably going to have to start at the very top, which means that their initial experience of trying to understand the system they're working on is going to be chock full of unpleasant surprises caused by lower layers they haven't even begun to understand doing things they would never have been able to predict. Working down from Visual Basic to bare-metal systems programming is surely going to be a years-long process of repeatedly throwing away everything you thought you understood to make way for a completely different understanding of the world which you will eventually come to see that your previous understanding vaguely approximated one small corner of. Much like learning chemistry at school, as I recall, with its endless series of layers of Lies-To-Children unfolding as you progress through the mutually contradictory syllabuses.

Well, some people do persevere through the various chemistry syllabuses and emerge at the far end of a university course with something which must by that time surely at least resemble a proper understanding of chemistry. So I suppose, by analogy, there might be people able to start from the very top of a Windows system and end up with a top-to-bottom comprehension of its entire software architecture; but I cannot believe that it would be as easy, or nearly as much fun, to learn that way as it was for me to learn from the bottom up. Knowing that you understand what's going on, and really understand it rather than working with a novice's approximation, is a good and motivating feeling, and a top-down student must surely continue to lack that until they finally hit bottom, whereas I had it for most of the way up.

(Meanwhile, Windows has the additional fun property that things you don't know about yet can seriously bite you. If you start by operating at the Windows GUI level and don't really understand the difference between programs and data yet, how are you possibly to know which e-mail attachments contain viruses which will take over your computer, and which are harmless data which can at worst show you offensive images or advertise useless products at you?)

So if someone were to hand me the kind of intelligent five-year-old that I once was, and explain that this kid would be in my charge for twenty years and could I please make a serious programmer out of them, I currently think that I would not start by sitting them down at a Windows machine at all. I'd probably start by digging out some kind of older machine – something at most as complicated as MS-DOS, although I might even go back as far as the BBC – and have them learn from the bottom up, as I did. Not from right at the bottom, of course; going straight into machine code and direct I/O port access would probably be excessive. But I'd start them with some kind of lowish-level language, perhaps C or perhaps Borland Pascal (though I'd certainly take the opportunity not to pollute them with BASIC!), and then I'd extend their education down through DOS and BIOS support functions to I/O ports and machine code, until they had an understanding of everything from Pascal down to the bare metal; and then, just as happened to me, I'd start piling additional layers on the top. Because I know that approach worked for me and for (as far as I'm aware) all the serious programmers I know; and I currently have no evidence at all, other than a tenuous analogy with school chemistry lessons, to make me believe that the top-down approach is a sensibly feasible way to create serious programmers.

Unfortunately, in the modern world the bottom-up approach takes a long time before it gets you any useful results. Knowing about I/O ports is really not going to be helpful when no machine except your learning-exercises box lets you access them directly because there's an OS in the way! In my day, bottom-up learning meant I could get useful stuff done on all the kinds of computer I gradually moved through. Do it today and you'd be looking at ten years before you had any directly usable skills, so you'd have to be very sure you wanted to be a programmer to go through all that.

So, can anyone reassure me by arguing that the top-down approach can work sensibly? Or suggest other plausible ways to learn all this stuff? I'm getting old enough now that there might actually be programmers in the younger generation who didn't start in the Spectrum/BBC era and didn't have the opportunity to learn from the bottom up to the extent I did; if any of those people are reading, I'd be interested if they'd share their experiences.

Comments

[senji.livejournal.com]

I'm reminded of Pham Newen, the Qeng Ho's "Programmer Archeologist". I don't have any answers though, since this is much of what I've been complaining about for years.

[dennyd.livejournal.com]

I don't know if I class as a serious programmer, but I sort of learned from the middle out... although I started on a ZX81, and then an Acorn Electron, I only ever poked around with BASIC on both of those. I didn't start learning about the underlying concepts until after I'd started learning higher-level languages, and the reason I learned about the OS-level stuff wasn't because I absolutely needed to at that time (although in retrospect it was helpful), but because I was interested.

I think people will learn just fine wherever they start, as long as they're interested. Their interest will take them up or down the stack as far as they need to go.

[aldabra]

Another barrier in the way of kids-of-today is how complex real-world programs are. Back when I were a lad, I could write shoot-em-ups and AI-type programs that were as good as the ones magazines published for you to type in, even if they weren't as good as the ones you had to pay real money for, and because they were as good as those people were impressed. These days it takes whole departments just to write a soundtrack, so why bother cobbling something together as a beginner in the evenings?

[cartesiandaemon.livejournal.com]

* IO port and transistor level does crop up *sometimes* as you no doubt know better than me, just being a layer or two below machine code, but you'll probably only come across it if you start low down.

* PS. Is that last sentence correct? Do we have a particularly good abstraction? Do you ever need to know stuff that low level? I guess you need to know the chance of memory being cosmic rayed, and latencies in processors and network cables?

* I also learnt reasonably bottom up, and got frustrated when doing otherhow (eg. plunging into windows programming without really knowing how the gui works, or frobbing perl scripts without knowing anything about unix), though was reasonably successful doing it.

* I learnt a lot of my lower level stuff for fun, and then it came in handy as background knowledge when I learnt higher level stuff for real. Maybe this should be encouraged, perhaps, frighteningly, by going back to approving low-level compsci degrees as training for real programming jobs.

* OTOH, I'm biased -- I want to think the way I learnt is valuble, so may be over optimistic in justifying that.

* Perhaps attempt to encapsulate where lies/abstractions leak. Teach someone how to make a database, and say "And whenever it runs slower than foo, or before you finish planning, building or releasing, go and ask someone who knows how it works on the disk what's wrong, and listen and undertand the answer." Admittedly, this is dodgy for security/reliability issues, you need to understand as you go along.

Sounds like complex engineering.

[davefish.livejournal.com]

When you put it like that, I'd say it was an engineering problem. I'm working with a team of people to make scientific instruments, and there are various disciplines present. A lot of the teamwork required is to communicate what is going on at the interfaces, and finding out which bit of adjacent disciplines are important to what you are doing.

You sound like you are after someone who would be functionally equivalent to the systems engineer on a more traditional engineering project, who has an overview of the system as a whole. One of the important things about that role is that it often requires that the sys. eng. doesn't get as involved in the detail of any particular section, but helps with the debugging when something goes wrong in the interface areas.

One question that might be interesting is how much adjacent knowledge does someone need to start being useful in a programming team. Can you assume that someone who appears bright and can plug together routines from an image processing library will be able do enough that he will be useful, or do they absolutely have to know down to the metal in most software engineering environments?

[atreic.livejournal.com]

I've often thought something much along those lines.

My solution is that I still have my beeb and my spectrum, ready to dust off if I ever start spawning mini-Vernons

[ptc24.livejournal.com]

I did some Z80 machine code when I was younger, but didn't really connect it to anything related to the BASIC I was using. Much later on I discovered C, which was something of a missing link for me. Even then, it's not so much a link to the assembler, where things are murky at best, but it does help me to understand higher-level languages a bit better.

[the-alchemist.livejournal.com]

This is interesting, though a little depressing. I hate not knowing much about how computers work, but I can't justify the time it would take to learn a useful amount.

[pjc50.livejournal.com]

I'd be interested to know what the answer is as well. There are two projects underway by friends of mine who are trying to bring simple programming to younger generations. One of these is Nifki (accessible through the web, allows user to make simple games etc), and the other is Eben Upton's project to build a sub-£10 programmable computer that plugs into the TV (basically a microcontroller with a video output). That system will run Lua, but have the option of direct hardware access for the curious.

I have a horrible suspicion that there's a horde of programmers somewhere, cranking out business objects and database reports in VBA and the like who not only don't know about the lower layers but whose minds recoil from the detail and mechanical simplicity of it.

[feanelwa.livejournal.com]

Based on the fact that I persevered through the chemistry and now I'm bosun on the good ship Tecnai sailing the stormy seas of quantum mechanics, I am now trying to do exactly that about a Windows system, but right from the bottom, as in what the electrons are doing. (Well what other bit am I going to start from? :)) My progress so far is tortoise-like, that is to say, slow but difficult to stop. It's inefficient and most people would probably consider it a waste of time compared to paying somebody else to doing it for me, but I don't care, I want to know. Maybe I should write a book when I've finished.

Though the Chemistry journey doesn't ever really end; it loops back to the beginning. You start off being told "it just is", then you get taught about atoms, then you get taught about orbitals, then you get taught about quantum mechanics and how to do it by maths, then once again "it just is" and this is just the best model we've got. But somehow maybe that fourteen year old intuition for what orbitals look like is the point of it all; so it comes full circle in the good and bad way.

[rmc28]

I learned to program in Excel macros, in the sense that there was something I wanted to do (or was told to in order to get a bit of my physics course ticked off) and was told "use macros, here's a guide to the commands you can put together" and I put them together in an order that did what I needed.

Then I worked at BAS for a summer and did shedloads of data analysis using Lotus 123 macros because that was there and I could transfer the concept of if/then, loop, etc across to the 'new' language.

Then wait a year, do the 2-week Java course to get into Part II General and spend a year going to lectures on everything from logic gates to operating systems to business studies.

And then work for (count them) 7 years as a programmer. I don't think I'm anywhere near your level, and I know I'm not a good system programmer or low-level programmer but I can break down a problem and solve it, even if not always quickly. I know how to find information and I generally learn 'just enough' to get a specific job done. I have got a sense of the existence of many layers of abstraction and roughly how they fit together and I don't ever remember the details but I do know who I work with who's good at different layers, and how to start diagnosing roughly where a problem might lie.

So for me, being given a good overview and a series of specific tasks and access to experts in various areas, and use of The Internet has done pretty well, and I like to think I'm at least an averagely-good programmer rather than an ignorant and dreadful one.

Virtual virtue

[hairyears.livejournal.com]

I doubt that you could go from the bottom up: the only route I could imagine would be imparting a thorough working knowledge - including programming exercises - on the Turing machine, then writing a simple compiler for a simple language on a simple device.

The problem, of course, is the real-life implementation of this exercise: the compiler would be written for an emulation of a real hardware device! Which is how almost all this work is done nowadays...

But at least there would be some understanding of what you're doing for a living, writing code for a Java or .NET virtual machine.

[oneplusme.livejournal.com]

This is a thought that's bothered me on occasion in the past as well. I learned programming in much the same way as you (and, indeed, significantly from you...), so I'm used to the idea that it's possible to comprehend (to some extent) all the layers in a system in which you're working. But modern PC-class hardware and system software is getting so fantastically complicated that it simply isn't practical to start at (or near) the bottom of the heap.

I suppose what I'm trying to say is: what's going to happen when we of the 8-bit generation are sufficiently Old Farts that only a vanishingly small percentage of working programmers have ever had the opportunity to play at that low a level? I'm sure there'll always be a subset of god-like hackers who are willing and able to handle the complexity at the bottom, but what would the effect be of most programmers being fundamentally unaware of how their high-level actions are implemented? (And are we there already?)

[pm215]

I think the problem is not so much that you need to start at the bottom as that you need to start small. So 8-bit BASICs are good because:

1. they have a fairly small feature set that's not too hard to get to grips with
2. they do reasonably shield you from the layers below
3. you can actually write something useful which isn't obviously just a toy program

Windows GUI programming fails on all counts. Console-based scripting in Python on Linux, say, is probably pretty good on all three.

I think point 3 is important because it means that you feel like you're actually getting somewhere and you don't have a huge mountain of study to be able to write anything worthwhile. [cf the difference between growing and building programs :-)] And in fact point 3 is where your idea of digging out a Beeb or even a DOS only PC fails -- you're just playing with toys, so there's no motivation. Linux is a system your student can actually use (and ideally is already using), and so the student is aware that small stdio utilities can actually do real useful jobs. The Windows universe is pretty much entirely monolithic GUI programs, which is why it's such a nasty place to bring up kids :-)

[douglas-reay.livejournal.com]

I've been in the situation of being given a child and told to teach them to program. I started him off with python, using a pre-written series of lessons that introduced things like variables, functions, etc.

I think a number of people nowdays don't start off trying to program the machine infront of them. They learn scripting languages for games like Second Life, which have the advantages of being simplified and object oriented. Nothing wrong with starting people off, not on real machine code, but on an emulator, building from logic gates upwards.

And what sort of programmers is the future going to demand anyway? Will 90% of paid programming still be maintenance/extension of internal corporate software? Will they be using grids, virtual servers, jini, clusters, swarms or pastry? I think future programmers are going to have a less analytical understanding of the lower layers and more the sort of understanding a naturalist has. They'll treat their computing environment as an ecology. Something growing and fluid, to be gardened rather than frozen and perfected.


Douglas - Dionysus' Adovacate

[philipstorry.livejournal.com]

The top down system can work, but only with some people.

I can't claim to have learnt from the top-down system really - I started with BASIC and peeks/pokes on a Spectrum, then moved to BASIC and x86 assembler on an IBM 8086 PC. Given that the most complex layer model there is effectively DOS -> BIOS -> Hardware, there's not much top to look down from...

Whether you call them abstractions, models or systems - we learn by building systems in our heads. Or at least, some people do.

I seem to recall that some researchers think that the same impulse which causes men to be train spotters also causes them to be good hunters. They learn what travels where and when, and gradually come to know why this is. They are effectively building a ground-up model of how a railway is run, much as good hunters build models of which animals eat/drink/forage/hunt. Trainspotting is an innate survival skill in some people... Mostly men, but still innate.

That ability to build a model is essential for these kinds of abstractions. I believe that for people with a strong modelling ability, it won't matter where you start them - top or bottom, they'll prod it and poke it until their model is usefully complete, and then move on.

For the rest of humanity, bottom up is probably easier. But they will also lack that "instinctive" ability to grasp what has gone wrong.

The same applies to support staff, by the way. This isn't just developers. I've trained plenty of support staff - some will be able to build those models and will do well, the others will be script readers. The script readers struggle with even basic scripting - the modellers will simply see scripting and basic programming as an extension of their work in support. A different use for the same model...

Bottom up is a better way to teach. But some people will just find their way anyway, because that's the way their minds work. Yes, we want to teach properly - but for those truly suited to working with computers, this won't be a long-term problem. They'll be able to switch between bottom-up and top-down models anyway...

[jvvw.livejournal.com]

The extent to which abstractions leak depends on the type of stuff you're programming. When I worked on router operating systems I had to understand lots about the innards of how things work all the time, now that I do web stuff mostly in PHP, I don't find abstractions leaking very often, though maybe I don't always notice it when they do.

So I don't think you can learn all that sort of stuff totally top-down. But I think there's useful stuff you can do without having to know all of that. I think evening when you're learning things bottom-up sometimes you need the occasional glimpse of higher layers where things are leading so you can see the big picture too, and maybe as motivation. Maybe you don't need to learn it all totally linearly bottom-up, but at some point you do need to make sure you understand the nuts and bolts if you want to write certain sorts of software.

[huggyrei.livejournal.com]

Hmmm.. I'm aware of the fact that there are lower levels when I'm writing my C++ stuff, but I'm afraid I don't really know much about how they all work. I'm not even sure how I would go about finding out.

And I never really had the chance to play around with spectrums and the like. I wasn't born until 1984 - and it's now far too difficult to both find such things, and also find manuals on how to use them.

I do have a short thingmy on machine code, which I really ought to finish reading, and reading through a book on Unix told me more than I knew before I did so.