simont | (no subject)

(no subject)

It has been a bit of a strange week.

Last night I released a new version of PuTTY, fixing a scary security hole which has apparently been present since the beginning of recorded history. We received notification of this hole a week ago, and I've been frantically running around trying to clear up the mess ever since.

This is only the second security incident PuTTY has had, and for the first one it was in good company (a rather large range of similar programs had related holes). This is the first time it's been us alone, and nobody's fault but mine.

Of course, everybody makes mistakes, and all programs contain bugs. Mostly you just quietly put out a new release, and anyone who cares about the bugs can upgrade. But when you become a security developer, the deal you accept is that suddenly your bugs are no longer things you can quietly fix without fuss; suddenly a bug can be a seriously big deal, and can affect people even when they haven't noticed anything wrong, and so you have to stand up and announce very loudly that you've been a muppet so people know they need to upgrade. You should not go into this field if public embarrassment is a problem for you.

On the plus side, this sort of thing happens to everybody else in the field as well, and you're more likely to be judged on the efficiency, promptness and honesty with which you deal with it than on the incompetence to let it happen in the first place. And on that basis I reckon we did OK; a response time of one week is quite a bit longer than I'd have liked (a fair amount of the delay was me trying to extract more information from the person who reported the problem, because the initial report was unclear), but is downright whizzy by comparison to a lot of security products.

All of that is the theory, anyway. I've been telling myself all that constantly for the whole of the last week, but it doesn't entirely stop the lingering guilt; the feeling that some number of million users have been depending on that code for years, including almost all my friends in particular, and that I, personally, have let every single one of them down. On Saturday, most of which I spent doing actual fixing, I had a fairly serious guilt trip at lunchtime and almost didn't manage to make it back to the keyboard.

Fortunately, after slogging on with it for a bit, I managed to recover some pride in my work by means of fixing the problems in a robust and sensible ‘now why didn't I think of that before?’ sort of way, and that seemed to get me over the worst of it. And now I've actually put out the fixed release, I mostly feel all right about it. Certainly it's a relief not to have it actually hanging over my head any more.

Apart from that, this week has been pretty good so far. Skipped the usual Pizza Express gathering on Monday because I had arranged to be feeding home-made ad-hoc pizza to lnr, which was great fun (if messy, in a throw-everything-randomly-around-the-kitchen sort of way) and I should make home-made ad-hoc pizza more often. Then Calling yesterday, with the surprising addition of the_alchemist, who it was particularly nice to see.

Flat | Top-Level Comments Only

Aw poor you! *hugs*

It's like building aeroplanes. Once you've made a fuselage and wings that don't fall apart on the ground, or on the first flight, people fly in it, but time still tells on all the little bits of grit hanging around in the refinery that might have fallen into the alloy and ended up in the fuselage, or the stress concentrations around the rivets by the windows, etc. - these things cause small cracks that grow more with each flight, until somebody spots them and you fix it. Or somebody doesn't spot them and it crashes. But either way there will be cracks, there is no such thing as a fatigue-immune aeroplane.

Planes are a huge capital investment, so once you've commissioned one you're pretty much stuck with flying it for 30 years. Plenty of time for even the unlikeliest of problems to surface.

Every year the engineering knowledge gets better, and older problems are (or at least should be) made impossible to happen again. But that doesn't remove them from existing stock, and of course new methods introduce problems of their own, and aviation is a very conservative discipline.

RISKS (http://catless.ncl.ac.uk/Risks) is good reading for this - many of the more public crashes get very detailed but accessible post-mortems here.

There is that. Unless it's a giant structural flaw you can usually get it fixed, or at least the part replaced, though, or I thought so.

I will have a read of that. Thank you!

Yep, or at least work around it. Individual parts are often redesigned to eliminate flaws, and will be replaced (depending on severity) at each plane's next scheduled inspection or on immediate recall. In many less severe cases the flaw will simply be documented on the maintenance schedule as a "check this other place for cracks too - if you see any replace part 1234ABC". As I said, very conservative, so if they can characterise the failure mode and have a reasonable idea of how long it will take to progress from first symptoms to critical failure, they'll stick to the devil they know rather than replace it with something they have no operational data on.

Oh yes, remind me to go to The Calling again sometime.

Go to The Calling again sometime.

Next one is August 17th.

Thanks so much for posting this, I wouldn't have known to update otherwise - instead I've been able to let work know too ;) (Who me, smug, noooo)

You mean you're not on the PUTTY-ANNOUNCE list? :)

( http://lists.tartarus.org/mailman/listinfo/putty-announce )

As I already get a noise-signal ratio in my email of something appalling and have only recently broken the back of the problem by diverting everything through gmail, um, no. I don't read the few lists I'm signed up, and contribute once in a blue moon :|

Well, I can empathise with that...

Although the only mail that I've ever got through that list was polite 'The next version of PuTTY is available'-type messages from the wonderful Mister Tatham. :)

(even if I am currently teasing him on Monochrome BBS (http://www.mono.org) about it)

One week is pretty shit hot in my not so very humble opinion, with at least a small basis in experience. Being a small dev team you can be exceedingly nimble, compared with even a medium-sized company...

Do you know when Core are going to publish their advisory?

They originally threatened to publish at the end of August (I think that was the "whether you like it or not" clause intended to force us to fix the problem in case we were thinking of sweeping it under the carpet), but when I said we'd have the fix out long before then they said they'd move the advisory forward. So it might well be out in the next few days.

Stuff was posted to the Full-Disclosure (http://lists.netsys.com/mailman/listinfo/full-disclosure) mailing list last night (http://lists.netsys.com/pipermail/full-disclosure/2004-August/024754.html).

I also remember that the formal advisory was posted to one of the many security-related mailing lists I'm on at some point last night, although I can't remember which one (may have been BugTraq, but I can't find any trace of it in their archives - its not FD I know that much).

Right, found it. It was posted to the Secunia advisories mailing list (http://secunia.com/secunia_security_advisories/) at some point last night; the advisory is available from here (http://secunia.com/advisories/12212/).

Hmmmm - useful stuff; they've got RSS feeds on LJ in the shape of

secuniasecurity for advisories and

secunia_viruses for virus alerts.

Hm, yes. That doesn't look like Core's advisory; that's just a translation into advisory format of the information on the PuTTY website. Though I suppose that's still useful, just for the benefit of people who watch advisory channels and expect to thereby be informed about what they need to upgrade.

The draft advisory Core sent me contained actual information about the precise problem (although it was somewhat incoherent - I'm vaguely hoping they'll let me send them additional text to help it make more sense), and that doesn't seem to have been published yet.

Well, Secunia do say that they are a clearing house for security adversies from all over the place so its not really surprising that they probably took the advisory from the PuTTY home page.

As for the Core advisory being a little incoherent, I'm not surprised given their track record on writing stuff which I've seen on FD and other places!

The Core advisory is now up (CORE-2004-0705), in case you're interested. Also I've published my own writeups of the two issues: vuln-modpow and vuln-ssh1-kex. I've mailed those links to both Core and Secunia, so with any luck they can issue revised versions of their advisories that actually say something useful.

hush - you're not superhuman, you can't be expected to have noticed something that's taken a number of years for someone else to notice.

so you have to stand up and announce very loudly that you've been a muppet so people know they need to upgrade.

IME, the anticipation of public embarrassment is always worse than the event itself. You've announced it now and you've not grown horns or had people blank you in the street. You're OK. I imagine you've had very few emails saying 'you idiot how could you miss this' and many more saying 'thank you for fixing this so quickly'.

feeling that some number of million users have been depending on that code for years, including almost all my friends in particular, and that I, personally, have let every single one of them down.

Guilt trips don't really help - though I know I hit myself with them a lot of the time. You made a mistake, you've fixed it, you're not going to make the same mistake again so let it go and move on. And, to be honest, every one of us who uses PuTTY does so with the knowledge that like all computer applications there may be bugs/security holes in it somewhere. We chose to take the risk that your code is less buggy/security risk prone than the alternatives (hmm I'm not entirely sure what the alternatives to PuTTY are, but that's beside the point).

*adds upgrading PuTTY to to-do list*

Thanks for writing it, and thanks for keeping it up-to-date. Obviously I hesitate to speak for your millions of users, but personally I'm grateful rather than annoyed.

If it's any help, my reaction was "Simon's fixed a security hole - that's good" rather than "Simon is a muppet". The fact that you got the fix out before people felt a need to post an advisory was damn good, and since the vulnerability involves spoofing to exploit under most circumstances, it's not too likely to be a worry unless someone is specifically targeting an attack at you.

Oh yes, and lest my position be clouded by my humour (here and elsewhere): You found a hole in your (non-commercial) software; You fixed it as quickly as humanly possible and you done a mighty fine job of letting us all know about it ASAP.

There nothing to be embarrassed about there - you've provided service well above and beyond both the call of 'duty', and frankly most commercial vendors I can think of.

Well done sir, say I!

Yup. What he said.

And, of course, this news popping out on #drum via #chiark has overloaded the download server...

But thanks to

bofhcam I got the download for the entirety of admin.cam.ac.uk anyway :)

Flat | Top-Level Comments Only