Nested comments would be a problem, I agree; in particular, they would make the number of states in the lexical analyser infinite. And one thing I've been so far trying to avoid is having variable-size annotations in each node, because that way lies garbage collection and all sorts of yuck.

Hmm. What exactly is your bracket counter, in that case? Was that the reason you were special casing it?

My bracket counter records, for a given chunk of text, how many levels of brackets/braces/whatever are closed within that text (and hence must presumably have been open at the start of it), and how many new levels are opened before the end of it. I'm not sure of the best way to deal with mismatches (an open paren matching a closing brace), but I suspect the answer is to match any open-thing with any close-thing and leave the top-level code to notice when they don't really match and complain to the user somehow.

The important thing about this is that it operates at the syntactic level, not the lexical level, which means that nothing else is conditional on it. Nothing in the syntax annotation for a chunk of text ever has to say "well, if there are three levels of brackets open at the start of this text then this happens, but if there are four then it's all totally different". The bracket counts themselves are conditional on the starting lexical state, but nothing in turn is conditional on them.

Hence, arbitrary nesting in the lexical analyser is a much worse problem than arbitrary nesting in the syntax, because lots of stuff is conditional on the opening lexer state, so allowing infinitely many distinguishable such states is asking for trouble.

Come to think of it, PostScript is going to have a problem with this, since it supports arbitrarily nested parentheses in strings...