Line comments versus line splicing

[simont]

Here's a thing I was pondering the other day about lexing.

Suppose you have a language containing two moderately common lexical features: a comment mechanism in which comments are newline-terminated (C++/C99 style //, shell #), and a line-splicing mechanism in which a backslash at the end of a line causes the next line to be glued on to it as if they were one long line. How should these interact?

C++ and sh have different answers. In C++, a backslash at the end of a // comment still line-splices, with the effect that the next line of the source file is considered to be part of the comment. (I've seen this trip up a user doing ASCII art in // comments.) In shell, it's the other way round: a comment renders everything to end-of-line irrelevant, including a trailing backslash. Either of these is defensible in terms of layering – the language specification has multiple layers of syntactic processing, with comments in one and line-splicing in another, and these behaviours drop out as a consequence of which order the layers appear in. (But it's kind of annoying that they do it differently.)

Anyway. Leaving aside how existing languages handle this interaction, how would we like to handle it, if we were designing a new syntax with both of these features? For the sake of illustration, I'll assume that the language also has syntactically significant newlines, so that line-splicing is a desirable thing to be attempting in the first place. (I can't really see how it isn't just an obvious misfeature in C/C++.)

A thing that's often annoyed me in make is that if you have a sprawling variable definition with lots of line-splices, you can't comment one particular line:

OBJECTS = one.o \
          two.o \ # putting a comment here doesn't work
          three.o \
          four.o # putting it here doesn't either, arrgh \
          five.o

So, for a start, it would be nice to arrange that at least one of the above syntaxes works. Either would be OK, but I prefer the first, because of the scenario I mentioned above about people doing ASCII art in comments – you might plausibly end a line comment with a backslash when line-splicing was the furthest thing from your mind.

Another credible use case is if you want to comment one of the line-spliced lines out:

OBJECTS = one.o \
          two.o \
#         three.o \
          four.o

(Also, you'd probably want to include a comment saying why it was commented out, which adds extra confusion.)

So I think that leads me to the following rules:

• A backslash causes line splicing if it is the last lexical token on a physical line, i.e. it still line-splices even if a comment-to-eol appears after it. (Also, while I'm here, it also shouldn't matter if whitespace appears after it. Syntactically significant whitespace is often unhelpful, but syntactically significant trailing whitespace is especially egregious.)
• A backslash appearing at the end of a comment-to-eol is just part of the comment and has no line-splicing effect.
• If a line with a line-splicing backslash is followed by one or more lines containing nothing but comments-to-eol (with optional initial whitespace), then those comment-only lines are completely ignored and the subsequent one will be spliced on to the original line instead.

So that permits all of the following usages:

OBJECTS = one.o \
          two.o \ # This object is notable for some reason
          three.o \
# the next two objects must not be reordered
          four_a.o \
          four_b.o \
# end of reordering constraint
          five.o \
#         six.o \ # removed until bug 1234 is fixed
          seven.o \
          eight.o

So. What use case have I missed with that analysis, and/or why is my modest proposal going to lead to crawling horrors in some totally different scenario which existing languages don't mess up so badly?

Comments

[gerald_duck]

Hmm.

For clarity, I'd prefer to think of three or more features interacting, not two:

1. Newline to terminate a comment
   
2. Newline for at least one other purpose
   
3. A token that can precede a newline to 'escape' it, token+nl turning into nothing

There are then three issues:

• Precedence
  
• Whether parsing transforms any of these syntactic structures into something that affects lower-precedence ones
  
• Whether or not we want a single newline to be able to serve multiple syntactic purposes

Having the escaping mechanism at lowest precedence is obviously pointless. So should escaping have higher precedence than all possible uses of a newline, or be lower than some but higher than others? To my way of thinking, the least mind-bending option is to say that escaping has unequivocally higher precedence than any syntax that uses newlines: either you see the escape token at the end of a line, or the newline isn't escaped. End of story.

End of that story, at any rate. There is still the question of how syntaxes that rely on newlines should compose when an unescaped newline occurs. C/C++ say that comment-to-nl has higher precedence than preprocessor directives, and that comment-to-nl is equivalent to newline, so '#define MOO baa//quack' defines MOO to baa. That, I think, is fair enough. Accepting, of course, your point that the syntax of preprocessor directives in C/C++ is an obvious misfeature in the first place. (If they could be freely interpolated, life would be much better in many ways, and the issue of whether comment-to-nl was deemed equivalent to whitespace or newline wouldn't arise. Though that makes comment-to-nl itself feel like a mistake in the language; hohum.)

So far as it goes, I'm fairly happy with C/C++'s rules. They let me write what I need to without tripping up, and when I do see something grotesque (which, naturally, was written by someone else) I can work out what it does. There are worse syntactic problems to worry about.

Looking at your proposed rules, by my understanding you're saying comment-to-nl should be treated as a newline, and should have higher precedence than the escape token, that newline as a statement delimiter should have lower precedence than the escape token. Then your rule 3 throws a curveball.

Picking things apart more carefully, I think the totality of what you're suggesting is:

• '//' to end of line is tokenised as comment
  
• '\' nl? comment+ is elided

…though you don't make explicit what you want this to do:

ten.o    \ # A comment
           # another comment
eleven.o

…I'm guessing you want it to treat ten.o and eleven.o as being on the same line?

While that has some heuristic benefits, it looks pretty clunky when formalised. Maybe a better solution is for shell to have a commenting style that sits within the line, so the problem goes away!

If designing a syntax from scratch, a far neater solution is to set aside a dedicated character, not used for anything else, and say that character to end of line is whitespace. Now your example becomes:

OBJECTS = one.o ♦
          two.o ♦ This object is notable for some reason
          three.o ♦
♦ the next two objects must not be reordered
          four_a.o ♦
          four_b.o ♦
♦ end of reordering constraint
          five.o ♦
♦         six.o ♦ removed until bug 1234 is fixed
          seven.o ♦
          eight.o

…which I find far more tolerable.

Even better, people should heed the moral, clear in hindsight, that newline is a syntactic element like any other and, like any other, you should avoid confusing nesting and interaction of syntactic constructs like the plague!

[simont]

To my way of thinking, the least mind-bending option is to say that escaping has unequivocally higher precedence than any syntax that uses newlines: either you see the escape token at the end of a line, or the newline isn't escaped.

I think this is precisely where pragmatism and conceptual simplicity diverge: the option you describe leads directly to the ASCII art trap, which it seems to me is a severe violation of the Principle of Least Astonishment (especially in C/C++'s strict implementation where invisible trailing whitespace can change the meaning) and imparts a fair amount of mindbendingness to my way of thinking.

your point that the syntax of preprocessor directives in C/C++ is an obvious misfeature in the first place. (If they could be freely interpolated, life would be much better in many ways [...])

That's a considerably broader point than the one I'd intended to make, but I'll cheerfully subscribe to it regardless. Being able to embed macro definitions anywhere, including in the expansions of other macros, would certainly offer scope for some exciting perversions :-)

Though that makes comment-to-nl itself feel like a mistake in the language

It is odd, now I think about it, that neither of C/C++'s comment syntaxes nests properly. If they were going to throw in a second comment syntax, they could have had it fix the bug in the first one!

you don't make explicit what you want this to do: [...] I'm guessing you want it to treat ten.o and eleven.o as being on the same line?

Yes, that was my intention. (I thought I had made that explicit, though perhaps I left an unintended loophole in the wording. The line containing ten.o has a line-splicing backslash by rule 1, and therefore rule 3 applies to the line following it.)

Rule 3 is a curveball, I agree, but then I was approaching the entire problem from a use-cases perspective to see what would happen, and since the conceptually clean approaches in C/C++ and shell both turn out to lead to weird use cases, it didn't surprise me that my use-case-focused reanalysis led to weird concepts :-)

a far neater solution is to set aside a dedicated character, not used for anything else, and say that character to end of line is whitespace

Hmm, yes, that is quite neat; a sort of comment-to-just-after-nl as opposed to (or rather, offered as an alternative to) the more usual comment-to-just-before-nl.

[gerald_duck]

The perversion of defining a macro in the expansion of another macro is orthogonal to the issue of whether or not preprocessor directives can be freely interpolated. The C/C++'s preprocessor could have allowed:

  #define MOO(define) #define x QUACK

…but they've prohibited it from working the way you might expect (and hijacked # inside macros for another purpose, which is why I was sick and used "define" as the macro parameter's name (-8 ).

I'm guessing they decided that macros defining macros was a can of worms best left unopened.

At this point I'm remembering Modula-3's pragma syntax: <* … *> to go with the Wirth (* … *) for comments. (How come everyone's happy using [ … ] for arrays in Pascal instead of (. … .), but still uses (* … *) instead of { … } for comments?) Had C used something similar, we could have avoided all this __attribute__ and __declspec nonsense for starters!

If I ever design a language in the C problem space, knowing what I know now, the pragma syntax will be something like:

void f() {
  #pragma #switch(COMPILER_VERSION) {
      RECENT_GCC:
          diagnostic push;
          diagnostic ignored "-Wuninitialized";
          break;
      MSVC:
          warning(push);
          warning(disable: C4700);
          break;
  }

  // Do some stuff...

  #pragma #switch(COMPILER_VERSION) {
      RECENT_GCC: diagnostic pop; break;
      MSVC: warning(pop); break;
  }
}

[bjh21.me.uk]

If designing a syntax from scratch, a far neater solution is to set aside a dedicated character, not used for anything else, and say that character to end of line is whitespace.

That looks suspiciously much like how TeX does comments (and line-splicing).

[fanf]

In a language with significant newlines you need both comment-with-splice and comment-without-splice - the latter so that you can add a comment without affecting the meaning of the layout. For example, I might have a makefile containing

OUTPS=${SRC:.tex=.ps}
OUTPDF=${SRC:.tex=.pdf}
OUTDVI=${SRC:.tex=.dvi}

And I should be able to comment out any of those lines without inadvertently concatenating them.

[gerald_duck]

It's not absolutely necessary, of course — you could just drop in a blank line afterwards.

But I do see what you mean. /-8

♦ smells like dnl

[digital-scurf.org]

Gerald Duck's idea of ♦ smells remarkably like M4's 'dnl' construct.

I would like it if more languages had similar.

[fanf]

Another option is to put the continuation marker at the start of the next line, and allow it to take effect across multiple preceding comment-only or whitespace-only lines. Eg:

OBJECTS = one.o
\\          two.o # This object is notable for some reason
\\          three.o
# the next two objects must not be reordered
\\          four_a.o
\\          four_b.o
# end of reordering constraint
\\          five.o
#\\         six.o # removed until bug 1234 is fixed
\\          seven.o
\\          eight.o

[fanf]

And another option (which I think BCPL has) is to pair continuation markers at the end and start of the spliced lines. Any white space and/or comments between the splice markers is deleted. You could also specify that bracketed comments such as /* */ have these semantics.

[fanf]

In fact, backslash-newline is redundant in C (if you don't use it to split tokens, which is evil) because its pertinent uses can be replaced by a /* \n */ comment or by a "..." \n "..." string continuation.

[gerald_duck]

Indeed. I made that observation when I bumped into Simon in the pub this evening.

There is just one place I can think of where backslash-newline is nonetheless needed: splitting a #include filename across multiple lines. Why doesn't the preprocessor concatenate all strings that follow the #include directive to make the filename? I concluded some hysterical raisins might be involved. (ISTR encountering a compiler once upon a time that had a non-standard optional second parameter.)

[fanf]

Good point - though if you need to split a #include like that then something has gone horribly wrong :-)

Also, #include delenda est: http://www.infoq.com/news/2012/11/llvm-modules

[simont]

I ought to have thought of that myself, since the disgusting mkfiles.pl script that I use to manage the many makefiles for my portable free software uses that technique. Ahem.

[deliberateblank.livejournal.com]

Ah, I see someone has been reading TDWTF. (YIK pretty much everyone has read that article.)

The article in question is perhaps more locally exemplified by:

#include <stdio.h>

#define FOO() \
printf("A\n"); \
// printf("B\n"); \
printf("C\n"); \
printf("D\n")


int main(int argc, char *argv[])
{
printf("1\n");
// foo \
printf("2\n");
printf("3\n");

printf("\n");

FOO();

return 0;
}

Which does two thing which one might not expect, one more so than the other.

I think your rules fix both of them, but do enable constructs where the start of the line splice is divorced by many comment lines from what it is spliced to, which might not be ideal. (If that makes a semantic difference, then perhaps one could consider that another argument against python...) Perhaps a requirement that all ignored comment line *must* also be splicing lines? Dunno.

[simont]

Actually, I don't know what TDWTF article you might be referring to! (I drop in there occasionally, but not regularly, and I haven't seen this one.) Presumably this was somebody else having the //...\ problem in C++? The instance of that I was thinking of was from someone I know personally, who had written something along the lines of

    printf("complicated format string",
           arg1,
           arg2, // \
           arg3, //  } comment about these three
           arg4, // /
           arg5);

and got mysterious compiler warnings about the format string mismatching the parameters, which turned out to be because arg3 had been dropped on the floor. (Mystery eventually solved by using a later version of gcc, which helpfully gave a warning about the line-continued comment.)

[deliberateblank.livejournal.com]

Yup. The story was this one:

http://thedailywtf.com/Articles/A-Bad-Leg.aspx

Which made a complete hash of telling the story, but is possible to figure out what was actually going on (and it fails just the same in gcc contrary to their telling), and surprised quite a few people reading it.