A lot of the problem with quotation syntax is that unquoting the string implies a rewrite, which leads to problems with repeated doubling of \\\\ and suchlike. Perhaps quotations should be passed down to the next layer verbatim, so that they are only unquoted at the last possible moment.

But then how can you tell at which level a \} (or \\\} or \\\\\}, etc.) is to be interpreted? (That is, where the "last possible moment" for unquoting that character sequence is: the bottom-most layer, or somewhere in between?)

If it isn't the bottom-most it isn't the last possible moment.

The problem of course is this leads to an un-unixy design where each program has to do unquoting of its arguments if necessary, rather than relying on the shell to handle all metacharacters, and this in turn inevitably leads to incompatibilities.

It's not really Unixy, but POSIX already has a kind of shared dequoting system in the form of getopt(), which handles the quoting of operands using --. Of course, this leads to precisely the kind of incompatibilities you refer to.