Suffixed string and char literals are accepted if they don't go through parser

[petrochenkov]

Things like "abc"suffix are lexically valid tokens, but they are reported as errors when parsed with the Rust Language Parser.

However, macro inputs are not parsed by the Rust Language Parser, each macro has its own language that it parses by itself.

It means that suffixed string and char literals can be passed to procedural macros (a procedural macro can process them as they want and make them legal and meaningful in its language), or declarative macros (if they are ignored).

Example:

macro_rules! blackhole { ($tt:tt) => () }

fn main() {
    blackhole!("string"suffix); // OK
}

Possible solutions:

• Alternative 1: Prohibit this during lexing, a string/character literal tokens won't be able to be suffixed, proc macros won't be able to use them in their DSLs.
• Alternative 2: Do nothing, string/character literal tokens can be suffixed (but not in the Rust Language proper), proc macros will be able to use them in their DSLs.

[petrochenkov]

This is listed as an unresolved question in the RFC introducing literal suffixes into lexer (rust-lang/rfcs#463).
@nrc argued for the currently implemented behavior in rust-lang/rfcs#463 (comment).

[petrochenkov]

@rust-lang/lang needs to make some decision and document it in the reference.

[Centril]

@petrochenkov What are the advantages to Alt 1. as you see it? Also, do we have some sort of crater data?

[petrochenkov]

What are the advantages to Alt 1. as you see it?

This is just a somewhat exceptional case of a token which is lexically valid, but always rejected by parser.
I think ~ is the only other example of such token right now (and also <-, but that's only because const generics like Type<-1> are not properly parsed yet).

After some thought, yeah, there's probably not much advantage.

[Centril]

Yeah my inclination is that this may be useful for macro authors somehow so I think we can leave this accepted.

[eddyb]

I regret not coming up with the "joint" system for composed tokens much earlier, and doing at least literal suffixes with it. But overall, I'm in favor of as much flexibility as possible for proc macro authors.

The only useful constraint I see, in terms of the token model, is having balanced ()/[]/{}, because they are used in the invocation syntax, and therefore allow unambiguous parsing around the invocation.
String literals and comments must have a fixed grammar specifically because they can contain those characters, and knowing whether it's part of the source or part of a literal/comment is crucial.

(Although I don't know if other @rust-lang/lang members share this view)

[nikomatsakis]

I agree with @eddyb as well =)

[petrochenkov]

Looks like there's some consensus to keep the existing behavior.
I made a PR to the reference documenting it - rust-lang/reference#612.