Tokens in an attribute macro receive wrong spacing

[dtolnay]

Originally reported by @CodeSandwich as dtolnay/syn#425.

We are seeing a functionlike procedural macro invocation m![< -] receive the expected spacing (Alone for both tokens) but preprocessing the same input with an attribute macro #[attribute] m![< -] receives incorrect spacing -- Joint for the < token even though it is followed by whitespace. Strangely using parentheses appears to fix the spacing so that #[attribute] m!(< -) correctly has Alone for both tokens.

Repro script

#!/bin/sh

cargo new --lib repro
cargo new --lib repro_macros

echo >repro/src/main.rs '
#![feature(proc_macro)]

extern crate repro_macros;
use repro_macros::{tokens, nothing};

// alone + alone
tokens![< -];

// joint + alone
#[nothing]
asdf![< -];

// alone + alone
#[nothing]
asdf!(< -);

fn main() {}
'

echo >>repro/Cargo.toml '
repro_macros = { path = "../repro_macros" }
'

echo >repro_macros/src/lib.rs '
#![feature(proc_macro)]

extern crate proc_macro;
use proc_macro::TokenStream;

#[proc_macro]
pub fn tokens(input: TokenStream) -> TokenStream {
    println!("{:#?}", input);
    TokenStream::empty()
}

#[proc_macro_attribute]
pub fn nothing(_: TokenStream, input: TokenStream) -> TokenStream {
    println!("{:#?}", input);
    TokenStream::empty()
}
'

echo >>repro_macros/Cargo.toml '
[lib]
proc-macro = true
'

cargo build --manifest-path repro/Cargo.toml

Output

TokenStream [
    Op {
        op: '<',
        spacing: Alone,
        span: #0 bytes(116..117)
    },
    Op {
        op: '-',
        spacing: Alone,
        span: #0 bytes(118..119)
    }
]
TokenStream [
    Term {
        sym: asdf,
        span: #0 bytes(7615976..7615980)
    },
    Op {
        op: '!',
        spacing: Alone,
        span: #0 bytes(0..0)
    },
    Group {
        delimiter: Parenthesis,
        stream: TokenStream [
            Op {
                op: '<',
                spacing: Joint,
                span: #0 bytes(0..0)
            },
            Op {
                op: '-',
                spacing: Alone,
                span: #0 bytes(0..0)
            }
        ],
        span: #0 bytes(0..0)
    },
    Op {
        op: ';',
        spacing: Alone,
        span: #0 bytes(0..0)
    }
]
TokenStream [
    Term {
        sym: asdf,
        span: #0 bytes(192..196)
    },
    Op {
        op: '!',
        spacing: Alone,
        span: #0 bytes(196..197)
    },
    Group {
        delimiter: Parenthesis,
        stream: TokenStream [
            Op {
                op: '<',
                spacing: Alone,
                span: #0 bytes(198..199)
            },
            Op {
                op: '-',
                spacing: Alone,
                span: #0 bytes(200..201)
            }
        ],
        span: #0 bytes(197..202)
    },
    Op {
        op: ';',
        spacing: Alone,
        span: #0 bytes(202..203)
    }
]

@alexcrichton

[alexcrichton]

So it looks like, as usual, there's a couple of bugs here.

Pretty-printed macros don't persist delimiters

When we print item-like macros we don't preserve the delimiter. This means that the pretty-printed version of foo![] and foo!{} both look like foo!().

That in turn causes this check to fail. which means in the second case here, asdf![< - ], we stringify the macro and reparse to get the tokens to hand to #[nothing]. This doesn't happen for asdf!(< - ) because the "probably equal" check passes as the pretty-printed version has the same delimiters.

Parsing stringified tokens gives wrong joint-ness

Joint-ness of a TokenTree when parsing is determined here and here. In both locations it's Span-based. During parsing a procedural macro's input spans get really weird and everything has hi/lo of 0. That means that everything which can be is considered joint.

---

I'll look into fixing both of these soon. The first one is likely not too hard, but the latter one I'll have to do some more investigation for.

[alexcrichton]

I've forked off the persistence of delimiters to #50840 and I've otherwise fixed the parser at #50838 (which will fix this issue)