Release 0.12

[dtolnay]

First I would like us to finish all of the AST changes that we know about, even if the parsing/printing isn't perfect for all of them.

cc @Arnavion - you're doing great so far! Thanks for all your help.

[dtolnay]

This is up in the air as Alex is helping rewrite syn to operate on token trees instead of strings: rust-lang/rust#40939.

[alexcrichton]

Next blocker for this is updating serde

[dtolnay]

Let's publish what we have under a different name for now. That way:

• the early adopters can use the new TokenStream without dealing with git dependencies,
• we can figure out the right idioms in Serde and futures-await and refer people to those when we eventually do 0.12,
• the large contingent happily using Macros 1.1 is not disrupted while we iterate on this.

Something like syn-unstable or syn-next.

[oli-obk]

You could also use *-pre versions instead of different crate names

[alexcrichton]

Oh to be clear I'm in no particular rush to release these crates, just figured it'd be good to know what's blocking it. I wouldn't mind using git for awhile.

[mystor]

I'm not sure whether or not we would want to start publishing it. I am currently inclined to publish it as 1.12.0-pre-1, assuming that cargo will never automatically select that version (I'm not sure what the state of cargo's support for pre-release versions is - I was reading rust-lang/cargo#2222 but couldn't determine what the current behavior is from it).

I would say that currently the API isn't stable, and is less ergonomic to use than syn 0.11, which makes me not want to replace syn 0.11 until we can make sure that the API is actually stable and usable, so I definitely don't think we want to publish it as 0.12.

If the only advantage then of 0.12 was that on nightly it could interact with the native proc_macro API, I would not bother publishing it, especially because due to rust-lang/rust#43081 it doesn't even give you better span information.

However, 0.12 also contains my fixes from #168 which means that it can correctly parse expressions now. This can be useful in some macros even today (e.g. https://github.com/mystor/unsauron - which is a silly macro I made, but represents a class of macros which are now possible with syn 0.12, but weren't possible with syn 0.11), It would be nice to be able to publish unsauron and future macros like it on crates.io, which requires that we publish 0.12 in some form.

That's why I feel we should publish it as a preview version, so that people don't start using it without knowing the trade-offs, but we don't make it impossible to publish crates which use it to crates.io.

[dtolnay]

I am not a fan of pre-release versions.

I think it can work well for "release candidate" or similar almost-complete versions. The deciding factor is whether time invested in upgrading a crate to use the release candidate is 100% applicable toward upgrading a crate to use the upcoming real release. Upgrading from 0.(x-1) to 0.x-rc1 to 0.x-rc2 to 0.x.0 should be the same amount of work as upgrading directly from 0.(x-1) to 0.x.0.

In my experience pre-release versions are no good for effectively "development snapshot" versions, which is the state syn would be in. We saw this with bincode releasing 1.0.0-alpha1 through 1.0.0-alpha7 over the course of several months. A sizeable fraction of people will pick those up and be frustrated with a treadmill of breaking changes over and over to the same code, even though they brought it upon themselves. And Cargo makes it even worse by upgrading across different pre-release versions even though there is no expectation that they are compatible (rust-lang/cargo#2222).

I would like to either publish a release under a different name like syn-next or make git dependencies work better (rust-lang/cargo#4247).

[alexcrichton]

I personally hope to avoid extra crates/prerelease if possible, and I think the only real blocker for git deps is the huge submodule, so I opened #183 to paper over that issue

[jimmycuadra]

+1 for just using Git and improving that experience if necessary.

Would it make sense to create a milestone for 0.12? I think it'd be helpful to get a better picture of what needs to be done for it. I'm in my usual state of not wanting to build a bunch of stuff on 0.11, knowing that an 0.12 with breaking changes is coming. I'd prefer to base my work on the latest stuff so I can give feedback on my experience, too! Right now the worst part about proc macros is the span information pointing to the invocation of the macro which is entirely useless to me. Sounds like that is more an issue with the proc macro feature than syn, though.

[mystor]

That problem isn't going to be fixed by 0.12 on stable, but might be fixable with 0.12 on nightly (due to the proc-macro2 shim).

I don't think there's a well-defined set of blockers right now, rather there's a general feeling that the new API is inferior to the current one, especially because of the need to specify every token when performing AST transformations. A milestone and a set of things which need to change might be good for driving us toward the release though.

[dtolnay]

I cleared some time over the next few weeks to do API design work and figure out a roadmap for 0.12.

[mystor]

Awesome!
I think the biggest hurdle in my experience for working with syn 0.12 is the tokens stuff that we were talking about at Rust Belt Rust, so hopefully we can come up with a good solution for that.

[dtolnay]

@mystor I assigned most of the open syntax issues to you. Feel free to decide which ones need to be in 0.12. Some of them may not require changes in the AST so those can wait. Later this month I will try to get through the ones that haven't been done by then. 😜

[dtolnay]

Tentatively scheduling this for next week Thursday or Friday, with mostly documentation work remaining. @mystor what is the chance you could get dtolnay/proc-macro2#36 across the finish line by then so we can put together some compelling demos involving Span?

[mystor]

I haven't really been working on it much. I don't think it'll be very hard if what we decide to do is simply have a build-time environment variable which controls access to the span APIs. I worry about the ergonomics of that, although it might be OK.

I've been considering more dramatic changes to the way syn and the token provider interact which would allow us to make the parser decoupled from the token provider and span format without hurting build times, which could be nice to have for things like build scripts, but I think that's outside the scope of syn 0.12 for sure.

[dtolnay]

I would like rust-lang/rust#47149 to land in a nightly before doing this release. I am finding it honestly pretty tough to come up with compelling demos of Span without that. The generated code cannot do anything interesting without it failing to compile or the error messages being worthless.

[dtolnay]

Bumping to next week (Monday or Tuesday) because the Span methods have been stuck in the post-vacation bors queue for 2 days and will not be in Friday's nightly.

[dtolnay]

I have some minor fixup and documentation work left (tracked here) but we should be good to go for Monday morning Pacific time. Everyone following this issue should cargo doc --all-features the docs locally and provide last-minute feedback on the new stuff. Thanks!

[mystor]

I'm not a big fan of the parse2 function name for parsing proc_macro2::TokenStream. Perhaps we want to instead make parse take a Into<proc_macro2::TokenStream> and get rid of it?

It might also be nice to only export the parser macros if a feature flag is enabled, so that if the user has nom in their project, but also wants to use the Tokens! macro, they don't have to worry about macro name conflicts?

[dtolnay]

Thanks! I preferred parse2 over one function that takes T: Into<proc_macro2::TokenStream> because turbofish becomes more complicated and unpleasant with Into -- was it syn::parse::<_, Expr> or syn::parse::<Expr, _>? It is really only two types that we ever expect users to pass in, and those are called proc_macro::TokenStream and proc_macro2::TokenStream which the user already needs to wrap their head around, so parse and parse2 seemed the least bad. This will go away when we no longer need proc-macro2.

I don't think feature gating the combinator macros would help because as soon as some other crate in your dependency graph needs syn without default-features = false, suddenly you get the macros whether you wanted them or not, and your crate doesn't compile. I would prefer if conflicts with nom are solved by writing #[macro_use(Token)].