Using thread-local state inside proc-macro causes ICE

[olegnn]

ICE will be produced on any channel.

Error:

thread 'rustc' panicked at 'use-after-free in `proc_macro` handle', src/libcore/option.rs:1190:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.39.0-beta.1 (968967007 2019-09-24) running on x86_64-apple-darwin

note: compiler flags: -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

error: custom attribute panicked
 --> proc_macro_bug/src/main.rs:7:1
  |
7 | #[some_macro(0)]
  | ^^^^^^^^^^^^^^^^
  |
  = help: message: use-after-free in `proc_macro` handle

error: aborting due to previous error

error: could not compile `proc_macro_bug`.

LocalKey used in macro generation:

extern crate proc_macro;
extern crate quote;
extern crate syn;

use proc_macro::TokenStream;
use quote::quote;
use syn::{parse_quote, Path};

thread_local! {
    static DEFAULT_CRATE_PATH: Path = parse_quote! { ::std };
}

#[proc_macro_attribute]
pub fn some_macro(_: TokenStream, _: TokenStream) -> TokenStream {
    TokenStream::from(DEFAULT_CRATE_PATH.with(|default_crate_path| {
        quote! {
            use #default_crate_path::boxed::Box;
        }
    }))
}

Macro calls (should be 2 at least) which produce error:

#[some_macro(0)]
struct Abc {}

#[some_macro(0)]
struct Cde {}

fn main() {}

Repo with minimal code sample: https://github.com/olegnn/proc-macro-ICE.

[Mark-Simulacrum]

Seems to have been fixed, at least, I cannot reproduce. Closing. I don't think a test is needed here.

[olegnn]

@Mark-Simulacrum latest nightly:

thread 'rustc' panicked at 'use-after-free in `proc_macro` handle', /rustc/433aae93e4ef866a1fdfefad136b32ed89acd3e7/src/libproc_macro/bridge/handle.rs:42:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.43.0-nightly (433aae93e 2020-02-14) running on x86_64-apple-darwin

note: compiler flags: -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

Check the repo https://github.com/olegnn/proc-macro-ICE if you can't reproduce on playground.

[Mark-Simulacrum]

I feel like this is intentional behavior, though I guess we probably want to produce an error rather than ICEing. Reopening.

[Centril]

cc @eddyb @petrochenkov

[eddyb]

It's generally hard to handle this gracefully due to how deep it is in the whole system - usually a panic inside a proc macro would turn into an error but here it doesn't seem to.
Maybe we just need to include more adjacent code in some catch_panic scope?

---

@olegnn This kind of code can't work, every time a proc macro is ran, the proc_macro types (e.g. TokenStream) have incompatible values with other runs.
Ideally we'd use multiple threads, or clear TLS from the proc macro somehow, but we'd need more optimizations to make the former viable, and I don't even know if the latter is possible.

It's probably better to replace DEFAULT_CRATE_PATH with a function, or even a struct DefaultCratePath; that implements ToTokens (if you want to be able to use #DefaultCratePath).

[estebank]

Current output:

error: custom attribute panicked
 --> proc-macro-bug/src/main.rs:7:1
  |
7 | #[some_macro(0)]
  | ^^^^^^^^^^^^^^^^
  |
  = help: message: use-after-free of `proc_macro` symbol

error: could not compile `proc-macro-bug` (bin "proc-macro-bug") due to previous error

[peter-lyons-kehl]

Hi Thank you for your investigation so far.

1. proc_macro stream parts are perfectly suited for thread local re-use, because they

• take long to parse, and
• are immutable, !Sync and !Send, and
• they are designed for re-use: their clone() is lightweight.

If (subsequent) proc_macro invocations are invoked in different threads, or shift-rota over a thread pool, that's fine with the above. It should also be fine with any well implemented proc macros (as user space-defined macros shouldn't have any side effects).

May I guess: The current restriction/ICE is in place to prevent/alert macro developers so as not to make a mess. Could we introduce a configuration option to suppress that check?

Yes, if a macro or its output depends on the thread (or the thread ID...), then the macro or its output/tests will break. But that's because such a macro is dirty.

2. The actual error message gets to be much worse than (reports) above. With #[proc_macro_attribute] this problem doesn't show up as an (obvious/well named) ICE, but much more confusing. It does mentions neither thread_local!, nor proc macro at all!

• cargo test
• Compiling allows v0.1.0 (/share/pkehl/GIT/allows)
• thread panicked while panicking. aborting.
• error: could not compile allows (test "simple")
• Caused by:
• process didn't exit successfully: rustc --crate-name simple --edition=2021 tests/simple.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=41 --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 --test -C metadata=825a0056779a2e30 -C extra-filename=-825a0056779a2e30 --out-dir /share/pkehl/GIT/allows/target/debug/deps -C incremental=/share/pkehl/GIT/allows/target/debug/incremental -L dependency=/share/pkehl/GIT/allows/target/debug/deps --extern allows=/share/pkehl/GIT/allows/target/debug/deps/liballows-cdc70b39f5946a9c.so -Ctarget-cpu=native (signal: 6, SIGABRT: process abort signal)

See minimized:

• https://github.com/coop-rs/allows branch minimized_thread_LocalKey, or:
• https://github.com/coop-rs/allows/blob/minimized_thread_LocalKey/src/lib.rs
• https://github.com/coop-rs/allows/blob/minimized_thread_LocalKey/tests/simple.rs

The above unhelpful error message is the same on both nightly and stable. My minimized example needs nightly, but only for the actual test that uses the proc macro - to keep it all simple. The proc macro itself compiles (but fails to run) on stable, too.

[peter-lyons-kehl]

(Updated): Producing ICE for me, too.

[ple1n]

It just happened for me

fn types() -> HashSet<Ident> {
    let idents: Punctuated<Ident, Comma> = parse_quote!(Select, End, Receive, Branch);
    idents.into_iter().collect()
}

thread_local! {
    static  TYPES: HashSet<Ident> = types();
}

error: custom attribute panicked
   --> netns-proxy-rs/src/tasks.rs:208:5
    |
208 |     #[session('k Self::Selector)]
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = help: message: use-after-free of `proc_macro` symbol

rustc 1.74.0-nightly (0288f2e 2023-09-25)