Eventual Concern: Send/Sync insufficient in the presence of multiple execution contexts.

[Lokathor]

Following up from rust-lang/rust#53639 (comment)

This came up as a side tangent in the "deprecate static mut(?)" issue, and I didn't want it to get buried and forgotten. It's something we will have to face up against eventually, and it feels like the Lang team would want to have input here.

Essentially: Send/Sync are only "true" when all values (including all pointers/references) live in a single global execution context (a single global address space).

Once you've got more than one execution context all of Rust's support in the type system flies right out the window.

Examples:

• multi-core bare metal (peripherals and/or memory can be exclusive to a core)
• multi-process (memory is shared between processes but each process might have it living in a different location)
• gpu/cpu interaction.

For now you can just "use raw pointers everywhere and be very careful", but that shouldn't be our answer forever.

(Note: This issue is not about a specific design proposal or change proposal, more like a "keep this on your radar" reminder.)

[jamesmunns]

I wrote up some notes on this topic in my lab notebook, after some further discussion in the embedded-wg. I would appreciate any comments on my notes (or my claims), and will follow up when I have the time to turn this into a blog post.

[nikomatsakis]

Heh, I've been waiting for someone to bring this up. =) I first noticed it when thinking about GPUs, but another context where it comes up (potentially) is trying to support a theoretical WASM GC, which kind of comes down to a similar desire to separate out types by "address space".

i toyed with some possible designs here at various points, it's definitely a rather painful thing to try and change.

[jamesmunns]

Yeah, it's been brought up in a couple contexts, and I hadn't been able to find an authoritative resource regarding this. I'm hoping to write a non-normative summary, sort of like Gankra's "Notes on Type Layouts and ABIs in Rust" post.

We often run into these problems in embedded systems, off the top of my head I can think of many kinds of systems with "different flavored pointers", including harvard architecture processors (like AVR), systems with 16 + 20 bit pointers (that are semi-compatible with each other) like the MSP430 family, flash addresses that can only be read from (but not written to), or MPU/MMU/TrustZone protected memory regions.

I'm inclined to state that within today's reality, a language like Rust cannot (at a language level) address this disparity, without a lot of case/platform specific behaviors, which I think might be better addressed by libraries, including libstd.

I think addressing these concerns BEYOND that starts getting in OS/Environment/Runtime/ABI sort of concerns which are certainly language-adjacent, but not "ruled" by the language itself.

[RalfJung]

We often run into these problems in embedded systems, off the top of my head I can think of many kinds of systems with "different flavored pointers", including harvard architecture processors (like AVR), [...]

I am inclined to draw a distinction here -- I think you are conflating two orthogonal issues. The cases you are listing are examples where a single thread of execution (I am using "thread" in a very general way here, it might be the same as what you call "execution context") has multiple address spaces at its disposal that it can all access to varying degree and with different instructions. The proper way to handle that (IMO) is to embed information the address space into the pointer. This can be modeled to some extend using newtypes around raw pointers. This is not at all in conflict with Send/Sync as long as all threads of execution have access to the same (multiple) address spaces.

Send/Sync come into play once you want to model communication between threads that don't have access to the same address spaces. Then you want to ensure that "this data (that is sent from one thread to another) does not contain pointers into address space X" or that it "only contains pointers into address spaces Y, Z", and that needs a separate trait. (I don't think we need two traits here like we do for Send and Sync; this is not a question of what kind of protocol is enforced on the data, it is purely a property of the reachable data.)

flash addresses that can only be read from (but not written to)

On its own that is not different from read-only memory in an MMU system, is it? Even in "normal" x86 systems, there are very different kinds of memory (RW, RO, RX, DMA, ...). But they all share a single address space, and the same instructions are used to access all of it, and that is why the Rust type system is sufficient to handle this.

[jamesmunns]

I think I sort of agree with you @RalfJung, but not entirely.

I do think that I need to be careful combining these issues, visibility of different pointer types (within a single PROCESS, rather than a single Thread), is definitely not overlapping with my main thrust regarding send and sync. I agree with you that this is a totally separate concern, unless we are talking about threads with divergent resources (which IMO shouldn't be 'modeled' as threads, but rather as separate processes, also being cavalier with language choice).

I have somewhat different feelings about this part:

On its own that is not different from read-only memory in an MMU system, is it? Even in "normal" x86 systems, there are very different kinds of memory (RW, RO, RX, DMA, ...). But they all share a single address space, and the same instructions are used to access all of it, and that is why the Rust type system is sufficient to handle this.

I agree the MMU analogy is correctly applied here. However, there are certainly a lot of operational/environmental concerns that can cause the program to act in unexpected ways (according to Rust), including doing things like writing to read-only memory, which will trigger a hardware exception. This comes down to the "environment" (OS or Hardware) not fulfilling the expectations the programming language has about the "environment".

Somewhat facetiously, this code trips over the same problem, even on x86 (though for slightly different reasons):

use core::sync::atomic::{AtomicUsize, Ordering};

#[link_section = ".rodata"] // Read-only mapped memory
static BAD: AtomicUsize = AtomicUsize::new(0);

fn main() {
    BAD.store(42, Ordering::SeqCst); // this segfaults
}

I don't know (yet) if this example is actually relevant, or just a "fun fact" that the environment can bite you in ways the compiler can't predict (if this was in the "correct" linker section, it would work). This application (ab-)uses the linker script to break a contract of Rust: mutable data shall be placed in a region where the data can be mutated (typically .data).

[RalfJung]

Somewhat facetiously, this code trips over the same problem, even on x86 (though for slightly different reasons):

That code is simply UB. When you have data that the Rust Abstract Machine considers mutable, the compiler will by default ensure that it truly is mutable; if you use link_section to override the compiler, you are violating some fundamental language assumptions and leaving the operational range that Rust promises to work in.

[jamesmunns]

@RalfJung this is getting off-topic for Send and Sync, but still relevant to my blog post. I'd be happy to follow up in jamesmunns/lab-notebook#3, if you're interested.

My comment on this is that "fundamental language assumptions" and "the operational range that Rust promises to work in" is exactly the sort of thing I am trying to capture, when I talk about the "environment". This environment includes everything that happens/exists after code leaves rustc, including linking, the placement on hardware, the operational state of hardware, etc.

I don't expect Rust as a language to help with this (similarly to how I don't expect Rust to help with managing multiple processes), because it's not a language/compiler concern, but rather an environmental/systems concern.

Whenever you leave the scope of "a single program running in a single process", you cross out of "rust" land, and into "environment" land, and Rust has preconditions it expects of it's environment to function correctly. You CAN use Rust to send IPC messages, but it's on a library/programmer to impedance-match the environment to Rust's expectations. You CAN use Rust to do FFI, but it's on a library/programmer to impedance-match the environment to Rust's expectations.

The thrust here is that Send and Sync are Rust preconditions, and they can't help you (by themselves) with problems that cross the Rust-to-environment barrier (but you can build libraries that use Send and Sync to help enforce part of this impedance-matching requirement).

Sorry for the off topic rant, I'd be happy to discuss the broader/semi-overlapping scope in my blog post issue.

[RalfJung]

My comment on this is that "fundamental language assumptions" and "the operational range that Rust promises to work in" is exactly the sort of thing I am trying to capture, when I talk about the "environment". This environment includes everything that happens/exists after code leaves rustc, including linking, the placement on hardware, the operational state of hardware, etc.

That makes a lot of sense, thanks for clarifying. :) Indeed, precisely documenting these assumptions will be an important part of a proper specification of the Rust Abstract Machine -- or rather, it will be part of how rustc implements that Abstract Machine.