Do not ignore uninhabited types for function-call ABI purposes.

[zachs18]

Background

Currently, #[repr(transparent)] types containing uninhabited 1-ZSTs as the non-transparent-wrapped field do not uphold the ABI-compatibility guarantee implied by #[repr(transparent)].

This is because currently, uninhabited types are ignored for function call ABI purposes¹. While this is probably never an issue for arguments (since a function whose arguments are uninhabited can by definition never be called), this can be an issue for return types.

For an example where this can be an issue: consider a struct Foo(u64, u64, u64); that is returned by "invisible reference"², and consider a #[repr(transparent)] struct FooWrapper(Foo, Void); where Void is enum Void {}. FooWrapper is uninhabited, but it can still be used as the return type of a function that diverges by panicking. Consider such a function fn foo(x: u64) -> FooWrapper { panic!("{x}") }. Calling this function should panic and print the u64 we passed in. We can take a fn(u64) -> FooWrapper fn pointer to this function, and transmute it to fn(u64) -> Foo. By the current documentation of ABI-compatibility, this transmuted fn pointer should be safe to call and should panic and print the u64 passed in. However, this is not the case, instead a stack address is printed; x does not have the correct value because the caller and callee disagree on the calling convention (the caller passes the return value pointer in %rdi and passes x in %rsi, the callee expects x in %rdi and does not expect a return value pointer).

See rust-lang/rust#135802 for more info, and see this playground link for an example of this behavior.

Proposal

Do not consider the inhabitedness of a type for function call ABI purposes.

• Remove the rustc_abi::BackendRepr::Uninhabited variant
  • Instead calculate the BackendRepr of uninhabited types "normally" (as though they were not uninhabited "at the top level", but still considering inhabitedness of variants to determine enum layout, etc)
• Add an uninhabited: bool field to rustc_abi::LayoutData so inhabitedness of a LayoutData can still be queried when necessary (e.g. when determining if an enum variant needs a tag value allocated to it).

Preliminary implementation: rust-lang/rust@master...zachs18:rust:backend-repr-remove-uninhabited (still need to add tests)

This should not affect type layouts (size/align/field offset); this should only affect function call ABI, and only of uninhabited types.

Mentors or Reviewers

cc @RalfJung from rust-lang/rust#135802 (comment)

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
  • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
  • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
• Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Footnotes

1. i.e. they have BackendRepr::Uninhabited, which lowers to PassMode::Ignore in most cases. ↩

2. e.g., on the x86_64 SysV ABI, a type with class MEMORY, where "the caller provides space for the return value and passes the address of this storage in %rdi as if it were the first argument to the function. In effect, this address becomes a “hidden” first argument" (System V Application Binary Interface AMD64 Architecture Processor Supplement (With LP64 and ILP32 Programming Models) Version 1.0) ↩

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections to the proposal should be discussed on Zulip and formally registered here by adding a comment with the following syntax:

@rfcbot concern reason-for-concern 
<description of the concern> 

Concerns can be lifted with:

@rfcbot resolve reason-for-concern 

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler

[RalfJung]

@rustbot second

[apiraino]

@rustbot label -final-comment-period +major-change-accepted