Enforce ownership in generated Swift code

[chinedufn]

Swift recently accepted a proposal for a consume operator.

The consume operator will let us declare that a function takes unique ownership of a type https://github.com/apple/swift-evolution/blob/main/proposals/0366-move-function.md?plain=1#L175-L184 .

Any attempts to use the type after it is consumed lead to a compile time error.

This new consume operator will make it possible for us to solve one of the memory safety issues described in the book

swift-bridge/book/src/safety/README.md

Lines 48 to 73 in d4f28b3

	#### Using an owned value after free
	Today, it is possible to pass ownership of a value from `Swift` to `Rust` and then
	try to use the value from `Swift`.
	This mean that a user can accidentally trigger undefined behavior.
	```rust
	#[swift_bridge::bridge]
	mod ffi {
	extern "Rust" {
	type MyOwnedType;
	fn drop(ty: MyOwnedType);
	}
	}
	```
	```swift
	let myOwnedType = MyOwnedType()
	drop(myOwnedType)
	// Undefined behavior since we no longer own this value.
	drop(myOwnedType)
	```

.

---

After the consume operator lands in stable Swift we can use it in our generated code to enforce ownership.

So, the following bridge module:

#[swift_bridge::bridge]
mod ffi {
    extern "Rust" {
        type Computer

        fn eat(&self, meal: Food);
        fn shutdown(self);
    }

    extern "Rust" {
        type Food;
    }
}

would generate Swift code along the lines of:

// Generated Swift (approximation)

class Computer {
    // Notice the `consume`.
    func eat(meal: consume Food) { /* */ }

    // Notice the `__consuming`, which consumes self.
    func __consuming shutdown() { /* */ }
}

I'm not sure when the consume operator will land in stable Swift, or how we can track its progress.

[chinedufn]

Swift 5.9 introduces some new ownership features. I haven't thought through what we can do with all of them yet. Linking to some resources here:

• Looks like a fairly comprehensive overview of some of the new ownership features https://www.hackingwithswift.com/articles/258/whats-new-in-swift-5-9

• A section of a WWDC talk that briefly discusses some ownership features https://developer.apple.com/videos/play/wwdc2023/10164/?time=1405

Some Early Thoughts

• We should investigate moving from using Swift classes as handles to Rust types to using Swift ~Copyable structs. This would remove an allocation, since instantiating a Swift class allocates memory but instantiating a Swift struct does not
  • Right now we use inheritance to allow owned Rust types to be passed to methods take that references. To support this behavior with structs we might need to generate a MyTypeProtocol MyTypeRefProtocol and MyTypeRefMutProtocol

    swift-bridge/book/src/bridge-module/opaque-types/README.md

    Lines 52 to 71 in 36565f2

    	## Owned, Ref and RefMut
    	When you define a type in an `extern "Rust"` block, three corresponding Swift classes get generated.
    	```swift
    	// Equivalent to `SomeType` in Rust
    	class SomeType: SomeTypeRefMut {
    	// ...
    	}
    	// Equivalent to `&mut SomeType` in Rust
    	class SomeTypeRefMut: SomeTypeRef {
    	// ...
    	}
    	// Equivalent to `&SomeType` in Rust
    	class SomeTypeRef {
    	// ...
    	}
    	```

    • We'll need to figure out whether or not there are any compile time implications to introducing these Swift protocols for every Rust type

[chinedufn]

More resources:

• https://github.com/apple/swift-evolution/blob/main/proposals/0390-noncopyable-structs-and-enums.md
• https://github.com/apple/swift-evolution/blob/main/proposals/0366-move-function.md
• https://github.com/apple/swift-evolution/blob/main/proposals/0377-parameter-ownership-modifiers.md

[chinedufn]

I think a good approach would be to introduce Swift compile-time enforced ownership support behind an attribute in the swift-bridge = "0.1.*", then after things stabilize we can remove the attribute, make Swift ownership the default behavior and remove our old runtime enforced ownership.
It's possible that this can be done without introducing any breaking changes. I'm not sure.

So, something like this would lead our code generators to emit Swift code that enforces ownership at compile time:

#[swift_bridge::bridge]
mod ffi {
    extern "Rust" {
        #[swift_bridge(__unstable__swift_compile_time_ownership)]
        type Computer

        fn eat(&self, meal: Food);
        fn shutdown(self);
    }
}

Then after things stabilize we'd remove the #[swift_bridge(__unstable__swift_compile_time_ownership)] attribute.

---

One approach would be to start by just diving in and writing some tests and implementations for some basic cases such as bridging a Rust struct that has a single primitive field, then using what we learn to get a better sense of the extent of the work that is needed and how to best approach it.

[NiwakaDev]

@chinedufn

One approach would be to start by just diving in and writing some tests and implementations for some basic cases such as bridging a Rust struct that has a single primitive field,

What is bridging a Rust struct? You mean opaque types?

[chinedufn]

Yeah I meant an opaque Rust type:

#[swift_bridge::bridge]
mod ffi {
    extern "Rust" {
        #[swift_bridge(__unstable__swift_compile_time_ownership)]
        type Computer
    }
}

struct Computer {
    field: u8
}

[chinedufn]

Swift recently accepted the "Noncopyable generics" proposal https://github.com/swiftlang/swift-evolution/blob/main/proposals/0427-noncopyable-generics.md

Once that lands we should have everything that we need to implement cross-language ownership.

[chinedufn]

Here's what I'm thinking in pseudocode.

// Rust

#[swift_bridge::bridge]
mod ffi {
    extern "Rust" {
        #[swift_bridge(__unstable__swift_compile_time_ownership)]
        type Counter;

        fn get(&self);
        fn increment(&mut self);
        fn take(self) -> u32;

        fn give_me_counter_owned(counter: Counter);
        fn give_me_counter_ref(counter: &Counter);
        fn give_me_counter_ref_mut(counter: &mut Counter);
    }
}

struct Counter {
    count: u32
}

// Generated Swift code (ROUGH SKETCH)

// NEW: The owned type is `~Copyable`
struct Counter: ~Copyable {
    ptr: UnsafeMutableRawPointer

    consuming func take() {
        __swift_bridge__$Counter$take(ptr)
    }
}

struct CounterRefMut: ~Copyable {
    ptr: UnsafeMutableRawPointer
}

struct CounterRef {
    ptr: UnsafeMutableRawPointer
}



protocol __swift_bridge__CounterRef {
    var ptr: UnsafeMutableRawPointer

    func get() -> UInt32 {
        __swift_bridge__$Counter$get(self.ptr)
    }
}
protocol __swift_bridge__CounterRefMut {
    var ptr: UnsafeMutableRawPointer

    func increment() {
        __swift_bridge__$Counter$increment(self.ptr)
    }
}


extension CounterRef: __swift_bridge__CounterRef {}

extension CounterRefMut: __swift_bridge__CounterRef {}

extension Counter: __swift_bridge__CounterRef {}
extension Counter: __swift_bridge__CounterRefMut {}

func give_me_counter_owned(consuming counter: Counter) {
    __swift_bridge__give_me_counter_owned(counter.ptr)
}

func give_me_counter_ref(borrowing counter: __swift_bridge__CounterRef) {
    __swift_bridge__give_me_counter_ref(counter.ptr)
}

func give_me_counter_ref_mut(borrowing counter: __swift_bridge__CounterRefMut) {
    __swift_bridge__give_me_counter_ref_mut(counter.ptr)
}

// Using the Counter from Swift

func foo() {
    let counter = Counter();

    // 0
    counter.get()

    counter.increment()
    // 1
    counter.get()

    // Ok, takes a reference
    give_me_counter_ref(counter)
    // Ok, takes a mutable reference
    give_me_counter_ref_mut(counter)

    // `counter` ownership has now been transferred to Rust
    counter.take()

    // COMPILE TIME ERROR
    give_me_counter_owned(counter)
}

I'm planning to experiment with a design like this using Swift (without swift-bridge) to sketch out signatures and make sure that they compile.

Then if that works I can land something in swift-bridge.

Not sure when I'll work on this though. Could be months from now.

[chinedufn]

Alright I have a working demo of how we can support ownership.

I downloaded the August 19, 2024 release of Swift from https://www.swift.org/download/

Then I ran the following script:

#!/bin/bash

# Downloaded a Swift main branch build from:
#  https://www.swift.org/download/

# Found out about this `--toolchain` flag here:
#  https://www.swift.org/install/macos/package_installer/
xcrun --toolchain swift swiftc \
    experiment.swift

./experiment

With the following experiment.swift

struct Counter: ~Copyable {
    var ptr: UInt

    consuming func take() {
        // __swift_bridge__$Counter$take(ptr)
    }
}

struct CounterRefMut: ~Copyable {
    var ptr: UInt
}

struct CounterRef {
    var ptr: UInt
}



protocol __swift_bridge__CounterRef: ~Copyable {
    var ptr: UInt { get  }
}
extension __swift_bridge__CounterRef where Self: ~Copyable {
	borrowing func get() -> UInt32 {
        // __swift_bridge__$Counter$get(self.ptr)
        return 12345
    }
}

protocol __swift_bridge__CounterRefMut: ~Copyable {
    var ptr: UInt { get }
}
extension __swift_bridge__CounterRefMut where Self: ~Copyable {
	borrowing func increment() {
        // __swift_bridge__$Counter$increment(self.ptr)
    }
}


extension CounterRef: __swift_bridge__CounterRef {}

extension CounterRefMut: __swift_bridge__CounterRef {}
extension CounterRefMut: __swift_bridge__CounterRefMut {}

extension Counter: __swift_bridge__CounterRef {}
extension Counter: __swift_bridge__CounterRefMut {}

func give_me_counter_owned(_ counter: consuming Counter) {
    // __swift_bridge__give_me_counter_owned(counter.ptr)
}

func give_me_counter_ref<T: __swift_bridge__CounterRef & ~Copyable>(_ counter: borrowing T) {
    // __swift_bridge__give_me_counter_ref(counter.ptr)
}

func give_me_counter_ref_mut<T: __swift_bridge__CounterRefMut & ~Copyable>(_ counter: borrowing T) {
    // __swift_bridge__give_me_counter_ref_mut(counter.ptr)
}

func foo() {
    let counter = Counter(ptr: 5);

    // 0
    _ = counter.get()

    counter.increment()
    // 1
    _ = counter.get()

    // Ok, takes a reference
    give_me_counter_ref(counter)
    // Ok, takes a mutable reference
    give_me_counter_ref_mut(counter)

    // `counter` ownership has now been transferred to Rust
    counter.take()

    // COMPILE TIME ERROR
    give_me_counter_owned(consume counter)
}
foo()

And I get the following error:

/{REDACTED}/experiment.swift:60:9: error: 'counter' consumed more than once
58 |
59 | func foo() {
60 |     let counter = Counter(ptr: 5);
   |         `- error: 'counter' consumed more than once
61 |
62 |     // 0
   :
73 |
74 |     // `counter` ownership has now been transferred to Rust
75 |     counter.take()
   |     `- note: consumed here
76 |
77 |     // COMPILE TIME ERROR
78 |     give_me_counter_owned(consume counter)
   |                                   `- note: consumed again here
79 | }
80 | foo()

[NiwakaDev]

@chinedufn
Your design looks good. Can I implement this feature? But I think it’s quite a challenging implementation, so I need to consult with you along the way.

[chinedufn]

I'm planning to work on this when Swift 6 is released. Looks like there's already a 6.0 release branch, so hopefully we aren't too far away from Swift 6.

I want to feel out what's needed to seamlessly transition users between the old no-ownership codegen to the new codegen that will have ownership.
As well as how to gradually implement it over time in a way that lets people easily opt-in to using it for some types, so that it can start being used before we get everything working.
As well as considering what our Minimum Supported Swift Version #288 policy should be. As in, how long should we support Swift 5?

After I do enough work to figure some of that stuff out we can split up the remaining work.

[chinedufn]

Swift 6 has been released https://www.swift.org/blog/announcing-swift-6/

[chinedufn]

Hmmm, Noncopyable Swift types are less promising than they appeared.

You can't implement Error, Equatable, Hashable or any other protocols on them.
Noncopyable types can only implement Sendable.

In my own codebase I only have 2 functions that attempt to return an extern Rust type as the error
in Result<T, E>, and I have almost no extern "Rust" types whose generated Swift code needs to implement
Swift protocols, but other peoples' codebases might be different.

At the very least it looks like if we supported generated Noncopyable Swift handles for extern "Rust" opaque
types we would still want to also support the current class-based system since the current system does not
have these limitations around implementing protocols.

Perhaps something like (very rough sketch):

#[swift_bridge::bridge]
mod ffi {
    extern "Rust" {
        #[swift_bridge(swift_repr = noncopyable)]
        type Foo;

        #[swift_bridge(swift_repr = class)]
        type Bar;
    }
}

So, the user would control whether the Rust type got emitted as a Swift class or a Swift ~Copyable noncopyable struct.

[colinmarc]

As I outlined in #309 (comment), it's okay for those types to be ~Copyable as long as the borrow types (eg RustStrRef) are Copyable. The implementation of Equatable, Hashable, etc would go on the reference types. That exactly matches Rust semantics, but Rust lets you borrow implicitly at calltime.

[chinedufn]

Hey quick update: Haven't ready any of your recent comments yet I'm mostly off the grid for another week and a half then I'll get caught up so that we can unblock your Linux test suite work.

[colinmarc]

By the way, it seems like there is work planned to lift the limitation that ~Copyable types can't conform to protocols. From the "future work" section of this proposal (emphasis mine):

As noted above, this proposal leaves most standard protocols as is, deferring their generalizations to subsequent future work. The single protocol we do generalize is ExpressibleByNilLiteral -- the nil syntax is so closely associated with the Optional type that it would not have been reasonable to omit it.

This of course is not tenable; we expect that many (or even most) of our standard protocols will need to eventually get generalized for noncopyable use.

For some protocols, this work is relatively straightforward. For example, we expect that generalizing Equatable, Hashable and Comparable would not be much of a technical challenge -- however, it will involve overhauling/refining Equatable's semantic requirements, which I do not expect to be an easy process. (Equatable currently requires that "equality implies substitutability"; if the two equal instances happen to be noncopyable, such unqualified, absolute statements no longer seem tenable.) The RawRepresentable protocol is also in this category.

In other cases, the generalization fundamentally requires additional language enhancements. For example, we may want to consider allowing noncopyable Error types -- but that implies that we'll also want to throw and catch noncopyable errors, and that will require a bit more work than adding a ~Copyable clause on the protocol. It makes sense to defer generalizing the protocol until we decide to do this; if/when we do, the generalizations of Result can and should be part of the associated discussion and proposal. Another example is ExpressibleByArrayLiteral, which is currently built around an initializer with a variadic parameter -- to generalize it, we need to either figure out how to generalize those, or we need to design some alternative interface.