Could loading a Rust dynamic library requires a (very up-to-date) special linker?

[robinmoussu]

When loading a dynamic library, the dynamic linker does some basics things like defining the address of the relocatable code, … For C and C++ those mechanisms are very basics, and standardized since a long time. This allow basically any library compiled with any compiler to be dynamically linked using any dynamic linker, without any regards to the versions of the various tools used.

If we could require a special linker for dynamically linking rust binaries, this would allow to apply custom logic during loading time. Requiring a (very) up-to-date Rust dynamic linker would also allow to change the ABI specification since the new linker would know what to do with it. However, this means that in order to start any binary using a Rust dynamic library, the user would need to have a compatible (ie. up-to-date) dynamic Rust linker.

I see 3 options:

• restrict ourself to the current capabilities of the current linkers.
• create a new linker, with added capabilities, then fix that new capability set in stone (so it would be possible to dynamically link a Rust library compiled in 2030 with a 2021 Rust dynamic linker).
• create a new linker, with added capabilities, and allow ourself to modify the set of capabilities in the future (for example for each Rust edition) which requires the user to update each time (ie. every 3 years with the current release cycle of Rust editions).

[bjorn3]

Implementing a dynamic linker is non-trivial. In addition things like TLS require special integration with the native libc, which may not even be possible outside of the libc version of the dynamic linker. (musl includes the dynamic linker in the libc itself. I don't know if the necessary functions are exported from libc and if it is allowed to use them if they are)

[ckaran]

I like the idea of creating a new linker. The issue is that we'd need both a linker and a loader, with the loader being the more difficult component from a purely politics/inertia point of view. Linkers are part of the compile-time toolchain; loaders need to be a part of the OS. To make that work, we'd need a highly portable loader, something that any operating system could integrate into their architecture with minimal effort. That will take significant amount of careful planning and design.

[bjorn3]

To make that work, we'd need a highly portable loader, something that any operating system could integrate into their architecture with minimal effort. That will take significant amount of careful planning and design.

That is impossible due to the tight integration between dynamic linker and libc. Traditionally both could easily be swapped, but over time they required more and more integration for features like TLS. Musl has completely given up on decoupling them and doesn't allow mixing different versions anymore.

[ckaran]

That is impossible due to the tight integration between dynamic linker and libc. Traditionally both could easily be swapped, but over time they required more and more integration for features like TLS. Musl has completely given up on decoupling them and doesn't allow mixing different versions anymore.

Hence the 'significant amount of careful planning and design.' 😉

I know that's flippant, but if we're thinking about this from the ground up, and what you're saying is a problem, then we need to think about how we could create a solution to it. Maybe the issue isn't just libc, but how libc is packaged. Perhaps a new ABI would give us more power. I honestly don't know if any of this is true, maybe it really is impossible. But before we give up, I'd like to figure out what the precise problems are, and document why they are impossible to solve, even if we rethink the issues from the ground up.

[carbotaniuman]

You'd need someone with the resources to drive a custom OS that wants to support this forward, have the influence to get it adopted as a starter. Same reason why Dart VM in Chrome never took traction, even though it "should" be better than JS. Designing a system with no backwards/forwards compat or without taking reality into account gets you something like GNU Hurd.

I realized that that paragraph may come off a little harsher than I intended, and I apologize - I don't mean to dismiss any ideas. What I was trying to say was that I think such a broad effort would be outside the scale and scope of this small, community-led project.

[ckaran]

No offense taken! I fully understand the issues involved. However, we need to start somewhere. The real trick is figuring out what makes this ABI design so much better than what is currently available that others want to use it. If there is some real 'killer feature', then others will adopt it on their own.

As for an OS, what about working with Redox to see what is possible/desirable? It's already a custom OS written in rust, so it could be a good place to experiment.

Other possible ways forwards:

• Write our own loader, and make it easy for other operating systems to adopt.
• Write a JIT that transpiles from our ABI format to the native format.
• Go with a full VM solution, at least until we figure out something better. Miri would be a good starting point here.

[bjorn3]

Go with a full VM solution, at least until we figure out something better. Miri would be a good starting point here.

Miri doesn't allow any interfacing with native code. It doesn't implement any ABI. Instead it just directly stores the arguments as regular locals in the callee frame. It is also not possible to change a dependency after compilation for miri, as rustc requires the SVH to match. Reworking Miri to allow for this is likely much more complex than the other options.

[ckaran]

Miri doesn't allow any interfacing with native code. It doesn't implement any ABI. Instead it just directly stores the arguments as regular locals in the callee frame. It is also not possible to change a dependency after compilation for miri, as rustc requires the SVH to match. Reworking Miri to allow for this is likely much more complex than the other options.

OK, in that case, some other VM entirely!

I'll fully admit, I'm more brain storming than anything else at the moment. I still don't know the exact bounds of what we're trying to accomplish here. Sometimes I think it's just a better calling convention, and other times I think we're trying to come up with the replacement for ELF. I think it'll be easier to really pin things down once we have a solid idea as to what the boundaries of the project are, which involves coming up with a solid, actionable list of goals.