RFC: move implementation of dlopen into Julia

[vtjnash]

This should give expanded flexibility, such as allowing for run-time search paths (currently named Base.DL_LOAD_PATH), should allow issue #949 to be implemented entirely in Julia, etc.

[vtjnash]

I put include("dlload.jl") here, so that it will be available for loading PCRE, but is after the very useful, basic string manipulation functions

[StefanKarpinski]

I dig it. Anything that can be moved into Julia probably should be. The only exception is the function to fork and exec a child process, which I've actually been meaning to move into C for some time now. But that's for performance reasons: in C you can make sure that you don't stomp on the heap at all, which boots the fork/exec out of the "fast lane" in the kernel. Of course, @JeffBezanson should take a look at it too.

[JeffBezanson]

Looks good, but there is a big wrinkle. The native dynamic linker is worse than this, and does not have the flexibility to specify libraries per-symbol AFAIK. It only supports symbol names and a global list of libraries to search. I was planning to move towards generating native object files, which would require us to go along with this limitation. We can probably specify libraries with absolute paths, in which case this code could be used to help find those paths. But then objects might have to be compiled on the deployment machine (or a clone of it), which may or may not be ok.

If we want to keep our current linking model, we'd have to do our own symbol resolution by emitting a global variable for each symbol, looking up the address of each at startup, and doing indirect calls through those globals. This strikes me as probably not a good idea.

[ViralBShah]

@JeffBezanson I couldn't understand what you said. Could you elaborate on "current linking model"? Is it possible to have something midway between what we currently have, and what @vtjnash has provided, so that there is more flexibility in the way we locate and use shared libraries?

[vtjnash]

Jeff, I couldn't follow you either. Are you looking for RTLD_DEEPBIND, which was suggested and merged in a pull request a while back?

[JeffBezanson]

The point is: we cannot continue using dlopen/dlsym in ccall. This results in call sites specialized for addresses in the current process. To generate object code, ccall needs to see the name of the symbol, and then it will be resolved by the dynamic linker as for any executable or shared object.

So if you really want to call dlopen, this code is perfectly fine. But the compiler will have to use some different mechanism. And we will need some declarative way to specify library dependencies, perhaps as part of a module preamble (along with imports and exports).

[ViralBShah]

Ok, I understand now. Calling dlopen in ccall is quite handy though, the way it is, and a more powerful dlopen would certainly be nice to have for interactive development, and writing one liner ccalls.

Could we continue supporting things the way they are right now, while requiring the user to do extra work in case they want to compile?

[JeffBezanson]

I don't like the idea of having two mechanisms. The only advantage of what we do now is that it lets you specify which library each symbol comes from, avoiding collisions and working around the standard library search path. Is that what you like about it, or just the syntax?

[ViralBShah]

The syntax lends itself to doing ccall as one liners. However, it is also nice that you know exactly where the symbols are coming from.

Perhaps I have not fully understood it, but I worth that it would be too much boilerplate, if one had to define things in a module preamble to be able to use ccall. I feel that any kind of boilerplate is a pain in the exploratory stage, but is ok when you are ready to compile/distribute/etc.

-viral

On 20-Aug-2012, at 8:36 AM, Jeff Bezanson wrote:

I don't like the idea of having two mechanisms. The only advantage of what we do now is that it lets you specify which library each symbol comes from, avoiding collisions and working around the standard library search path. Is that what you like about it, or just the syntax?

—
Reply to this email directly or view it on GitHub.

[JeffBezanson]

Well, we can certainly have concise syntax either way. Stefan and I discussed making the syntax ccall((:func, :lib), ...).

[StefanKarpinski]

Yes, the idea was that you'd write one of the following:

ccall(:memcpy, ...)               # in the julia binary's main address space
ccall((:lgamma, :libfdm), ...)    # in the dynamically loaded library "libfdm.so"
ccall((:lgamma, :libfdm, 1), ...) # in the dynamically loaded library "libfdm.1.so"

That is no more verbose than using a variable called "libfdm" and it avoids defining all thes global libfoo variables which then have to be redefined after image generation (because libraries get mapped to different locations in memory).