Reimplement Git using libgit2

[kmsquire]

Right now, when a large number of packages are installed, updating is sloooowww. The bottleneck seems to be the frequent calls to git. It would be nice if Git were reimplemented using libgit2 (http://libgit2.github.com/, source at https://github.com/libgit2/libgit2).

The only possible issue concerns licensing: libgit2 is a GPL2, with a linking exception that states:

In addition to the permissions in the GNU General Public License,
the authors give you unlimited permission to link the compiled
version of this library into combinations with other programs,
and to distribute those combinations without any restriction
coming from the use of this file.

See: https://github.com/libgit2/libgit2/blob/development/COPYING

Is this permissive enough to include with Julia?

cc: @StefanKarpinski

[StefanKarpinski]

I believe that linking exception is sufficiently liberal although I wonder why they didn't use LGPL? This is definitely a good idea and it would be great if someone wants to tackle it. I've started to a couple of times, but it's a little daunting.

[anthgur]

I would like to attempt to do this. I have been reading about libgit2 and looking through implementations of it in other languages. It seems like it will be more straightforward considering how Julia can call C directly.

It looks like external dependencies are in deps/, so that's where the link to libgit2 would go if I understand correctly. I would then need to set up the Makefile to compile libgit2 along with the rest of the external dependencies.

This is where I'm not sure what to do.
I've read through the Julia manual and this seems to be the most relevant chapter to this endeavor:
Calling C and Fortran Code

How would I structure the implementation of the git commands? Could it be a pure Julia implementation rather than requiring extra C for bindings like Rugged?

I see that there is already a module for git in Julia's base: git.jl
Would I just create functions within that module that call libgit2 directly? That seems to be the most direct approach to me.

[Keno]

Yes, the idea is to have it be in pure Julia and provide a nice Julia-Level API on top of the raw C api. Probably the best way to go for you, would be to start working in a LibGit2.jl package outside of Base, an then once that matures, we can import it into Base eventually. Awesome that you're working on this!

[anthgur]

That actually makes a lot of sense, thanks for the advice! I'll look through the existing packages to find one that has an external dependency to guide myself. Are there any that you know of off of the top of your head?

[quinnj]

You can check out the ODBC package, which has been around a while now (in
terms of Julia packages!) there's also HDF5, NetCDF, and SQLite.

I'd be willing to help out as well.

-Jacob
On Oct 9, 2013 12:55 PM, "anthonyurena" [email protected] wrote:

That actually makes a lot of sense, thanks for the advice! I'll look
through the existing packages to find one that has an external dependency
to guide myself. Are there any that you know of off of the top of your head?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4158#issuecomment-25988134
.

[kmsquire]

Great! Glad you're taking this on.

It seems like it will be more straightforward considering how Julia can

call C directly.

Hopefully that will be the case. Some things can still be tricky,
especially passing C structs back and forth. If you have issues, the dev
list is a good resource.

It looks like external dependencies are in deps/https://github.com/JuliaLang/julia/tree/master/deps,
so that's where the link to libgit2 would go if I understand correctly. I
would then need to set up the Makefile to compile libgit2 along with the
rest of the external dependencies.

That's correct.

This is where I'm not sure what to do.
I've read through the Julia manual and this seems to be the most relevant
chapter to this endeavor:
Calling C and Fortran Codehttp://docs.julialang.org/en/release-0.1-0/manual/calling-c-and-fortran-code/

How would I structure the implementation of the git commands? Could it be
a pure Julia implementation rather than requiring extra C for bindings like
Rugged https://github.com/libgit2/rugged/tree/development/ext/rugged?

Again, that depends, but interfacing with most C libraries can be done
directly in Julia.

I see that there is already a module for git in Julia's base: git.jlhttps://github.com/anthonyurena/julia/blob/master/base/git.jl
Would I just create functions within that module that call libgit2
directly? That seems to be the most direct approach to me.

I would suggest creating a separate Git2 module in a separate file (say,
git2.jl).

Feel free to touch base here or the dev list if you have questions. Have
fun!

Kevin

[StefanKarpinski]

Yes, this is an excellent project. It will make Pkg like 100 times faster for many operations.

[Keno]

Other than the packages @karbarcca mentioned, GnuTLS, Nettle or ZMQ might be good guides as well.

[StefanKarpinski]

Now substantially easier as I have cut down the size of the Git module by quite a lot: 995138c.

[tkelman]

closing, being implemented by #11196