How to make authenticated clones/fetches

[rsdy]

My goal would be to clone a private GitHub repository by running my own credential management system that's never stored externally.

After much tinkering, I've found the following way to make this possible:

use gix::{
    credentials::{helper::NextAction, protocol::Outcome},
    remote::{Connection, Direction},
    sec::identity::Account,
};

#[tokio::main]
async fn main() {
    let token = "github_personal_token";
    let clone = gix::prepare_clone_bare("https://github.com/user/private_repository", "/path/to/clone").unwrap();
    _ = clone
        .configure_remote(|remote| {
            let connect = remote
                .connect(Direction::Fetch, gix::progress::Discard)?
                .with_credentials(|action| match action {
                    gix::credentials::helper::Action::Get(ctx) => Ok(Some(Outcome {
                        identity: Account {
                            username: ctx.username.clone().unwrap_or("x-access-token".into()).clone(),
                            password: token.into(),
                        },
                        next: NextAction::from(ctx),
                    })),
                    gix::credentials::helper::Action::Store(_) => Ok(None),
                    gix::credentials::helper::Action::Erase(_) => Ok(None),
                });

	    let out = connect.remote().clone();
            _ = connect.prepare_fetch(Default::default())?.receive(&false.into())?;

	    Ok(out)
        }).persist();
}

This construct is not very optimal.

First, I have little way to influence error handling.
Second, it seems like unless I opt into using the *-async-std or blocking-* features, I'm not able to use the connect method at all, which seems counter-intuitive as reqwest should support tokio equally well.

After trying to find a better way, however, I've yet to find a good way to achieve what I want.

For comparison, the git2 I use currently looks like this, which allows me to detect errors gracefully, although it obviously also lacks async and interrupt support.

    let url = url.to_owned();
    let target = target.to_owned();

    tokio::task::spawn_blocking(move || {
        let options = {
            let mut callbacks = RemoteCallbacks::new();
            callbacks.credentials(auth);

            let mut fo = git2::FetchOptions::new();
            fo.remote_callbacks(callbacks);
            fo
        };

        let mut builder = git2::build::RepoBuilder::new();
        builder.fetch_options(options);
        builder.clone(&url, &target)
    })
    .await
    .expect("git failed")?;

    Ok(())

I would appreciate any advice on improving this code.

[Byron]

I recommend taking a look at the tests or at the implementation of gix clone for examples on how to use the API.

As for how the above can be improved, I recommend using fetch_then_checkout() instead of persist, while avoiding to fetch yourself in the configure_remote method which is meant to do only that.

The configuration of credentials is a bit cumbersome right now as there is no adapter to abstract the interface - right now it's built for credential helper integration. Error handling is possible even though custom errors are harder to deal with - it should be possible to improve that by working on the 'completely customized' authentication use-case. git2 is primarily an API which wants to hand off everything to the caller, whereas gix works like git by default, while allow the caller to override and configure everything, and it shows, for better or worse.

Async is implemented in a way to 'make it work', but inherently most of what's happening in the client is blocking. Async is clearly a second class citizen and goes only as far as there is demand for it. The implementation thus far was completely sponsored for instance and based on the needs of funded applications. I am open to seeing async support improved, but would love to see such improvements around becoming runtime agnostic instead of adding more. Maybe it won't take much to support tokio as well, I didn't research it. Contributions are welcome.

In the meantime, I recommend integrating gitoxide into tokio applications by means of spawn_blocking(|| …gitoxide…).

I hope that helps.

[rsdy]

One issue that I found trying to call fetch (or fetch_then_checkout) externally was that the connection's configuration doesn't survive the cloning of the remote(). So the with_credentials callback will not have any effect, and instead of my own configuration, it will just use the default, which prompts the user for the credentials.

[Byron]

I validated that within the clone machinery there is no clone happening, and if remote in configure_remote is returned as Ok(remote) after setting its authentication function, that will be effective when connecting.

[rsdy]

Unfortunately, I can't reproduce this. Is this what you mean?

use gix::{
    credentials::{helper::NextAction, protocol::Outcome},
    remote::{Connection, Direction},
    sec::identity::Account,
};

#[tokio::main]
async fn main() {
    let token = "github_personal_token";
    let clone = gix::prepare_clone_bare("https://github.com/user/private_repository", "/path/to/clone").unwrap();

    _ = clone
        .configure_remote(|remote| {
            _ = remote.connect(Direction::Fetch, gix::progress::Discard)?
                .with_credentials(|action| match action {
                    gix::credentials::helper::Action::Get(ctx) => Ok(Some(Outcome {
                        identity: Account {
                            username: ctx.username.clone().unwrap_or("x-access-token".into()).clone(),
                            password: token.into(),
                        },
                        next: NextAction::from(ctx),
                    })),
                    gix::credentials::helper::Action::Store(_) => Ok(None),
                    gix::credentials::helper::Action::Erase(_) => Ok(None),
                });

	    Ok(remote)
        }).fetch_only(gix::progress::Discard, &false.into()).unwrap();
}

Instead of using the provided credentials, this example still defaults to asking the user on stdin for an answer.

I really appreciate your responsiveness on this!

[Byron]

That's exactly what should work, but after looking once more I think I have found the culprit. I will be back with a fix.

Edit: I must be blind, it's clear that it won't work as the authentication configuration is done on the connection, but that's not accessible on the remote. Hence it can't work. The solution here is to make it possible to configure the connection.

[Byron]

Alright, if you look at the latest main you will find some (breaking) changes to support controlling the authentication when cloning.

It works like this (even though I didn't try to compile this code):

use gix::{
    credentials::{helper::NextAction, protocol::Outcome},
    remote::{Connection, Direction},
    sec::identity::Account,
};

#[tokio::main]
async fn main() {
    let token = "github_personal_token";
    let clone = gix::prepare_clone_bare("https://github.com/user/private_repository", "/path/to/clone").unwrap();

    _ = clone
        .configure_connection(|con| {
            con
                .set_credentials(|action| match action {
                    gix::credentials::helper::Action::Get(ctx) => Ok(Some(Outcome {
                        identity: Account {
                            username: ctx.username.clone().unwrap_or("x-access-token".into()).clone(),
                            password: token.into(),
                        },
                        next: NextAction::from(ctx),
                    })),
                    gix::credentials::helper::Action::Store(_) => Ok(None),
                    gix::credentials::helper::Action::Erase(_) => Ok(None),
                });

	    Ok(())
        }).fetch_only(gix::progress::Discard, &false.into()).unwrap();
}

Please let me know if this works for you, or if there are other issues that need ironing out.

By the way, the tool you are writing explicitly wants to control authentication, right? Because without override it would pick up git credentials, or store them and then pick them up just like git would, as it's the same system. The idea is that gix powered applications work just like git out of the box. I think what I am trying to understand better is your particular usecase :). Thanks for your consideration.

[rsdy]

Thanks for this, this resolves my issue perfectly. I really appreciate your work!

The reason I'm looking to control authentication to Git myself, is because the app I'm working on (https://bloop.ai) will manage Github OAuth tokens for itself, and I explicitly do not want to share these credentials with the user's existing contexts. This would allow the user to clean up the relevant folders and the authorization tokens would never interfere or linger around with the scopes they grant.

The motivation to use gix is because the pure Rust interfaces would hopefully give us a bit more control over the cloning process, and reduce our dependence on native libraries such as openssl.

[Byron]

Thanks for this, this resolves my issue perfectly. I really appreciate your work!

And thank you for the trust in gitoxide! I will do my best to make it work for you.

The motivation to use gix is because the pure Rust interfaces would hopefully give us a bit more control over the cloning process, and reduce our dependence on native libraries such as openssl.

Pure-Rust support is currently quite bare if compared to the curl backend and all of its configuration options, but the more the reqwest backend is used, the more it has a chance to evolve. @paulyoung can be thanked for its existence :).