Document how to use hub CLI on GitHub Actions

[fregante]

Apparently it should work now but it's not clear how to get there: #2149

Can you document it somewhere?

[mislav]

Good idea. I haven't used hub from within Actions yet, so I'm summoning @bdougie @Geertvdc @rtimush @CLowenthal-zmi @LRWeber in case any of you have anything to add (but feel free to ignore). Off the top of my head, these are the steps that we'd need to document:

1. First, your action needs to obtain the hub executable somehow in its environment.

   In your Dockerfile you could use a package manager such as apt-get install for Debian, but that is likely to get you an outdated version of hub.

   Another approach is to vendor hub in your project and set it up manually like demonstrated here LLNL/llnl.github.io@2046d13.

   There is also this, which looks promising but that I still need to look into: https://github.com/marketplace/actions/setup-hub

2. Second, hub needs the GITHUB_TOKEN environment variable. GitHub can generate and provide this token for your actions, but you need to pull in the secret manually in your steps: https://help.github.com/en/articles/virtual-environments-for-github-actions#github_token-secret

3. Third, you should set the GITHUB_USER environment variable to reference the actor for this action. It can simply be set to the owner of the current repo. Make hub easier to use from GitHub Actions #2256

[Geertvdc]

Hi @mislav, tnx for the mention. I'm the creator of the setup-hub action which installs the hub cli on your worker.

https://github.com/marketplace/actions/setup-hub

In that Readme it explains how to set it up. Basically you run the action to setup hub once once and afterwards you can just use it in your script steps.

If any questions on this action let me know (it was the first github action I created)

[LRWeber]

It may be worth noting; Since the recent GitHub Action workflow migration from HCL to YAML, we've been encountering authentication issues on our action's git push commands back to the repo.

As a Docker container based action, the sudden change in behavior seems odd. I'm not sure if this is due to a coincidental change in how the provided GITHUB_TOKEN works or something else we're now missing, but that's something to test / keep an eye on while authoring this documentation.

(If anyone already has a working solution under the new YAML system, any advice would be appreciated! I've already sent a message to GitHub support to see if they can shed any light on this issue, but haven't heard back as yet.)

[bdougie]

It may be worth noting; Since the recent GitHub Action workflow migration from HCL to YAML, we've been encountering authentication issues on our action's git push commands back to the repo.

@LRWeber is this what you are running into, Actions Event Restrictions?

cc/ @ethomson and @chrispat for awareness

[LRWeber]

It may be worth noting; Since the recent GitHub Action workflow migration from HCL to YAML, we've been encountering authentication issues on our action's git push commands back to the repo.

@LRWeber is this what you are running into, Actions Event Restrictions?

No, our issue is unrelated; We aren't trying to trigger anything else with the push from our action. Rather, we are just trying to get a push from our action to work at all!

The error message we've been seeing is from the script that runs in our Docker action:
fatal: could not read Username for 'https://github.com': No such device or address

This only started happening after we were forced to migrate our workflow from .github/main.workflow to .github/workflows/schedule.yml. Nothing within the Docker container was changed, which is why I suspect something related to the variables that get fed into it.

We have both GITHUB_TOKEN and GITHUB_USER set, per the instructions as @mislav noted above.

[rtimush]

@LRWeber I was able to solve this issue by creating a simple askpass.sh script and setting a GIT_ASKPASS environment variable action Dockerfile. Hope this helps.

[ethomson]

@LRWeber Thanks for the information. I'm not entirely sure what changed here - I wouldn't have thought that there was anything in the git authentication model that we were doing special for HCL-based workflows that we've stopped doing in YAML based workflows.

Independent of that, to get this working, you should be able to tell git to use the GITHUB_TOKEN.

@bryanmacfarlane understands some of the tricky bits we do with git auth better than I do, he may have insight here on how to both specify this and do so in a way that is optimal for keeping the secrets secret.

[bryanmacfarlane]

we've been encountering authentication issues on our action's git push commands back to the repo.

We have feature work in flight right now so git push back to the triggering repo will just work. It will put the token in the gitconfig for that url for the life of the job.

What we currently do is use the extra header with the GITHUB_TOKEN option added to git core. e.g.

git -c http.extraheader="AUTHORIZATION: basic ***" fetch ...

I was able to solve this issue by creating a simple askpass.sh script and setting a GIT_ASKPASS

That will work on hosted machines (single use) but be careful. We're about to release self hosted runners and remembering a token that's time bombed per run (which github token is) becomes problematic. The issues we've seen in CI are machines with cred managers where it asks you the first time but cred managers store and then the next run in a few hours on that machine fails because askpass isn't called.

That's why we worked with git core to add the extra header option. The bearer token is passed for that invocation and never stored and doesn't trigger cred managers. Which is what you want in CI with time bombed tokens (the opposite of what you want on a dev machine with a long lived token).

[LRWeber]

We have feature work in flight right now so git push back to the triggering repo will just work.

@bryanmacfarlane That's great news! It seems like a common use case, so that will be extremely helpful. If we don't make progress towards resolving our particular issue, we'll keep an eye out for that update.

What we currently do is use the extra header with the GITHUB_TOKEN option added to git core.

I tried changing our push command to:
git -c http.extraheader="AUTHORIZATION: basic ${GITHUB_TOKEN}" push --set-upstream origin $BRANCH_NAME
It still fails on that line, but the error message has changed:
fatal: unable to access 'https://github.com/LLNL/llnl.github.io.git/': The requested URL returned error: 400

We are tracking our specific issue here LLNL/llnl.github.io#43 (In case further discussion/troubleshooting would be more appropriate there instead!)

[mislav]

In #2264 it was pointed out to me that none of the hub operations that hit the API (which is most of them) will work from Actions unless we change the hub codebase to send the Accept: application/vnd.github.machine-man-preview+json header, since the OAuth token provided to Actions via the GITHUB_TOKEN header is technically a GitHub App token.

Does this match other people's experience when trying to run hub <command> in your Action, i.e. do you get HTTP 401 responses because this header isn't set? If so, we need to make another release that sets the correct Accept value when hub is run in the context of an Action.

[fregante]

The error message we've been seeing is from the script that runs in our Docker action:
fatal: could not read Username for 'https://github.com': No such device or address

I'm getting the same error with both hub and git when trying git push, it just means that the call is non authenticated. I ended up using git + HTTP auth embedded in the remote:

name: No banned git tags
on:
  push:
    tags:
      - hotfix
jobs:
  delete:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout repo
      uses: actions/checkout@v1
    - name: Delete extra tag
      env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      run: |
        git remote set-url origin "https://fregante:[email protected]/$GITHUB_REPOSITORY.git"
        git push --delete origin hotfix

EDIT: this is now available as an action: https://github.com/marketplace/actions/setup-git-token

[artificialsoph]

Just want to add that with hub version 2.12.4 I am still receiving the fatal: could not read Username for 'https://github.com': No such device or address error when I have set each GITHUB_USER, GITHUB_PASSWORD and GITHUB_TOKEN environment variables.

I was running to an extra annoying problem with an inner submodule which I was able to fix by applying @fregante's solution above. I've generalized that solution and added the lines for the submodule fix below:

git clone "https://$GITHUB_USER:[email protected]/$GITHUB_REPOSITORY.git" outer_repo
cd outer_repo
git config -f .gitmodules submodule.out.url "https://$GITHUB_USER:[email protected]/$GITHUB_SUBMODULE.git"
git submodule update --init --recursive

[mislav]

@fregante @artificialsoph @bryanmacfarlane I wonder if, instead of rewriting git clone URLs to stick a token in them, one could create a netrc file as such:

cat >>~/.netrc <<EOF
machine github.com
  login $GITHUB_ACTOR
  password $GITHUB_TOKEN
EOF

I haven't tried this, but there's a chance it might work and is probably cleaner.

However, it seems that most people on this thread don't really have an issue with hub per se (i.e. with commands such as hub api or hub pull-request), but with git cloning or pushing. If someone is successfully using hub from Actions, please feel encouraged to share what worked and what didn't work for you; thanks!

[fregante]

I had also tried adding -c http.extraheader= directly on the git push line but it required a not-very-readable base64'ed user+token.

I went through a few iterations and decided that adding a single line to re-set the remote was the simplest-looking one (1 LOC).

Now I kinda wish GitHub would add offer a trusted action that includes that origin setup without having to pass the token manually (e.g. action/checkout has implicit access to it)

[LRWeber]

Adding ${GITHUB_USER}:${GITHUB_TOKEN} directly to our git clone URL solved our push authentication problem! (Without using http.extraheader at all.)

It looks like our hub pull-request command that follows is working perfectly.
For the record, we still have GITHUB_USER set explicitly in order for our git clone command to use it.

[fregante]

Just to clarify, GITHUB_USER is not provided by GitHub and technically envs starting with GITHUB_ shouldn't even be allowed, according to the docs:

Note: GitHub reserves the GITHUB_ environment variable prefix for internal use by GitHub. Setting an environment variable or secret with the GITHUB_ prefix will result in an error.

That's why my workflow above hardcodes the repo owner's username: you still have to provide it in the workflow somewhere.

Alternatively, it can be extracted from $GITHUB_REPOSITORY itself (code not tested on GHA, only locally)

echo "https://${GITHUB_REPOSITORY%/*}:$GITHUB_TOKEN@github.com"  > ~/.git-credentials
git push

Docs:
https://git-scm.com/docs/git-credential-store#_storage_format
https://stackoverflow.com/a/4170409/288906

EDIT: this is now available as an action: https://github.com/marketplace/actions/setup-git-token

[LRWeber]

That's a good point. The new workflow authoring stopped enforcing that rule. Even when it did, we had been intentionally circumventing it to get hub working.

I updated our code again, and I'm happy to say that all of our git and hub commands are working fine without any GITHUB_USER variable set.

I am curious about the GITHUB_REPOSITORY solution; How does this affect an organization-owned repository using hub? Does the actual "username" value not matter? Or does that vary based on the specific command? Do organizations count as valid GitHub users in these cases?

[bryanmacfarlane]

I think most of these scenarios will happen to work now but when we offer self-hosted runners later this year there will be problems with solutions that trigger the git auth flows leading to things like apple git storing a time bombed token, not challenging for auth again on the next job. The -c extra header thing is ugly but avoids all those problems.

Also note that we have feature work that's almost done where push (to the triggering repo) will just work.

Haven't tried the netrc option.

[fregante]

Also note that we have feature work that's almost done where push (to the triggering repo) will just work.

The user still has to provide the token via env or with, right?

By the way, I packaged up the GIT credentials setup via ~/.netrc as an action:

    steps:
    - uses: actions/checkout@master
    - uses: fregante/setup-git-token@v1
      with:
        token: ${{ secrets.GITHUB_TOKEN }}
    - run: git push ... # Or hub push if you use a container with hub

Source at https://github.com/fregante/setup-git-token. No dependencies, just a handful of clear lines.

[bryanmacfarlane]

@DrSensor - can you create a separate issue for getting the cli on the images? mention me and I'll get it routed to the compute team that creates the images. Include your wget workaround. Thanks!

[peaceiris]

I requested adding hub to GitHub Actions virtual environments at actions/runner-images#1

[mislav]

@DrSensor Thank you for sharing your approach! I'm worried about downloading hub from a 3rd party: https://glare.now.sh. Right now, glare.now.sh might proxy to the latest release of hub from GitHub, but in the future it can get compromised and inject code into people's Actions, which could be catastrophic because an action (particularly ones that utilize hub) might have access to GITHUB_TOKEN.

For the interest of security and stability, I would much rather suggest that people download hub from GitHub's own servers and to pin to a specific version of hub. Future major hub releases might change how hub release operates (this is just my speculation; no such changes are planned yet) and you wouldn't want your actions to start failing just because hub had a new major release.

To download hub for Linux and place the executable into bin/hub of current directory (no need to install it system-wide just to be used in an action):

wget -q -O- \
  https://github.com/github/hub/releases/download/v2.12.8/hub-linux-amd64-2.12.8.tgz  | \
  tar xz --strip-components=1 --wildcards '*/bin/hub'

[Geertvdc]

@DrSensor and @mislav this is why i made the "setup-hub" Github Action to install the official hub cli from the latest release available.

This way it is super easy to use it and is in line with other dependencies you need for your builds. You can use it as follows:

steps:
- uses: actions/checkout@latest
- uses: geertvdc/setup-hub@master

- run: hub --version

more info here: https://github.com/Geertvdc/setup-hub. or marketplace link: https://github.com/marketplace/actions/setup-hub

Let me know if you have any feedback on this Action. Always open for it ❤️

[DrSensor]

I am actually a bit worried about GitHub action that is written in Javascript because there is a chance that the transitive dependencies might get compromised. That's why I'm using glare for temporary. Thanks @mislav for the input, I think I should host the proxy by myself or maybe use jobs.proxy.services in my workflow file. By the way, the proxy can also pin down the download to a specific version.

I hope published actions has something like manifest file for security permission.

[chrispat]

@DrSensor we do not pull dependencies at runtime by default so the potential for compromise with upstream dependencies for JavaScript is no different than any other packaging technology.

Of course, someone could choose to do that using a docker container or Dockerfile but there is no way we can gruard against that.

[fregante]

transitive dependencies might get compromised

In setup-hub both dependencies can/should be dropped:

• typed-rest-client just fetches a JSON, global.fetch should be enough
• semver is only used to validate the version, but hub does not necessarily follow semver

[mislav]

Here's a simple script to download hub for the current environment.

curl -fsSL https://gist.github.com/mislav/941a4edd3b63c9d9b07f7901b27e8b23.txt | bash -s 2.12.8
bin/hub <command>

It works on all Actions environments: Ubuntu, macOS, Windows.

I'm considering "promoting" this script to the github/hub repo so that the download URL for the script reflect our (hopefully trusted!) repo and doesn't go through any 3rd parties.

[bryanmacfarlane]

Note that the role of setup- actions is to ensure that all the actions configurations options just work by using a toolkit (e.g. proxies for self-hosted etc). It also ensures caching (self-hosted runners will only download once and continue to run other jobs.) and prepending the path for the next out of proc steps to reference as just hub. Auth also just works so the user doesn't have to map in the token.

The instructions here will download everytime correct?

Another option is to offer setup-hub under the actions org so there is full chain of trust if that's your concern. I think we can also make auth "just work" similar to our checkout action. Push just works without having the user map anything. I would be open to that and help out if @Geertvdc is interested.

Regarding security ...

Setup actions do need the http-client dependency. actions -> tool-cache -> http-client

The toolkit is maintained by github employees including the http client. Also note that recent releases has the http-client one we use completely under actions org locked down to GH employees.

As @chrispat mentioned, it's a build time dependency (not runtime) so the actions are self contained. Once built, someone compromising a dependency will not affect existing actions. The http-client has one tiny external dependency (tunnel) which we (github) have inspected a part of our security audits. Our CI also runs audits.

[mislav]

@bryanmacfarlane Thank you for the detailed explanation!

The instructions here will download everytime correct?

Correct. I thought this might be acceptable for casual usage for hub during an Action run.

Another option is to offer setup-hub under the actions org so there is full chain of trust if that's your concern. I think we can also make auth "just work" similar to our checkout action.

That would be terrific. Please feel free to loop me into that work. (I suspect that will happen in the actions org.) For now I'm closing this thread because people have enough resources and options to start using hub in their actions, but of course I'm always open to improvements re: stability & security.

After other options are available, I will amend the documentation.

[bryanmacfarlane]

@mislav - will do. Thanks!