Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support repos that use LFS #929

Open
RothAndrew opened this issue Oct 18, 2022 · 4 comments
Open

Support repos that use LFS #929

RothAndrew opened this issue Oct 18, 2022 · 4 comments
Labels
enhancement ✨ New feature or request

Comments

@RothAndrew
Copy link
Contributor

Is your feature request related to a problem? Please describe.
An error is thrown if a repo is declared in zarf.yaml that uses LFS

Enumerating objects: 843, done.
Counting objects: 100% (843/843), done.
Delta compression using up to 8 threads
Compressing objects: 100% (793/793), done.
error: unable to rewind rpc post data - try increasing http.postBuffer
error: RPC failed; curl 56 LibreSSL SSL_read: Operation timed out, errno 60
send-pack: unexpected disconnect while reading sideband packet
Writing objects: 100% (842/842), 1.21 GiB | 1.28 MiB/s, done.
Total 842 (delta 109), reused 0 (delta 0), pack-reused 0
fatal: the remote end hung up unexpectedly

Describe the solution you'd like
I'd like the ability to add a repo to a zarf package that uses LFS, and have the repo land in Gitea with the LFS stuff working such that it respects my .gitattributes and continues to server files using LFS over in the air gap

Describe alternatives you've considered
Stripping the LFS and having super mega repo size

Additional context
#13 talked about LFS but it was specifically mentioning Git Bundle. Do we still use Git Bundle?

@RothAndrew
Copy link
Contributor Author

cc @brianrexrode pls provide any additional detail that you might have

@jeff-mccoy jeff-mccoy added enhancement ✨ New feature or request git labels Nov 16, 2022
@corang
Copy link
Contributor

corang commented Jan 19, 2024

@Racer159 bump

@corang
Copy link
Contributor

corang commented Jan 19, 2024

Example package:

kind: ZarfPackageConfig
metadata:
  name: repo-lfs-test

components:
  - name: lfs-test
    repos:
      - https://gitlab.com/defense-unicorns/image-intake

command to mirror to gitlab

zarf package mirror-resources zarf-package-repo-lfs-test-amd64.tar.zst \
        --git-url https://gitlab.com/ \
        --git-push-username <your username> \
        --git-push-password <you GL PAT> \
        -l trace

output when trying with my account:

  DEBUG   2024-01-19T19:51:54Z  -  Rewrite git URL: https://gitlab.com/defense-unicorns/image-intake -> https://gitlab.com//corang/image-intake-4124452048                          
└ (/home/runner/work/zarf/zarf/src/internal/packager/git/push.go:100)
  DEBUG   2024-01-19T19:51:54Z  -  Repo not yet available offline, skipping fetch...                                                                                                
└ (/home/runner/work/zarf/zarf/src/internal/packager/git/push.go:134)
  ⠼  GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all".

 WARNING  Retrying (1/3): failed to push the git repo "image-intake-740757006": unable to push repo
          to the gitops service: command error on refs/heads/lfs-test: pre-receive hook declined
  DEBUG   2024-01-19T19:52:02Z  -  Rewrite git URL: https://gitlab.com/defense-unicorns/image-intake -> https://gitlab.com//corang/image-intake-4124452048                          
└ (/home/runner/work/zarf/zarf/src/internal/packager/git/push.go:100)
  ⠹  Processing git repo https://gitlab.com/defense-unicorns/image-intake (0s)

 WARNING  Retrying (2/3): failed to push the git repo "image-intake-740757006": unable to fetch the
          git repo prior to push: remote repository is empty
  DEBUG   2024-01-19T19:52:07Z  -  Rewrite git URL: https://gitlab.com/defense-unicorns/image-intake -> https://gitlab.com//corang/image-intake-4124452048                          
└ (/home/runner/work/zarf/zarf/src/internal/packager/git/push.go:100)
  ⠹  Processing git repo https://gitlab.com/defense-unicorns/image-intake (0s)

 WARNING  Retrying (3/3): failed to push the git repo "image-intake-740757006": unable to fetch the
          git repo prior to push: remote repository is empty
  DEBUG   2024-01-19T19:52:13Z  -  unable to push the repos to the repository: unable to push repo https://gitlab.com/defense-unicorns/image-intake to the Git Server: failed to push the git repo "image-intake-740757006": unable to fetch the git repo prior to push: remote repository is empty
└ (/home/runner/work/zarf/zarf/src/pkg/message/message.go:192)
     ERROR:  Failed to deploy package: unable to push the repos to the repository: unable to push repo
             https://gitlab.com/defense-unicorns/image-intake to the Git Server: failed to push the git repo
             "image-intake-740757006": unable to fetch the git repo prior to push: remote repository is empty
└ (/home/runner/work/zarf/zarf/src/cmd/package.go:110)
  DEBUG   2024-01-19T19:52:13Z  -  goroutine 1 [running]:
          runtime/debug.Stack()
                /opt/hostedtoolcache/go/1.21.5/x64/src/runtime/debug/stack.go:24 +0x5e
          github.com/defenseunicorns/zarf/src/pkg/message.Fatal({0x3f57720?, 0xc0016d7320?}, {0xc0012f6b40, 0x11a})
                /home/runner/work/zarf/zarf/src/pkg/message/message.go:185 +0x14a
          github.com/defenseunicorns/zarf/src/pkg/message.Fatalf({0x3f57720, 0xc0016d7320}, {0x46b6236?, 0x0?}, {0xc001ac1c70?, 0x0?, 0x0?})
                /home/runner/work/zarf/zarf/src/pkg/message/message.go:192 +0x66
          github.com/defenseunicorns/zarf/src/cmd.glob..func23(0x7e4c9e0?, {0xc00097e7e0?, 0x4?, 0x463de10?})
                /home/runner/work/zarf/zarf/src/cmd/package.go:110 +0x115
          github.com/spf13/cobra.(*Command).execute(0x7e4c9e0, {0xc00097e750, 0x9, 0x9})
                /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:987 +0xaa3
          github.com/spf13/cobra.(*Command).ExecuteC(0x7e4d840)
                /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1115 +0x3ff
          github.com/spf13/cobra.(*Command).Execute(...)
                /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1039
          github.com/defenseunicorns/zarf/src/cmd.Execute()
                /home/runner/work/zarf/zarf/src/cmd/root.go:63 +0x1a
          main.main()
                /home/runner/work/zarf/zarf/main.go:24 +0x71
└ (/home/runner/work/zarf/zarf/src/pkg/message/message.go:192)

@corang
Copy link
Contributor

corang commented Jan 19, 2024

@MxNxPx This is why the image-intake repo started breaking, zarf doesn't support repos with LFS objects which the image-intake repo has

@Noxsios Noxsios moved this to Backlog in Zarf (old) Mar 8, 2024
@salaxander salaxander added this to Zarf Jul 22, 2024
@github-project-automation github-project-automation bot moved this to Backlog in Zarf Jul 22, 2024
@salaxander salaxander removed the status in Zarf Jul 22, 2024
@salaxander salaxander moved this to Triage in Zarf Sep 10, 2024
@schristoff schristoff removed the git label Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement ✨ New feature or request
Projects
Status: Triage
Development

No branches or pull requests

4 participants