Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RemoteSha being added to internally-create packages causing renv::status to fail #2014

Closed
fh-mthomson opened this issue Oct 11, 2024 · 11 comments

Comments

@fh-mthomson
Copy link
Contributor

I'm finding a subtle regression b/w renv 1.0.7 -> 1.0.10 that I suspect is related to RemoteSha changes discussed in #1961.

When running in:

  1. Workbench: renv::status() shows no issues
  2. GitLab CI: renv::status() returns
- The project is out-of-sync -- use `renv::status()` for details.
The following package(s) are out of sync [lockfile != library]:
# RSPM -----------------------------------------------------------------------
- searchr             [0.2.6: unchanged]
- myinternals         [0.4.0: unchanged]

When I downgrade to 1.0.7, then renv::snapshot() removes the RemoteSha entries for each, which allows the CI to again pass with renv::status() showing no issues.

Details:

  • Both Workbench/GitLab CI are using the same Docker image + version, and I've cleared the CI cache to confirm that re-installing (from binary) doesn't resolve the issue.
  • Each of searchr and myinternals (masked names for internal packages) are installed from tag-based git builders on Package Manager.
  • I'm not currently using pak with renv.
@fh-mthomson fh-mthomson changed the title RemoteSha being added to internally-create packages causing RemoteSha being added to internally-create packages causing renv::status to fail Oct 11, 2024
@kevinushey
Copy link
Collaborator

Any chance you can prepare a reproducible example, just so I can investigate more easily?

@fh-mthomson
Copy link
Contributor Author

Any chance you can prepare a reproducible example, just so I can investigate more easily?

Not easily, unfortunately, since it's only happening for private packages on our private PM instance.

Do you have a PM instance with git-backed packages that would be publicly accessible? That's the only way I could think of getting closer to something to help me give you more details.

If you prefer, I can submit a Posit Support ticket with more detailed logs, but figured I'd start here in case you have thoughts.

@kevinushey
Copy link
Collaborator

kevinushey commented Oct 11, 2024

Can you try running the following:

trace(renv:::renv_lockfile_diff_record, quote({
  if (identical(before$Package, "searchr")) {
    writeLines("Before:")  
    str(before)
    writeLines("")

    writeLines("After:")
    str(after)
    writeLines("")
  }
}), print = FALSE)

renv::status()

and let me know what output you see?

@fh-mthomson
Copy link
Contributor Author

fh-mthomson commented Oct 14, 2024

I got a bit closer to ~reproducible:

  1. configured the following git-builder on our dev PM:
rspm create git-builder --url=https://github.com/tidyverse/googledrive.git --source=OTHER_Source --build-trigger=tags
  1. Installed (avoiding the cache):
renv::install(
  "googledrive",
  repos = c(OTHER = "https://<internal PM>/OTHER/__linux__/jammy/latest"),
  rebuild = TRUE
)
  1. Created a lockfile - relevant chunk:
    "googledrive": {
      "Package": "googledrive",
      "Version": "2.1.1",
      "Source": "Repository",
      "Repository": "RSPM",
      "RemoteSha": "a33c4ddfbc68e399c6373771e689310791def85c",
      "Requirements": [
        "R",
        "cli",
        "gargle",
        "glue",
        "httr",
        "jsonlite",
        "lifecycle",
        "magrittr",
        "pillar",
        "purrr",
        "rlang",
        "tibble",
        "utils",
        "uuid",
        "vctrs",
        "withr"
      ],
      "Hash": "e99641edef03e2a5e87f0a0b1fcc97f4"
    },
full file (masking our dev URL)

{
  "R": {
    "Version": "4.1.3",
    "Repositories": [
      {
        "Name": "CRAN",
        "URL": "https://packagemanager.posit.co/cran/latest"
      },
      {
        "Name": "OTHER",
        "URL": "https://<internal>/OTHER/latest"
      }
    ]
  },
  "Packages": {
    "R6": {
      "Package": "R6",
      "Version": "2.5.1",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R"
      ],
      "Hash": "470851b6d5d0ac559e9d01bb352b4021"
    },
    "askpass": {
      "Package": "askpass",
      "Version": "1.2.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "sys"
      ],
      "Hash": "c39f4155b3ceb1a9a2799d700fbd4b6a"
    },
    "base64enc": {
      "Package": "base64enc",
      "Version": "0.1-3",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R"
      ],
      "Hash": "543776ae6848fde2f48ff3816d0628bc"
    },
    "bslib": {
      "Package": "bslib",
      "Version": "0.8.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R",
        "base64enc",
        "cachem",
        "fastmap",
        "grDevices",
        "htmltools",
        "jquerylib",
        "jsonlite",
        "lifecycle",
        "memoise",
        "mime",
        "rlang",
        "sass"
      ],
      "Hash": "b299c6741ca9746fb227debcb0f9fb6c"
    },
    "cachem": {
      "Package": "cachem",
      "Version": "1.1.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "fastmap",
        "rlang"
      ],
      "Hash": "cd9a672193789068eb5a2aad65a0dedf"
    },
    "cli": {
      "Package": "cli",
      "Version": "3.6.3",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "utils"
      ],
      "Hash": "b21916dd77a27642b447374a5d30ecf3"
    },
    "curl": {
      "Package": "curl",
      "Version": "5.2.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R"
      ],
      "Hash": "d91263322a58af798f6cf3b13fd56dde"
    },
    "digest": {
      "Package": "digest",
      "Version": "0.6.37",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "utils"
      ],
      "Hash": "33698c4b3127fc9f506654607fb73676"
    },
    "evaluate": {
      "Package": "evaluate",
      "Version": "1.0.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R"
      ],
      "Hash": "3fd29944b231036ad67c3edb32e02201"
    },
    "fansi": {
      "Package": "fansi",
      "Version": "1.0.6",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "grDevices",
        "utils"
      ],
      "Hash": "962174cf2aeb5b9eea581522286a911f"
    },
    "fastmap": {
      "Package": "fastmap",
      "Version": "1.2.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "aa5e1cd11c2d15497494c5292d7ffcc8"
    },
    "fontawesome": {
      "Package": "fontawesome",
      "Version": "0.5.2",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "htmltools",
        "rlang"
      ],
      "Hash": "c2efdd5f0bcd1ea861c2d4e2a883a67d"
    },
    "fs": {
      "Package": "fs",
      "Version": "1.6.4",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "methods"
      ],
      "Hash": "15aeb8c27f5ea5161f9f6a641fafd93a"
    },
    "gargle": {
      "Package": "gargle",
      "Version": "1.5.2",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "cli",
        "fs",
        "glue",
        "httr",
        "jsonlite",
        "lifecycle",
        "openssl",
        "rappdirs",
        "rlang",
        "stats",
        "utils",
        "withr"
      ],
      "Hash": "fc0b272e5847c58cd5da9b20eedbd026"
    },
    "glue": {
      "Package": "glue",
      "Version": "1.8.0",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R",
        "methods"
      ],
      "Hash": "5899f1eaa825580172bb56c08266f37c"
    },
    "googledrive": {
      "Package": "googledrive",
      "Version": "2.1.1",
      "Source": "Repository",
      "Repository": "RSPM",
      "RemoteSha": "a33c4ddfbc68e399c6373771e689310791def85c",
      "Requirements": [
        "R",
        "cli",
        "gargle",
        "glue",
        "httr",
        "jsonlite",
        "lifecycle",
        "magrittr",
        "pillar",
        "purrr",
        "rlang",
        "tibble",
        "utils",
        "uuid",
        "vctrs",
        "withr"
      ],
      "Hash": "e99641edef03e2a5e87f0a0b1fcc97f4"
    },
    "highr": {
      "Package": "highr",
      "Version": "0.11",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R",
        "xfun"
      ],
      "Hash": "d65ba49117ca223614f71b60d85b8ab7"
    },
    "htmltools": {
      "Package": "htmltools",
      "Version": "0.5.8.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R",
        "base64enc",
        "digest",
        "fastmap",
        "grDevices",
        "rlang",
        "utils"
      ],
      "Hash": "81d371a9cc60640e74e4ab6ac46dcedc"
    },
    "httr": {
      "Package": "httr",
      "Version": "1.4.7",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "R6",
        "curl",
        "jsonlite",
        "mime",
        "openssl"
      ],
      "Hash": "ac107251d9d9fd72f0ca8049988f1d7f"
    },
    "jquerylib": {
      "Package": "jquerylib",
      "Version": "0.1.4",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "htmltools"
      ],
      "Hash": "5aab57a3bd297eee1c1d862735972182"
    },
    "jsonlite": {
      "Package": "jsonlite",
      "Version": "1.8.9",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "methods"
      ],
      "Hash": "4e993b65c2c3ffbffce7bb3e2c6f832b"
    },
    "knitr": {
      "Package": "knitr",
      "Version": "1.48",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R",
        "evaluate",
        "highr",
        "methods",
        "tools",
        "xfun",
        "yaml"
      ],
      "Hash": "acf380f300c721da9fde7df115a5f86f"
    },
    "lifecycle": {
      "Package": "lifecycle",
      "Version": "1.0.4",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "cli",
        "glue",
        "rlang"
      ],
      "Hash": "b8552d117e1b808b09a832f589b79035"
    },
    "magrittr": {
      "Package": "magrittr",
      "Version": "2.0.3",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R"
      ],
      "Hash": "7ce2733a9826b3aeb1775d56fd305472"
    },
    "memoise": {
      "Package": "memoise",
      "Version": "2.0.1",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "cachem",
        "rlang"
      ],
      "Hash": "e2817ccf4a065c5d9d7f2cfbe7c1d78c"
    },
    "mime": {
      "Package": "mime",
      "Version": "0.12",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "tools"
      ],
      "Hash": "18e9c28c1d3ca1560ce30658b22ce104"
    },
    "openssl": {
      "Package": "openssl",
      "Version": "2.2.2",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "askpass"
      ],
      "Hash": "d413e0fef796c9401a4419485f709ca1"
    },
    "pillar": {
      "Package": "pillar",
      "Version": "1.9.0",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "cli",
        "fansi",
        "glue",
        "lifecycle",
        "rlang",
        "utf8",
        "utils",
        "vctrs"
      ],
      "Hash": "15da5a8412f317beeee6175fbc76f4bb"
    },
    "pkgconfig": {
      "Package": "pkgconfig",
      "Version": "2.0.3",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "utils"
      ],
      "Hash": "01f28d4278f15c76cddbea05899c5d6f"
    },
    "purrr": {
      "Package": "purrr",
      "Version": "1.0.2",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "cli",
        "lifecycle",
        "magrittr",
        "rlang",
        "vctrs"
      ],
      "Hash": "1cba04a4e9414bdefc9dcaa99649a8dc"
    },
    "rappdirs": {
      "Package": "rappdirs",
      "Version": "0.3.3",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R"
      ],
      "Hash": "5e3c5dc0b071b21fa128676560dbe94d"
    },
    "renv": {
      "Package": "renv",
      "Version": "1.0.10",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "utils"
      ],
      "Hash": "d0387d5687ec933dd7587efd4cfa2d85"
    },
    "rlang": {
      "Package": "rlang",
      "Version": "1.1.4",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R",
        "utils"
      ],
      "Hash": "3eec01f8b1dee337674b2e34ab1f9bc1"
    },
    "rmarkdown": {
      "Package": "rmarkdown",
      "Version": "2.28",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R",
        "bslib",
        "evaluate",
        "fontawesome",
        "htmltools",
        "jquerylib",
        "jsonlite",
        "knitr",
        "methods",
        "tinytex",
        "tools",
        "utils",
        "xfun",
        "yaml"
      ],
      "Hash": "062470668513dcda416927085ee9bdc7"
    },
    "sass": {
      "Package": "sass",
      "Version": "0.4.9",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R6",
        "fs",
        "htmltools",
        "rappdirs",
        "rlang"
      ],
      "Hash": "d53dbfddf695303ea4ad66f86e99b95d"
    },
    "sys": {
      "Package": "sys",
      "Version": "3.4.3",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "de342ebfebdbf40477d0758d05426646"
    },
    "tibble": {
      "Package": "tibble",
      "Version": "3.2.1",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "fansi",
        "lifecycle",
        "magrittr",
        "methods",
        "pillar",
        "pkgconfig",
        "rlang",
        "utils",
        "vctrs"
      ],
      "Hash": "a84e2cc86d07289b3b6f5069df7a004c"
    },
    "tinytex": {
      "Package": "tinytex",
      "Version": "0.53",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "xfun"
      ],
      "Hash": "9db859e8aabbb474293dde3097839420"
    },
    "utf8": {
      "Package": "utf8",
      "Version": "1.2.4",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R"
      ],
      "Hash": "62b65c52671e6665f803ff02954446e9"
    },
    "uuid": {
      "Package": "uuid",
      "Version": "1.2-1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R"
      ],
      "Hash": "34e965e62a41fcafb1ca60e9b142085b"
    },
    "vctrs": {
      "Package": "vctrs",
      "Version": "0.6.5",
      "Source": "Repository",
      "Repository": "RSPM",
      "Requirements": [
        "R",
        "cli",
        "glue",
        "lifecycle",
        "rlang"
      ],
      "Hash": "c03fa420630029418f7e6da3667aac4a"
    },
    "withr": {
      "Package": "withr",
      "Version": "3.0.1",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R",
        "grDevices",
        "graphics"
      ],
      "Hash": "07909200e8bbe90426fbfeb73e1e27aa"
    },
    "xfun": {
      "Package": "xfun",
      "Version": "0.48",
      "Source": "Repository",
      "Repository": "CRAN",
      "Requirements": [
        "R",
        "grDevices",
        "stats",
        "tools"
      ],
      "Hash": "89e455b87c84e227eb7f60a1b4e5fe1f"
    },
    "yaml": {
      "Package": "yaml",
      "Version": "2.3.10",
      "Source": "Repository",
      "Repository": "CRAN",
      "Hash": "51dab85c6c98e50a18d7551e9d49f76c"
    }
  }
}

  1. CI Error:
$ Rscript -e ' # collapsed multi-line command
- The project is out-of-sync -- use `renv::status()` for details.
[1] "hello"
Tracing function "renv_lockfile_diff_record" in package "renv
(not-exported)"
[1] "renv_lockfile_diff_record"
Before:
List of 7
 $ Package     : chr "googledrive"
 $ Version     : chr "2.1.1"
 $ Source      : chr "Repository"
 $ Repository  : chr "RSPM"
 $ RemoteSha   : chr "a33c4ddfbc68e399c6373771e689310791def85c"
 $ Requirements: chr [1:16] "R" "cli" "gargle" "glue" ...
 $ Hash        : chr "e99641edef03e2a5e87f0a0b1fcc97f4"
After:
List of 8
 $ Package     : chr "googledrive"
 $ Version     : chr "2.1.1"
 $ Source      : chr "Repository"
 $ Repository  : chr "RSPM"
 $ RemoteType  : chr "repository"
 $ RemoteSha   : chr "a33c4ddfbc68e399c6373771e689310791def85c"
 $ Requirements: chr [1:16] "R" "cli" "gargle" "glue" ...
 $ Hash        : chr "e135beaf7a1b46c08f9389c56af5df38"
The following package(s) are out of sync [lockfile != library]:
# RSPM -----------------------------------------------------------------------
- googledrive   [2.1.1: unchanged]
See `?renv::status` for advice on resolving these issues.
Error: renv_status$synchronized is not TRUE
Execution halted
GitLab CI config

default:
  image: rocker/tidyverse:4.1.3
  before_script:
  - |
    Rscript -e '
    install.packages("renv", repos = "https://packagemanager.posit.co/cran/__linux__/jammy/latest")
    renv::restore()
    '

stages:
  - test

lint:
  stage: test
  script:
  - |
    Rscript -e '
    print("hello")

    trace(
      renv:::renv_lockfile_diff_record,
      quote({
        if (identical(before$Package, "googledrive")) {
          writeLines("Before:")
          str(before)
          writeLines("")

          writeLines("After:")
          str(after)
          writeLines("")
        }
      }),
      print = FALSE
      )

    renv_status <- renv::status()
    stopifnot(renv_status$synchronized)
    '

For comparison, RemoteSha displayed in the lock file does align with the git-builder output, per the release: https://github.com/tidyverse/googledrive/releases/tag/v2.1.1

- googledrive (Trigger Type: tags)
    Transaction ID: 4634453
    Repository: https://github.com/tidyverse/googledrive.git
    SHA: a33c4ddfbc68e399c6373771e689310791def85c
    Tag: v2.1.1
    Package Version: 2.1.1
    Status: succeeded
    Time: Mon Oct 14 05:54:46 UTC 2024

Perhaps the issue is actually with Hash, not RemoteSha?

@kevinushey
Copy link
Collaborator

Sorry for the late response. I think the issue here is actually the differing remote types -- in particular, the second record has:

$ RemoteType : chr "repository"

I've pushed a change in 1c8c64c that might alleviate the issue; please let me know if it makes a difference.

@fh-mthomson
Copy link
Contributor Author

Thanks for taking a look! I just ran:

  1. renv::install('rstudio/renv')
  2. renv::snapshot()

Then pushed, and getting a different error trying to install via CI. Any suggestions?

# Bootstrapping renv 1.0.11.9000 [sha: b531b02] ------------------------------
- Downloading renv ... FAILED
Error in h(simpleError(msg, call)) : failed to download:
All download methods failed
Calls: source ... bootstrap -> withCallingHandlers -> renv_bootstrap_download

Relevant snippets (lmk if the full files would help):

renv/activate.R:

  # the requested version of renv
  version <- "1.0.11.9000"
  attr(version, "sha") <- "b531b0215fa8edcd80a05af22ec0f85a4dcb027a"

renv.lock:

    "renv": {
      "Package": "renv",
      "Version": "1.0.11.9000",
      "Source": "GitHub",
      "RemoteType": "github",
      "RemoteHost": "api.github.com",
      "RemoteUsername": "rstudio",
      "RemoteRepo": "renv",
      "RemoteRef": "main",
      "RemoteSha": "b531b0215fa8edcd80a05af22ec0f85a4dcb027a",
      "Requirements": [
        "utils"
      ],
      "Hash": "9002c894649542a4ca23e1e6f25f3f82"
    },

@kevinushey
Copy link
Collaborator

Hmm, I wasn't able to reproduce locally:

# Bootstrapping renv 1.0.11.9000 [sha: b531b02] ------------------------------
- Downloading renv ... OK
- Installing renv  ... OK

- Project '~/scratch/blah' loaded. [renv 1.0.11.9000; sha: b531b02]

Is it possible that your CI system doesn't have access to GitHub for some reason? (renv is trying to bootstrap itself by downloading the appropriate version of renv from GitHub, but that is apparently failing)

@fh-mthomson
Copy link
Contributor Author

fh-mthomson commented Nov 13, 2024

hmm looks to be a common issue with remotes - r-lib/remotes#641 (comment)

More minimal example, running this in CI fails:

$ Rscript -e 'remotes::install_github("rstudio/renv")'
Using bundled GitHub PAT. Please add your own PAT using `gitcreds::gitcreds_set()`
Error: Failed to install 'unknown package' from GitHub:
  HTTP error 401.
  Bad credentials
  Rate limit remaining: 59/60
  Rate limit reset at: 2024-11-14 00:22:12 UTC
  
Execution halted

I tried using pak via renv in GitLab CI:

default:
  image: rocker/tidyverse:4.4.2
  before_script:
  - |
    Rscript -e '
    pak::pak('rstudio/renv')
    options(renv.config.pak.enabled = TRUE)
    renv::restore()
    '

but, I think it's still failing from .Rprofile calling renv/activate.R to bootstrap install from GH.

Error:

# Bootstrapping renv 1.0.11.9000 [sha: b531b02] ------------------------------
- Downloading renv ... FAILED
Error in h(simpleError(msg, call)) : failed to download:
All download methods failed
Calls: source ... bootstrap -> withCallingHandlers -> renv_bootstrap_download
Execution halted

Any suggestions?

@fh-mthomson
Copy link
Contributor Author

Output of:

    Sys.getenv("GITHUB_TOKEN", unset = NA)
    Sys.getenv("GITHUB_PAT", unset = NA)
    Sys.getenv("GH_TOKEN", unset = NA)

Is:

[1] NA
[1] NA
[1] NA

Setting GITHUB_PAT manually in the CI config allows it to successfully bootstrap install!

# Bootstrapping renv 1.0.11.9000 [sha: b531b02] ------------------------------
- Downloading renv ... 
OK
- Installing renv  ... 
OK

(seems like a separate bug to investigate, since I wouldn't expect GITHUB_PAT to be necessary)

BUT the good news is that I confirmed the latest version of renv does resolve the original reported issue. thank you!!

@kevinushey
Copy link
Collaborator

BUT the good news is that I confirmed the latest version of renv does resolve the original reported issue. thank you!!

Phew, that's awesome to hear. Thanks for confirming!

@fh-mthomson
Copy link
Contributor Author

Thanks again for making this change in this release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants