Can’t fetch pipelines from merge requests because of mismatched ref
#280
Comments
|
Thank you for the speedy response! I did see #259 but it didn’t seem like the same thing. My mistake. I’m glad I helped you find the issue, at any rate. I updated to the latest Docker image (and confirmed that the digests match), but I’m afraid I’m still seeing the same issue: |
|
I don't think this is fixed still. See my comment - 83ca5f2#r51677228 |
|
I still think something isn't quite right.. with the latest change I get Before the patch I would get |
|
@mpickering, indeed this is following some breaking changes I am about to release as part of basically you will now need to set the following to get it to work with merge requests refs: project_defaults:
pull:
refs:
merge_requests:
enabled: true |
|
I tried making that change. Still the same result, here is the config. |
|
@mpickering can you give it a try with the following config? ---
server:
listen_address: ':9887'
webhook:
enabled: true
secret_token: <>
gitlab:
token: <>
url: 'https://gitlab.haskell.org'
projects_defaults:
pull:
refs:
branches:
regexp: .*
merge_requests:
enabled: true
pull:
environments_from_projects:
on_init: false
scheduled: false
metrics:
on_init: false
scheduled: false
projects_from_wildcards:
on_init: false
scheduled: false
refs_from_projects:
on_init: false
scheduled: false
wildcards: [{}] |
|
Still seems broken on all MRs for me. There are now no errors in the log but no metrics are populating the DB. |
hmm interesting, I cannot manage to reproduce 🤔 I added some further logging details in 366cd92 if you feel like trying it out. |
|
|
and you confirm the wildcard is a global one ( |
|
The issue was the config you gave me had |
|
ah right! my bad 🤦 yep that would be great 👍 |
|
I'm still seeing this issue on 0.5.2-39-g287e04c Here's a log from a webhook request: Here's my config.yml: Any progress on this bug? |
|
Hey @mvisonneau, I've dug through the code a bit to try to help. I think the issue is related to GitLab's weird refs related to merge-requests, and the implementation for fetching a pipeline based on the ref. I'm not experienced with Go, but here's how I understand the execution flow, what I suspect is the problem, and my proposed solution. Execution flow
{
"object_kind": "pipeline",
"object_attributes": {
"id": 1864,
"ref": "MR/usr/branchName",
"tag": false,
"sha": "1234567890",
"before_sha": "0987654321",
"source": "merge_request_event",
"status": "running",
"detailed_status": "running",
"stages": [
"prechecks",
"build",
"test"
],
"created_at": "2022-02-01 9:00:00 UTC",
"finished_at": null,
"duration": null,
"queued_duration": 9,
"variables": []
},
"merge_request": {
"id": 6302,
"iid": 3340,
"title": "Merge Request Title",
"source_branch": "MR/usr/branchName",
"source_project_id": 123,
"target_branch": "master",
"target_project_id": 123,
"state": "opened",
"merge_status": "can_be_merged",
"url": "https://gitlab_url/path/to/project/-/merge_requests/3340"
},
"user": {
"..."
},
"project": {
"id": 123,
"name": "project",
"...",
"path_with_namespace": "path/to/project",
"default_branch": "master",
"..."
},
"commit": {
"..."
}
},
"builds": [
{
"..."
ProblemIf you can't find the pipeline from the ref using the ListProjectPipelines() function, the code fails. I think the merge-request ref isn't pure git, so it behaves a bit strange. I tried manually querying the API using the merge-request ref and it worked, but I suspect something strange is happening with the goGitlab dependency. Ultimately I think using merge-request refs aren't a good idea. Proposed SolutionModify the condition on /pkg/controller/pipelines.go, lines 26-54. The idea is instead of reformatting the refName string, just call // We need to handle merge-request refs differently
if ref.Kind == schemas.RefKindMergeRequest {
// Get the pipeline directly using the pipeline ID from the Webhook payload
pipeline, err := c.Gitlab.GetRefPipeline(ref, pipelineIdFromWebhookPayload)
if err != nil {
return err
}
} else {
// Otherwise get the pipeline using the list approach
pipelines, _, err := c.Gitlab.GetProjectPipelines(ref.Project.Name, &goGitlab.ListProjectPipelinesOptions{
ListOptions: goGitlab.ListOptions{
PerPage: 1,
Page: 1,
},
Ref: ref.Name, // Don't need the refName value anymore.
})
{. . .} // handle errors
// Get pipeline from list
pipeline, err := c.Gitlab.GetRefPipeline(ref, pipelines[0].ID)
if err != nil {
return err
}
}I'm not sure if this would break functionality for the non-webhook data fetching or not. it seems like it would require some re-work for fetch events too. |
|
I’ve got my stack up and running again. Sorry for leaving this in a state of limbo for so long. Unfortunately, though @davidtlascelles’s detailed comments are extremely helpful, I had to move away from the webhook approach for unrelated reasons, and as such am unable to test either the current version or the proposed solution! |
I’m not sure whether GitLab changed something recently, in the run up to version 14, but I have the exporter set up with a Webhook and it’s unable to fetch any pipelines. For example (with my project details edited):
I did some debugging and what I found is that GitLab’s pipeline details (now?) use
reflike this:If you search for pipelines using the name of the branch, it returns nothing. If you use this ref, it does show up. However, it’s not possible to generate it from the Webhook request. Instead, you can take the
shafrom the Webhook request and retrieve pipelines based on that instead (by passing ashaparameter instead of arefparameter). I confirmed manually that it works. No matter what therefis, theshawill always match.The text was updated successfully, but these errors were encountered: