Add OIDC authentication customizer for GitLab #1052

jhoward-lm · 2025-02-04T14:37:49Z

Description

This PR adds an implementation of the OidcAuthenticationCustomizer service provider interface specific to GitLab. It is meant to synchronize a user's projects and roles/max access levels per project within GitLab to a Hyades instance.

Depends on stevespringett/Alpine#755.

Note

WIP: a rough to-do list with outstanding items is in a comment block in GitLabSyncTask.java

Addressed Issue

Closes DependencyTrack/hyades#1632

Additional Details

This PR is a request for early review while still in development in order to change direction if necessary.

Checklist

I have read and understand the contributing guidelines
~~This PR fixes a defect, and I have provided tests to verify that the fix is effective~~
This PR implements an enhancement, and I have provided tests to verify that it works as intended
This PR introduces changes to the database model, and I have updated the migration changelog accordingly
This PR introduces new or alters existing behavior, and I have updated the documentation accordingly

jhoward-lm · 2025-02-04T14:38:34Z

Tagging for awareness:
@ashearin @lmphil @pkwiatkowski1 @Strakeln @jmayer-lm @EphraimEM

jhoward-lm · 2025-02-04T14:40:59Z

@nscuro Would you mind doing a preliminary review and letting me know if this implementation/direction needs adjustment? Thanks!

nscuro

Looks good so far. I think concurrency control of the task will become interesting.

The workflow orchestration functionality we're currently working on will provide the ability to serialize based on a concurrency group, similar to what GitHub Actions has. So this undertaking will become easier in the future, but until then we need another solution.

src/main/java/org/dependencytrack/integrations/gitlab/GitLabAuthenticationCustomizer.java

nscuro · 2025-02-04T15:43:26Z

src/main/java/org/dependencytrack/tasks/GitLabSyncTask.java

+
+import static org.dependencytrack.model.ConfigPropertyConstants.GITLAB_ENABLED;
+
+public class GitLabSyncTask implements LoggableSubscriber {


I presume we need a way to serialize execution of these tasks? If two or more users with overlapping GitLab project accesses login in close succession, we will encounter race conditions.

Before we go into the how (there are multiple options), can you think of any property/properties of the OIDC profile that can act as event key? i.e. what would you use as concurrency group?

Nothing in the OIDC profile comes to mind. The only thing I can think of is maybe the groups claim, if they can be locked individually, but that doesn't give us full GitLab project paths.

The name of each GitLab project as it is read from the GitLab API response can be used as well, but that isn't available in the profile

pom.xml

jhoward-lm · 2025-02-04T18:31:57Z

@nscuro We noticed that changes in the alpine schema aren't yet reflected in hyades or the apiserver, so we created a changeset that seems to work. The changeset is now pushed to this PR. Do we need to open a PR in hyades to update schema.sql or anything?

/cc @pkwiatkowski1

nscuro · 2025-02-05T10:03:18Z

The changeset is now pushed to this PR. Do we need to open a PR in hyades to update schema.sql or anything?

Leave it for now. We will add that migration in a separate PR anyway, since it involves a bit more non-SQL migration logic. You can keep the migration in your PR so you can test, but will likely need to rebase later down the road.

jhoward-lm · 2025-02-06T16:11:43Z

@nscuro Just wanted to see if you had any ideas on a couple of bugs we're seeing. It looks like the event is somehow firing off twice, and getting a weird error about classes not being found in the CLASSPATH. For example:

2025-02-06 16:06:00,793 INFO [GitLabSyncTask] Starting GitLab sync task
2025-02-06 16:06:00,801 INFO [GitLabSyncTask] Starting GitLab sync task
2025-02-06 16:06:00,822 INFO [GitLabSyncer] Creating project example-project
2025-02-06 16:06:00,823 INFO [GitLabSyncer] Creating project example-project
2025-02-06 16:06:00,832 ERROR [Schema] An exception was thrown while adding/validating class(es) :
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. 
Please check your specification and your CLASSPATH.
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. 
Please check your specification and your CLASSPATH.

I'll continue debugging, just seeing if anything stands out as immediately obvious to you. Thanks!

nscuro · 2025-02-06T16:20:51Z

@jhoward-lm For the redundant invokation, see my comment here: stevespringett/Alpine#755 (comment)

2025-02-06 16:06:00,832 ERROR [Schema] An exception was thrown while adding/validating class(es) :
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. 
Please check your specification and your CLASSPATH.
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. 
Please check your specification and your CLASSPATH.

Admittedly not sure, I'd have to checkout the code and see if I can reproduce it.

jhoward-lm · 2025-02-06T18:18:05Z

@nscuro Here is the full stack trace if it helps

Full error output

2025-02-06 12:15:25,284 ERROR [Schema] An exception was thrown while adding/validating class(es) : Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. Please check your specification and your CLASSPATH.
Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. Please check your specification and your CLASSPATH.
org.datanucleus.exceptions.ClassNotResolvedException: Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. Please check your specification and your CLASSPATH.
        at org.datanucleus.ClassLoaderResolverImpl.classForName(ClassLoaderResolverImpl.java:219)
        at org.datanucleus.ClassLoaderResolverImpl.classForName(ClassLoaderResolverImpl.java:347)
        at org.datanucleus.store.rdbms.mapping.MappingManagerImpl.getMappingType(MappingManagerImpl.java:291)
        at org.datanucleus.store.rdbms.mapping.MappingManagerImpl.getDefaultJavaTypeMapping(MappingManagerImpl.java:1348)
        at org.datanucleus.store.rdbms.mapping.MappingManagerImpl.getMappingClass(MappingManagerImpl.java:829)
        at org.datanucleus.store.rdbms.mapping.MappingManagerImpl.getMapping(MappingManagerImpl.java:631)
        at org.datanucleus.store.rdbms.table.ClassTable.manageMembers(ClassTable.java:648)
        at org.datanucleus.store.rdbms.table.ClassTable.manageClass(ClassTable.java:554)
        at org.datanucleus.store.rdbms.table.ClassTable.initializeForClass(ClassTable.java:1380)
        at org.datanucleus.store.rdbms.table.ClassTable.initialize(ClassTable.java:273)
        at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.initializeClassTables(RDBMSStoreManager.java:3460)
        at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:3083)
        at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:118)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.manageClasses(RDBMSStoreManager.java:1666)
        at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:670)
        at org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getStatementForCandidates(RDBMSQueryUtils.java:430)
        at org.datanucleus.store.rdbms.query.JDOQLQuery.compileQueryFull(JDOQLQuery.java:911)
        at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:377)
        at org.datanucleus.store.query.Query.executeQuery(Query.java:1965)
        at org.datanucleus.store.query.Query.executeWithMap(Query.java:1911)
        at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:437)
        at org.datanucleus.api.jdo.JDOQuery.executeResultUnique(JDOQuery.java:393)
        at org.dependencytrack.persistence.ProjectQueryManager.doesProjectExist(ProjectQueryManager.java:1301)
        at org.dependencytrack.persistence.QueryManager.doesProjectExist(QueryManager.java:607)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.createProjectStructure(GitLabSyncer.java:174)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.synchronize(GitLabSyncer.java:112)
        at org.dependencytrack.tasks.GitLabSyncTask.inform(GitLabSyncTask.java:78)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1717)
        at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1458)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2034)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)

2025-02-06 12:15:25,284 WARN [Query] Query for candidates of org.dependencytrack.model.Project and subclasses resulted in no possible candidates : Class "org.dependencytrack.model.Classifier" was not found in the CLASSPATH. Please check your specification and your CLASSPATH.
Exception in thread "ForkJoinPool-1-worker-1" javax.jdo.JDOUserException: Don't support result clause of count(this) with resultClass of java.lang.Long
        at org.datanucleus.api.jdo.JDOAdapter.getJDOExceptionForNucleusException(JDOAdapter.java:701)
        at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:456)
        at org.datanucleus.api.jdo.JDOQuery.executeResultUnique(JDOQuery.java:393)
        at org.dependencytrack.persistence.ProjectQueryManager.doesProjectExist(ProjectQueryManager.java:1301)
        at org.dependencytrack.persistence.QueryManager.doesProjectExist(QueryManager.java:607)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.createProjectStructure(GitLabSyncer.java:174)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.synchronize(GitLabSyncer.java:112)
        at org.dependencytrack.tasks.GitLabSyncTask.inform(GitLabSyncTask.java:78)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1717)
        at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1458)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2034)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)
NestedThrowablesStackTrace:
Don't support result clause of count(this) with resultClass of java.lang.Long
org.datanucleus.exceptions.NucleusUserException: Don't support result clause of count(this) with resultClass of java.lang.Long
        at org.datanucleus.store.rdbms.query.JDOQLQuery.compileInternal(JDOQLQuery.java:442)
        at org.datanucleus.store.query.Query.executeQuery(Query.java:1965)
        at org.datanucleus.store.query.Query.executeWithMap(Query.java:1911)
        at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:437)
        at org.datanucleus.api.jdo.JDOQuery.executeResultUnique(JDOQuery.java:393)
        at org.dependencytrack.persistence.ProjectQueryManager.doesProjectExist(ProjectQueryManager.java:1301)
        at org.dependencytrack.persistence.QueryManager.doesProjectExist(QueryManager.java:607)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.createProjectStructure(GitLabSyncer.java:174)
        at org.dependencytrack.integrations.gitlab.GitLabSyncer.synchronize(GitLabSyncer.java:112)
        at org.dependencytrack.tasks.GitLabSyncTask.inform(GitLabSyncTask.java:78)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:110)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1726)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.compute(ForkJoinTask.java:1717)
        at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1458)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2034)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)

jhoward-lm · 2025-02-06T19:01:43Z

@nscuro In src/main/resources/META-INF/persistence.xml, it looks like org.dependencytrack.model.Classifier is missing. Does it need to be added manually, or is this file generated from somewhere?

nscuro · 2025-02-06T19:49:37Z

Adding Classifier there should not be necessary, because Classifier is only ever stored as simple String.

persistence.xml only needs to list classes that are "persistence capable" (i.e. represent a table, for a lack of better terms).

So either there's a bug in the ORM, or perhaps you have some changes locally that are not included in this PR?

nscuro · 2025-02-07T11:23:37Z

src/main/java/org/dependencytrack/integrations/gitlab/GitLabSyncer.java

+            for (String group : toCreate) {
+                LOGGER.debug("Creating project " + group);
+
+                Project existingProject = qm.getProject(group, null);


I can't find any place where the qm being used here is set. The code should currently throw a NPE. Is there some code you haven't yet pushed?

Yes I have some local code that sets the qm and some other fixes. We think the use of UnblockedEvent might have something to do with the Classifier not being found. I'll push what I have shortly

Is the change here acceptable?

Signed-off-by: Jonathan Howard <[email protected]>

Signed-off-by: Patrick Kwiatkowski <[email protected]>

Signed-off-by: Jonathan Howard <[email protected]>

Signed-off-by: Patrick Kwiatkowski <[email protected]>

Signed-off-by: Jonathan Howard <[email protected]>

Signed-off-by: Allen Shearin <[email protected]>

Signed-off-by: Ephraim Mensah <[email protected]>

Signed-off-by: Allen Shearin <[email protected]>

Signed-off-by: Jonathan Howard <[email protected]>

Signed-off-by: Allen Shearin <[email protected]>

Signed-off-by: Jonathan Howard <[email protected]>

Signed-off-by: Allen Shearin <[email protected]>

Signed-off-by: Jonathan Howard <[email protected]>

…erver into gitlab-integration

Signed-off-by: Philippe <[email protected]>

Signed-off-by: Jonathan Howard <[email protected]>

nscuro reviewed Feb 4, 2025

View reviewed changes

nscuro reviewed Feb 7, 2025

View reviewed changes

jhoward-lm force-pushed the gitlab-integration branch 3 times, most recently from fa505bb to 6cad00c Compare February 13, 2025 19:56

jhoward-lm and others added 16 commits February 20, 2025 10:59

feat: initial GitLab integration (WIP)

ff4d92f

Signed-off-by: Jonathan Howard <[email protected]>

fix: added gitlab auth class

d5e2679

Signed-off-by: Patrick Kwiatkowski <[email protected]>

fixed pom merge issue

a1a1035

Signed-off-by: Jonathan Howard <[email protected]>

refactor: implement customizer, register event

b52c077

Signed-off-by: Jonathan Howard <[email protected]>

refactor: remove token class member from sync task

0c81541

Signed-off-by: Jonathan Howard <[email protected]>

refactor: make user available to syncer

8669568

Signed-off-by: Jonathan Howard <[email protected]>

fix: remove scheduled task registration

c44b77b

Signed-off-by: Jonathan Howard <[email protected]>

style: javadoc list

7b29e8b

Signed-off-by: Jonathan Howard <[email protected]>

refactor: use Event.dispatch, remove pom repository

ed4486e

Signed-off-by: Jonathan Howard <[email protected]>

chore: fix pom alpine-parent version and formatting

9d28cea

Signed-off-by: Jonathan Howard <[email protected]>

fix: add migration changeset

64ee9f0

Signed-off-by: Jonathan Howard <[email protected]>

added user as syncer class var

5844fbe

Signed-off-by: Patrick Kwiatkowski <[email protected]>

refactor: add GitLabRole enum

9637c34

Signed-off-by: Jonathan Howard <[email protected]>

fix: remove redundant superinterface

799ceb1

Signed-off-by: Jonathan Howard <[email protected]>

fix: ensure query manager instantiated

7ccf3ac

Signed-off-by: Jonathan Howard <[email protected]>

fix: implement base Event interface

68a1268

Signed-off-by: Jonathan Howard <[email protected]>

ashearin and others added 18 commits February 20, 2025 11:04

feat: add GitLab Project Class and GitLab graphql query functionality

0f65d81

Signed-off-by: Allen Shearin <[email protected]>

chore: formatting and visibility fixes

9c321c2

Signed-off-by: Allen Shearin <[email protected]>

add map permission to gitlab default roles

d8e6f19

Signed-off-by: Ephraim Mensah <[email protected]>

add GitlabClientTest file

11cec33

Signed-off-by: Ephraim Mensah <[email protected]>

modify permission to static member of the class

e056cfc

Signed-off-by: Ephraim Mensah <[email protected]>

supress warning on mapPermissionToRoles set to uncheck

5788010

Signed-off-by: Ephraim Mensah <[email protected]>

clean up remove unused import

c18b124

Signed-off-by: Ephraim Mensah <[email protected]>

add the planner role with permission

eaeef42

Signed-off-by: Ephraim Mensah <[email protected]>

chore: refine gitlabsyncer class variables

f8ff2d4

Signed-off-by: Allen Shearin <[email protected]>

refactor: flatten created project structure

f553daa

Signed-off-by: Jonathan Howard <[email protected]>

fix: add user to GitLab role teams

7b12ec0

Signed-off-by: Jonathan Howard <[email protected]>

tests: add gitlabsyncer tests

0c144fe

Signed-off-by: Allen Shearin <[email protected]>

style: use modern Java conventions

a53ca68

Signed-off-by: Jonathan Howard <[email protected]>

perf: exclude archived GitLab projects

47f4795

Signed-off-by: Jonathan Howard <[email protected]>

refactor: configurable GraphQL query inputs

4406575

Signed-off-by: Jonathan Howard <[email protected]>

chore: move baseURL initialization to gitlab client constructor

8c489de

Signed-off-by: Allen Shearin <[email protected]>

chore: remove duplicate accesstoken member

e5a18dc

Signed-off-by: Allen Shearin <[email protected]>

refactor: locks to manage concurrent project access

f952260

Signed-off-by: Jonathan Howard <[email protected]>

jhoward-lm force-pushed the gitlab-integration branch from d6ecd8c to f952260 Compare February 20, 2025 17:07

jhoward-lm and others added 3 commits February 24, 2025 09:49

Merge branch 'main' of https://github.com/DependencyTrack/hyades-apis…

4f2ede7

…erver into gitlab-integration

add getGitLabProjects unit test

1a0f052

Signed-off-by: Philippe <[email protected]>

refactor: remove GitLabProject name field

cc785c8

Signed-off-by: Jonathan Howard <[email protected]>

jhoward-lm force-pushed the gitlab-integration branch from 29f5a81 to cc785c8 Compare February 24, 2025 18:26

ashearin mentioned this pull request Feb 24, 2025

Add roles #1069

Draft

5 tasks

refactor: add GitLab integration constants

265b9a1

Signed-off-by: Jonathan Howard <[email protected]>

EphraimEM mentioned this pull request Feb 27, 2025

Implement GitLab Project Integration with Topics Management DependencyTrack/hyades-frontend#247

Draft

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add OIDC authentication customizer for GitLab #1052

Add OIDC authentication customizer for GitLab #1052

jhoward-lm commented Feb 4, 2025

jhoward-lm commented Feb 4, 2025

jhoward-lm commented Feb 4, 2025

nscuro left a comment

nscuro Feb 4, 2025

jhoward-lm Feb 4, 2025

jhoward-lm commented Feb 4, 2025

nscuro commented Feb 5, 2025

jhoward-lm commented Feb 6, 2025

nscuro commented Feb 6, 2025

jhoward-lm commented Feb 6, 2025

jhoward-lm commented Feb 6, 2025

nscuro commented Feb 6, 2025

nscuro Feb 7, 2025

jhoward-lm Feb 7, 2025

jhoward-lm Feb 7, 2025


		import static org.dependencytrack.model.ConfigPropertyConstants.GITLAB_ENABLED;

		public class GitLabSyncTask implements LoggableSubscriber {

Add OIDC authentication customizer for GitLab #1052

Are you sure you want to change the base?

Add OIDC authentication customizer for GitLab #1052

Conversation

jhoward-lm commented Feb 4, 2025

Description

Addressed Issue

Additional Details

Checklist

jhoward-lm commented Feb 4, 2025

jhoward-lm commented Feb 4, 2025

nscuro left a comment

Choose a reason for hiding this comment

nscuro Feb 4, 2025

Choose a reason for hiding this comment

jhoward-lm Feb 4, 2025

Choose a reason for hiding this comment

jhoward-lm commented Feb 4, 2025

nscuro commented Feb 5, 2025

jhoward-lm commented Feb 6, 2025

nscuro commented Feb 6, 2025

jhoward-lm commented Feb 6, 2025

jhoward-lm commented Feb 6, 2025

nscuro commented Feb 6, 2025

nscuro Feb 7, 2025

Choose a reason for hiding this comment

jhoward-lm Feb 7, 2025

Choose a reason for hiding this comment

jhoward-lm Feb 7, 2025

Choose a reason for hiding this comment