Merge pull request #10972 from IQSS/10959-bearer-token-auth-ext

Handle unregistered users in BearerTokenAuthMechanism and implement user registration mechanism

doc/release-notes/10959-oidc-api-auth-ext.md

@@ -0,0 +1,14 @@
+Extends the OIDC API auth mechanism (available through feature flag ``api-bearer-auth``) to properly handle cases
+where ``BearerTokenAuthMechanism`` successfully validates the token but cannot identify any Dataverse user because there
+is no account associated with the token.
+
+To register a new user who has authenticated via an OIDC provider, a new endpoint has been
+implemented (``/users/register``). A feature flag named ``api-bearer-auth-provide-missing-claims`` has been implemented
+to allow
+sending missing user claims in the request JSON. This is useful when the identity provider does not supply the necessary
+claims. However, this flag will only be considered if the ``api-bearer-auth`` feature flag is enabled. If the latter is
+not enabled, the ``api-bearer-auth-provide-missing-claims`` flag will be ignored.
+
+A feature flag named ``api-bearer-auth-handle-tos-acceptance-in-idp`` has been implemented. When enabled, it specifies
+that Terms of Service acceptance is managed by the identity provider, eliminating the need to explicitly include the
+acceptance in the user registration request JSON.

doc/sphinx-guides/source/api/auth.rst

@@ -81,6 +81,29 @@ To test if bearer tokens are working, you can try something like the following (
   
   curl -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/users/:me
 
+To register a new user who has authenticated via an OIDC provider, the following endpoint should be used:
+
+.. code-block:: bash
+
+  curl  -H "Authorization: Bearer $TOKEN" -X POST http://localhost:8080/api/users/register --data '{"termsAccepted":true}'
+
+If the feature flag ``api-bearer-auth-handle-tos-acceptance-in-idp``` is disabled, it is essential to send a JSON that includes the property ``termsAccepted``` set to true, indicating that you accept the Terms of Use of the installation. Otherwise, you will not be able to create an account. However, if the feature flag is enabled, Terms of Service acceptance is handled by the identity provider, and it is no longer necessary to include the ``termsAccepted``` parameter in the JSON.
+
+In this JSON, we can also include the fields ``position`` or ``affiliation``, in the same way as when we register a user through the Dataverse UI. These fields are optional, and if not provided, they will be persisted as empty in Dataverse.
+
+There is another flag called ``api-bearer-auth-provide-missing-claims`` that can be enabled to allow sending missing user claims in the registration JSON. This is useful when the identity provider does not supply the necessary claims. However, this flag will only be considered if the ``api-bearer-auth`` feature flag is enabled. If the latter is not enabled, the ``api-bearer-auth-provide-missing-claims`` flag will be ignored.
+
+With the ``api-bearer-auth-provide-missing-claims`` feature flag enabled, you can include the following properties in the request JSON:
+
+- ``username``
+- ``firstName``
+- ``lastName``
+- ``emailAddress``
+
+If properties are provided in the JSON, but corresponding claims already exist in the identity provider, an error will be thrown, outlining the conflicting properties.
+
+This functionality is included under a feature flag because using it may introduce user impersonation issues, for example if the identity provider does not provide an email field and the user submits an email address they do not own.
+
 Signed URLs
 -----------
 

doc/sphinx-guides/source/installation/config.rst

@@ -3343,6 +3343,15 @@ please find all known feature flags below. Any of these flags can be activated u
     * - api-session-auth
       - Enables API authentication via session cookie (JSESSIONID). **Caution: Enabling this feature flag exposes the installation to CSRF risks!** We expect this feature flag to be temporary (only used by frontend developers, see `#9063 <https://github.com/IQSS/dataverse/issues/9063>`_) and for the feature to be removed in the future.
       - ``Off``
+    * - api-bearer-auth
+      - Enables API authentication via Bearer Token.
+      - ``Off``
+    * - api-bearer-auth-provide-missing-claims
+      - Enables sending missing user claims in the request JSON provided during OIDC user registration, when these claims are not returned by the identity provider and are required for registration. This feature only works when the feature flag ``api-bearer-auth`` is also enabled. **Caution: Enabling this feature flag exposes the installation to potential user impersonation issues.**
+      - ``Off``
+    * - api-bearer-auth-handle-tos-acceptance-in-idp
+      - Specifies that Terms of Service acceptance is handled by the IdP, eliminating the need to include ToS acceptance boolean parameter (termsAccepted) in the OIDC user registration request body. This feature only works when the feature flag ``api-bearer-auth`` is also enabled.
+      - ``Off``
     * - avoid-expensive-solr-join
       - Changes the way Solr queries are constructed for public content (published Collections, Datasets and Files). It removes a very expensive Solr join on all such documents, improving overall performance, especially for large instances under heavy load. Before this feature flag is enabled, the corresponding indexing feature (see next feature flag) must be turned on and a full reindex performed (otherwise public objects are not going to be shown in search results). See :doc:`/admin/solr-search-index`. 
       - ``Off``

docker-compose-dev.yml

@@ -17,6 +17,7 @@ services:
       SKIP_DEPLOY: "${SKIP_DEPLOY}"
       DATAVERSE_JSF_REFRESH_PERIOD: "1"
       DATAVERSE_FEATURE_API_BEARER_AUTH: "1"
+      DATAVERSE_FEATURE_API_BEARER_AUTH_PROVIDE_MISSING_CLAIMS: "1"
       DATAVERSE_MAIL_SYSTEM_EMAIL: "dataverse@localhost"
       DATAVERSE_MAIL_MTA_HOST: "smtp"
       DATAVERSE_AUTH_OIDC_ENABLED: "1"

src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java

@@ -14,14 +14,11 @@
 import edu.harvard.iq.dataverse.dataset.DatasetTypeServiceBean;
 import edu.harvard.iq.dataverse.engine.command.Command;
 import edu.harvard.iq.dataverse.engine.command.DataverseRequest;
-import edu.harvard.iq.dataverse.engine.command.exception.CommandException;
-import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException;
-import edu.harvard.iq.dataverse.engine.command.exception.PermissionException;
+import edu.harvard.iq.dataverse.engine.command.exception.*;
 import edu.harvard.iq.dataverse.engine.command.impl.GetDraftDatasetVersionCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetLatestAccessibleDatasetVersionCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetLatestPublishedDatasetVersionCommand;
 import edu.harvard.iq.dataverse.engine.command.impl.GetSpecificPublishedDatasetVersionCommand;
-import edu.harvard.iq.dataverse.engine.command.exception.RateLimitCommandException;
 import edu.harvard.iq.dataverse.externaltools.ExternalToolServiceBean;
 import edu.harvard.iq.dataverse.license.LicenseServiceBean;
 import edu.harvard.iq.dataverse.pidproviders.PidUtil;
@@ -56,6 +53,7 @@
 import java.net.URI;
 import java.util.Arrays;
 import java.util.Collections;
+import java.util.Map;
 import java.util.UUID;
 import java.util.concurrent.Callable;
 import java.util.logging.Level;
@@ -631,10 +629,22 @@ protected <T> T execCommand( Command<T> cmd ) throws WrappedResponse {
              * sometimes?) doesn't have much information in it:
              *
              * "User @jsmith is not permitted to perform requested action."
+             *
+             * Update (11/11/2024):
+             *
+             * An {@code isDetailedMessageRequired} flag has been added to {@code PermissionException} to selectively return more
+             * specific error messages when the generic message (e.g. "User :guest is not permitted to perform requested action")
+             * lacks sufficient context. This approach aims to provide valuable permission-related details in cases where it
+             * could help users better understand their permission issues without exposing unnecessary internal information.
              */
-            throw new WrappedResponse(error(Response.Status.UNAUTHORIZED,
-                                                    "User " + cmd.getRequest().getUser().getIdentifier() + " is not permitted to perform requested action.") );
-
+            if (ex.isDetailedMessageRequired()) {
+                throw new WrappedResponse(error(Response.Status.UNAUTHORIZED, ex.getMessage()));
+            } else {
+                throw new WrappedResponse(error(Response.Status.UNAUTHORIZED,
+                        "User " + cmd.getRequest().getUser().getIdentifier() + " is not permitted to perform requested action."));
+            }
+        } catch (InvalidFieldsCommandException ex) {
+            throw new WrappedResponse(ex, badRequest(ex.getMessage(), ex.getFieldErrors()));
         } catch (CommandException ex) {
             Logger.getLogger(AbstractApiBean.class.getName()).log(Level.SEVERE, "Error while executing command " + cmd, ex);
             throw new WrappedResponse(ex, error(Status.INTERNAL_SERVER_ERROR, ex.getMessage()));
@@ -809,6 +819,18 @@ protected Response badRequest( String msg ) {
         return error( Status.BAD_REQUEST, msg );
     }
 
+    protected Response badRequest(String msg, Map<String, String> fieldErrors) {
+        return Response.status(Status.BAD_REQUEST)
+                .entity(NullSafeJsonBuilder.jsonObjectBuilder()
+                        .add("status", ApiConstants.STATUS_ERROR)
+                        .add("message", msg)
+                        .add("fieldErrors", Json.createObjectBuilder(fieldErrors).build())
+                        .build()
+                )
+                .type(MediaType.APPLICATION_JSON_TYPE)
+                .build();
+    }
+
     protected Response forbidden( String msg ) {
         return error( Status.FORBIDDEN, msg );
     }

src/main/java/edu/harvard/iq/dataverse/api/Users.java

@@ -8,29 +8,33 @@
 import edu.harvard.iq.dataverse.api.auth.AuthRequired;
 import edu.harvard.iq.dataverse.authorization.users.ApiToken;
 import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser;
+import edu.harvard.iq.dataverse.authorization.users.GuestUser;
 import edu.harvard.iq.dataverse.authorization.users.User;
-import edu.harvard.iq.dataverse.engine.command.impl.ChangeUserIdentifierCommand;
-import edu.harvard.iq.dataverse.engine.command.impl.GetUserTracesCommand;
-import edu.harvard.iq.dataverse.engine.command.impl.MergeInAccountCommand;
-import edu.harvard.iq.dataverse.engine.command.impl.RevokeAllRolesCommand;
+import edu.harvard.iq.dataverse.engine.command.impl.*;
+import edu.harvard.iq.dataverse.settings.FeatureFlags;
+import edu.harvard.iq.dataverse.util.BundleUtil;
 import edu.harvard.iq.dataverse.util.FileUtil;
 
+import static edu.harvard.iq.dataverse.api.auth.AuthUtil.extractBearerTokenFromHeaderParam;
 import static edu.harvard.iq.dataverse.util.json.JsonPrinter.json;
 
+import java.text.MessageFormat;
 import java.util.Arrays;
 import java.util.List;
+import java.util.Optional;
 import java.util.logging.Level;
 import java.util.logging.Logger;
+
+import edu.harvard.iq.dataverse.util.json.JsonParseException;
+import edu.harvard.iq.dataverse.util.json.JsonUtil;
 import jakarta.ejb.Stateless;
 import jakarta.json.JsonArray;
+import jakarta.json.JsonObject;
 import jakarta.json.JsonObjectBuilder;
+import jakarta.json.stream.JsonParsingException;
 import jakarta.ws.rs.*;
 import jakarta.ws.rs.container.ContainerRequestContext;
-import jakarta.ws.rs.core.Context;
-import jakarta.ws.rs.core.MediaType;
-import jakarta.ws.rs.core.Request;
-import jakarta.ws.rs.core.Response;
-import jakarta.ws.rs.core.Variant;
+import jakarta.ws.rs.core.*;
 
 /**
  *
@@ -266,4 +270,24 @@ public Response getTracesElement(@Context ContainerRequestContext crc, @Context
         }
     }
 
+    @POST
+    @Path("register")
+    public Response registerOIDCUser(String body) {
+        if (!FeatureFlags.API_BEARER_AUTH.enabled()) {
+            return error(Response.Status.INTERNAL_SERVER_ERROR, BundleUtil.getStringFromBundle("users.api.errors.bearerAuthFeatureFlagDisabled"));
+        }
+        Optional<String> bearerToken = extractBearerTokenFromHeaderParam(httpRequest.getHeader(HttpHeaders.AUTHORIZATION));
+        if (bearerToken.isEmpty()) {
+            return error(Response.Status.BAD_REQUEST, BundleUtil.getStringFromBundle("users.api.errors.bearerTokenRequired"));
+        }
+        try {
+            JsonObject userJson = JsonUtil.getJsonObject(body);
+            execCommand(new RegisterOIDCUserCommand(createDataverseRequest(GuestUser.get()), bearerToken.get(), jsonParser().parseUserDTO(userJson)));
+        } catch (JsonParseException | JsonParsingException e) {
+            return error(Response.Status.BAD_REQUEST, MessageFormat.format(BundleUtil.getStringFromBundle("users.api.errors.jsonParseToUserDTO"), e.getMessage()));
+        } catch (WrappedResponse e) {
+            return e.getResponse();
+        }
+        return ok(BundleUtil.getStringFromBundle("users.api.userRegistered"));
+    }
 }

src/main/java/edu/harvard/iq/dataverse/api/auth/ApiKeyAuthMechanism.java

@@ -9,6 +9,7 @@
 
 import jakarta.inject.Inject;
 import jakarta.ws.rs.container.ContainerRequestContext;
+
 import java.util.logging.Logger;
 
 /**
@@ -49,7 +50,7 @@ public User findUserFromRequest(ContainerRequestContext containerRequestContext)
             authUser = userSvc.updateLastApiUseTime(authUser);
             return authUser;
         }
-        throw new WrappedAuthErrorResponse(RESPONSE_MESSAGE_BAD_API_KEY);
+        throw new WrappedUnauthorizedAuthErrorResponse(RESPONSE_MESSAGE_BAD_API_KEY);
     }
 
     private String getRequestApiKey(ContainerRequestContext containerRequestContext) {
@@ -59,15 +60,15 @@ private String getRequestApiKey(ContainerRequestContext containerRequestContext)
         return headerParamApiKey != null ? headerParamApiKey : queryParamApiKey;
     }
 
-    private void checkAnonymizedAccessToRequestPath(String requestPath, PrivateUrlUser privateUrlUser) throws WrappedAuthErrorResponse {
+    private void checkAnonymizedAccessToRequestPath(String requestPath, PrivateUrlUser privateUrlUser) throws WrappedUnauthorizedAuthErrorResponse {
         if (!privateUrlUser.hasAnonymizedAccess()) {
             return;
         }
         // For privateUrlUsers restricted to anonymized access, all api calls are off-limits except for those used in the UI
         // to download the file or image thumbs
         if (!(requestPath.startsWith(ACCESS_DATAFILE_PATH_PREFIX) && !requestPath.substring(ACCESS_DATAFILE_PATH_PREFIX.length()).contains("/"))) {
             logger.info("Anonymized access request for " + requestPath);
-            throw new WrappedAuthErrorResponse(RESPONSE_MESSAGE_BAD_API_KEY);
+            throw new WrappedUnauthorizedAuthErrorResponse(RESPONSE_MESSAGE_BAD_API_KEY);
         }
     }
 }

src/main/java/edu/harvard/iq/dataverse/api/auth/AuthUtil.java

@@ -0,0 +1,24 @@
+package edu.harvard.iq.dataverse.api.auth;
+
+import java.util.Optional;
+
+public class AuthUtil {
+
+    private static final String BEARER_AUTH_SCHEME = "Bearer";
+
+    /**
+     * Extracts the Bearer token from the provided HTTP Authorization header value.
+     * <p>
+     * Validates that the header value starts with the "Bearer" scheme as defined in RFC 6750.
+     * If the header is null, empty, or does not start with "Bearer ", an empty {@link Optional} is returned.
+     *
+     * @param headerParamBearerToken the raw HTTP Authorization header value containing the Bearer token
+     * @return An {@link Optional} containing the raw Bearer token if present and valid; otherwise, an empty {@link Optional}
+     */
+    public static Optional<String> extractBearerTokenFromHeaderParam(String headerParamBearerToken) {
+        if (headerParamBearerToken != null && headerParamBearerToken.toLowerCase().startsWith(BEARER_AUTH_SCHEME.toLowerCase() + " ")) {
+            return Optional.of(headerParamBearerToken);
+        }
+        return Optional.empty();
+    }
+}