Auth Manager API part 4: RESTClient, HTTPClient #11992

adutra · 2025-01-17T16:43:51Z

4th PR for the Auth Manager API. Previous ones:

This PR introduces the required changes to RESTClient and HTTPClient. It also introduces a BaseHTTPClient abstract class to facilitate the creation and execution of HTTPRequests in a consistent way.

The biggest change in actually in TestRESTCatalog, because almost every Mockito.verify() call needs to be adapted.

The AuthManager API is still "unplugged" at this point.

\cc @nastra @danielcweeks

core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java

core/src/main/java/org/apache/iceberg/rest/HTTPClient.java

core/src/main/java/org/apache/iceberg/rest/BaseHTTPClient.java

danielcweeks · 2025-01-22T23:28:33Z

core/src/main/java/org/apache/iceberg/rest/HTTPClient.java

    this.mapper = objectMapper;
+    this.authSession = AuthSession.EMPTY;


I think defaulting to empty opens up a risk that requests will be made without an explicit session. This somewhat hides the case where we missed a call that should have auth. I would think we would leave this a null and require that a session be used (e.g. validate the auth was set in the execute call). Thoughts @nastra?

I think that makes sense to enforce/validate that a proper auth session is being used

We are going to run into a chicken and egg problem: to create an auth session I need an HTTP client; but now, to create an HTTP client, I would need an auth session.

I understand that may seem like the case here, but I think we're just saying that you create it initially as set to null, but validate when execute is called that it is not null. You can still create the HTTPClient, but you can't make a request unless there is an active auth session. This just prevents unintentional use of an "empty" auth session (it needs to be intentional).

Ah, now I get it. Let me try your suggestion.

core/src/main/java/org/apache/iceberg/rest/HTTPClient.java

nastra · 2025-01-28T13:03:15Z

aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java

-                              .build())));
+              id -> {
+                RESTClient client =
+                    httpClient().withAuthSession(org.apache.iceberg.rest.auth.AuthSession.EMPTY);


wouldn't this create a new http client every time this is called?

Indeed. The child client is lightweight though, but it's probably better to move this call to httpClient().

nastra · 2025-01-28T15:20:20Z

aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java

@@ -192,7 +192,8 @@ private RESTClient httpClient() {
              HTTPClient.builder(properties())
                  .uri(baseSignerUri())
                  .withObjectMapper(S3ObjectMapper.mapper())
-                  .build();
+                  .build()
+                  .withAuthSession(org.apache.iceberg.rest.auth.AuthSession.EMPTY);


maybe worth having a withAuthSession on the builder class itself(similar to withObjectMapper)

Builder method added.

nastra · 2025-01-28T15:20:48Z

aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java

@@ -338,6 +339,7 @@ public SdkHttpFullRequest sign(
      Consumer<Map<String, String>> responseHeadersConsumer = responseHeaders::putAll;
      S3SignResponse s3SignResponse =
          httpClient()
+              .withAuthSession(org.apache.iceberg.rest.auth.AuthSession.EMPTY)


I believe this change can be reverted now that the session is set in httpClient()

gaborkaszab · 2025-01-28T16:11:08Z

aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java

@@ -84,6 +85,7 @@ private RESTClient httpClient() {

  private LoadCredentialsResponse fetchCredentials() {
    return httpClient()
+        .withAuthSession(AuthSession.EMPTY)


I just took a quick look because I'm also working on the HttpClient code and was curious about this PR.

This might have been commented, but wouldn't it be possible to pass the AuthSession to the HttpClient during creation, maybe through a Builder? Then we could avoid adding these 'withAuthSession' calls when we use the httpClient.

Different sessions are used depending on the context, so it can't be restricted to a single one.

nastra · 2025-01-29T12:02:17Z

core/src/main/java/org/apache/iceberg/rest/HTTPClient.java

+      baseHeaders.forEach(allHeaders::putIfAbsent);
+    }
+
+    Preconditions.checkState(authSession != null, "no AuthSession available");


Suggested change

Preconditions.checkState(authSession != null, "no AuthSession available");

Preconditions.checkState(authSession != null, "Invalid auth session: null");

adutra · 2025-01-30T13:40:19Z

@danielcweeks @nastra is there anything else I need to change?

jbonofre · 2025-02-04T15:23:38Z

I see @nastra approved. I did a new pass and it looks good to me. @danielcweeks wdyt ?

danielcweeks · 2025-02-04T23:57:59Z

aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java

@@ -192,6 +192,7 @@ private RESTClient httpClient() {
              HTTPClient.builder(properties())
                  .uri(baseSignerUri())
                  .withObjectMapper(S3ObjectMapper.mapper())
+                  .withAuthSession(org.apache.iceberg.rest.auth.AuthSession.EMPTY)


Why are we setting this to EMPTY? In order to sign we need an auth session. Wouldn't this result in no headers/sigs being produced if a call was made without explicitly setting the auth session? We want enforce that an auth session is used, but this results in falling back to nothing.

I agree with your suggestions for other components, but here I disagree: the HTTP client must be configured with an empty auth session because it will be used for token refreshes. If it contains, say, token t1, and token t1 gets refreshed and exchanged for t2, there would be a clash between the token from the HTTP client (t1) and the one from the refreshed auth session (t2).

I will however move this call to the authSession() method and add a comment to explain why an EMPTY session is needed in this specific case.

danielcweeks · 2025-02-04T23:58:41Z

aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java

+          client =
+              HTTPClient.builder(properties)
+                  .uri(properties.get(URI))
+                  .withAuthSession(AuthSession.EMPTY)


Same here? The default should not be EMPTY, that just cases it to fall through.

danielcweeks · 2025-02-05T00:00:55Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

+    try (RESTClient initClient =
+        clientBuilder
+            .apply(props)
+            .withAuthSession(org.apache.iceberg.rest.auth.AuthSession.EMPTY)) {


Same here, we shouldn't default to EMPTY

danielcweeks · 2025-02-05T00:01:05Z

core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java

+    this.client =
+        clientBuilder
+            .apply(mergedProps)
+            .withAuthSession(org.apache.iceberg.rest.auth.AuthSession.EMPTY);


Here as well.

danielcweeks · 2025-02-05T00:06:44Z

I feel like there may have been a small understanding with how we handle the initial authSession. We don't want to default it to empty otherwise if someone adds a request like client.get(...) it will proceed without an auth session as opposed to getting an error because they didn't use the proper client.withAuthSession(session).get(...).

It's better to fail early here so that we know the issue was a lack of auth info as part of the request because we'll only get a 401/403 back without knowing that we just failed to apply auth.

danielcweeks · 2025-02-06T17:29:52Z

aws/src/main/java/org/apache/iceberg/aws/s3/VendedCredentialsProvider.java

@@ -75,10 +76,13 @@ private RESTClient httpClient() {
    if (null == client) {
      synchronized (this) {
        if (null == client) {
+          DefaultAuthSession authSession =
+              DefaultAuthSession.of(
+                  HTTPHeaders.of(OAuth2Util.authHeaders(properties.get(OAuth2Properties.TOKEN))));


@adutra It's beyond the scope of this PR since this is an existing problem, but somehow we need the AuthManager/AuthSession to be able to refresh OAuth tokens here as well. The token here will expire at which point the credential provider will not be able to get new vended credentials.

Yes. I don't mind looking into it once we have the whole API merged 👍

Auth Manager API part 4: RESTClient, HTTPClient

81b16de

github-actions bot added the core label Jan 17, 2025

adutra mentioned this pull request Jan 17, 2025

Auth Manager API part 5: SigV4 Auth Manager #11995

Merged

nastra reviewed Jan 20, 2025

View reviewed changes