feat: enable setting quota_project_id #1128

summer-ji-eng · 2020-06-15T08:03:08Z

As part of extensible client options, enable quota_project_id in GAPIC. So that client is able to set quota project ID, like "Settings.NewBuilder().SetQuotaProjectID(String)"

A quota project id might set up from 4 ways:

set up from setting directly: `settings.newBuilder().SetQuotaProjectID
set up from header provider
set up from internal header provider
set up from credential

Open Question:
1st priority of getting value is from SetQuotaProjectID, if it exists, would overwrite all value set from other ways.
So, what is the priority of other 3 ways, if SetQuotaProjectID is absent.
The current implementation is based on the order of calling the method. Let me know if you have any other preference. @chingor13

Integration Test: Tested in local java-TextToSpeech repo and using the local snapshot, test passes.

@Test
  public void quotaProjectIDIntegrationTest() throws IOException {
    final String QUOTA_PROJECT_ID_KEY = "x-google-user-project";
    final String QUOTA_PROJECT_ID = "test_quota_project_id";
    final String QUOTA_PORJECT_ID_HEADER_PROVIDER = "test_quota_project_id_from_header_provider";

    final HeaderProvider headerProvider = new HeaderProvider() {
      @Override
      public Map<String, String> getHeaders() {
        return Collections.singletonMap(QUOTA_PROJECT_ID_KEY, QUOTA_PORJECT_ID_HEADER_PROVIDER);
      }
    };

    TextToSpeechSettings settingsWithQuotaOnly = TextToSpeechSettings.newBuilder()
        .setTransportChannelProvider(channelProvider)
        .setCredentialsProvider(NoCredentialsProvider.create())
        .setQuotaProjectID(QUOTA_PROJECT_ID)
        .build();
    TextToSpeechClient clientWithQuotaOnly = TextToSpeechClient.create(settingsWithQuotaOnly);

    TextToSpeechSettings settingsWithAllSource = TextToSpeechSettings.newBuilder()
        .setTransportChannelProvider(channelProvider)
        .setHeaderProvider(headerProvider)
        .setQuotaProjectID(QUOTA_PROJECT_ID)
        .build();
    TextToSpeechClient clientWithAllSource = TextToSpeechClient.create(settingsWithAllSource);

    Assert.assertEquals(clientWithQuotaOnly.getSettings().getQuotaProjectID(), QUOTA_PROJECT_ID);
    Assert.assertEquals(clientWithAllSource.getSettings().getHeaderProvider().getHeaders().get(QUOTA_PROJECT_ID_KEY), QUOTA_PORJECT_ID_HEADER_PROVIDER);
    Assert.assertEquals(clientWithQuotaOnly.getSettings().getQuotaProjectID(), QUOTA_PROJECT_ID);
  }

codecov · 2020-06-15T08:03:14Z

Codecov Report

Merging #1128 into master will increase coverage by 0.41%.
The diff coverage is 92.00%.

@@             Coverage Diff              @@
##             master    #1128      +/-   ##
============================================
+ Coverage     78.65%   79.06%   +0.41%     
- Complexity     1170     1193      +23     
============================================
  Files           204      205       +1     
  Lines          5195     5264      +69     
  Branches        417      434      +17     
============================================
+ Hits           4086     4162      +76     
+ Misses          935      930       -5     
+ Partials        174      172       -2

Impacted Files	Coverage Δ	Complexity Δ
...main/java/com/google/api/gax/rpc/StubSettings.java	`75.32% <80.64%> (+1.34%)`	`12.00 <1.00> (+1.00)`
...ain/java/com/google/api/gax/rpc/ClientContext.java	`92.85% <100.00%> (+11.97%)`	`19.00 <6.00> (+10.00)`
...in/java/com/google/api/gax/rpc/ClientSettings.java	`93.75% <100.00%> (+0.50%)`	`13.00 <1.00> (+1.00)`
.../rpc/internal/QuotaProjectIdHidingCredentials.java	`100.00% <100.00%> (ø)`	`9.00 <9.00> (?)`
...le/api/gax/core/InstantiatingExecutorProvider.java	`100.00% <0.00%> (+37.50%)`	`5.00% <0.00%> (+2.00%)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ce0c30...7f91763. Read the comment docs.

bshaffer · 2020-06-17T19:10:57Z

gax/src/main/java/com/google/api/gax/rpc/StubSettings.java

      this.streamWatchdogProvider = settings.streamWatchdogProvider;
      this.streamWatchdogCheckInterval = settings.streamWatchdogCheckInterval;
      this.tracerFactory = settings.tracerFactory;
    }

+    /** Get Quota Project ID from Client Context * */
+    private static String getQuotaProjectIDFromClientContext(ClientContext clientContext) {


Seems like we use ID in methods and variables, and Id in class names. Is this idiomatic?

ID used in all methods and variables, which is really just my preference. I haven't find any guideline for naming.
Let me know if any rule need to be follow here, I could make change.

We should use quotaProjectId. See https://google.github.io/styleguide/javaguide.html#s5-naming

I will make change to quotaProjectId

bshaffer · 2020-06-17T19:13:27Z

gax/src/main/java/com/google/api/gax/rpc/StubSettings.java

+      if (clientContext.getHeaders().containsKey(QUOTA_PROJECT_ID_HEADER_KEY)) {
+        return clientContext.getHeaders().get(QUOTA_PROJECT_ID_HEADER_KEY);
+      }
+      if (clientContext.getInternalHeaders().containsKey(QUOTA_PROJECT_ID_HEADER_KEY)) {


What is the difference between headers and internal headers? And is it possible for quota project to come in on either one?

Also, what is the use case for pulling it from the header? I think maybe it is not necessary to detect this (@broady ?)

quota project id can be gotten from header and internal header. double confirm with @chingor13

but I am not sure the use case. @broady

I believe the internal header provider is reserved for gapic and handwritten layers to inject headers. The general header provider is to allow library users to specify arbitrary extra headers.

It's not desired for library users to set the quota project via header provider, but previously that was the only mechanism for doing so. We should have an explicit strategy for how to handle that case.

Thanks for clarifying.

It seems ok to support this then, for backwards compatibility. And also to ensure any quota project behavior stays consistent regardless of how it's set.

chingor13

Maybe it's just me, but I'm not seeing how the quota project id when configured from ClientSettings is actually making it into the request or connection.

summer-ji-eng · 2020-06-18T21:17:34Z

Maybe it's just me, but I'm not seeing how the quota project id when configured from ClientSettings is actually making it into the request or connection.

Can I have more detail about how do you use configure from ClientSetting. Try to reproduce on my side.
BTW, googleapis/java-texttospeech#176 is the draft PR to test using ClientSettings

chingor13 · 2020-06-18T22:05:25Z

Can I have more detail about how do you use configure from ClientSetting. Try to reproduce on my side.

Just the direct configuration method:

TextToSpeechSettings settingsWithAllSource = TextToSpeechSettings.newBuilder()
        .setQuotaProjectId(QUOTA_PROJECT_ID)
        .build();

Reading the code, I see how the quotaProjectId instance variable is populated on ClientContext, but not how that actually makes it to the gRPC connection or request as a header.

For example, the credentials authorization header is supplied by providing the credentials to the ApiCallContext which is used for setting up the channel and all options.

summer-ji-eng · 2020-06-30T06:55:52Z

@chingor13 @bshaffer
For integration testing, I used gapic-generator generate a showcase[schema] test client library. And use local gax-java SNAPSHOP(code) to test SetQuotaProjectID feature. It passed all the tests. I push the client into my personal repo(please feel free to suggest if you have you other recommend solution). Here is the testing code.

chingor13 · 2020-07-06T21:36:49Z

I'm still concerned that the headers are not actually configured for the TransportProvider or the CallContext. Somewhere in

gax-java/gax/src/main/java/com/google/api/gax/rpc/ClientContext.java

Lines 151 to 166 in cea6118

    
           if (transportChannelProvider.needsHeaders()) { 
        
             transportChannelProvider = transportChannelProvider.withHeaders(headers); 
        
           } 
        
           if (transportChannelProvider.needsEndpoint()) { 
        
             transportChannelProvider = transportChannelProvider.withEndpoint(settings.getEndpoint()); 
        
           } 
        
           if (transportChannelProvider.needsCredentials() && credentials != null) { 
        
             transportChannelProvider = transportChannelProvider.withCredentials(credentials); 
        
           } 
        
           TransportChannel transportChannel = transportChannelProvider.getTransportChannel(); 
        
           ApiCallContext defaultCallContext = 
        
               transportChannel.getEmptyCallContext().withTransportChannel(transportChannel); 
        
           if (credentials != null) { 
        
             defaultCallContext = defaultCallContext.withCredentials(credentials); 
        
           }

I'd expect to see how we're telling gRPC to set the x-goog-user-project header.

The testing code in the generated showcase client is inspecting the settings object (which is correct), but it's not ensuring that the header is being set on the gRPC request.

bshaffer · 2020-07-13T18:02:15Z

gax/src/main/java/com/google/api/gax/rpc/StubSettings.java

@@ -247,6 +287,10 @@ public B setCredentialsProvider(CredentialsProvider credentialsProvider) {
    @BetaApi("The surface for customizing headers is not stable yet and may change in the future.")
    public B setHeaderProvider(HeaderProvider headerProvider) {
      this.headerProvider = headerProvider;
+      if (this.quotaProjectId == null


@chingor13 is it necessary to support setting quota project automatically through the incoming headers? This is not something I am familiar with, or something done in any other languages AFAIK. It also makes the precedent of these operations less intuitive.

chingor13

I think what we want to do:

In ClientContext.create(), we want to add "x-goog-user-project" => settings.getQuotaProjectId() to the Map<String, String> headers that we provide to the transportChannelProvider. This will populate the x-goog-user-project on all the requests using that transport.

If a quotaProjectId is set, we also need to tell the Credentials to no longer send the x-goog-user-project header (it could be an incorrect value as the explicitly configured quotaProjectId setting should take precedence over the ones provided by the credentials). This will be harder and likely we'll have to do something ugly like providing our a wrapper Credentials implementation:

static class QuotaProjectHidingCredentials implements Credentials {
  private Credentials wrappedCredentials;
  QuotaProjectHidingCredentials(Credentials wrappedCredentials) {
    this.wrappedCredentials = wrappedCredentials;
  }

  public Map<String,List<String>> getRequestMetadata(URI uri) {
    Map<String,List<String>> headers = wrappedCredentials.getRequestMetadata(URI uri);
    // return new map without "x-goog-user-project" header
  }

  // ... other abstract methods delegate to the wrapped credentials
}

summer-ji-eng · 2020-07-17T18:26:25Z

build.gradle

@@ -24,7 +24,7 @@ apply plugin: 'com.github.sherter.google-java-format'
 apply plugin: 'io.codearte.nexus-staging'

 // TODO: Populate this from dependencies.properties version property (for proper Gradle-Bazel sync)
-project.version = "1.57.1" // {x-version-update:gax:current}
+project.version = "1.57.2" // {x-version-update:gax:current}


@chingor13 not sure if I should upgrade a version. let me know what you think. Thanks

gax/src/main/java/com/google/api/gax/rpc/ClientContext.java

bshaffer

One question for clarification, and a minor request. Otherwise LGTM!

bshaffer · 2020-07-21T21:16:14Z

gax/src/main/java/com/google/api/gax/rpc/StubSettings.java

+      if (clientContext.getHeaders().containsKey(QUOTA_PROJECT_ID_HEADER_KEY)) {
+        return clientContext.getHeaders().get(QUOTA_PROJECT_ID_HEADER_KEY);
+      }
+      if (clientContext.getInternalHeaders().containsKey(QUOTA_PROJECT_ID_HEADER_KEY)) {


It seems ok to support this then, for backwards compatibility. And also to ensure any quota project behavior stays consistent regardless of how it's set.

gax/src/main/java/com/google/api/gax/rpc/ClientContext.java

gax/src/main/java/com/google/api/gax/rpc/internal/QuotaProjectIdHidingCredentials.java

chingor13

One more header to fix, then LGTM

gax/src/test/java/com/google/api/gax/rpc/internal/QuotaProjectIdHidingCredentialsTest.java

…IdHidingCredentialsTest.java Co-authored-by: Jeff Ching <[email protected]>

summer-ji-eng added 2 commits June 15, 2020 00:49

Enable Quota Project ID in Client setting

8e7b92f

Add quota project Id in clientContext

c641572

summer-ji-eng requested review from chingor13 and bshaffer June 15, 2020 08:03

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 15, 2020

bshaffer reviewed Jun 17, 2020

View reviewed changes

chingor13 suggested changes Jun 18, 2020

View reviewed changes

change from quotaProjectID to quotaProjectId

cea6118

summer-ji-eng requested a review from chingor13 June 30, 2020 06:56

summer-ji-eng mentioned this pull request Jul 1, 2020

Test quota project googleapis/gapic-generator#3245

Closed

bshaffer reviewed Jul 13, 2020

View reviewed changes

chingor13 reviewed Jul 14, 2020

View reviewed changes

summer-ji-eng added 6 commits July 14, 2020 23:54

Add Executor

a157f44

Add quotaProjectHidingCredentialsTest

1151fdc

correct getRequestMetadata

c6b9d47

Add unit test for QuotaProjectIdHidingCredentials

0a8d179

add unit test for client context

a63ffae

Merge branch 'master' into quota_project

74541a5

summer-ji-eng requested a review from chingor13 July 17, 2020 17:25

summer-ji-eng commented Jul 17, 2020

View reviewed changes

gax/src/main/java/com/google/api/gax/rpc/ClientContext.java Show resolved Hide resolved

summer-ji-eng requested a review from bshaffer July 17, 2020 18:27

Merge branch 'master' into quota_project

6a56e6b

bshaffer approved these changes Jul 21, 2020

View reviewed changes

chingor13 suggested changes Jul 21, 2020

View reviewed changes

Add comment and beta annotation

0e1e201

summer-ji-eng requested a review from chingor13 July 22, 2020 22:00

chingor13 approved these changes Jul 23, 2020

View reviewed changes

gax/src/test/java/com/google/api/gax/rpc/internal/QuotaProjectIdHidingCredentialsTest.java Outdated Show resolved Hide resolved

Update gax/src/test/java/com/google/api/gax/rpc/internal/QuotaProject…

7f91763

…IdHidingCredentialsTest.java Co-authored-by: Jeff Ching <[email protected]>

summer-ji-eng merged commit 20bb200 into master Jul 24, 2020

summer-ji-eng deleted the quota_project branch July 24, 2020 05:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: enable setting quota_project_id #1128

feat: enable setting quota_project_id #1128

summer-ji-eng commented Jun 15, 2020

codecov bot commented Jun 15, 2020 •

edited

Loading

bshaffer Jun 17, 2020

summer-ji-eng Jun 17, 2020

chingor13 Jun 18, 2020

summer-ji-eng Jun 18, 2020

bshaffer Jun 17, 2020

summer-ji-eng Jun 17, 2020

chingor13 Jun 18, 2020

summer-ji-eng Jun 18, 2020

bshaffer Jul 21, 2020

chingor13 left a comment

summer-ji-eng commented Jun 18, 2020 •

edited

Loading

chingor13 commented Jun 18, 2020

summer-ji-eng commented Jun 30, 2020 •

edited

Loading

chingor13 commented Jul 6, 2020

bshaffer Jul 13, 2020

chingor13 left a comment •

edited

Loading

summer-ji-eng Jul 17, 2020

bshaffer left a comment

bshaffer Jul 21, 2020

chingor13 left a comment

feat: enable setting quota_project_id #1128

feat: enable setting quota_project_id #1128

Conversation

summer-ji-eng commented Jun 15, 2020

codecov bot commented Jun 15, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chingor13 left a comment

Choose a reason for hiding this comment

summer-ji-eng commented Jun 18, 2020 • edited Loading

chingor13 commented Jun 18, 2020

summer-ji-eng commented Jun 30, 2020 • edited Loading

chingor13 commented Jul 6, 2020

Choose a reason for hiding this comment

chingor13 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bshaffer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chingor13 left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 15, 2020 •

edited

Loading

summer-ji-eng commented Jun 18, 2020 •

edited

Loading

summer-ji-eng commented Jun 30, 2020 •

edited

Loading

chingor13 left a comment •

edited

Loading