feat: submodule for cloud memorystore (memcache) #22

pkatsovich · 2020-07-15T06:36:57Z

This PR is mostly working with 2 exceptions:

When ever I apply or destroy the resource, the response from the API is always a 403 claiming the API isn't enabled.
Error: Error waiting for Deleting Instance: error while retrieving operation: googleapi: Error 403: Cloud Memorystore for Memcached API has not been used in project ######## before or it is disabled

However, I assure you the API is enabled, and the GSA creating the resource has permissions.. because the instance actually gets created! (or destroyed if I need be). it's just the API seems to respond with a 403 no matter what.

When I attempt to set memcache_parameters attribute, it plans fine, but then fails to apply w/ a different error from the API:

Error: Error waiting for Deleting Instance: error while retrieving operation: googleapi: Error 403: Cloud Memorystore for Memcached API has not been used in project 1018575646121 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/memcache.googleapis.com/overview?project=1018575646121 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.Help",
    "links": [
      {
        "description": "Google developers console API activation",
        "url": "https://console.developers.google.com/apis/api/memcache.googleapis.com/overview?project=1018575646121"
      }
    ]
  }
]

I suspect exception 2 is going to be resolved w/ the next release of the google-beta terraform provider, because I see a fix was merged into master but is not in the latest available release tag.

@morgante hoping you can provide some guidance here, thanks!

Jberlinsky · 2020-07-16T02:18:05Z

Hi @pkatsovich -- thanks for your contribution!

Regarding (1), can you tell us more about the setup you're executing the module in when you see that behavior (403 error)? Is the service account Terraform is running as housed in the same GCP project you're trying to create the Memorystore instance in?

pkatsovich · 2020-07-16T02:29:01Z

Hi @Jberlinsky thanks for taking a look at this.
Yes, the service account used to create the memcache instance is in a separate project within the same org. The service account has the owner role set for the project via a folder IAM policy.

Jberlinsky · 2020-07-16T02:36:15Z

@pkatsovich Is the Cloud Memorystore for Memcached API enabled in the project that houses the service account? If not, please enable that and try again.

pkatsovich · 2020-07-16T15:08:29Z

@Jberlinsky thanks for the guidance. Enabling the cloudmemstore memcache api in the SA host project has fixed the 403 response. Are we expecting this behavior to remain, or will the API enablement only be required for the project to which cloudmemstore belongs? Are you able to verify my assumption re (2)?

Jberlinsky · 2020-07-17T00:25:03Z

@pkatsovich API enablement is required on both the project the service account resides in, and the project targeted by the API action.

The error message that you provided RE: (2) seems to exhibit the API enablement issue identified in (1).

pkatsovich · 2020-07-17T02:43:46Z

@Jberlinsky sorry copy/paste error in my original description, this is the actual error when try to use the memcache_parameters attribute

Error: Error creating Instance: googleapi: Error 400: Invalid JSON payload received. Unknown name "memcacheParameters" at 'resource': Cannot find field.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.BadRequest",
    "fieldViolations": [
      {
        "description": "Invalid JSON payload received. Unknown name \"memcacheParameters\" at 'resource': Cannot find field.",
        "field": "resource"
      }
    ]
  }
]

Jberlinsky · 2020-07-17T03:21:00Z

@pkatsovich Could you provide an example resource call that reproduces the issue?

pkatsovich · 2020-07-17T15:06:54Z

@Jberlinsky I'm getting this when running a terraform apply, using this branch.

Below is the full output, including the plan

module.enable_apis.google_project_service.project_services["memcache.googleapis.com"]: Refreshing state... [id=REDACTED/memcache.googleapis.com]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_memcache_instance.self will be created
  + resource "google_memcache_instance" "self" {
      + authorized_network = "projects/REDACTED/global/networks/shared"
      + create_time        = (known after apply)
      + display_name       = "REDACTED memcache"
      + id                 = (known after apply)
      + name               = "REDACTED"
      + node_count         = 1
      + project            = "REDACTED"
      + region             = "us-central1"
      + zones              = (known after apply)

      + memcache_parameters {
          + id     = (known after apply)
          + params = {
              + "listen-backlog" = "2048"
              + "max-item-size"  = "8388608"
            }
        }

      + node_config {
          + cpu_count      = 1
          + memory_size_mb = 1024
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_memcache_instance.self: Creating...

Error: Error creating Instance: googleapi: Error 400: Invalid JSON payload received. Unknown name "memcacheParameters" at 'resource': Cannot find field.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.BadRequest",
    "fieldViolations": [
      {
        "description": "Invalid JSON payload received. Unknown name \"memcacheParameters\" at 'resource': Cannot find field.",
        "field": "resource"
      }
    ]
  }
]

  on main.tf line 17, in resource "google_memcache_instance" "self":
  17: resource "google_memcache_instance" "self" {


[terragrunt] 2020/07/17 11:04:19 Hit multiple errors:
exit status 1

Jberlinsky · 2020-07-19T19:52:09Z

@pkatsovich It does indeed look like the issue you're pointing out and the pull request in the beta provider are related. I can't speak to when that might be released.

pkatsovich · 2020-07-21T04:44:56Z

google-beta 3.31.0 has been released with the referenced fix. I've updated this PR to include an example and tests which I have passing locally. @Jberlinsky. would you be able to review/merge or provide the errors from google's CI?

pkatsovich · 2020-07-24T03:24:14Z

@morgante or @Jberlinsky could I please get some feedback on why the int check is failing?

morgante · 2020-07-24T05:04:51Z

@pkatsovich This is the error from CI:

Error: Provider produced inconsistent result after apply

When applying changes to
module.project.module.project-factory.google_service_account.default_service_account,
provider "google" produced an unexpected new value for was present, but now
absent.

Unfortunately this isn't terribly useful. My recommendation would be trying to upgrade the version of project factory used in the test/setup to the latest.

bharathkkb · 2020-07-24T05:47:36Z

@pkatsovich I would also bump the provider in setup to 3.x. I have noticed this in some other modules with 2.x

pkatsovich · 2020-07-24T17:13:12Z

@bharathkkb @morgante Thanks for the guidance. I bumped the provider in setup to 3.x, bumped project factory to latest (8.1.0) and set default_service_account = "delete". which seemed to be causing some churn. Test still seems to be failing but I can't reproduce locally, I'm hoping it's a more descriptive error now?

morgante · 2020-07-24T17:16:41Z

Yes, much clearer error now. Setup/converge succeeds; it fails on verify:

Profile: memcache
Version: (not specified)
Target:  local://

  ×  memcache-instance: Command: `gcloud beta memcache instances describe test-memcache --project=ci-memory-store-3eb3 --region=us-east1 --format=json` (4 failed)
     ×  Command: `gcloud beta memcache instances describe test-memcache --project=ci-memory-store-3eb3 --region=us-east1 --format=json` 
     undefined method `[]' for nil:NilClass
     ×  Command: `gcloud beta memcache instances describe test-memcache --project=ci-memory-store-3eb3 --region=us-east1 --format=json` 
     undefined method `[]' for nil:NilClass
     ×  Command: `gcloud beta memcache instances describe test-memcache --project=ci-memory-store-3eb3 --region=us-east1 --format=json` exit_status should eq 0
     
     expected: 0
          got: 2
     
     (compared using ==)

     ×  Command: `gcloud beta memcache instances describe test-memcache --project=ci-memory-store-3eb3 --region=us-east1 --format=json` stderr should eq ""
     
     expected: ""
          got: "ERROR: (gcloud.beta) Invalid choice: 'memcache'.\nMaybe you meant:\n  gcloud redis instances describ...lates describe\n\nTo search the help text of gcloud commands, run:\n  gcloud help -- SEARCH_TERMS\n"
     
     (compared using ==)
     
     Diff:
     @@ -1 +1,16 @@
     +ERROR: (gcloud.beta) Invalid choice: 'memcache'.
     +Maybe you meant:
     +  gcloud redis instances describe
     +  gcloud app instances describe
     +  gcloud bigtable instances describe
     +  gcloud compute instances describe
     +  gcloud filestore instances describe
     +  gcloud spanner instances describe
     +  gcloud sql instances describe
     +  gcloud bigtable instances get-iam-policy
     +  gcloud compute instance-groups describe
     +  gcloud compute instance-templates describe
     +
     +To search the help text of gcloud commands, run:
     +  gcloud help -- SEARCH_TERMS

pkatsovich · 2020-07-24T17:28:17Z

thanks @morgante, This is an error because of the gcloud sdk installed on the test instance. I was able to fix that error locally by updating the Makefile to use the latest
developers tools docker image. Is there anything else I can do to correct this?

morgante · 2020-07-24T19:24:35Z

The image version is specified here: https://github.com/terraform-google-modules/terraform-google-memorystore/blob/master/build/int.cloudbuild.yaml

pkatsovich · 2020-07-24T22:20:49Z

Thanks for the pointers @morgante. That looks to have been the last bit and the PR is passing all checks. Hoping this can be merged and included in the next module version!

morgante

Thanks for the contribution and patience with our CI. Just a few notes.

examples/memcache/memcache.tf

modules/memcache/README.md

morgante · 2020-07-24T22:33:22Z

modules/memcache/main.tf

+  region             = var.region
+  authorized_network = var.authorized_network
+  node_count         = var.node_count
+  display_name       = var.display_name


Can this default to var.name if not provided?

modules/memcache/variables.tf

morgante · 2020-07-24T22:35:07Z

test/fixtures/memcache/main.tf

+ */
+
+module "memcache" {
+  source = "../../../modules/memcache"


Fixtures should call examples, not the module directly.

test/fixtures/memcache/variables.tf

test/setup/main.tf

morgante · 2020-07-28T14:29:05Z

Tests are failing with this error:

       Error: Error creating Instance: googleapi: Error 400: Google private service access is not enabled. Enable private service access and try again. For information on enabling private service access, see https://cloud.google.com/service-infrastructure/docs/enabling-private-services-access
       com.google.cloud.memcache.action.validator.sn.ServiceNetworkingException: <eye3 title='FAILED_PRECONDITION'/> generic::FAILED_PRECONDITION: Google private service access is not enabled. Enable private service access and try again. For information on enabling private service access, see https://cloud.google.com/service-infrastructure/docs/enabling-private-services-access
       
         on ../../../modules/memcache/main.tf line 17, in resource "google_memcache_instance" "self":
         17: resource "google_memcache_instance" "self" {

pkatsovich · 2020-07-28T19:30:37Z

hi @morgante thanks. I think this might be a race condition. I attempted to handle the private service access using the same method that's use in the cloudsql module .

https://github.com/terraform-google-modules/terraform-google-memorystore/pull/22/files#diff-c635286e9cc2991efe0732e02ba0d268R21
and module_depends_on
https://github.com/terraform-google-modules/terraform-google-memorystore/pull/22/files#diff-c635286e9cc2991efe0732e02ba0d268R36

This was the reason I had the private-service-access in the setup originally. Any guidance is appreciated.

bharathkkb · 2020-07-28T20:11:15Z

@pkatsovich if it is a race condition issue, for module_depends_on to work you would need to use a null_resource within module
example: https://github.com/terraform-google-modules/terraform-google-sql-db/blob/295ca1842b4bd4121b0618fedf773c7e5363f40a/modules/mysql/main.tf#L168
and then use that throughout within resources.

Recently when I had a similar issue, I ended up using the can function. @morgante what are your thoughts on this alternative?

module "private-service-access" {
  source      = "GoogleCloudPlatform/sql-db/google//modules/private_service_access"
  version     = "3.2.0"
  project_id  = var.project
  vpc_network = "default"
}

module "memcache" {
  source            = "../../modules/memcache"
  name              = var.name
  project           = can(module.private-service-access.peering_completed) : var.project ? ""
  memory_size_mb    = var.memory_size_mb
  enable_apis       = var.enable_apis
  cpu_count         = var.cpu_count
  region            = var.region
}

This is hacky but again native support is coming with 0.13.

morgante · 2020-07-28T20:23:59Z

The other way to do this is with an implicit dependency. Specifically we should update the private_service_access module the project ID and reference that.

pkatsovich · 2020-07-28T21:45:49Z

@bharathkkb thanks for the suggestion! @morgante please let me know if there's anything else on this.

pkatsovich marked this pull request as draft July 15, 2020 06:37

pkatsovich force-pushed the cloudmemcache branch from af4d1bf to a9e096f Compare July 21, 2020 04:28

pkatsovich marked this pull request as ready for review July 21, 2020 04:42

feat: submodule for cloud memorystore (memcache)

0c9e81f

pkatsovich force-pushed the cloudmemcache branch from b7b6513 to 0c9e81f Compare July 24, 2020 03:17

pkatsovich added 2 commits July 24, 2020 10:05

bump test setup providers and project-factory

d94396e

delete default service account on ci project setup

b66a228

morgante suggested changes Jul 24, 2020

View reviewed changes

bump cloudbuild DOCKER_TAG_VERSION_DEVELOPER_TOOLS to 0.11.6

cb37609

pkatsovich force-pushed the cloudmemcache branch from 00915b9 to 3013377 Compare July 28, 2020 07:08

address comments; use private-service-access submodule, update test

af2b67b

pkatsovich force-pushed the cloudmemcache branch from 3013377 to af2b67b Compare July 28, 2020 07:13

pkatsovich requested a review from morgante July 28, 2020 08:04

remove module_depends_on

4595771

morgante approved these changes Jul 28, 2020

View reviewed changes

morgante merged commit 6067568 into terraform-google-modules:master Jul 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: submodule for cloud memorystore (memcache) #22

feat: submodule for cloud memorystore (memcache) #22

pkatsovich commented Jul 15, 2020 •

edited

Loading

Jberlinsky commented Jul 16, 2020

pkatsovich commented Jul 16, 2020

Jberlinsky commented Jul 16, 2020

pkatsovich commented Jul 16, 2020

Jberlinsky commented Jul 17, 2020

pkatsovich commented Jul 17, 2020

Jberlinsky commented Jul 17, 2020

pkatsovich commented Jul 17, 2020 •

edited

Loading

Jberlinsky commented Jul 19, 2020

pkatsovich commented Jul 21, 2020 •

edited

Loading

pkatsovich commented Jul 24, 2020

morgante commented Jul 24, 2020

bharathkkb commented Jul 24, 2020

pkatsovich commented Jul 24, 2020

morgante commented Jul 24, 2020

pkatsovich commented Jul 24, 2020

morgante commented Jul 24, 2020

pkatsovich commented Jul 24, 2020

morgante left a comment

morgante Jul 24, 2020

morgante Jul 24, 2020

morgante commented Jul 28, 2020

pkatsovich commented Jul 28, 2020

bharathkkb commented Jul 28, 2020 •

edited

Loading

morgante commented Jul 28, 2020

pkatsovich commented Jul 28, 2020

feat: submodule for cloud memorystore (memcache) #22

feat: submodule for cloud memorystore (memcache) #22

Conversation

pkatsovich commented Jul 15, 2020 • edited Loading

Jberlinsky commented Jul 16, 2020

pkatsovich commented Jul 16, 2020

Jberlinsky commented Jul 16, 2020

pkatsovich commented Jul 16, 2020

Jberlinsky commented Jul 17, 2020

pkatsovich commented Jul 17, 2020

Jberlinsky commented Jul 17, 2020

pkatsovich commented Jul 17, 2020 • edited Loading

Jberlinsky commented Jul 19, 2020

pkatsovich commented Jul 21, 2020 • edited Loading

pkatsovich commented Jul 24, 2020

morgante commented Jul 24, 2020

bharathkkb commented Jul 24, 2020

pkatsovich commented Jul 24, 2020

morgante commented Jul 24, 2020

pkatsovich commented Jul 24, 2020

morgante commented Jul 24, 2020

pkatsovich commented Jul 24, 2020

morgante left a comment

Choose a reason for hiding this comment

morgante Jul 24, 2020

Choose a reason for hiding this comment

morgante Jul 24, 2020

Choose a reason for hiding this comment

morgante commented Jul 28, 2020

pkatsovich commented Jul 28, 2020

bharathkkb commented Jul 28, 2020 • edited Loading

morgante commented Jul 28, 2020

pkatsovich commented Jul 28, 2020

pkatsovich commented Jul 15, 2020 •

edited

Loading

pkatsovich commented Jul 17, 2020 •

edited

Loading

pkatsovich commented Jul 21, 2020 •

edited

Loading

bharathkkb commented Jul 28, 2020 •

edited

Loading