Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_lambda_function.lambda-name: unexpected EOF #2509

Closed
wips opened this issue Dec 1, 2017 · 5 comments · Fixed by #5798
Closed

aws_lambda_function.lambda-name: unexpected EOF #2509

wips opened this issue Dec 1, 2017 · 5 comments · Fixed by #5798
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/lambda Issues and PRs that pertain to the lambda service.
Milestone

Comments

@wips
Copy link

wips commented Dec 1, 2017

Hi there,

When I add or delete vpc_config into my lambda settings terraform fails (see the error below). A part of my config related to lambda:

Terraform Version

Terraform v0.10.8. This is not the last one, but I can't upgrade terraform on our environments.

Terraform Configuration Files

# ...
resource "aws_lambda_function" "lambda-name" {
  filename          = "build.zip"
  function_name     = "${module.naming.aws_lambda}-lambda-name"
  handler           = "src/app/lambda/get-voucher/lambda-name.handler"
  runtime           = "nodejs6.10"
  role              = "${aws_iam_role.project-name.arn}"
  timeout           = "${var.lambda_timeout}"
  memory_size       = "${var.lambda_memory_size}"
  source_code_hash  = "${base64sha256(file("build/src/app/lambda/project-name/lambda-name.js"))}"

# I'm deleting adding this section
  vpc_config {
    subnet_ids         = ["${split(",", module.networks.subnets[format("%s.%s.%s", var.aws_region, var.vpc, "app")])}"]
    security_group_ids = ["${module.networks.security_groups[format("%s.%s.%s", var.aws_region, var.vpc, "app")]}"]
  }

  tags {
    ProductCode   = "${var.product_code_tag}"
    Environment   = "${var.environment_tag}"
    InventoryCode = "${var.inventory_code_tag}"
  }

  environment {
    variables {
      a_table_name = "${var.dynamodb_table_name}"
      a_table_field_status = "${var.dynamodb_table_attribute_status}"
      a_table_field_campaign = "${var.dynamodb_table_attribute_hash_key}"
      TABLE_NAME_PREFIX = "${lower(format("%s-%s-%s", var.product_code_tag, var.environment_tag, var.aws_region))}"
      statistics_table_name = "${var.statistics_table_name}"
      statistics_db_host = "${var.aurora_connection_host}"
      statistics_db_username = "${var.aurora_username}"
      statistics_db_password = "${var.aurora_password}"
      statistics_db_port = "${var.aurora_port}"
      statistics_db_schema = "${var.statistics_schema_name}"
    }
  }
}
# ...

Debug Output

aws_lambda_function.lambda-name: Modifying... (ID: my-env-prefix-lambda-name)
  environment.0.variables.%:                      "10" => "4"
  environment.0.variables.statistics_db_host:     "my-host.us-east-1.rds.amazonaws.com" => ""
  environment.0.variables.statistics_db_password: "AQICAHiIVCeABktkzHIQ82GGqiNK90vKUQnSYmA/MtYIiGKFgwECo/Cex6RG+Dw66yr9+KhvAAAAbjBsBgkqhkiG9w0BBwagXzBdAgEAMFgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMI79ziVpi4OLec3B8AgEQgCsXgtkwgIYuS6Gl5cqGt07BJqZru23sU1EFwlUxVZOfWfSKXLeW4DhoZgHE" => ""
  environment.0.variables.statistics_db_port:     "3306" => ""
  environment.0.variables.statistics_db_schema:   "shema_name" => ""
  environment.0.variables.statistics_db_username: "db_user" => ""
  environment.0.variables.statistics_table_name:  "table_name" => ""
  source_code_hash:                               "p9+O/hGu0Lvcbx00EcwcGPIL2oVEX+8yT1PqstaBHWk=" => "pYZ1JrAzAz6Jc9ttDj6IaJhZ+VZCE/Ro2yHXM/XZf3M="
  vpc_config.#:                                   "1" => "0"
  vpc_config.0.security_group_ids.#:              "1" => "0"
  vpc_config.0.security_group_ids.1234567890:     "sg-foobarfoo" => ""
  vpc_config.0.subnet_ids.#:                      "3" => "0"
  vpc_config.0.subnet_ids.1234567892:             "subnet-barfooba" => ""
  vpc_config.0.subnet_ids.1234567893:             "subnet-rwerewre" => ""
  vpc_config.0.subnet_ids.1234567894:             "subnet-cvbvcbvb" => ""

Error: Error applying plan:

14 error(s) occurred:

* aws_lambda_function.lambda-name: unexpected EOF
* aws_lambda_function.another-lambda-name: unexpected EOF

Panic Output

No

Expected Behavior

VPC config should be deleted

Actual Behavior

Terraform failed with the error above

Steps to Reproduce

  1. terraform apply

Is it a known bug or maybe I'm doing smth wrong?
Unfortunately, I can't provide complete logs and/or debug info at the moment, but still any help is much appreciated.

@apparentlymart
Copy link
Contributor

Hi @wips,

The "unexpected EOF" feature suggests that either there was a network problem reaching the lambda API or, alternatively, the AWS provider crashed and caused Terraform to lose its connection to the plugin.

Unfortunately it's not possible to guess what failed here with the information given. If you saw anything that looks like a crash (stack trace, etc) in the output it would helpful to see that to get an idea of what failed. I understand that you are not able to share full detailed logs, but if you run Terraform with TF_LOG=trace you should be able to see in the log output the underlying API requests being made to the Lambda API along with any responses, so if you can see valid-looking responses coming back that would at least help to eliminate API connectivity from the equation.

@apparentlymart apparentlymart added bug Addresses a defect in current functionality. waiting-response Maintainers are waiting on response from community or contributor. labels Dec 19, 2017
@grahamlyons
Copy link

grahamlyons commented Dec 21, 2017

Hi @apparentlymart,

I found this when investigating #1341.

# terraform version
Terraform v0.11.1
+ provider.aws v1.6.0

With this config:

resource "aws_security_group" "test" {                                          
  vpc_id = "vpc-abcdef"                                                       
  name = "radek-test"                                                           
}                                                                               
                                                                                
resource "aws_lambda_function" "test" {                                         
    filename      = "lambda.zip"                                                
    function_name = "radek-test"                                                
    role          = "${aws_iam_role.iam_for_lambda.arn}"                        
    handler       = "index.example"                                             
    runtime       = "nodejs4.3"                                                 
                                                                                
    vpc_config {                                                                
        subnet_ids         = ["subnet-654321"]                                
        security_group_ids = ["${aws_security_group.test.id}"]                  
    }                                                                           
}                                                                               
                                                                                
resource "aws_iam_role_policy" "iam_policy_for_lambda" {                        
    name = "iam_policy_for_lambda_radek"                                        
    role = "${aws_iam_role.iam_for_lambda.id}"                                  
    policy = <<EOF                                                              
{                                                                               
  "Version": "2012-10-17",                                                      
  "Statement": [                                                                
        {                                                                       
            "Effect": "Allow",                                                  
            "Action": [                                                         
                "logs:CreateLogGroup",                                          
                "logs:CreateLogStream",                                         
                "logs:PutLogEvents"                                             
            ],                                                                  
            "Resource": "arn:aws:logs:*:*:*"                                    
        },                                                                      
    {                                                                           
      "Effect": "Allow",                                                        
      "Action": [                                                               
        "ec2:CreateNetworkInterface",                                           
        "ec2:DescribeNetworkInterfaces",                                        
        "ec2:DeleteNetworkInterface"                                            
      ],                                                                        
      "Resource": [                                                             
        "*"                                                                     
      ]                                                                         
    }                                                                           
  ]                                                                             
}                                                                               
EOF                                                                             
}                                                                               
                                                                                
resource "aws_iam_role" "iam_for_lambda" {                                      
    name = "iam_for_lambda_radek"                                               
    assume_role_policy = <<EOF                                                  
{                                                                               
  "Version": "2012-10-17",                                                      
  "Statement": [                                                                
    {                                                                           
      "Action": "sts:AssumeRole",                                               
      "Principal": {                                                            
        "Service": "lambda.amazonaws.com"                                       
      },                                                                        
      "Effect": "Allow",                                                        
      "Sid": ""                                                                 
    }                                                                           
  ]                                                                             
}                                                                               
EO

Running this:

echo 'exports.example = (event, context)=>(null)' > index.js && zip lambda.zip index.js
terraform apply -auto-approve

Then removing the vcp_config block and running terraform apply -auto-approve again, I get this:

aws_security_group.test: Refreshing state... (ID: sg-123456)
aws_iam_role.iam_for_lambda: Refreshing state... (ID: iam_for_lambda_radek)
aws_lambda_function.test: Refreshing state... (ID: radek-test)
aws_iam_role_policy.iam_policy_for_lambda: Refreshing state... (ID: iam_for_lambda_radek:iam_policy_for_lambda_radek)
aws_lambda_function.test: Modifying... (ID: radek-test)
  vpc_config.#:                             "1" => "0"
  vpc_config.0.security_group_ids.#:        "1" => "0"
  vpc_config.0.security_group_ids.23335267: "sg-123456" => ""
  vpc_config.0.subnet_ids.#:                "1" => "0"
  vpc_config.0.subnet_ids.3928636147:       "subnet-64321" => ""

Error: Error applying plan:

1 error(s) occurred:

* aws_lambda_function.test: 1 error(s) occurred:

* aws_lambda_function.test: unexpected EOF

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.


panic: runtime error: index out of range
2017-12-21T12:21:13.719Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 
2017-12-21T12:21:13.719Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: goroutine 232 [running]:
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsLambdaFunctionUpdate(0xc420e420e0, 0x2380a00, 0xc4201d3200, 0x0, 0x0)
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_lambda_function.go:588 +0x3120
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Apply(0xc420918fc0, 0xc420ab3040, 0xc420afa7c0, 0x2380a00, 0xc4201d3200, 0x1, 0xc420e3a720, 0xc420b4b1d0)
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:233 +0x2ab
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Apply(0xc4208f2850, 0xc420ab2ff0, 0xc420ab3040, 0xc420afa7c0, 0x7fbe7180e6c8, 0x0, 0x18)
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:283 +0xa4
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Apply(0xc420a5bba0, 0xc420afa340, 0xc420662d90, 0x0, 0x0)
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       /opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:527 +0x57
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: reflect.Value.call(0xc4201185a0, 0xc420071218, 0x13, 0x27421dc, 0x4, 0xc420b5df20, 0x3, 0x3, 0x0, 0x0, ...)
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       /usr/local/go/src/reflect/value.go:434 +0x906
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: reflect.Value.Call(0xc4201185a0, 0xc420071218, 0x13, 0xc420027f20, 0x3, 0x3, 0xc420b6f7a0, 0xc420027f28, 0xc420499f80)
2017-12-21T12:21:13.720Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       /usr/local/go/src/reflect/value.go:302 +0xa4
2017-12-21T12:21:13.721Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: net/rpc.(*service).call(0xc420ae6500, 0xc42016d1d0, 0xc420a46568, 0xc420173b80, 0xc420a740c0, 0x1f362c0, 0xc420afa340, 0x16, 0x1f36300, 0xc420662d90, ...)
2017-12-21T12:21:13.721Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       /usr/local/go/src/net/rpc/server.go:381 +0x142
2017-12-21T12:21:13.721Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: created by net/rpc.(*Server).ServeCodec
2017-12-21T12:21:13.721Z [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       /usr/local/go/src/net/rpc/server.go:475 +0x36b
2017/12/21 12:21:13 [TRACE] root: eval: *terraform.EvalWriteState
2017/12/21 12:21:13 [TRACE] root: eval: *terraform.EvalApplyProvisioners
2017/12/21 12:21:13 [TRACE] root: eval: *terraform.EvalIf
2017/12/21 12:21:13 [TRACE] root: eval: *terraform.EvalWriteState
2017/12/21 12:21:13 [TRACE] root: eval: *terraform.EvalWriteDiff
2017/12/21 12:21:13 [TRACE] root: eval: *terraform.EvalApplyPost
2017/12/21 12:21:13 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* aws_lambda_function.test: unexpected EOF
2017/12/21 12:21:13 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* aws_lambda_function.test: unexpected EOF
2017/12/21 12:21:13 [TRACE] [walkApply] Exiting eval tree: aws_lambda_function.test
2017/12/21 12:21:13 [TRACE] dag/walk: upstream errored, not walking "meta.count-boundary (count boundary fixup)"
2017/12/21 12:21:13 [TRACE] dag/walk: upstream errored, not walking "provider.aws (close)"
2017/12/21 12:21:13 [TRACE] dag/walk: upstream errored, not walking "root"
2017-12-21T12:21:13.724Z [DEBUG] plugin: plugin process exited: path=/home/vagrant/workspace/acuris/tflambdabug/.terraform/plugins/linux_amd64/terraform-provider-aws_v1.6.0_x4
2017/12/21 12:21:13 [TRACE] Preserving existing state lineage "39213a94-8666-4c37-95be-5cc9f3ed183d"
2017/12/21 12:21:13 [TRACE] Preserving existing state lineage "39213a94-8666-4c37-95be-5cc9f3ed183d"
2017/12/21 12:21:13 [TRACE] Preserving existing state lineage "39213a94-8666-4c37-95be-5cc9f3ed183d"
2017/12/21 12:21:13 [TRACE] Preserving existing state lineage "39213a94-8666-4c37-95be-5cc9f3ed183d"
2017/12/21 12:21:13 [DEBUG] plugin: waiting for all plugin processes to complete...
2017-12-21T12:21:13.730Z [WARN ] plugin: error closing client during Kill: err="connection is shut down"



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

I've got the full trace log for plan and apply in a gist: https://gist.github.com/grahamlyons/fb8dd4664a1769f6cc008ff6d252e1f9

(stdout and stderr are mixed i.e. TF_LOG=trace terraform plan &> lambda_vpc_panic_plan.output)

@bflad bflad added the service/lambda Issues and PRs that pertain to the lambda service. label Jan 18, 2018
@radeksimko radeksimko added crash Results from or addresses a Terraform crash or kernel panic. and removed waiting-response Maintainers are waiting on response from community or contributor. labels Jan 23, 2018
@bflad bflad added this to the v1.35.0 milestone Sep 5, 2018
@bflad
Copy link
Contributor

bflad commented Sep 5, 2018

The fix for this crash along with properly handling aws_lambda_function resource vpc_config removals by actually removing the VPC configuration from the Lambda function has been merged into master and will be released with version 1.35.0 of the AWS provider, likely later today.

@bflad
Copy link
Contributor

bflad commented Sep 6, 2018

This has been released in version 1.35.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@ghost
Copy link

ghost commented Apr 3, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 3, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/lambda Issues and PRs that pertain to the lambda service.
Projects
None yet
5 participants