Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS network ACL's and security groups keep loosing their rules on each alternating plan/apply. #8129

Closed
bernielomax opened this issue Aug 11, 2016 · 8 comments

Comments

@bernielomax
Copy link

bernielomax commented Aug 11, 2016

Hi,

I keep experiencing an issue where terraform keeps modifying my AWS ACL and SG's on every run. It keeps alternating between applying the ACL/SG rule or removing them. Heres some of the plan out put below. On this particular example it will then add my rules for application_acl and will remove the rules for the database_acl... It just keeps alternating between applying rules and removing them on each run! This was also happening on a previous version for me. Looks like this could possible be caused by the route tables refreshing every run?

~ aws_network_acl.application_acl
    subnet_ids.#:          "0" => "3"
    subnet_ids.2063153051: "" => "subnet-b3cba9c5"
    subnet_ids.3899139489: "" => "subnet-30ae0068"
    subnet_ids.4142336080: "" => "subnet-4e24712a"

~ aws_network_acl.database_acl
    ingress.#:                     "7" => "1"
    ingress.1047571647.action:     "allow" => ""
    ingress.1047571647.cidr_block: "172.25.16.0/24" => ""
    ingress.1047571647.from_port:  "0" => "0"
    ingress.1047571647.icmp_code:  "0" => "0"
    ingress.1047571647.icmp_type:  "0" => "0"
    ingress.1047571647.protocol:   "-1" => ""
    ingress.1047571647.rule_no:    "100" => "0"
    ingress.1047571647.to_port:    "0" => "0"
    ingress.1736622700.action:     "allow" => "allow"
    ingress.1736622700.cidr_block: "0.0.0.0/0" => "0.0.0.0/0"
    ingress.1736622700.from_port:  "32768" => "32768"
    ingress.1736622700.icmp_code:  "0" => "0"
    ingress.1736622700.icmp_type:  "0" => "0"
    ingress.1736622700.protocol:   "6" => "tcp"
    ingress.1736622700.rule_no:    "500" => "500"
    ingress.1736622700.to_port:    "60999" => "60999"
    ingress.1786450.action:        "allow" => ""
    ingress.1786450.cidr_block:    "172.25.17.0/24" => ""
    ingress.1786450.from_port:     "22" => "0"
    ingress.1786450.icmp_code:     "0" => "0"
    ingress.1786450.icmp_type:     "0" => "0"
    ingress.1786450.protocol:      "6" => ""
    ingress.1786450.rule_no:       "21" => "0"
    ingress.1786450.to_port:       "22" => "0"
    ingress.2727494838.action:     "allow" => ""
    ingress.2727494838.cidr_block: "172.25.16.0/24" => ""
    ingress.2727494838.from_port:  "22" => "0"
    ingress.2727494838.icmp_code:  "0" => "0"
    ingress.2727494838.icmp_type:  "0" => "0"
    ingress.2727494838.protocol:   "6" => ""
    ingress.2727494838.rule_no:    "20" => "0"
    ingress.2727494838.to_port:    "22" => "0"
    ingress.3281948791.action:     "allow" => ""
    ingress.3281948791.cidr_block: "172.25.17.0/24" => ""
    ingress.3281948791.from_port:  "0" => "0"
    ingress.3281948791.icmp_code:  "0" => "0"
    ingress.3281948791.icmp_type:  "0" => "0"
    ingress.3281948791.protocol:   "-1" => ""
    ingress.3281948791.rule_no:    "101" => "0"
    ingress.3281948791.to_port:    "0" => "0"
    ingress.348602793.action:      "allow" => ""
    ingress.348602793.cidr_block:  "172.25.18.0/24" => ""
    ingress.348602793.from_port:   "0" => "0"
    ingress.348602793.icmp_code:   "0" => "0"
    ingress.348602793.icmp_type:   "0" => "0"
    ingress.348602793.protocol:    "-1" => ""
    ingress.348602793.rule_no:     "102" => "0"
    ingress.348602793.to_port:     "0" => "0"
    ingress.921427448.action:      "allow" => ""
    ingress.921427448.cidr_block:  "172.25.18.0/24" => ""
    ingress.921427448.from_port:   "22" => "0"
    ingress.921427448.icmp_code:   "0" => "0"
    ingress.921427448.icmp_type:   "0" => "0"
    ingress.921427448.protocol:    "6" => ""
    ingress.921427448.rule_no:     "22" => "0"
    ingress.921427448.to_port:     "22" => "0"

+ aws_network_acl_rule.allow_all_application_subnets.0
    cidr_block:     "172.25.8.0/23"
    egress:         "false"
    from_port:      "0"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "-1"
    rule_action:    "allow"
    rule_number:    "100"
    to_port:        "0"

+ aws_network_acl_rule.allow_all_application_subnets.1
    cidr_block:     "172.25.10.0/23"
    egress:         "false"
    from_port:      "0"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "-1"
    rule_action:    "allow"
    rule_number:    "101"
    to_port:        "0"

+ aws_network_acl_rule.allow_all_application_subnets.2
    cidr_block:     "172.25.12.0/23"
    egress:         "false"
    from_port:      "0"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "-1"
    rule_action:    "allow"
    rule_number:    "102"
    to_port:        "0"

-/+ aws_network_acl_rule.allow_all_database_subnets.0
    cidr_block:     "172.25.16.0/24" => "172.25.16.0/24"
    egress:         "false" => "false"
    from_port:      "0" => "0"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "all" => "-1" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "100" => "100"
    to_port:        "0" => "0"

-/+ aws_network_acl_rule.allow_all_database_subnets.1
    cidr_block:     "172.25.17.0/24" => "172.25.17.0/24"
    egress:         "false" => "false"
    from_port:      "0" => "0"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "all" => "-1" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "101" => "101"
    to_port:        "0" => "0"

-/+ aws_network_acl_rule.allow_all_database_subnets.2
    cidr_block:     "172.25.18.0/24" => "172.25.18.0/24"
    egress:         "false" => "false"
    from_port:      "0" => "0"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "all" => "-1" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "102" => "102"
    to_port:        "0" => "0"

+ aws_network_acl_rule.allow_dmz_ssh_to_application_subnets.0
    cidr_block:     "172.25.0.0/23"
    egress:         "false"
    from_port:      "22"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "6"
    rule_action:    "allow"
    rule_number:    "20"
    to_port:        "22"

+ aws_network_acl_rule.allow_dmz_ssh_to_application_subnets.1
    cidr_block:     "172.25.2.0/23"
    egress:         "false"
    from_port:      "22"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "6"
    rule_action:    "allow"
    rule_number:    "21"
    to_port:        "22"

+ aws_network_acl_rule.allow_dmz_ssh_to_application_subnets.2
    cidr_block:     "172.25.4.0/23"
    egress:         "false"
    from_port:      "22"
    network_acl_id: "acl-9d3f06f9"
    protocol:       "6"
    rule_action:    "allow"
    rule_number:    "22"
    to_port:        "22"

-/+ aws_network_acl_rule.allow_dmz_ssh_to_database_subnets.0
    cidr_block:     "172.25.16.0/24" => "172.25.16.0/24"
    egress:         "false" => "false"
    from_port:      "22" => "22"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "tcp" => "6" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "20" => "20"
    to_port:        "22" => "22"

-/+ aws_network_acl_rule.allow_dmz_ssh_to_database_subnets.1
    cidr_block:     "172.25.17.0/24" => "172.25.17.0/24"
    egress:         "false" => "false"
    from_port:      "22" => "22"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "tcp" => "6" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "21" => "21"
    to_port:        "22" => "22"

-/+ aws_network_acl_rule.allow_dmz_ssh_to_database_subnets.2
    cidr_block:     "172.25.18.0/24" => "172.25.18.0/24"
    egress:         "false" => "false"
    from_port:      "22" => "22"
    network_acl_id: "acl-983f06fc" => "acl-983f06fc"
    protocol:       "tcp" => "6" (forces new resource)
    rule_action:    "allow" => "allow"
    rule_number:    "22" => "22"
    to_port:        "22" => "22"

~ aws_route_table.private_route_table.0
    route.3165135897.cidr_block:                "" => "0.0.0.0/0"
    route.3165135897.gateway_id:                "" => "nat-0f398e86a54620af5"
    route.3165135897.instance_id:               "" => ""
    route.3165135897.nat_gateway_id:            "" => ""
    route.3165135897.network_interface_id:      "" => ""
    route.3165135897.vpc_peering_connection_id: "" => ""
    route.667727132.cidr_block:                 "0.0.0.0/0" => ""
    route.667727132.gateway_id:                 "" => ""
    route.667727132.instance_id:                "" => ""
    route.667727132.nat_gateway_id:             "nat-0f398e86a54620af5" => ""
    route.667727132.network_interface_id:       "" => ""
    route.667727132.vpc_peering_connection_id:  "" => ""

~ aws_route_table.private_route_table.1
    route.1925574287.cidr_block:                "0.0.0.0/0" => ""
    route.1925574287.gateway_id:                "" => ""
    route.1925574287.instance_id:               "" => ""
    route.1925574287.nat_gateway_id:            "nat-0db98bf955d2eceb8" => ""
    route.1925574287.network_interface_id:      "" => ""
    route.1925574287.vpc_peering_connection_id: "" => ""
    route.444407989.cidr_block:                 "" => "0.0.0.0/0"
    route.444407989.gateway_id:                 "" => "nat-0db98bf955d2eceb8"
    route.444407989.instance_id:                "" => ""
    route.444407989.nat_gateway_id:             "" => ""
    route.444407989.network_interface_id:       "" => ""
    route.444407989.vpc_peering_connection_id:  "" => ""

~ aws_route_table.private_route_table.2
    route.2589993565.cidr_block:                "0.0.0.0/0" => ""
    route.2589993565.gateway_id:                "" => ""
    route.2589993565.instance_id:               "" => ""
    route.2589993565.nat_gateway_id:            "nat-0a8044e6b5e90fe5d" => ""
    route.2589993565.network_interface_id:      "" => ""
    route.2589993565.vpc_peering_connection_id: "" => ""
    route.897214756.cidr_block:                 "" => "0.0.0.0/0"
    route.897214756.gateway_id:                 "" => "nat-0a8044e6b5e90fe5d"
    route.897214756.instance_id:                "" => ""
    route.897214756.nat_gateway_id:             "" => ""
    route.897214756.network_interface_id:       "" => ""
    route.897214756.vpc_peering_connection_id:  "" => ""

~ aws_security_group.application_sg
    ingress.#:                                    "3" => "1"
    ingress.2963200659.cidr_blocks.#:             "0" => "0"
    ingress.2963200659.from_port:                 "22" => "0"
    ingress.2963200659.protocol:                  "tcp" => ""
    ingress.2963200659.security_groups.#:         "1" => "0"
    ingress.2963200659.security_groups.446249355: "sg-26dfee40" => ""
    ingress.2963200659.self:                      "false" => "false"
    ingress.2963200659.to_port:                   "22" => "0"
    ingress.3575264873.cidr_blocks.#:             "1" => "1"
    ingress.3575264873.cidr_blocks.0:             "0.0.0.0/0" => "0.0.0.0/0"
    ingress.3575264873.from_port:                 "32768" => "32768"
    ingress.3575264873.protocol:                  "tcp" => "tcp"
    ingress.3575264873.security_groups.#:         "0" => "0"
    ingress.3575264873.self:                      "false" => "false"
    ingress.3575264873.to_port:                   "60999" => "60999"
    ingress.753360330.cidr_blocks.#:              "0" => "0"
    ingress.753360330.from_port:                  "0" => "0"
    ingress.753360330.protocol:                   "-1" => ""
    ingress.753360330.security_groups.#:          "0" => "0"
    ingress.753360330.self:                       "true" => "false"
    ingress.753360330.to_port:                    "0" => "0"

~ aws_security_group.database_sg
    ingress.#:                                    "3" => "1"
    ingress.2963200659.cidr_blocks.#:             "0" => "0"
    ingress.2963200659.from_port:                 "22" => "0"
    ingress.2963200659.protocol:                  "tcp" => ""
    ingress.2963200659.security_groups.#:         "1" => "0"
    ingress.2963200659.security_groups.446249355: "sg-26dfee40" => ""
    ingress.2963200659.self:                      "false" => "false"
    ingress.2963200659.to_port:                   "22" => "0"
    ingress.3575264873.cidr_blocks.#:             "1" => "1"
    ingress.3575264873.cidr_blocks.0:             "0.0.0.0/0" => "0.0.0.0/0"
    ingress.3575264873.from_port:                 "32768" => "32768"
    ingress.3575264873.protocol:                  "tcp" => "tcp"
    ingress.3575264873.security_groups.#:         "0" => "0"
    ingress.3575264873.self:                      "false" => "false"
    ingress.3575264873.to_port:                   "60999" => "60999"
    ingress.753360330.cidr_blocks.#:              "0" => "0"
    ingress.753360330.from_port:                  "0" => "0"
    ingress.753360330.protocol:                   "-1" => ""
    ingress.753360330.security_groups.#:          "0" => "0"
    ingress.753360330.self:                       "true" => "false"
    ingress.753360330.to_port:                    "0" => "0"

My template looks like the following for these items:

resource "aws_network_acl_rule" "allow_dmz_ssh_to_application_subnets" {
  count = "${length(split(",",lookup(var.availability_zone_map, var.region)))}"
  network_acl_id = "${aws_network_acl.application_acl.id}"
  cidr_block = "${element(split(",",var.dmz_subnets), count.index)}"
  rule_action = "allow"
  protocol = "6"
  from_port = "22"
  to_port = "22"
  rule_number = "${count.index + 20}"
}

resource "aws_network_acl_rule" "allow_all_application_subnets" {
  count = "${length(split(",",lookup(var.availability_zone_map, var.region)))}"
  network_acl_id = "${aws_network_acl.application_acl.id}"
  cidr_block = "${element(split(",",var.application_subnets), count.index)}"
  rule_action = "allow"
  protocol = "-1"
  from_port = "0"
  to_port = "0"
  rule_number = "${count.index + 100}"
}

resource "aws_network_acl" "database_acl" {
  vpc_id = "${aws_vpc.vpc.id}"
  subnet_ids = ["${aws_subnet.database_subnet.*.id}"]
  ingress {
    protocol   = "tcp"
    rule_no    = "500"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = "32768"
    to_port    = "60999"
  }
  egress {
    protocol   = "-1"
    rule_no    = "100"
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = "0"
    to_port    = "0"
  }
  tags {
    Name = "${var.environment}-${var.application}-database-acl"
  }
}

resource "aws_network_acl_rule" "allow_dmz_ssh_to_database_subnets" {
  count = "${length(split(",",lookup(var.availability_zone_map, var.region)))}"
  network_acl_id = "${aws_network_acl.database_acl.id}"
  cidr_block = "${element(split(",",var.database_subnets), count.index)}"
  rule_action = "allow"
  protocol = "6"
  from_port = "22"
  to_port = "22"
  rule_number = "${count.index + 20}"
}

resource "aws_network_acl_rule" "allow_all_database_subnets" {
  count = "${length(split(",",lookup(var.availability_zone_map, var.region)))}"
  network_acl_id = "${aws_network_acl.database_acl.id}"
  cidr_block = "${element(split(",",var.database_subnets), count.index)}"
  rule_action = "allow"
  protocol = "-1"
  from_port = "0"
  to_port = "0"
  rule_number = "${count.index + 100}"
}

Terraform Version

Terraform v0.7.0

Affected Resource(s)

Please list the resources as a list, for example:

  • aws_network_acl
  • aws_network_acl_rule
  • aws_security_group
  • aws_security_group_rule
@bernielomax bernielomax changed the title AWS network ACL and security groups keep loosing their rules on each alternating plan/apply. AWS network ACL's and security groups keep loosing their rules on each alternating plan/apply. Aug 11, 2016
@cbarbour
Copy link

It looks like Amazon is translating the values that Terraform is inserting, and Terraform isn't determining that the values are synonymous.

protocol: "all" => "-1" (forces new resource)
protocol: "tcp" => "6" (forces new resource)

Try changing the protocol to "all" and "tcp" instead of "-1" and "6" in your configs, and see if that sidesteps the issue.

@Bjorn248
Copy link

Bjorn248 commented Aug 11, 2016

I can confirm that I have been experiencing the same behavior with security groups. The first run works as expected, and then the 2nd run removes the rules that were applied using aws_security_group_rule. Then during the 3rd run, it adds the rules back that were removed during the 2nd run. This goes on forever, all odd numbered runs put the infrastructure in the desired state, and the even numbered runs break it.

This might also be alleviated if we had the ability to attach security groups to running instances instead of having aws_security_group_rule we could just define another aws_security_group that we could attach to a previously created instance. Just a thought, as it is supported by the AWS API under ModifyInstanceAttribute and via the cli with ec2-modify-instance-attribute.

It would appear that terraform is comparing the aws_security_group alone and not a combination of that and any relevant aws_security_group_rule when checking against the remote state. This is an understandable bug. Terraform sees that the aws_security_group_rule is in place during plan number 2 but that the remote state of the aws_security_group does not match what is in the .tf file, because of that rule.

Here's a relevant snippet of my .tf files, which I have used to recreate the issue. I've redacted paths to my private keys and any other sensitive information, but should be usable to recreate the issue locally using this, once the REDACTED strings are replaced.

https://gist.github.com/Bjorn248/892e7d7563cb86403f026dc99970ccd8

@bernielomax
Copy link
Author

@Bjorn248 ,

My problem where rules were alternating seem to be caused by using a mix of inline and seperate rules. I have since moved to all seperate rules and it now just refreshes the existing ruleset over every time i apply. I.e. destroys and creates the entire SG/ACL on each apply.

@Bjorn248
Copy link

@bernielomax, interesting...so would the solution for me be to create a placeholder empty security group, then attach that to the instance, and then define all required rules using aws_security_group_rule and define no rules in the aws_security_group itself? I don't think I can have the SG be deleted every time, because the instance depends on that SG existing.

@bernielomax
Copy link
Author

bernielomax commented Aug 26, 2016

Yeah thats what seemed to have worked for me. The other suggestion is take note of what the values are that are causing the refresh. I.e. "from value" => "to value". From what I have seen Terraform has a hard time determining differences between data types. i.e. "all" != "-1" or "-1" != -1 even though they are the same as far as AWS is concerned. So you need to make sure your to value (which is what is set in your TF template) matches what the from value which is what was set on the previous apply. I hope that makes sense. As far as this issue is concerned. The following is still relavent and seems to be a bug:

~ aws_network_acl.application_acl
subnet_ids.#: "0" => "3"
subnet_ids.2063153051: "" => "subnet-b3cba9c5"
subnet_ids.3899139489: "" => "subnet-30ae0068"
subnet_ids.4142336080: "" => "subnet-4e24712a"

All my route tables continually being refreshed. I may have solved the ACL/SG problem using the method described above.

@bernielomax
Copy link
Author

bernielomax commented Aug 28, 2016

I was able to resolve the route table issue using a similar fix. I declared my route table using a single inline default rule. It seems this was causing the route table to be continually refreshed with random diffs. I removed all inline rules and just used seperate "aws_route" resource's to declare all my route table entries.

I think there are 3 options to fix this:

  • Create a fix that allows for both inline and separate resource declarations.
  • Add extra validation to check for both inline and separate resource declarations and fail if both are found.
  • Remove inline declarations entirely.

@Bjorn248
Copy link

@bernielomax, I can confirm that the workaround you suggested does work. While this approach is not intuitive, it does at least serve as a working approach to get around this bug. Thanks for the suggestion!

@ghost
Copy link

ghost commented Apr 10, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Apr 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants