feat: Added VPC Endpoint Services and Configurations

[mlozoya2]

🎉 Thank you for making CloudQuery awesome by submitting a PR 🎉

Summary

This PR adds Ec2VpcEndpointServices and Ec2VpcEndpointServiceConfigurations resources.

Ideally, these new resources would be added as relation tables under the Ec2VpcEndpoint resource, but there are issues with the AWS-sdk when fetching them using the ServiceName/ServiceID parameters. The fetch throws errors related to invalid formats for ServiceName and ServiceID. The workaround was to omit the parameter and grab all of the resources.

---

Use the following steps to ensure your PR is ready to be reviewed

• Read the contribution guidelines 🧑‍🎓
• Run go fmt to format your code 🖊
• Lint your changes via golangci-lint run 🚨 (install golangci-lint here)
• Update or add tests. Learn more about testing here 🧪
• Update the docs by running go run ./docs/docs.go and committing the changes 📃
• If adding a new resource, add relevant Terraform files in a separate PR 📂
• Ensure the status checks below are successful ✅

[bbernays]

Looks really great! A few minor comments

[bbernays]

@mlozoya2 This pr looks good, can you expand a little bit on the issues you faced with the AWS SDK?

[erezrokah]

/test sha=66f9682cc9d8bdb04966fe10fe02e405488d6791

[erezrokah]

Same as cloudquery/cq-provider-azure#331 (comment).
Great work on the PR @mlozoya2, I'll look into adding the relevant resources to our Terraform files

Update:

Ended up ignoring specific columns in tests instead of creating a specific configuration for those

[erezrokah]

/test sha=07eb9141e3621441a64041fd170e39deec1abed9

[mlozoya2]

@bbernays Sure thing. In my first attempt, I had both of these new tables under vpc_endpoints as relation tables in the way that you would expect (vpc_endpoints -> vpc_endpoint_services -> vpc_endpoint_service_configurations). With this approach, I was able to setup the inputs as follows...

config := ec2.DescribeVpcEndpointServicesInput{
	ServiceNames: []string{*parent.Item.(types.VpcEndpoint).ServiceName},
}

config := ec2.DescribeVpcEndpointServiceConfigurationsInput{
	ServiceIds: []string{*parent.Item.(types.ServiceDetail).ServiceId},
}

This produced the following error for almost all of the fetches...

failed to resolve table "aws_ec2_vpc_endpoint_services": error at github.com/cloudquery/cq-provider-aws/resources/services/ec2.fetchEc2VpcEndpointServices[vpc_endpoints.go:292] operation error EC2: DescribeVpcEndpointServices, https response error StatusCode: 400, RequestID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, api error InvalidServiceName: The Vpc Endpoint Service 'com.amazonaws.vpce.us-east-1.vpce-svc-xxxxxxxxxxxxxxxxx' does not exist

We were able to pull some results into the tables, but far fewer than expected. For example, we only got 1 entry into the service_configuration table when we would expect 200+. The only way I was able to pull all of our expected results was to make the fetches without specifying any parameters(FindAll).

[bbernays]

@mlozoya2 - Thank you for that very thorough explanation, I fully agree that your initial proposed heirarchy is definitely the ideal solution. Do you happen to have the code from that initial attempt? I want to make sure that there wasn't an issue on the CQ SDK side of things that blocked you from finding all of the results you expected.

[mlozoya2]

@bbernays This is what I had for vpc_endpoints.go on the first attempt (excludes non-relevant columns on vpc_endpoint_services and vpc_endpoint_service_configurations)

package ec2

import (
	"context"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/ec2"
	"github.com/aws/aws-sdk-go-v2/service/ec2/types"
	"github.com/cloudquery/cq-provider-aws/client"
	"github.com/cloudquery/cq-provider-sdk/provider/diag"
	"github.com/cloudquery/cq-provider-sdk/provider/schema"
)

func Ec2VpcEndpoints() *schema.Table {
	return &schema.Table{
		Name:          "aws_ec2_vpc_endpoints",
		Description:   "Describes a VPC endpoint.",
		Resolver:      fetchEc2VpcEndpoints,
		Multiplex:     client.ServiceAccountRegionMultiplexer("ec2"),
		IgnoreError:   client.IgnoreCommonErrors,
		DeleteFilter:  client.DeleteAccountRegionFilter,
		Options:       schema.TableCreationOptions{PrimaryKeys: []string{"account_id", "id"}},
		IgnoreInTests: true,
		Columns: []schema.Column{
			{
				Name:        "account_id",
				Description: "The AWS Account ID of the resource.",
				Type:        schema.TypeString,
				Resolver:    client.ResolveAWSAccount,
			},
			{
				Name:        "region",
				Description: "The AWS Region of the resource.",
				Type:        schema.TypeString,
				Resolver:    client.ResolveAWSRegion,
			},
			{
				Name:        "arn",
				Description: "The Amazon Resource Name (ARN) for the resource.",
				Type:        schema.TypeString,
				Resolver: client.ResolveARN(client.EC2Service, func(resource *schema.Resource) ([]string, error) {
					return []string{"vpc-endpoint", *resource.Item.(types.VpcEndpoint).VpcEndpointId}, nil
				}),
			},
			{
				Name:        "creation_timestamp",
				Description: "The date and time that the VPC endpoint was created.",
				Type:        schema.TypeTimestamp,
			},
			{
				Name:        "last_error_code",
				Description: "The error code for the VPC endpoint error.",
				Type:        schema.TypeString,
				Resolver:    schema.PathResolver("LastError.Code"),
			},
			{
				Name:        "last_error_message",
				Description: "The error message for the VPC endpoint error.",
				Type:        schema.TypeString,
				Resolver:    schema.PathResolver("LastError.Message"),
			},
			{
				Name:        "network_interface_ids",
				Description: "(Interface endpoint) One or more network interfaces for the endpoint.",
				Type:        schema.TypeStringArray,
			},
			{
				Name:        "owner_id",
				Description: "The ID of the AWS account that owns the VPC endpoint.",
				Type:        schema.TypeString,
			},
			{
				Name:        "policy_document",
				Description: "The policy document associated with the endpoint, if applicable.",
				Type:        schema.TypeString,
			},
			{
				Name:        "private_dns_enabled",
				Description: "(Interface endpoint) Indicates whether the VPC is associated with a private hosted zone.",
				Type:        schema.TypeBool,
			},
			{
				Name:        "requester_managed",
				Description: "Indicates whether the VPC endpoint is being managed by its service.",
				Type:        schema.TypeBool,
			},
			{
				Name:        "route_table_ids",
				Description: "(Gateway endpoint) One or more route tables associated with the endpoint.",
				Type:        schema.TypeStringArray,
			},
			{
				Name:        "service_name",
				Description: "The name of the service to which the endpoint is associated.",
				Type:        schema.TypeString,
			},
			{
				Name:        "state",
				Description: "The state of the VPC endpoint.",
				Type:        schema.TypeString,
			},
			{
				Name:        "subnet_ids",
				Description: "(Interface endpoint) One or more subnets in which the endpoint is located.",
				Type:        schema.TypeStringArray,
			},
			{
				Name:        "tags",
				Description: "Any tags assigned to the VPC endpoint.",
				Type:        schema.TypeJSON,
				Resolver:    resolveEc2vpcEndpointTags,
			},
			{
				Name:        "id",
				Description: "The ID of the VPC endpoint.",
				Type:        schema.TypeString,
				Resolver:    schema.PathResolver("VpcEndpointId"),
			},
			{
				Name:        "vpc_endpoint_type",
				Description: "The type of endpoint.",
				Type:        schema.TypeString,
			},
			{
				Name:        "vpc_id",
				Description: "The ID of the VPC to which the endpoint is associated.",
				Type:        schema.TypeString,
			},
		},
		Relations: []*schema.Table{
			{
				Name:          "aws_ec2_vpc_endpoint_dns_entries",
				Description:   "Describes a DNS entry.",
				Resolver:      fetchEc2VpcEndpointDnsEntries,
				IgnoreInTests: true,
				Columns: []schema.Column{
					{
						Name:        "vpc_endpoint_cq_id",
						Description: "Unique CloudQuery ID of aws_ec2_vpc_endpoints table (FK)",
						Type:        schema.TypeUUID,
						Resolver:    schema.ParentIdResolver,
					},
					{
						Name:        "dns_name",
						Description: "The DNS name.",
						Type:        schema.TypeString,
					},
					{
						Name:        "hosted_zone_id",
						Description: "The ID of the private hosted zone.",
						Type:        schema.TypeString,
					},
				},
			},
			{
				Name:          "aws_ec2_vpc_endpoint_groups",
				Description:   "Describes a security group.",
				Resolver:      fetchEc2VpcEndpointGroups,
				IgnoreInTests: true,
				Columns: []schema.Column{
					{
						Name:        "vpc_endpoint_cq_id",
						Description: "Unique CloudQuery ID of aws_ec2_vpc_endpoints table (FK)",
						Type:        schema.TypeUUID,
						Resolver:    schema.ParentIdResolver,
					},
					{
						Name:        "group_id",
						Description: "The ID of the security group.",
						Type:        schema.TypeString,
					},
					{
						Name:        "group_name",
						Description: "The name of the security group.",
						Type:        schema.TypeString,
					},
				},
			},
			{
				Name:          "aws_ec2_vpc_endpoint_services",
				Description:   "Describes a VPC endpoint service.",
				Resolver:      fetchEc2VpcEndpointServices,
				IgnoreInTests: true,
				Columns: []schema.Column{
					{
						Name:        "vpc_endpoint_cq_id",
						Description: "Unique CloudQuery ID of aws_ec2_vpc_endpoints table (FK)",
						Type:        schema.TypeUUID,
						Resolver:    schema.ParentIdResolver,
					},
					{
						Name:        "service_name",
						Description: "The Amazon Resource Name (ARN) of the service.",
						Type:        schema.TypeString,
					},
					{
						Name:        "id",
						Description: "The ID of the endpoint service.",
						Type:        schema.TypeString,
						Resolver:    schema.PathResolver("ServiceId"),
					},
				},
				Relations: []*schema.Table{
					{
						Name:          "aws_ec2_vpc_endpoint_service_configurations",
						Description:   "Describes a service configuration for a VPC endpoint service.",
						Resolver:      fetchEc2VpcEndpointServiceConfigurations,
						IgnoreInTests: true,
						Columns: []schema.Column{
							{
								Name:        "vpc_endpoint_service_cq_id",
								Description: "Unique CloudQuery ID of aws_ec2_vpc_endpoint_services table (FK)",
								Type:        schema.TypeUUID,
								Resolver:    schema.ParentIdResolver,
							},
							{
								Name:        "service_id",
								Description: "The ID of the service.",
								Type:        schema.TypeString,
							},
							{
								Name:        "service_name",
								Description: "The name of the service.",
								Type:        schema.TypeString,
							},
							{
								Name:        "service_state",
								Description: "The service state.",
								Type:        schema.TypeString,
							},
						},
					},
				},
			},
		},
	}
}

// ====================================================================================================================
//                                               Table Resolver Functions
// ====================================================================================================================
func fetchEc2VpcEndpoints(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- interface{}) error {
	var config ec2.DescribeVpcEndpointsInput
	c := meta.(*client.Client)
	svc := c.Services().EC2
	for {
		output, err := svc.DescribeVpcEndpoints(ctx, &config, func(o *ec2.Options) {
			o.Region = c.Region
		})
		if err != nil {
			return diag.WrapError(err)
		}
		res <- output.VpcEndpoints
		if aws.ToString(output.NextToken) == "" {
			break
		}
		config.NextToken = output.NextToken
	}
	return nil
}
func resolveEc2vpcEndpointTags(ctx context.Context, meta schema.ClientMeta, resource *schema.Resource, c schema.Column) error {
	r := resource.Item.(types.VpcEndpoint)
	tags := map[string]*string{}
	for _, t := range r.Tags {
		tags[*t.Key] = t.Value
	}
	return diag.WrapError(resource.Set("tags", tags))
}
func fetchEc2VpcEndpointDnsEntries(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- interface{}) error {
	endpoint := parent.Item.(types.VpcEndpoint)
	res <- endpoint.DnsEntries

	return nil
}
func fetchEc2VpcEndpointGroups(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- interface{}) error {
	endpoint := parent.Item.(types.VpcEndpoint)
	res <- endpoint.Groups

	return nil
}
func fetchEc2VpcEndpointServices(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- interface{}) error {
	config := ec2.DescribeVpcEndpointServicesInput{
		ServiceNames: []string{*parent.Item.(types.VpcEndpoint).ServiceName},
	}
	c := meta.(*client.Client)
	svc := c.Services().EC2
	for {
		output, err := svc.DescribeVpcEndpointServices(ctx, &config, func(options *ec2.Options) {
			options.Region = c.Region
		})
		if err != nil {
			return diag.WrapError(err)
		}
		res <- output.ServiceDetails
		if aws.ToString(output.NextToken) == "" {
			break
		}
		config.NextToken = output.NextToken
	}
	return nil
}
func fetchEc2VpcEndpointServiceConfigurations(ctx context.Context, meta schema.ClientMeta, parent *schema.Resource, res chan<- interface{}) error {
	config := ec2.DescribeVpcEndpointServiceConfigurationsInput{
		ServiceIds: []string{*parent.Item.(types.ServiceDetail).ServiceId},
	}
	c := meta.(*client.Client)
	svc := c.Services().EC2
	for {
		output, err := svc.DescribeVpcEndpointServiceConfigurations(ctx, &config, func(options *ec2.Options) {
			options.Region = c.Region
		})
		if err != nil {
			return diag.WrapError(err)
		}
		res <- output.ServiceConfigurations
		if aws.ToString(output.NextToken) == "" {
			break
		}
		config.NextToken = output.NextToken
	}
	return nil
}

[bbernays]

@mlozoya2 - That looks good and it would work on small accounts, but I think the issue you ran into was probably throttling at the SDK level for large accounts. As every single VPC Endpoint Service would require 3+ API calls to resolve. Great job working around that issue!