Error ingesting resources #34

xino12 · 2021-04-12T09:39:35Z

Since April 8th we see an error while ingesting AWS resources.

Error: rpc error: code = Unknown desc = ERROR: foreign key constraint "aws_ec2_subnet_ipv6_cidr_block_association_sets_subnet_id_fkey" cannot be implemented (SQLSTATE 42804)
Usage:
  cloudquery fetch [flags]

Flags:
      --driver string   database driver postgresql/neo4j (env: CQ_DRIVER) (default "postgresql")
      --dsn string      database connection string (env: CQ_DSN) (example: 'host=localhost user=postgres password=pass DB.name=postgres port=5432')
  -h, --help            help for fetch
      --path string     path to configuration file. can be generated with 'gen config' command (env: CQ_CONFIG_PATH) (default "./config.yml")
      --version         version for fetch

Global Flags:
      --enableConsoleLog      Enable console logging (default true)
      --enableFileLogging     enableFileLogging makes the framework logging to a file (default true)
      --encodeLogsAsJson      EncodeLogsAsJson makes the logging framework logging JSON
      --logDirectory string   Directory to logging to to when file logging is enabled (default ".")
      --logFile string        Filename is the name of the logfile which will be placed inside the directory (default "cloudquery.log")
      --maxAge int            MaxAge the max age in days to keep a logfile (default 3)
      --maxBackups int        MaxBackups the max number of rolled files to keep (default 3)
      --maxSize int           MaxSize the max size in MB of the logfile before it's rolled (default 30)
      --plugin-dir string     Directory to save and load Cloudquery plugins from (env: CQ_PLUGIN_DIR) (default "/home/jenkins/agent/workspace/cloudquery-ingest")
  -v, --verbose               Enable Verbose logging

2021/04/12 02:06:35 rpc error: code = Unknown desc = ERROR: foreign key constraint "aws_ec2_subnet_ipv6_cidr_block_association_sets_subnet_id_fkey" cannot be implemented (SQLSTATE 42804)

I guess that's part of the migration to the new SDK? Thanks for the good job as always :D

The text was updated successfully, but these errors were encountered:

yevgenypats · 2021-04-12T11:21:57Z

Hi @xino12 , Thanks for posting it. Yes, I think it's part of the migration issues. Can you try dropping the Postgres table and re-running cloudquery?

xino12 · 2021-04-21T06:45:38Z

Hi @yevgenypats I dropped the whole database and recreate it but I got this error now:


{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.625307Z","table":"aws_iam_policies"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.644644Z","table":"aws_iam_policies"}
{"@level":"debug","@message":"creating table relations","@timestamp":"2021-04-21T06:43:58.654870Z","table":"aws_iam_policies"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.654908Z","table":"aws_iam_policy_versions"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.654946Z","table":"aws_iam_policy_versions"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.669052Z","table":"aws_iam_policy_versions"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.672978Z","table":"aws_cloudwatch_alarms"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.690490Z","table":"aws_cloudwatch_alarms"}
{"@level":"debug","@message":"creating table relations","@timestamp":"2021-04-21T06:43:58.694381Z","table":"aws_cloudwatch_alarms"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.694413Z","table":"aws_cloudwatch_alarm_metrics"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.694467Z","table":"aws_cloudwatch_alarm_metrics"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.707296Z","table":"aws_cloudwatch_alarm_metrics"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.710757Z","table":"aws_ec2_customer_gateways"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.722538Z","table":"aws_ec2_customer_gateways"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.725804Z","table":"aws_ec2_vpcs"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.738342Z","table":"aws_ec2_vpcs"}
{"@level":"debug","@message":"creating table relations","@timestamp":"2021-04-21T06:43:58.741868Z","table":"aws_ec2_vpcs"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.741894Z","table":"aws_ec2_vpc_cidr_block_association_sets"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.741940Z","table":"aws_ec2_vpc_cidr_block_association_sets"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.755371Z","table":"aws_ec2_vpc_cidr_block_association_sets"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.757017Z","table":"aws_ec2_vpc_ipv6_cidr_block_association_sets"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.757102Z","table":"aws_ec2_vpc_ipv6_cidr_block_association_sets"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.770433Z","table":"aws_ec2_vpc_ipv6_cidr_block_association_sets"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.772561Z","table":"aws_ec2_instances"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.786801Z","table":"aws_ec2_instances"}
{"@level":"debug","@message":"creating table relations","@timestamp":"2021-04-21T06:43:58.788892Z","table":"aws_ec2_instances"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.788920Z","table":"aws_ec2_instance_block_device_mappings"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.788964Z","table":"aws_ec2_instance_block_device_mappings"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.804696Z","table":"aws_ec2_instance_block_device_mappings"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.806524Z","table":"aws_ec2_instance_elastic_gpu_associations"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.806588Z","table":"aws_ec2_instance_elastic_gpu_associations"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.821353Z","table":"aws_ec2_instance_elastic_gpu_associations"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.823346Z","table":"aws_ec2_instance_elastic_inference_accelerator_associations"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.823426Z","table":"aws_ec2_instance_elastic_inference_accelerator_associations"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.836681Z","table":"aws_ec2_instance_elastic_inference_accelerator_associations"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.838496Z","table":"aws_ec2_instance_licenses"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.838538Z","table":"aws_ec2_instance_licenses"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.852233Z","table":"aws_ec2_instance_licenses"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.853703Z","table":"aws_ec2_instance_network_interfaces"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.853762Z","table":"aws_ec2_instance_network_interfaces"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.943872Z","table":"aws_ec2_instance_network_interfaces"}
{"@level":"debug","@message":"creating table relations","@timestamp":"2021-04-21T06:43:58.945728Z","table":"aws_ec2_instance_network_interfaces"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.945759Z","table":"aws_ec2_instance_network_interface_groups"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.945788Z","table":"aws_ec2_instance_network_interface_groups"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.959327Z","table":"aws_ec2_instance_network_interface_groups"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.960600Z","table":"aws_ec2_instance_network_interface_ipv6_addresses"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.960676Z","table":"aws_ec2_instance_network_interface_ipv6_addresses"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.973655Z","table":"aws_ec2_instance_network_interface_ipv6_addresses"}
{"@level":"debug","@message":"creating table relation","@timestamp":"2021-04-21T06:43:58.975159Z","table":"aws_ec2_instance_network_interface_instance_private_ip_addresses"}
{"@level":"debug","@message":"creating table if not exists","@timestamp":"2021-04-21T06:43:58.975227Z","table":"aws_ec2_instance_network_interface_instance_private_ip_addresses"}
{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-21T06:43:58.989329Z","table":"aws_ec2_instance_network_interface_instance_private_ip_addresses"}
6:43AM DBG panic: runtime error: invalid memory address or nil pointer dereference
6:43AM DBG [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x2e3d105]
6:43AM DBG 
6:43AM DBG goroutine 6 [running]:
6:43AM DBG github.com/cloudquery/cq-provider-sdk/provider.Migrator.upgradeTable(0x3bc59c0, 0xc0001acc58, 0x3bda380, 0xc0000e21c0, 0x3bc5700, 0xc000198000, 0xc000411680, 0x0, 0x0)
6:43AM DBG 	/home/runner/go/pkg/mod/github.com/cloudquery/[email protected]/provider/migrator.go:52 +0x445
6:43AM DBG github.com/cloudquery/cq-provider-sdk/provider.Migrator.CreateTable(0x3bc59c0, 0xc0001acc58, 0x3bda380, 0xc0000e21c0, 0x3bc5700, 0xc000198000, 0xc000411680, 0xc000411500, 0x0, 0x0)
6:43AM DBG 	/home/runner/go/pkg/mod/github.com/cloudquery/[email protected]/provider/migrator.go:74 +0x408
6:43AM DBG github.com/cloudquery/cq-provider-sdk/provider.Migrator.CreateTable(0x3bc59c0, 0xc0001acc58, 0x3bda380, 0xc0000e21c0, 0x3bc5700, 0xc000198000, 0xc000411500, 0xc000411280, 0x0, 0x0)
6:43AM DBG 	/home/runner/go/pkg/mod/github.com/cloudquery/[email protected]/provider/migrator.go:86 +0x66b
6:43AM DBG github.com/cloudquery/cq-provider-sdk/provider.Migrator.CreateTable(0x3bc59c0, 0xc0001acc58, 0x3bda380, 0xc0000e21c0, 0x3bc5700, 0xc000198000, 0xc000411280, 0x0, 0x0, 0x0)
6:43AM DBG 	/home/runner/go/pkg/mod/github.com/cloudquery/[email protected]/provider/migrator.go:86 +0x66b
6:43AM DBG github.com/cloudquery/cq-provider-sdk/provider.(*Provider).Init(0xc0004180a0, 0xc00007e110, 0xa, 0xc000080300, 0x5e, 0x40b701, 0x344fa00, 0x347b480)
6:43AM DBG 	/home/runner/go/pkg/mod/github.com/cloudquery/[email protected]/provider/provider.go:63 +0x19c
6:43AM DBG github.com/cloudquery/cq-provider-sdk/proto.(*GRPCServer).Init(0xc000454a20, 0x3bc5780, 0xc0000ba720, 0xc0000be280, 0xc000454a20, 0xc0000ba720, 0xc00018cba0)
6:43AM DBG 	/home/runner/go/pkg/mod/github.com/cloudquery/[email protected]/proto/grpc.go:43 +0x68
6:43AM DBG github.com/cloudquery/cq-provider-sdk/proto/internal._Provider_Init_Handler(0x347b480, 0xc000454a20, 0x3bc5780, 0xc0000ba720, 0xc0000b0480, 0x0, 0x3bc5780, 0xc0000ba720, 0xc0000821c0, 0x6e)
6:43AM DBG 	/home/runner/go/pkg/mod/github.com/cloudquery/[email protected]/proto/internal/plugin_grpc.pb.go:104 +0x214
6:43AM DBG google.golang.org/grpc.(*Server).processUnaryRPC(0xc00041cc40, 0x3bd69c0, 0xc000484900, 0xc0000e0120, 0xc000402ae0, 0x49c75c0, 0x0, 0x0, 0x0)
6:43AM DBG 	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1217 +0x522
6:43AM DBG google.golang.org/grpc.(*Server).handleStream(0xc00041cc40, 0x3bd69c0, 0xc000484900, 0xc0000e0120, 0x0)
6:43AM DBG 	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1540 +0xd05
6:43AM DBG google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc00040c150, 0xc00041cc40, 0x3bd69c0, 0xc000484900, 0xc0000e0120)
6:43AM DBG 	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:878 +0xa5
6:43AM DBG created by google.golang.org/grpc.(*Server).serveStreams.func1
6:43AM DBG 	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:876 +0x1fd
2021/04/21 06:43:58 Client closed local connection on 127.0.0.1:5432
6:43AM DBG received EOF, stopping recv loop err="rpc error: code = Unavailable desc = transport is closing"
6:43AM DBG plugin process exited error="exit status 2" path=/home/jenkins/agent/workspace/cloudquery-ingest/.cq/providers/cloudquery/aws/latest-linux-amd64 pid=98
Error: rpc error: code = Unavailable desc = transport is closing
Usage:
  cloudquery fetch [flags]

Flags:
      --driver string   database driver postgresql/neo4j (env: CQ_DRIVER) (default "postgresql")
      --dsn string      database connection string (env: CQ_DSN) (example: 'host=localhost user=postgres password=pass DB.name=postgres port=5432')
  -h, --help            help for fetch
      --path string     path to configuration file. can be generated with 'gen config' command (env: CQ_CONFIG_PATH) (default "./config.yml")
      --version         version for fetch

Global Flags:
      --enableConsoleLog            Enable console logging (default true)
      --enableFileLogging           enableFileLogging makes the framework logging to a file (default true)
      --encodeLogsAsJson            EncodeLogsAsJson makes the logging framework logging JSON
      --logDirectory string         Directory to logging to to when file logging is enabled (default ".")
      --logFile string              Filename is the name of the logfile which will be placed inside the directory (default "cloudquery.log")
      --maxAge int                  MaxAge the max age in days to keep a logfile (default 3)
      --maxBackups int              MaxBackups the max number of rolled files to keep (default 3)
      --maxSize int                 MaxSize the max size in MB of the logfile before it's rolled (default 30)
      --plugin-dir string           Directory to save and load CloudQuery plugins from (env: CQ_PLUGIN_DIR) (default "/home/jenkins/agent/workspace/cloudquery-ingest")
      --reattach-providers string   Path to reattach unmanaged plugins, mostly used for testing purposes (env: CQ_REATTACH_PROVIDERS)
  -v, --verbose                     Enable Verbose logging

2021/04/21 06:43:58 rpc error: code = Unavailable desc = transport is closing

roneli · 2021-04-22T07:58:20Z

Hi @xino12, thanks for reporting the issue, we fixed the panic in 0.1.4 of the SDK and updated with v0.3.5 of the provider. Please have a try, it shouldn't crash now. I couldn't replicate the error on my system (I used postgres 13). It should print warning messages instead of errors, so if you see them when the provider initializes, post them here so I can investigate it further.

xino12 · 2021-04-22T08:11:51Z

I deleted again all aws tables and I see this error now:

{"@level":"debug","@message":"migrating table columns if required","@timestamp":"2021-04-22T08:09:09.468221Z","table":"aws_ec2_security_group_ip_permissions_egress_user_id_group_pairs"}
{"@level":"debug","@message":"adding column","@timestamp":"2021-04-22T08:09:09.469443Z","column":"id"}
{"@level":"warn","@message":"column missing from table, not adding it","@timestamp":"2021-04-22T08:09:09.469469Z","column":"id","table":"aws_ec2_security_group_ip_permissions_egress_user_id_group_pairs"}
{"@level":"debug","@message":"adding column","@timestamp":"2021-04-22T08:09:09.469490Z","column":"security_groupip_permissions_egress_id"}
{"@level":"error","@message":"failed to create table","@timestamp":"2021-04-22T08:09:09.470590Z","error":"ERROR: column \"security_groupip_permissions_egress_id\" of relation \"aws_ec2_security_group_ip_permissions_egress_user_id_group_pair\" already exists (SQLSTATE 42701)","table":"aws_ec2_security_groups"}

roneli · 2021-04-22T08:20:03Z

Okay I see the issue, thanks for the log. I will update a hot fix very soon and update when ready.

roneli · 2021-04-22T08:47:22Z

@xino12 can you please tell me the version of postgres you are using? can you also try to drop the specific table aws_ec2_security_group_ip_permissions_egress_user_id_group_pairs? i.e DROP TABLE aws_ec2_security_group_ip_permissions_egress_user_id_group_pairs

xino12 · 2021-04-22T09:25:45Z

I'm using postgres 11, tried deleting but still have the same issue.

xino12 · 2021-04-22T09:43:43Z

I've tried dropping all tables, let's see if now it does not fail, but it's certainly confusing

xino12 · 2021-04-22T09:45:49Z

it seems there might be a dependency when creating the tables that somehow you have it relaxed in your local setup and it does fail in other envs

xino12 · 2021-04-22T10:38:43Z

Okay, I think I found a pattern, it's very long table names what created this mess

xino12 · 2021-04-22T10:41:31Z

Probably any table name with more than 63 characters will create issues while creating the tables in pgsql 11.

roneli · 2021-04-22T11:15:12Z

Yes, I was suspecting it had to do with the pg version. Postgres 13 is a little more flexible, we are currently testing our providers on pg 12/13. We have an open task cloudquery/cq-provider-sdk#9 since this can also occur on long column names. Ill update the table names so they are shorter than 63 bytes.

xino12 · 2021-04-22T11:24:58Z

🙇🏼

roneli · 2021-04-22T11:41:28Z

Fixed #38 and released in v0.3.6. Added pg 11 to the matrix testing as well, so if this occurs again we will catch it before it happens. Thanks again for helping out in finding this issue! Tell me if it is resolved on your side so I can close this issue :)

xino12 · 2021-04-22T12:53:08Z

So I see that now all migrations work :) I see errors with assuming roles but I'm closing this issue.

Thanks @roneli

roneli assigned yevgenypats Apr 12, 2021

yevgenypats assigned roneli and unassigned yevgenypats Apr 21, 2021

roneli added the bug Something isn't working label Apr 21, 2021

roneli mentioned this issue Apr 21, 2021

Fix migrator panic cloudquery/cq-provider-sdk#14

Merged

roneli closed this as completed Apr 22, 2021

roneli reopened this Apr 22, 2021

xino12 closed this as completed Apr 22, 2021

roneli mentioned this issue Apr 28, 2021

Foreign Key constraint cannot be implemented #50

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error ingesting resources #34

Error ingesting resources #34

xino12 commented Apr 12, 2021

yevgenypats commented Apr 12, 2021

xino12 commented Apr 21, 2021 •

edited

Loading

roneli commented Apr 22, 2021

xino12 commented Apr 22, 2021 •

edited

Loading

roneli commented Apr 22, 2021

roneli commented Apr 22, 2021 •

edited

Loading

xino12 commented Apr 22, 2021

xino12 commented Apr 22, 2021

xino12 commented Apr 22, 2021

xino12 commented Apr 22, 2021

xino12 commented Apr 22, 2021

roneli commented Apr 22, 2021

xino12 commented Apr 22, 2021

roneli commented Apr 22, 2021 •

edited

Loading

xino12 commented Apr 22, 2021

Error ingesting resources #34

Error ingesting resources #34

Comments

xino12 commented Apr 12, 2021

yevgenypats commented Apr 12, 2021

xino12 commented Apr 21, 2021 • edited Loading

roneli commented Apr 22, 2021

xino12 commented Apr 22, 2021 • edited Loading

roneli commented Apr 22, 2021

roneli commented Apr 22, 2021 • edited Loading

xino12 commented Apr 22, 2021

xino12 commented Apr 22, 2021

xino12 commented Apr 22, 2021

xino12 commented Apr 22, 2021

xino12 commented Apr 22, 2021

roneli commented Apr 22, 2021

xino12 commented Apr 22, 2021

roneli commented Apr 22, 2021 • edited Loading

xino12 commented Apr 22, 2021

xino12 commented Apr 21, 2021 •

edited

Loading

xino12 commented Apr 22, 2021 •

edited

Loading

roneli commented Apr 22, 2021 •

edited

Loading

roneli commented Apr 22, 2021 •

edited

Loading