Skip to content

Commit

Permalink
Merge #72056 #74214 #74243
Browse files Browse the repository at this point in the history 
72056: cdc: add pubsub sink r=wongio123 a=wongio123

changefeedccl: Pubsub Sink Integration
    
  Previously there was no changefeed sink for pubsub.
  This was inadequate because there are use cases for a pubsub sink when
  a Kafka sink is overkill and there are advantages with easier cloud
  compatibility. To address this I created a generic pubsub sink system
  that will allow for different pubsub sink integrations across different
  cloud providers to be easily implemented in the future. Currently GCP
  pubsub and in memory pubsub are implemented.
  
  Release note (enterprise change): Changefeeds now support gcp pubsub as a sink.

74214: cmd/reduce: fix logging and handle parsing errors r=mgartner a=mgartner

#### cmd/reduce: move pkg/testutils/reduce to pkg/cmd/reduce

Release note: None

#### cmd/reduce: fix logging and handle parsing errors

Previously, the `reduce` tool could unexpectedly exit without printing a
reduce form of the input SQL if a transformation pass resulted in SQL
that could not be successfully parsed. This commit ignores these errors
so that the tool can continue to reduce.

Also, the logger has been changed to a goroutine-safe `log.Logger`.

Release note: None

#### cmd/reduce: unset errexit

The `reduce` tool now unsets `errexit` in CRDB shells so that the entire
SQL input is processed, even if there are multiple errors.

Release note: None

#### cmd/reduce: simplify pretty printed SQL output

Previously, the `reduce` tool obnoxiously formatted the reduced SQL with
many parentheses, line breaks, and indents. It has been simplified so
that the output is condensed and easier to read.

Release note: None


74243: git: clean untracked files when switching branches r=irfansharif a=irfansharif

As our set of gitignored files change, switching to older branches can
often lead to untracked files left around that need manual cleanup. We
could perhaps do this automatically.

Release note: None

Co-authored-by: Alex Wong <[email protected]>
Co-authored-by: Marcus Gartner <[email protected]>
Co-authored-by: irfan sharif <[email protected]>
  • Loading branch information
@mgartner @irfansharif
4 people committed Dec 23, 2021
4 parents 62c80fa + dbc190b + 719b313 + 9e46bee commit 31438b5
Show file tree
Hide file tree
Showing 39 changed files with 1,136 additions and 178 deletions.
1 change: 1 addition & 0 deletions BUILD.bazel
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ load("//build/bazelutil/staticcheckanalyzers:def.bzl", "STATICCHECK_CHECKS")
# gazelle:resolve go github.com/cockroachdb/cockroach/pkg/sql/colflow_test @cockroach//pkg/sql/colflow:colflow_test
# gazelle:resolve go github.com/cockroachdb/cockroach/pkg/util/caller_test @cockroach//pkg/util/caller:caller_test
# gazelle:resolve go github.com/cockroachdb/cockroach/pkg/util/json_test @cockroach//pkg/util/json:json_test
# gazelle:resolve go google.golang.org/genproto/googleapis/pubsub/v1 @org_golang_google_genproto//googleapis/pubsub/v1:pubsub

# These packages use github.com/golang/mock to generate mocks. The target
# generating mocks for a given package necessarily depends on the package's
Expand Down
176 changes: 98 additions & 78 deletions DEPS.bzl
View file

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions githooks/post-checkout
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@
set -e

if [ "$1" != "$2" ]; then # previous ref != new ref.
git clean pkg -fd
exec git submodule update --init --recursive
fi
46 changes: 26 additions & 20 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,12 @@ module github.com/cockroachdb/cockroach
go 1.17

require (
cloud.google.com/go/storage v1.16.0
cloud.google.com/go/pubsub v1.16.0
cloud.google.com/go/storage v1.18.2
github.com/Azure/azure-sdk-for-go v57.1.0+incompatible
github.com/Azure/azure-storage-blob-go v0.14.0
github.com/Azure/go-autorest/autorest v0.11.20
github.com/Azure/go-autorest/autorest/azure/auth v0.4.2
github.com/Azure/go-autorest/autorest/azure/auth v0.5.8
github.com/Azure/go-autorest/autorest/to v0.4.0
github.com/BurntSushi/toml v0.4.1
github.com/DataDog/zstd v1.4.8 // indirect
Expand Down Expand Up @@ -67,7 +68,7 @@ require (
github.com/fraugster/parquet-go v0.4.0
github.com/ghemawat/stream v0.0.0-20171120220530-696b145b53b9
github.com/go-ole/go-ole v1.2.4 // indirect
github.com/go-sql-driver/mysql v1.5.0
github.com/go-sql-driver/mysql v1.6.0
github.com/go-swagger/go-swagger v0.26.1
github.com/gogo/protobuf v1.3.2
github.com/golang-commonmark/html v0.0.0-20180910111043-7d7c804e1d46 // indirect
Expand Down Expand Up @@ -125,6 +126,7 @@ require (
github.com/petermattis/goid v0.0.0-20180202154549-b0b1615b78e5
github.com/pierrre/geohash v1.0.0
github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0
github.com/prometheus/client_golang v1.11.0
github.com/prometheus/client_model v0.2.1-0.20210607210712-147c58e9608a
Expand Down Expand Up @@ -154,22 +156,22 @@ require (
go.opentelemetry.io/otel/exporters/zipkin v1.0.0-RC3
go.opentelemetry.io/otel/sdk v1.0.0-RC3
go.opentelemetry.io/otel/trace v1.0.0-RC3
golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97
golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
golang.org/x/exp v0.0.0-20210514180818-737f94c0881e
golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
golang.org/x/mod v0.5.1 // indirect
golang.org/x/net v0.0.0-20210913180222-943fd674d43e
golang.org/x/oauth2 v0.0.0-20210819190943-2bc19b11175f
golang.org/x/oauth2 v0.0.0-20211104180415-d3ed0bb246c8
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
golang.org/x/sys v0.0.0-20211102061401-a2f17f7b995c
golang.org/x/sys v0.0.0-20211210111614-af8b64212486
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d
golang.org/x/text v0.3.7
golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac
golang.org/x/tools v0.1.7
google.golang.org/api v0.56.0
google.golang.org/genproto v0.0.0-20210909211513-a8c4777a87af
google.golang.org/grpc v1.40.0
google.golang.org/api v0.63.0
google.golang.org/genproto v0.0.0-20211222154725-9823f7ba7562
google.golang.org/grpc v1.43.0
google.golang.org/grpc/examples v0.0.0-20210324172016-702608ffae4d // indirect
google.golang.org/protobuf v1.27.1
gopkg.in/yaml.v2 v2.4.0
Expand All @@ -179,12 +181,13 @@ require (
)

require (
cloud.google.com/go v0.93.3 // indirect
cloud.google.com/go v0.99.0 // indirect
cloud.google.com/go/kms v1.1.0 // indirect
github.com/Azure/azure-pipeline-go v0.2.3 // indirect
github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/adal v0.9.15 // indirect
github.com/Azure/go-autorest/autorest/azure/cli v0.3.1 // indirect
github.com/Azure/go-autorest/autorest/azure/cli v0.4.3 // indirect
github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
github.com/Azure/go-autorest/autorest/validation v0.3.1 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
Expand All @@ -208,17 +211,21 @@ require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d // indirect
github.com/cenkalti/backoff/v4 v4.1.1 // indirect
github.com/census-instrumentation/opencensus-proto v0.3.0 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4 // indirect
github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dimchansky/utfbom v1.1.0 // indirect
github.com/dimchansky/utfbom v1.1.1 // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/eapache/go-resiliency v1.2.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/envoyproxy/protoc-gen-validate v0.6.1 // indirect
github.com/envoyproxy/go-control-plane v0.10.1 // indirect
github.com/envoyproxy/protoc-gen-validate v0.6.2 // indirect
github.com/felixge/httpsnoop v1.0.1 // indirect
github.com/fsnotify/fsnotify v1.4.9 // indirect
github.com/fsnotify/fsnotify v1.5.1 // indirect
github.com/ghodss/yaml v1.0.0 // indirect
github.com/go-kit/log v0.1.0 // indirect
github.com/go-logfmt/logfmt v0.5.1 // indirect
Expand All @@ -241,8 +248,8 @@ require (
github.com/golang-jwt/jwt/v4 v4.0.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/google/go-querystring v1.0.0 // indirect
github.com/google/uuid v1.2.0 // indirect
github.com/googleapis/gax-go/v2 v2.1.0 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/googleapis/gax-go/v2 v2.1.1 // indirect
github.com/gorilla/handlers v1.5.1 // indirect
github.com/gorilla/websocket v1.4.2 // indirect
github.com/hashicorp/go-uuid v1.0.2 // indirect
Expand All @@ -267,7 +274,7 @@ require (
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.11 // indirect
github.com/klauspost/compress v1.12.2 // indirect
github.com/klauspost/compress v1.13.5 // indirect
github.com/klauspost/pgzip v1.2.5 // indirect
github.com/magiconair/properties v1.8.4 // indirect
github.com/mailru/easyjson v0.7.6 // indirect
Expand All @@ -287,15 +294,14 @@ require (
github.com/openzipkin/zipkin-go v0.2.5 // indirect
github.com/pelletier/go-toml v1.8.1 // indirect
github.com/pierrec/lz4 v2.6.0+incompatible // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pkg/profile v1.6.0 // indirect
github.com/pquerna/cachecontrol v0.0.0-20200921180117-858c6e7e6b7e // indirect
github.com/prometheus/procfs v0.6.0 // indirect
github.com/pseudomuto/protokit v0.2.0 // indirect
github.com/russross/blackfriday/v2 v2.0.1 // indirect
github.com/shurcooL/sanitized_anchor_name v1.0.0 // indirect
github.com/sirupsen/logrus v1.8.1 // indirect
github.com/spf13/afero v1.4.1 // indirect
github.com/spf13/afero v1.6.0 // indirect
github.com/spf13/cast v1.3.1 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/spf13/viper v1.7.1 // indirect
Expand All @@ -310,7 +316,7 @@ require (
go.opentelemetry.io/proto/otlp v0.9.0 // indirect
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.7.0 // indirect
go.uber.org/zap v1.16.0 // indirect
go.uber.org/zap v1.19.0 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
google.golang.org/appengine v1.6.7 // indirect
gopkg.in/ini.v1 v1.62.0 // indirect
Expand Down
Loading

0 comments on commit 31438b5

Please sign in to comment.