Update dependencies and remove juju/errors #79

kennytm · 2018-10-17T14:11:35Z

Replaced juju/errors by pingcap/errors (exported as pkg/errors due to how pingcap/tidb imports it) (LGPL-v3 → BSD-2-clause)
Updated pingcap/tidb to v2.1.0-rc.4 to entirely remove juju/errors from the vendor.
- Updated pingcap/pd to v2.1.0-rc.4
- Updated pingcap/kvproto to certain master
- Updated pingcap/tipb to certain master
- Replaced golang/protobuf by gogo/protobuf (BSD-3-clause)
- Added opentracing/basictracer-go (Apache-2.0)
Removed the golang.org/x/net dependency as we can use the built-in context package (the two are interchangeable after Go 1.7 anyway)
Removed the explicit dependency on pingcap/tidb-tools and siddontang/go, we're not using glide anymore
Updated some direct dependencies:
- Updated BurntSushi/toml from v0.3.0 to v0.3.1 (WTFPL → MIT)
- Updated prometheus/client_golang from v0.8.0 to v0.9.0
- Updated sirupsen/logrus from v0.11.6 to v1.1.1
- Updated golang.org/x/sys to certain master
- Updated google.golang.org/grpc from v1.12.0 to v1.15.0
Added the commercial license

…kage

sre-bot · 2018-10-17T14:11:38Z

Hi contributor, thanks for your PR.

This patch needs to be approved by someone of admins. They should reply with "/ok-to-test" to accept this PR for running test automatically.

kennytm · 2018-10-17T14:19:57Z

/run-all-tests

kennytm · 2018-10-17T14:28:31Z

@csuzhangxc @WangXiangUSTC PTAL; also cc @iamxy @GregoryIan for TOOL-390.

IANTHEREAL · 2018-10-18T06:12:16Z

@gregwebs PTAL

gregwebs · 2018-10-18T14:15:22Z

LICENSE.txt

+PingCAP CONFIDENTIAL
+____________________
+
+ [2015] - [2018] PingCAP Incorporated


IANAL, but I think this should say "Copyright" somewhere.

Also, I think this project was created after 2015.

There's still no update on TOOL-390 so far. We could still change it (if needed) before the 2.1 GA release I think.

This is much better now. But you should change 2015 to the date the project started.

gregwebs · 2018-10-18T14:40:18Z

Thank you! Getting easier to distribute properly now. I will be putting license check tooling in the SRE repo.

~/tidb/compliance/licenses/licenses $(go list ./... | grep cmd) | sed 's|github.com/pingcap/tidb-lightning||' | sed 's|/vendor/||'
                                                                 ? (BSD 3-clause "New" or "Revised" License, 26%)
github.com/BurntSushi/toml                               MIT License
github.com/apache/thrift/lib/go/thrift                   Apache License 2.0 (95%)
github.com/beorn7/perks/quantile                         MIT License (98%)
github.com/codahale/hdrhistogram                         MIT License
github.com/coreos/etcd                                   Apache License 2.0
github.com/coreos/go-semver/semver                       Apache License 2.0
github.com/cznic/mathutil                                BSD 3-clause "New" or "Revised" License (96%)
github.com/cznic/sortutil                                BSD 3-clause "New" or "Revised" License (96%)
github.com/go-sql-driver/mysql                           Mozilla Public License 2.0
github.com/gogo/protobuf                                 BSD 3-clause "New" or "Revised" License (90%)
github.com/golang/protobuf                               BSD 3-clause "New" or "Revised" License (96%)
github.com/golang/snappy                                 BSD 3-clause "New" or "Revised" License (96%)
github.com/google/btree                                  Apache License 2.0
github.com/grpc-ecosystem/go-grpc-middleware             Apache License 2.0
github.com/grpc-ecosystem/go-grpc-prometheus             Apache License 2.0
github.com/joho/sqltocsv                                 ? (BSD 3-clause Clear License, 82%)
github.com/matttproud/golang_protobuf_extensions/pbutil  Apache License 2.0
github.com/opentracing/basictracer-go/wire               MIT License
github.com/opentracing/opentracing-go                    MIT License
github.com/pingcap/goleveldb/leveldb                     BSD 2-clause "Simplified" License
github.com/pingcap/kvproto/pkg                           Apache License 2.0
github.com/pingcap/pd/pd-client                          Apache License 2.0
github.com/pingcap/tidb                                  Apache License 2.0
github.com/pingcap/tipb                                  Apache License 2.0
github.com/pkg/errors                                    BSD 2-clause "Simplified" License
github.com/prometheus/client_golang/prometheus/internal  Apache License 2.0
github.com/prometheus/client_model/go                    Apache License 2.0
github.com/prometheus/common                             Apache License 2.0
github.com/prometheus/procfs                             Apache License 2.0
github.com/satori/go.uuid                                MIT License (98%)
github.com/sirupsen/logrus                               MIT License
github.com/spaolacci/murmur3                             BSD 3-clause "New" or "Revised" License
github.com/twinj/uuid                                    MIT License (98%)
github.com/uber/jaeger-client-go                         MIT License
github.com/uber/jaeger-lib/metrics                       Apache License 2.0
golang.org/x/crypto/ssh/terminal                         BSD 3-clause "New" or "Revised" License (96%)
golang.org/x/net                                         BSD 3-clause "New" or "Revised" License (96%)
golang.org/x/sys/unix                                    BSD 3-clause "New" or "Revised" License (96%)
golang.org/x/text                                        BSD 3-clause "New" or "Revised" License (96%)
google.golang.org/genproto/googleapis                    Apache License 2.0
google.golang.org/grpc                                   Apache License 2.0
gopkg.in/natefinch/lumberjack.v2                         MIT License

gregwebs · 2018-10-18T14:40:58Z

There is just one lib (go-sql-driver/mysql) that requires making its source code available. It seems like we use this just for an error type. Its odd to me that we would check an error type from that package rather than from tidb.

kennytm · 2018-10-18T15:46:13Z

@gregwebs Thanks for the check!

While go-sql-driver/mysql is only explicitly used for checking error types, it is also implicitly used to register the MySQL driver for database/sql, i.e. the line sql.Open("mysql", ...) also relies on go-sql-driver/mysql.

If we need to avoid MPL-2.0, the only alternative I could find is github.com/ziutek/mymysql/godrv which is BSD-3-clause.

I'm not sure if "making its source available" means the library itself or its dependency. Lemme check the license itself.

kennytm · 2018-10-18T16:03:18Z

So I believe the relevant parts are:

3.3. Distribution of a Larger Work

You may create and distribute a Larger Work under terms of Your choice, provided that You also comply with the requirements of this License for the Covered Software. ...

Here Larger Work = pingcap/tidb-lightning, Covered Software = go-sql-driver/mysql. Meaning it is fine we use MPL-2.0 dependency here.

Additional info:

gregwebs · 2018-10-18T16:04:31Z

It would also be nice to send a similar vendor update PR to tidb-inspect-tools. It looks like the mysql driver is imported there but not used?

gregwebs · 2018-10-18T16:06:34Z

Oh, so same thing in tidb-inspect-tools where it uses sql.Open

gregwebs · 2018-10-18T16:15:05Z

We can use MPL code or LGPL code, just not GPL. LGPL/MPL requires source distribution with the binary, which in general we are not in compliance with right now (although where we offer downloads from github we probably are). It would have been nice to not be concerned about source distribution since at this point we essentially just have the mysql driver and one k8s dependency that require source distribution.

But it looks like we do need the MySQL driver, so we will have to develop our MPL/LGPL source distribution process.

gregwebs · 2018-10-18T17:46:58Z

https://internal.pingcap.net/confluence/display/PIN/Software+License+-+Compliance

kennytm · 2018-10-18T18:19:54Z

Just checked tidb-inspect-tools. Like pingcap/tidb-binlog#360, it also uses ngaut/log and calls SetRotateByHour() in many places 😑

IANTHEREAL · 2018-10-29T05:43:45Z

LGTM

Removed the goyacc step since the parser has been moved out. For now we rely on `git checkout` to recover the parser until v3.0.0-alpha is tagged

kennytm · 2018-10-30T18:42:52Z

(Updated TiDB from RC3 to RC4)

Jenkin's open file limit is somehow reduced to 65536, and the default tikv requirement is 40960*2+1000 = 82920, causing CI failing to start the integration tests. Here we reduce the number to 4096*2+1000 = 9192 via explicit config.

kennytm · 2018-10-31T03:52:17Z

/run-all-tests

IANTHEREAL · 2018-10-31T10:17:43Z

@WangXiangUSTC PTAL

kennytm · 2018-10-31T14:22:25Z

@gregwebs @iamxy so for https://internal.pingcap.net/confluence/display/PIN/Software+License+-+Compliance, what exactly we should do to be compliant?

The current distributed package https://download.pingcap.org/tidb-lightning-latest-linux-amd64.tar.gz contains just 3 executable files. Do we copy the MPL source code into the tarball as well?
We need to cat the NOTICE and LICENSE files of the vendored dependencies into the tarball?
We need to put an EULA into the tarball? Is this "PingCAP CONFIDENTIAL" file the correct one?

gregwebs · 2018-10-31T14:50:23Z

PingCAP CONFIDENTIAL applies to the source code which we do not distribute and is not a EULA. I don't think we have a EULA for our binaries, that is a separate issue from compliance. We should not include PingCAP CONFIDENTIAL in the release.

gregwebs · 2018-10-31T15:00:01Z

Yes, we should concat all license and notice files.

We can put the source in the tarball or we can put it in a publicly available location, perhaps a S3/GCS referenced in the release tarball.
I don't like putting the source in the tarball because it bloats it, but we do have a small dependency here. However, consider a docker image distribution rather than a tarball: we don't want to have to include MPL source code in the runtime. In that case, pointing to a publicly available location is preferable

kennytm · 2018-10-31T15:26:03Z

We could just point to https://github.com/go-sql-driver/mysql/tree/v1.4.0 in the NOTICE file I guess, similar to that of prometheus/client_golang

The following components are included in this product:

Go-MySQL-Driver 
Copyright 2013 The Go-MySQL-Driver Authors. All rights reserved.
Licensed under the Mozilla Public License Version 2.0.
Source code is available at <https://github.com/go-sql-driver/mysql/tree/v1.4.0>

gregwebs · 2018-10-31T16:00:14Z

It's easy in go to concat all the license and notice from vendor.

Just pointing to the source code for MPL is not enough. We need to distribute the source ourselves with the binary.

july2993 · 2018-11-06T03:03:41Z

LGTM

kennytm added 3 commits October 17, 2018 21:23

*: replaced juju/errors by pingcap/errors and use builtin context pac…

419de46

…kage

vendor: update tidb to v2.1.0-rc.3 and update its dependencies

72a0388

vendor: update the rest of the direct dependencies

a8a26d1

license: added commercial license

e5bdf82

gregwebs reviewed Oct 18, 2018

View reviewed changes

vendor: update TiDB and PD to v2.1.0-rc.4

80000fa

Removed the goyacc step since the parser has been moved out. For now we rely on `git checkout` to recover the parser until v3.0.0-alpha is tagged

kennytm added the status/WIP Work in progress label Oct 30, 2018

kennytm force-pushed the kennytm/update-dependencies-and-remove-juju-error branch 3 times, most recently from 6d01ce3 to 2a67cc6 Compare October 31, 2018 03:38

kennytm force-pushed the kennytm/update-dependencies-and-remove-juju-error branch from 2a67cc6 to d249213 Compare October 31, 2018 03:49

kennytm removed the status/WIP Work in progress label Oct 31, 2018

license: fix the year range

2b36dc6

kennytm requested a review from WangXiangUSTC November 1, 2018 04:38

kennytm mentioned this pull request Nov 1, 2018

Rewrite data file parser; Insert _tidb_rowid when needed; Update checkpoint structure #82

Merged

15 tasks

kennytm merged commit ba9350e into master Nov 6, 2018

kennytm deleted the kennytm/update-dependencies-and-remove-juju-error branch November 6, 2018 03:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update dependencies and remove juju/errors #79

Update dependencies and remove juju/errors #79

kennytm commented Oct 17, 2018 •

edited

Loading

sre-bot commented Oct 17, 2018

kennytm commented Oct 17, 2018

kennytm commented Oct 17, 2018

IANTHEREAL commented Oct 18, 2018

gregwebs Oct 18, 2018

gregwebs Oct 18, 2018

kennytm Oct 31, 2018 •

edited

Loading

gregwebs Oct 31, 2018

kennytm Oct 31, 2018

gregwebs Oct 31, 2018

gregwebs commented Oct 18, 2018

gregwebs commented Oct 18, 2018

kennytm commented Oct 18, 2018

kennytm commented Oct 18, 2018

3.3. Distribution of a Larger Work

gregwebs commented Oct 18, 2018

gregwebs commented Oct 18, 2018

gregwebs commented Oct 18, 2018

gregwebs commented Oct 18, 2018

kennytm commented Oct 18, 2018 •

edited

Loading

IANTHEREAL commented Oct 29, 2018

kennytm commented Oct 30, 2018

kennytm commented Oct 31, 2018

IANTHEREAL commented Oct 31, 2018

kennytm commented Oct 31, 2018

gregwebs commented Oct 31, 2018

gregwebs commented Oct 31, 2018

kennytm commented Oct 31, 2018

gregwebs commented Oct 31, 2018

july2993 commented Nov 6, 2018

Update dependencies and remove juju/errors #79

Update dependencies and remove juju/errors #79

Conversation

kennytm commented Oct 17, 2018 • edited Loading

sre-bot commented Oct 17, 2018

kennytm commented Oct 17, 2018

kennytm commented Oct 17, 2018

IANTHEREAL commented Oct 18, 2018

gregwebs Oct 18, 2018

Choose a reason for hiding this comment

gregwebs Oct 18, 2018

Choose a reason for hiding this comment

kennytm Oct 31, 2018 • edited Loading

Choose a reason for hiding this comment

gregwebs Oct 31, 2018

Choose a reason for hiding this comment

kennytm Oct 31, 2018

Choose a reason for hiding this comment

gregwebs Oct 31, 2018

Choose a reason for hiding this comment

gregwebs commented Oct 18, 2018

gregwebs commented Oct 18, 2018

kennytm commented Oct 18, 2018

kennytm commented Oct 18, 2018

3.3. Distribution of a Larger Work

gregwebs commented Oct 18, 2018

gregwebs commented Oct 18, 2018

gregwebs commented Oct 18, 2018

gregwebs commented Oct 18, 2018

kennytm commented Oct 18, 2018 • edited Loading

IANTHEREAL commented Oct 29, 2018

kennytm commented Oct 30, 2018

kennytm commented Oct 31, 2018

IANTHEREAL commented Oct 31, 2018

kennytm commented Oct 31, 2018

gregwebs commented Oct 31, 2018

gregwebs commented Oct 31, 2018

kennytm commented Oct 31, 2018

gregwebs commented Oct 31, 2018

july2993 commented Nov 6, 2018

kennytm commented Oct 17, 2018 •

edited

Loading

kennytm Oct 31, 2018 •

edited

Loading

kennytm commented Oct 18, 2018 •

edited

Loading