Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

syncer: support TLS for region syncer #1728

Merged
merged 4 commits into from
Sep 5, 2019
Merged

syncer: support TLS for region syncer #1728

merged 4 commits into from
Sep 5, 2019

Conversation

rleungx
Copy link
Member

@rleungx rleungx commented Sep 4, 2019 •
edited
Loading

What problem does this PR solve?

Currently, the region syncer establish the connection by using WithInsecure() which doesn't use TLS. Closes #1682.

What is changed and how it works?

This PR adds TLS support for the region syncer.

Check List

Tests

  • Manual test

Related changes

  • Need to cherry-pick to the release branch
  • Need to be included in the release notes

@rleungx rleungx added the needs-cherry-pick-release-3.0 The PR needs to cherry pick to release-3.0 branch. label Sep 4, 2019
@codecov-io
Copy link

codecov-io commented Sep 4, 2019 •
edited
Loading

Codecov Report

Merging #1728 into master will increase coverage by 0.16%.
The diff coverage is 36.11%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1728      +/-   ##
==========================================
+ Coverage   76.95%   77.12%   +0.16%     
==========================================
  Files         160      161       +1     
  Lines       15731    15734       +3     
==========================================
+ Hits        12106    12135      +29     
+ Misses       2608     2593      -15     
+ Partials     1017     1006      -11
Impacted Files Coverage Δ
client/client.go 69.76% <100%> (+3.98%) ⬆️
server/region_syncer/server.go 82.05% <100%> (+0.15%) ⬆️
pkg/grpcutil/grpcutil.go 18.51% <18.51%> (ø)
server/region_syncer/client.go 82.89% <50%> (+6.94%) ⬆️
server/tso/tso.go 77.06% <0%> (-2.76%) ⬇️
server/grpc_service.go 57.91% <0%> (-0.44%) ⬇️
server/cluster.go 83.35% <0%> (+0.25%) ⬆️
server/member/leader.go 80.61% <0%> (+0.51%) ⬆️
server/server.go 82.79% <0%> (+0.57%) ⬆️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ca5cae6...ef5fce3. Read the comment docs.

@rleungx rleungx requested a review from Connor1996 September 5, 2019 05:25
shafreeck
shafreeck approved these changes Sep 5, 2019
View reviewed changes
Copy link
Contributor

@shafreeck shafreeck left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nolouch nolouch added the status/can-merge Indicates a PR has been approved by a committer. label Sep 5, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Sep 5, 2019

Your auto merge job has been accepted, waiting for #1729

@sre-bot
Copy link
Contributor

sre-bot commented Sep 5, 2019

/run-all-tests

@sre-bot sre-bot merged commit f2487bf into tikv:master Sep 5, 2019
@sre-bot
Copy link
Contributor

sre-bot commented Sep 5, 2019

cherry pick to release-3.0 failed

rleungx added a commit to rleungx/pd that referenced this pull request Sep 9, 2019
@rleungx
syncer: support TLS for region syncer (tikv#1728)
9d773d0 
Signed-off-by: Ryan Leung <[email protected]>
@rleungx rleungx mentioned this pull request Sep 9, 2019
Merged
Luffbee added a commit that referenced this pull request Sep 9, 2019
@Luffbee
[new-hotspot-scheduler] add operator influence and more metrics (#1713)
83ab4c2 
* *: unify get store function everywhere (#1671)

Signed-off-by: Ryan Leung <[email protected]>

* remove unnecessary parentheses

*  server: use leader lease to determine tso service validity (#1676)

Signed-off-by: disksing <[email protected]>

* change internal stat values to float64

* add pending operator influence

* add metrics of pending influence

* fix metrics

* fix panic

* adjust pending influence of balanceHotWrite

* change weight of pending influence

* test: fix tests (#1696)

* test: fix region syncer test

Signed-off-by: disksing <[email protected]>

* decrease region rolling window; store pending influence in scheduler

* add config-check flag for pd-server (#1695)

Signed-off-by: cwen0 <[email protected]>

* decrease possiblility transfer hot write leader

* change pending influence weight

* add unstarted op metrics

* add logs for debug

* add log for debug

* add logs for debug

* add logs for debug

* add logs for debug

* add logs for debug

* add logs for debug

* add logs for debug

* Revert "add logs for debug"

This reverts commit e74c7a9.

* add metrics for hotspot operators

* operator: rewrite move region related functions (#1667)

* add metrics for pending operators

* *: support setting endKey for ScanRange (#1700)

Signed-off-by: disksing <[email protected]>

* fix bug

* fix bug

* fix bug

* fix metrics thread-safe bug

* fix logic bug

* *: reduce some unnecessary parameters (#1698)

Signed-off-by: Ryan Leung <[email protected]>

* schedule: Do not send an operator of a region wth a stale epoch (#1659)

* schedule: Do not send an operator of a region wth a stale epoch

Signed-off-by: Shafreeck Sea <[email protected]>

* schedule: check the version changed by the operator self

Signed-off-by: Shafreeck Sea <[email protected]>

* schedule: fix unit test

Signed-off-by: Shafreeck Sea <[email protected]>

* schedule: fix to avoid dispatching a stale opstep

Signed-off-by: Shafreeck Sea <[email protected]>

* dispatch: refactor "ConsumeConfVer() int" to "ExpectConfVerChange() bool"

Signed-off-by: Shafreeck Sea <[email protected]>

* dispatch: fix typo in comment

Signed-off-by: Shafreeck Sea <[email protected]>

* fix typo

Co-Authored-By: Ryan Leung <[email protected]>

* dispatch: fix unittest

Signed-off-by: Shafreeck Sea <[email protected]>

* dispatch: refine format

Signed-off-by: Shafreeck Sea <[email protected]>

* server: fix the dead lock in scatter region (#1706)

Signed-off-by: Ryan Leung <[email protected]>

* add drop time for operator

* use IsDropped to recognize canceled ops

* try to fix trans leader burst

* try to fix trans leader burst

* add zombie influence

* change select src dst strategy; improve op_controller

* change select src strategy

* fix bug

* tools: fix set namespace in pd-ctl (#1701)

Signed-off-by: Ryan Leung <[email protected]>

* tools: fix parse url without http prefix (#1703)

Signed-off-by: Ryan Leung <[email protected]>

* tests: support deadlock detection in make test (#1704)

Signed-off-by: Ryan Leung <[email protected]>

* Makefile: fix failpoint enable (#1722)

Signed-off-by: nolouch <[email protected]>

* checker: fix the issue that a region does not merge to the sibling with smaller size (#1723)

Signed-off-by: disksing <[email protected]>

* tools: balance region simulator (#1708)

* scheduler: do not remove the operator when the step does not finish (#1715)

Signed-off-by: Shafreeck Sea <[email protected]>

* operator: fix the AddLearner config version judgment (#1732)

Signed-off-by: nolouch <[email protected]>

* tools: fix TLS in pd control (#1729)

Signed-off-by: Ryan Leung <[email protected]>

* syncer: support TLS for region syncer (#1728)

Signed-off-by: Ryan Leung <[email protected]>

* schedule: fix a thread-safe bug and improve code (#1719)
Luffbee added a commit that referenced this pull request Sep 11, 2019
@Luffbee
[new-hotspot-scheduler] merge master (#1752)
bcfcada 
* *: unify get store function everywhere (#1671)

Signed-off-by: Ryan Leung <[email protected]>

*  server: use leader lease to determine tso service validity (#1676)

Signed-off-by: disksing <[email protected]>

* test: fix tests (#1696)

* test: fix region syncer test

Signed-off-by: disksing <[email protected]>

* add config-check flag for pd-server (#1695)

Signed-off-by: cwen0 <[email protected]>

* operator: rewrite move region related functions (#1667)

* *: support setting endKey for ScanRange (#1700)

Signed-off-by: disksing <[email protected]>

* *: reduce some unnecessary parameters (#1698)

Signed-off-by: Ryan Leung <[email protected]>

* schedule: Do not send an operator of a region wth a stale epoch (#1659)

* schedule: Do not send an operator of a region wth a stale epoch

Signed-off-by: Shafreeck Sea <[email protected]>

* schedule: check the version changed by the operator self

Signed-off-by: Shafreeck Sea <[email protected]>

* schedule: fix unit test

Signed-off-by: Shafreeck Sea <[email protected]>

* schedule: fix to avoid dispatching a stale opstep

Signed-off-by: Shafreeck Sea <[email protected]>

* dispatch: refactor "ConsumeConfVer() int" to "ExpectConfVerChange() bool"

Signed-off-by: Shafreeck Sea <[email protected]>

* dispatch: fix typo in comment

Signed-off-by: Shafreeck Sea <[email protected]>

* fix typo

Co-Authored-By: Ryan Leung <[email protected]>

* dispatch: fix unittest

Signed-off-by: Shafreeck Sea <[email protected]>

* dispatch: refine format

Signed-off-by: Shafreeck Sea <[email protected]>

* server: fix the dead lock in scatter region (#1706)

Signed-off-by: Ryan Leung <[email protected]>

* tools: fix set namespace in pd-ctl (#1701)

Signed-off-by: Ryan Leung <[email protected]>

* tools: fix parse url without http prefix (#1703)

Signed-off-by: Ryan Leung <[email protected]>

* tests: support deadlock detection in make test (#1704)

Signed-off-by: Ryan Leung <[email protected]>

* Makefile: fix failpoint enable (#1722)

Signed-off-by: nolouch <[email protected]>

* checker: fix the issue that a region does not merge to the sibling with smaller size (#1723)

Signed-off-by: disksing <[email protected]>

* tools: balance region simulator (#1708)

* scheduler: do not remove the operator when the step does not finish (#1715)

Signed-off-by: Shafreeck Sea <[email protected]>

* operator: fix the AddLearner config version judgment (#1732)

Signed-off-by: nolouch <[email protected]>

* tools: fix TLS in pd control (#1729)

Signed-off-by: Ryan Leung <[email protected]>

* syncer: support TLS for region syncer (#1728)

Signed-off-by: Ryan Leung <[email protected]>

* schedule: fix a thread-safe bug and improve code (#1719)

* statistics: fix region flow calculation (#1688)

Signed-off-by: jiyingtk <[email protected]>

* makefile: improve deadlock-enable/disable (#1736)

* api: fix missing keys statistic in region information (#1741)

Signed-off-by: nolouch <[email protected]>

* *: update go version to 1.13 (#1742)

Signed-off-by: disksing <[email protected]>

* coordinator: add the operator cost time in log field (#1748)

Signed-off-by: nolouch <[email protected]>
sre-bot pushed a commit that referenced this pull request Sep 12, 2019
@rleungx @sre-bot
syncer: support TLS for region syncer (#1728) (#1739)
dc03c83 
Signed-off-by: Ryan Leung <[email protected]>
nolouch pushed a commit to nolouch/pd that referenced this pull request Nov 12, 2019
@rleungx @nolouch
syncer: support TLS for region syncer (tikv#1728) (tikv#1739)
49a1bc0 
Signed-off-by: Ryan Leung <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Connor1996 Connor1996 Connor1996 left review comments

@shafreeck shafreeck shafreeck approved these changes

@nolouch nolouch nolouch approved these changes

@lhy1024 lhy1024 lhy1024 approved these changes

@disksing disksing Awaiting requested review from disksing

Assignees
No one assigned
Labels
needs-cherry-pick-release-3.0 The PR needs to cherry pick to release-3.0 branch. status/can-merge Indicates a PR has been approved by a committer.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

gRPC connection fail when --join with TLS enabled
7 participants
@rleungx @codecov-io @sre-bot @shafreeck @nolouch @Connor1996 @lhy1024