v3 support #121

MegaByte875 · 2022-03-28T16:34:44Z

upgrade nebula client
update apis/template configuration

wey-gu · 2022-03-29T02:24:57Z

Dear @porscheme
Here is the v3 support of the operator, if you are interested to try it ;)

Thanks.
BR//Wey

kqzh · 2022-03-29T06:23:42Z

pkg/controller/component/storaged_cluster.go

+		err := metaClient.Disconnect()
+		if err != nil {
+			return
+		}


log err or just _ = metaClient.Disconnect() ?

kqzh · 2022-03-29T06:50:20Z

pkg/nebula/meta_client.go

 }

-func (m *metaClient) Balance(req *meta.BalanceReq) (*meta.BalanceResp, error) {
-	return m.client.Balance(req)
+func (m *metaClient) Balance(req *meta.AdminJobReq) (*meta.AdminJobResp, error) {


I think we can remove this func because we don't recommend use it, user can use BalanceData or BalanceLeader instead

kqzh · 2022-03-29T06:56:51Z

pkg/nebula/meta_client.go

+		Cmd:   meta.AdminCmd_LEADER_BALANCE,
+		Paras: [][]byte{space},
+	}
+	resp, err := m.client.RunAdminJob(req)


use m.balance(req) ?

kqzh · 2022-03-29T07:12:49Z

pkg/nebula/meta_client.go

-			log.Info("balance plan running now")
-			return m.BalanceStatus(resp.Id)
+			log.Info("balance job running now")
+			return m.BalanceStatus(*resp.GetResult_().JobID, space)


It seems balanceStatus is a long task, do we need use async？

Yes, this's not a ideal implementation. I will save the last job in status field.

kqzh · 2022-03-29T07:16:37Z

pkg/nebula/meta_client.go

 		}
+		resp, err := m.client.RunAdminJob(req)
+		if err != nil {
+			return false, err


if meta done or leader changed? balanceStatus will never stop?

kqzh · 2022-03-29T07:21:23Z

pkg/util/retry/retry.go

+	return wait.PollImmediateUntil(interval, func() (bool, error) {
+		done, err := fn(ctx)
+		if err != nil {
+			if done {
+				return false, err
+			}
+		} else if done {
+			return true, nil
+		}
+		return false, nil


how about

done, err := fn(ctx) if err != nil { return false, err } return done, nil

veezhang · 2022-03-29T07:22:17Z

apis/apps/v1alpha1/nebulacluster_types.go

-	Workload WorkloadStatus `json:"workload,omitempty"`
+	Version    string         `json:"version,omitempty"`
+	Phase      ComponentPhase `json:"phase,omitempty"`
+	HostsAdded bool           `json:"hostsAdded,omitempty"`


The HostsAdded only for Storage?

veezhang · 2022-03-29T07:27:24Z

pkg/controller/component/storaged_cluster.go

+	oldReplicas := extender.GetReplicas(oldWorkload)
+	newReplicas := extender.GetReplicas(newWorkload)
+	if !nc.Status.Storaged.HostsAdded || *newReplicas > *oldReplicas {
+		if err := c.addStorageHosts(nc, *oldReplicas, *newReplicas); err != nil {


It will not error when the same host add more than once?

Here only executed once

For example:

addStorageHosts successfully.

syncComponentStatus failed.

reconcile, and call addStorageHosts again.

veezhang · 2022-03-29T07:28:28Z

pkg/controller/component/storaged_cluster.go

+		if err := c.addStorageHosts(nc, *oldReplicas, *newReplicas); err != nil {
+			_, ok := err.(*net.DNSError)
+			if ok {
+				return nil


Why? It's sync successfully when DNSError?

Here capture DNSError will not block subsequent reconcile logic

So, it's the DNSError a normal behavior?
If all the following succeed, then add host succeeded?

veezhang · 2022-03-29T07:30:11Z

pkg/controller/component/storaged_scaler.go

+	empty := len(spaces) == 0
+	if !empty {


How about if len(spaces) > 0 ?

veezhang · 2022-03-29T07:30:33Z

pkg/controller/component/storaged_scaler.go

-		if len(hosts) > 0 {
-			if err := metaClient.RemoveHost(hosts); err != nil {
-				return err
+		if !empty {


How about if len(spaces) > 0 ?

veezhang · 2022-03-29T07:31:26Z

pkg/controller/component/storaged_updater.go

@@ -26,7 +26,8 @@ import (
 	"k8s.io/apimachinery/pkg/runtime/schema"
 	"sigs.k8s.io/controller-runtime/pkg/client"

-	ng "github.com/vesoft-inc/nebula-go/v2/nebula"
+	ng "github.com/vesoft-inc/nebula-go/v3/nebula"


How about nebulago to unify the alias.

veezhang · 2022-03-29T07:36:03Z

pkg/util/retry/retry.go

+	var stop <-chan struct{}
+	if ctx != nil {
+		stop = ctx.Done()
+	}


Will it block whenstop is nil?

Done may return nil if this context never be canceled

How use ctx.Done() directly, ctx generally cannot be nil.

veezhang · 2022-03-29T07:36:57Z

tests/e2e/nebulacluster/util.go

@@ -39,7 +39,7 @@ import (
 	"k8s.io/utils/pointer"
 	"sigs.k8s.io/controller-runtime/pkg/client"

-	nebula "github.com/vesoft-inc/nebula-go/v2"
+	nebula "github.com/vesoft-inc/nebula-go/v3"


How about nebulago to unify the alias.

veezhang · 2022-03-29T08:00:19Z

pkg/controller/component/storaged_cluster.go

+		err := metaClient.Disconnect()
+		if err != nil {
+			return
+		}


_ = metaClient.Disconnect() ?

veezhang · 2022-03-29T08:09:14Z

How do you think about it when it's needed multiple versions at the same kubernetes?

2. add balance job status

porscheme · 2022-03-30T22:31:33Z

@MegaByte875 @wey-gu how I test this?

wey-gu · 2022-03-30T23:57:20Z

@MegaByte875 could you help @porscheme with a step by step guide?
I think to clone your branch should be ok but guides to help from the build to deploy may help a lot to him/me.
Thanks!

MegaByte875 · 2022-03-31T02:06:49Z

I will update helm charts later, you can deploy nebula v3 by helm @porscheme

veezhang

LGTM

porscheme · 2022-03-31T23:25:05Z

operator scheduler is crashing, see that attached

---my helm values file---

image:
  nebulaOperator:
    image: vesoft/nebula-operator:v1.0.0
    imagePullPolicy: Always
  kubeRBACProxy:
    image: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
    imagePullPolicy: Always
  kubeScheduler:
    image: k8s.gcr.io/kube-scheduler:v1.18.8
    imagePullPolicy: Always

imagePullSecrets: []
kubernetesClusterDomain: ""

controllerManager:
  create: true
  replicas: 2
  env: []
  resources:
    limits:
      cpu: 200m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 100Mi

admissionWebhook:
  create: true

scheduler:
  create: true
  schedulerName: nebula-scheduler
  replicas: 1
  env: []
  resources:
    limits:
      cpu: 200m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 100Mi

# Enable openkruise scheme for controller manager. (default false)
enableKruise: false

# Period at which the controller forces the repopulation of its local object stores. (default 1h0m0s)
syncPeriod:

# Namespace the controller watches for updates to Kubernetes objects, If empty, all namespaces are watched.
watchNamespace:

# The address the metric endpoint binds to. (default ":8080")
metricsBindAddr: ":8080"

# The address the probe endpoint binds to. (default: ":8081")
healthProbeBindAddr: ":8081"

# The TCP port the Webhook server binds to. (default 9443)
webhookBindPort: 9443

# Maximum number of concurrently running reconcile loops for nebula cluster (default 3)
maxConcurrentReconciles: 3

nodeSelector:
  agentpool: metad

tolerations: []

affinity: {}

v3 support

7a5b2a4

MegaByte875 requested review from veezhang and kqzh March 28, 2022 16:34

wey-gu mentioned this pull request Mar 29, 2022

sequence number of the column in the CSV file doesn't work vesoft-inc/nebula-importer#202

Closed

kqzh reviewed Mar 29, 2022

View reviewed changes

veezhang reviewed Mar 29, 2022

View reviewed changes

MegaByte875 added 2 commits March 30, 2022 21:40

1. remove scale interface

e5470e0

2. add balance job status

fix comments

3a0e5a2

MegaByte875 added 2 commits March 31, 2022 14:02

remove unused code

dc55993

remove DNS error capture

31847cb

veezhang approved these changes Mar 31, 2022

View reviewed changes

veezhang merged commit facd74e into vesoft-inc:master Mar 31, 2022

jamieliu1023 mentioned this pull request Apr 2, 2022

Weekly Report 2022-04-01 vesoft-inc/nebula-community#102

Closed

MegaByte875 deleted the support_3.0 branch April 24, 2022 05:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v3 support #121

v3 support #121

MegaByte875 commented Mar 28, 2022 •

edited

Loading

wey-gu commented Mar 29, 2022

kqzh Mar 29, 2022

kqzh Mar 29, 2022

kqzh Mar 29, 2022

kqzh Mar 29, 2022

MegaByte875 Mar 30, 2022

kqzh Mar 29, 2022

kqzh Mar 29, 2022

veezhang Mar 29, 2022

veezhang Mar 29, 2022

MegaByte875 Mar 30, 2022

veezhang Mar 31, 2022

veezhang Mar 29, 2022

MegaByte875 Mar 30, 2022

veezhang Mar 31, 2022

veezhang Mar 29, 2022

veezhang Mar 29, 2022

veezhang Mar 29, 2022

veezhang Mar 29, 2022

MegaByte875 Mar 30, 2022

veezhang Mar 31, 2022

veezhang Mar 29, 2022

veezhang Mar 29, 2022

veezhang commented Mar 29, 2022

porscheme commented Mar 30, 2022

wey-gu commented Mar 30, 2022 •

edited

Loading

MegaByte875 commented Mar 31, 2022

veezhang left a comment

porscheme commented Mar 31, 2022

v3 support #121

v3 support #121

Conversation

MegaByte875 commented Mar 28, 2022 • edited Loading

wey-gu commented Mar 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

veezhang commented Mar 29, 2022

porscheme commented Mar 30, 2022

wey-gu commented Mar 30, 2022 • edited Loading

MegaByte875 commented Mar 31, 2022

veezhang left a comment

Choose a reason for hiding this comment

porscheme commented Mar 31, 2022

MegaByte875 commented Mar 28, 2022 •

edited

Loading

wey-gu commented Mar 30, 2022 •

edited

Loading