feat: refactor metering ,support Third-party resource access and idempotent #2258

xiao-jay · 2022-12-13T08:05:27Z

Signed-off-by: 晓杰 [email protected]

podResource，Resource，Metering，ExtensionResourcePrice api type （one day）
metering watch extensionResourcePrice（half day）
metering watch ns and create （half day）
pod-controller create resurce CR logic（half day）
metering watch Resource create by resource-Controller and update Metering （one day）
metering create AccountBalance and account will watch it and deduction balancde（half day）
Modify inappropriate places in the code by github Comment（one day）
e2e test （two day）

one module one commit

计量计费流程图

用户创建NS时操作和第三方资源控制器接入流程图

codecov · 2022-12-13T08:11:52Z

Codecov Report

Base: 66.72% // Head: 66.72% // No change to project coverage 👍

Coverage data is based on head (9197106) compared to base (75c1fe1).
Patch has no changes to coverable lines.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2258   +/-   ##
=======================================
  Coverage   66.72%   66.72%           
=======================================
  Files           6        6           
  Lines         535      535           
=======================================
  Hits          357      357           
  Misses        148      148           
  Partials       30       30

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

lingdie · 2022-12-24T11:11:39Z

controllers/metering/controllers/metering_controller.go

 const (
-	FinalizerName          = "metering.sealos.io/finalizer"
-	UserAnnotationOwnerKey = "user.sealos.io/creator"
+	MeteringPrefix           = "metering-"
+	ResourceQuotaPrefix      = "quota-"
+	METERINGNAMESPACEENV     = "METERING_SYSTEM_NAMESPACE"
+	DEFAULTMETERINGNAMESPACE = "metering-system"
 )


place these consts in to api/common_types.go is batter?

and notice spile format.

if METERINGNAMESPACEENV will not change, switch cr metering to cluster scpoe is ok?

place these consts in to api/common_types.go is batter?

i agree

if METERINGNAMESPACEENV will not change, switch cr metering to cluster scpoe is ok?

What is the purpose switch cr metering to cluster scpoe

lingdie · 2022-12-24T11:14:32Z

controllers/metering/controllers/metering_controller.go

+	MeteringSystemNameSpace string
+	MeteringInterval        int


use globe var is enough?

personal preference，both can realize function.

lingdie · 2022-12-24T11:20:50Z

controllers/metering/controllers/metering_controller.go

+	return nil
+}
+
+func (r *MeteringReconcile) DelMetering(ctx context.Context, name, namespace string) error {


no need to export this func.

a sample encapsulation，else need write more code in reconcile

lingdie · 2022-12-24T11:24:24Z

controllers/metering/controllers/metering_controller.go

 	return ctrl.NewControllerManagedBy(mgr).
 		For(&meteringv1.Metering{}).
 		Watches(&source.Kind{Type: &corev1.Namespace{}}, &handler.EnqueueRequestForObject{}).
+		Watches(&source.Kind{Type: &meteringv1.ExtensionResourcePrice{}}, &handler.EnqueueRequestForObject{}).
+		Watches(&source.Kind{Type: &meteringv1.Resource{}}, &handler.EnqueueRequestForObject{}).


IMO, why not split these watchs to defirent controllers?

What does IMO mean

because one CR should changed by one controller，if split controller will lead to 2 controller change same CR，lead to some confilct。

lingdie · 2022-12-24T11:34:06Z

controllers/metering/controllers/metering_controller.go

+	if _, err := controllerutil.CreateOrUpdate(ctx, r.Client, metering, func() error {
+		for _, v := range metering.Spec.Resources {
+			used := resource.MustParse(v.Used.String())
+			v.Used.Sub(used)
+		}
 		return nil
 	}); err != nil {
-		return fmt.Errorf("sync resource quota failed: %v", err)
+		return 0, err
 	}


calculate func just do calculate is ok.

lingdie · 2022-12-24T11:34:42Z

controllers/metering/controllers/metering_controller.go

+	var totalAmount float64
+	for resourceName, resourceValue := range metering.Spec.Resources {
+		if _, ok := metering.Spec.Resources[resourceName]; ok {
+			amount := float64(resourceValue.Used.MilliValue()) * float64(resourceValue.Price) / float64(resourceValue.Unit.MilliValue())


maybe MilliValue is use less?

like cpu，if use value it can not expressed as 800mi

lingdie · 2022-12-24T11:42:12Z

controllers/metering/controllers/metering_controller.go

-				TotalAmount:      0,
-				LatestUpdateTime: time.Now().Unix(),
-			},
+	if time.Now().Unix()-metering.Status.LatestUpdateTime >= int64(time.Minute.Seconds())*int64(metering.Spec.TimeInterval) {


Add comment for this, split to a check func. IMO use second as TimeInterval is batter.

no need to used second，usually will 1minute or 60minute ，not used often like 90s .

lingdie · 2022-12-24T11:54:11Z

controllers/user/controllers/account_controller.go

+const ACCOUNTNAMESPACEENV = "ACCOUNT_NAMESPACE"
+const DEFAULTACCOUNTNAMESPACE = "sealos-system"


can we switch account to cluster scope?

sure，user only have one account，i agree you

but not in a hurry，what good if account change to cluster scope？

fanux · 2022-12-27T08:59:52Z

controllers/metering/README.md


-### How it works
-This project aims to follow the Kubernetes [Operator pattern](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
+sealos cloud是一个多租户的，以k8s为内核的云操作系统，传统的资源隔离级别是虚拟机，而sealos cloud的资源隔离级别是namespace，每个用户都至少有一个自己的namespace用来使用，这样就给怎么计费带来挑战。怎么样计费k8s中 用户使用的cpu、memory等资源？怎么样计费流量等 metering 不可见的资源。


资源隔离级别是namespace, 这是不是不太贴切。

那该如何形容，隔离粒度怎么样

Signed-off-by: 晓杰 <[email protected]>

… Metering Signed-off-by: 晓杰 <[email protected]>

…uction balancde Signed-off-by: 晓杰 <[email protected]>

Signed-off-by: 晓杰 <[email protected]>

xiao-jay · 2022-12-28T08:51:00Z

i will update readme until I complete the docs

fanux · 2022-12-28T10:36:10Z

controllers/metering/Makefile

@@ -1,6 +1,6 @@

 # Image URL to use all building/pushing image targets
-IMG ?= ghcr.io/labring/sealos-metering-controller:dev
+IMG ?= ghcr.io/xiao-jay/sealos-metering-controller:dev


better not use personal account

zzjin · 2022-12-28T13:49:15Z

controllers/metering/api/v1/metering_types.go

+		//corev1.ResourceRequestsCPU:    resource.MustParse("100"),
+		corev1.ResourceLimitsCPU: resource.MustParse("10000"),
+		//corev1.ResourceRequestsMemory: resource.MustParse("100"),
+		corev1.ResourceLimitsMemory: resource.MustParse("10000Gi"),


Too large for test purpose?

zzjin · 2022-12-28T13:54:06Z

controllers/metering/main.go

@@ -90,6 +91,14 @@ func main() {
 		setupLog.Error(err, "unable to create controller", "controller", "Metering")
 		os.Exit(1)
 	}
+	if err = (&controllers.PodResourceReconciler{


Maybe pod resource controller can split into self-hosting controllers?

The next iteration will change to facilitate decoupling

Signed-off-by: 晓杰 <[email protected]>

xiao-jay force-pushed the metering-flowway branch 5 times, most recently from 35a8705 to c2151b2 Compare December 19, 2022 13:09

xiao-jay marked this pull request as ready for review December 19, 2022 13:10

xiao-jay force-pushed the metering-flowway branch 14 times, most recently from 93febac to 576b8b2 Compare December 23, 2022 12:03

lingdie reviewed Dec 24, 2022

View reviewed changes

fanux reviewed Dec 27, 2022

View reviewed changes

fanux approved these changes Dec 27, 2022

View reviewed changes

fanux approved these changes Dec 28, 2022

View reviewed changes

xiao-jay added 5 commits December 28, 2022 15:06

feat:design metering、podresource、resource api type

d36e84f

Signed-off-by: 晓杰 <[email protected]>

metering watch user ns and create

1e29ea8

Signed-off-by: 晓杰 <[email protected]>

feat:add e2e test about metering watch ns

da9b0f0

Signed-off-by: 晓杰 <[email protected]>

feat:metering watch extensionResourcePrice

529ab5b

Signed-off-by: 晓杰 <[email protected]>

feat:pod-controller create resurce CR logic

98712c6

Signed-off-by: 晓杰 <[email protected]>

xiao-jay added 9 commits December 28, 2022 15:06

feat:metering watch Resource create by resource-Controller and update…

fdbe7b3

… Metering Signed-off-by: 晓杰 <[email protected]>

feat:metering create AccountBalance and account will watch it and ded…

d05ddc5

…uction balancde Signed-off-by: 晓杰 <[email protected]>

feat:add infra calculate test

24ce631

Signed-off-by: 晓杰 <[email protected]>

feat:add deploy.yaml and temporary ci

2303e0f

Signed-off-by: 晓杰 <[email protected]>

feat:change account ns to sealos-system

27d919a

Signed-off-by: 晓杰 <[email protected]>

fix:update resourceName

a4901db

Signed-off-by: 晓杰 <[email protected]>

feat:add deduction info to account chargelist

d1a5403

Signed-off-by: 晓杰 <[email protected]>

fix:payment error

3a835a5

Signed-off-by: 晓杰 <[email protected]>

fix:update resource name

b075eac

Signed-off-by: 晓杰 <[email protected]>

xiao-jay force-pushed the metering-flowway branch from 1795e93 to b075eac Compare December 28, 2022 07:08

fanux reviewed Dec 28, 2022

View reviewed changes

fanux approved these changes Dec 28, 2022

View reviewed changes

xiao-jay force-pushed the metering-flowway branch from 02bddff to 669612f Compare December 28, 2022 11:04

zzjin reviewed Dec 28, 2022

View reviewed changes

xiao-jay force-pushed the metering-flowway branch from 669612f to 20a9452 Compare December 29, 2022 03:13

Modify inappropriate places in the code by github Comment

9197106

Signed-off-by: 晓杰 <[email protected]>

xiao-jay force-pushed the metering-flowway branch from 20a9452 to 9197106 Compare December 29, 2022 06:20

fanux merged commit fd8ae0d into labring:main Dec 29, 2022

xiao-jay mentioned this pull request Mar 12, 2023

feat: KubeArmor based Github Action kubearmor/KubeArmor#1128

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: refactor metering ,support Third-party resource access and idempotent #2258

feat: refactor metering ,support Third-party resource access and idempotent #2258

xiao-jay commented Dec 13, 2022 •

edited

Loading

codecov bot commented Dec 13, 2022 •

edited

Loading

lingdie Dec 24, 2022

lingdie Dec 24, 2022

lingdie Dec 24, 2022

xiao-jay Dec 27, 2022

xiao-jay Dec 27, 2022

lingdie Dec 24, 2022

xiao-jay Dec 28, 2022

lingdie Dec 24, 2022

xiao-jay Dec 27, 2022

lingdie Dec 24, 2022

xiao-jay Dec 27, 2022

xiao-jay Dec 27, 2022 •

edited

Loading

lingdie Dec 24, 2022

xiao-jay Dec 28, 2022

lingdie Dec 24, 2022

xiao-jay Dec 27, 2022

lingdie Dec 24, 2022

xiao-jay Dec 28, 2022

lingdie Dec 24, 2022

xiao-jay Dec 27, 2022

xiao-jay Dec 28, 2022

fanux Dec 27, 2022

xiao-jay Dec 27, 2022

xiao-jay commented Dec 28, 2022 •

edited

Loading

fanux Dec 28, 2022

zzjin Dec 28, 2022

xiao-jay Dec 29, 2022

zzjin Dec 28, 2022

xiao-jay Dec 29, 2022

		const ACCOUNTNAMESPACEENV = "ACCOUNT_NAMESPACE"
		const DEFAULTACCOUNTNAMESPACE = "sealos-system"

feat: refactor metering ,support Third-party resource access and idempotent #2258

feat: refactor metering ,support Third-party resource access and idempotent #2258

Conversation

xiao-jay commented Dec 13, 2022 • edited Loading

codecov bot commented Dec 13, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiao-jay Dec 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiao-jay commented Dec 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiao-jay commented Dec 13, 2022 •

edited

Loading

codecov bot commented Dec 13, 2022 •

edited

Loading

xiao-jay Dec 27, 2022 •

edited

Loading

xiao-jay commented Dec 28, 2022 •

edited

Loading