Make per-namespace Quotas configurable

[blairdrummond]

As per this: https://www.youtube.com/watch?v=KUyEuY5ZSqI&t=653s

It would make sense for the metadata to be stored within the Profile metadata? Maybe?

The exact location of the metadata isn't super important, as long as it cannot be modified by users. We should decide on and implement this, incorporating it with the design of aaw-kubeflow-profiles to keep the quotas in CI.

[zachomedia]

@blairdrummond My thought for the easiest implementation was to use a label on the Profile to store "overrides" to the resource quotas, so like quotas.statcan.gc.ca/gpu-override: 2 for example.

(reason for a label over an annotation is we could do filters in the future to see what profiles have overridden limits)

[chuckbelisle]

@rohank07
As discussed earlier, the block of code that needs to be changed is located in this statcan/aaw-kubeflow-profiles-controller/cmd/quotas.go

// generateResourceQuotas generates resource quotas for the given profile.
// TODO: Allow overrides in a namespace
func generateResourceQuotas(profile *kubeflowv1.Profile) []*corev1.ResourceQuota {
	quotas := []*corev1.ResourceQuota{}

	quotas = append(quotas, &corev1.ResourceQuota{
		ObjectMeta: metav1.ObjectMeta{
			Name:      "quotas",
			Namespace: profile.Name,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(profile, kubeflowv1.SchemeGroupVersion.WithKind("Profile")),
			},
		},
		Spec: corev1.ResourceQuotaSpec{
			Hard: corev1.ResourceList{
				// CPU
				"requests.cpu": *resource.NewQuantity(70, resource.DecimalSI),
				"limits.cpu":   *resource.NewQuantity(70, resource.DecimalSI),

				// Memory
				"requests.memory": *resource.NewScaledQuantity(368, resource.Giga),
				"limits.memory":   *resource.NewScaledQuantity(368, resource.Giga),

				// Storage
				"requests.storage": *resource.NewScaledQuantity(4, resource.Tera),

				// GPU
				"requests.nvidia.com/gpu": *resource.NewQuantity(2, resource.DecimalSI),

				// Pods
				"pods": *resource.NewQuantity(100, resource.DecimalSI),

				// Services
				"services.nodeports":     *resource.NewQuantity(0, resource.DecimalSI),
				"services.loadbalancers": *resource.NewQuantity(0, resource.DecimalSI),
			},
		},
	})

	return quotas
}

When the profile is created, the default quotas are applied and the the labels included in the profile.jsonnet are also added/override those affected defaults.

Storage label name: quotas.statcan.gc.ca/requests.storage

All that would then be required would be to add quotas.statcan.gc.ca/requests.storage: 8 to requested profile in order to allow 8TB of storage.