From 50bb208119baa39c1f11205f7955d24320bd56ca Mon Sep 17 00:00:00 2001 From: Hector Fernandez Date: Sun, 17 May 2020 12:41:03 +0200 Subject: [PATCH 1/5] doc: proposal custom cloud provider over gRPC --- .../proposals/plugable-provider-grpc.md | 123 ++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 cluster-autoscaler/proposals/plugable-provider-grpc.md diff --git a/cluster-autoscaler/proposals/plugable-provider-grpc.md b/cluster-autoscaler/proposals/plugable-provider-grpc.md new file mode 100644 index 000000000000..b39ab9fd09ae --- /dev/null +++ b/cluster-autoscaler/proposals/plugable-provider-grpc.md @@ -0,0 +1,123 @@ +# Plugable Cloud Provider over gRPC + +## Motivation + +CA is released as a bundle which includes a hardcoded list of supported cloud providers. +Whenever users want to implement the logic of their own cloud provider, they need to fork the CA and add their own implementation. + +In particular users need to follow these steps to support a custom private cloud: + +* Write a client for your private cloud in Go, implementing CloudProvider interface. + +* Add constructing it to cloud provider builder. + +* Build a custom image of Cluster Autoscaler that includes those changes and configure it to start with your cloud provider. + +This is a concern that has been raised in the past [PR953](https://github.com/kubernetes/autoscaler/issues/953) and [PR1060](https://github.com/kubernetes/autoscaler/issues/1060). + +Therefore a new implemetation should be added to CA in order to extend it without breaking any backwards compatibility or +the current cloud provider implementations. + +## Goals + +* Support custom cloud provider implementations without changing the current `CloudProvider` interface. +* Make CA extendable, so users do not need to fork the CA repository. + +## Proposal + +There are couple of examples of plugable designs using Go SDKs that would guide us on how to extend CA +to support custom providers as plugins. +This approach is inspired based on [Hashicorp go-plugin](https://github.com/hashicorp/go-plugin) and [Grafana Go SDK for plugins](https://github.com/grafana/grafana-plugin-sdk-go). + +There are two alternatives on how to plug custom cloud providers: + +* **Option1:** Install the plugin as a binary that would be mounted into the CA container +and invoked by CA server. +In this option CA server launches each provider plugin as a subprocess and communicates with it over gRPC. + +* **Option2:** A custom cloud provider server is deployed along side CA and both communicates via gRPC with TLS/SSL. + +Regardless of the chosen option, both solutions have to expose a common gRPC API +with the following operations: + +```go +type clusterAutoscalerCustomProviderServer struct { + ... +} + +func (s *clusterAutoscalerCustomProviderServer) NodeGroups(ctx context.Context) ([]pb.NodeGroup, error) { + ... +} +... + +func (s *clusterAutoscalerCustomProviderServer) NodeGroupForNode(ctx context.Context, *apiv1.Node) (*pb.NodeGroup, error) { + ... +} +... + +func (s *clusterAutoscalerCustomProviderServer) Refresh() error { + ... +} +... + +func (s *clusterAutoscalerCustomProviderServer) GetAvailableMachineTypes() ([]string, error) { + ... +} + +... + +``` + +Obviously, these API calls implement the `CloudProvider` interface of the [CA](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloud_provider.go#L50): + +```go +type CloudProvider interface { + Name() string + + NodeGroups() []NodeGroup + + NodeGroupForNode(*apiv1.Node) (NodeGroup, error) + + Pricing() (PricingModel, errors.AutoscalerError) + + GetAvailableMachineTypes() ([]string, error) + + NewNodeGroup(machineType string, labels map[string]string, systemLabels map[string]string, + taints []apiv1.Taint, extraResources map[string]resource.Quantity) (NodeGroup, error) + + GetResourceLimiter() (*ResourceLimiter, error) + + GPULabel() string + + GetAvailableGPUTypes() map[string]struct{} + + Cleanup() error + + Refresh() error + + ... +} +``` + +In order to talk to the custom cloud provider server, this new cloud provider has to be registered +when bootstrapping the CA. +Consequently, the CA needs to expose new flags to specify the cloud provider and all the required properties +to reach the remote gRPC server. + +A new flag, named +`--cloud-provider-url=https://local.svc.io/mycloudprovider/server`, determines the URL to reach the custom provider implementation. +In addition this approach reuses the existing flag that defines the name of the cloud provider `--cloud-provider=mycloudprovider` +https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider. + +Obviously, this new apprach needs to use TLS to ensure a secure communication between CA and this CA provider server. +The flags need to be defined [TODO]. + +## User Stories + +### Story1 + +When using CA only a reduced list of cloud providers are supported, if users want to use their own private cloud provider (e.g. Openstack, OpenNebula,...), they need to implement its cloud provider interface for that environment. +This locked design limits the extensibility of CA and goes against certain native Kubernetes primitives. + +This approach aims to make CA plugable, so any user can implement the logic of their own provider +following a pre-defined gRPC interface. From 26f941852c9098e0a443e9c162c149b4dde91ede Mon Sep 17 00:00:00 2001 From: Hector Fernandez Date: Sun, 17 May 2020 12:49:48 +0200 Subject: [PATCH 2/5] chore: mention the new cloud provider --- cluster-autoscaler/proposals/plugable-provider-grpc.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/cluster-autoscaler/proposals/plugable-provider-grpc.md b/cluster-autoscaler/proposals/plugable-provider-grpc.md index b39ab9fd09ae..e5b0d30d6a3e 100644 --- a/cluster-autoscaler/proposals/plugable-provider-grpc.md +++ b/cluster-autoscaler/proposals/plugable-provider-grpc.md @@ -109,6 +109,10 @@ A new flag, named In addition this approach reuses the existing flag that defines the name of the cloud provider `--cloud-provider=mycloudprovider` https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider. +To connect the CA core with this new remote cloud provider, this approach needs to implement a new generic cloud provider +as part of the CA core code. +This new provider, named `CustomCloudProvider` simply makes gRPC calls to the remote functions exposed by the custom cloud provider server. In other words, it forwards the calls and handle the errors analogously how done in other existing providers. + Obviously, this new apprach needs to use TLS to ensure a secure communication between CA and this CA provider server. The flags need to be defined [TODO]. From c11665bec83ee40f3315fca93720ccbf6efc06fa Mon Sep 17 00:00:00 2001 From: Hector Fernandez Date: Wed, 2 Dec 2020 19:40:58 +0100 Subject: [PATCH 3/5] chore: add gRPC contracts Signed-off-by: Hector Fernandez --- .../proposals/plugable-provider-grpc.md | 534 +++++++++++++++++- 1 file changed, 519 insertions(+), 15 deletions(-) diff --git a/cluster-autoscaler/proposals/plugable-provider-grpc.md b/cluster-autoscaler/proposals/plugable-provider-grpc.md index e5b0d30d6a3e..8bc1e3c1e264 100644 --- a/cluster-autoscaler/proposals/plugable-provider-grpc.md +++ b/cluster-autoscaler/proposals/plugable-provider-grpc.md @@ -15,7 +15,7 @@ In particular users need to follow these steps to support a custom private cloud This is a concern that has been raised in the past [PR953](https://github.com/kubernetes/autoscaler/issues/953) and [PR1060](https://github.com/kubernetes/autoscaler/issues/1060). -Therefore a new implemetation should be added to CA in order to extend it without breaking any backwards compatibility or +Therefore a new implemetation should be added to CA in order to extend it without breaking any backwards compatibility or the current cloud provider implementations. ## Goals @@ -25,7 +25,7 @@ the current cloud provider implementations. ## Proposal -There are couple of examples of plugable designs using Go SDKs that would guide us on how to extend CA +There are couple of examples of pluggable designs using Go SDKs that would guide us on how to extend CA to support custom providers as plugins. This approach is inspired based on [Hashicorp go-plugin](https://github.com/hashicorp/go-plugin) and [Grafana Go SDK for plugins](https://github.com/grafana/grafana-plugin-sdk-go). @@ -68,7 +68,7 @@ func (s *clusterAutoscalerCustomProviderServer) GetAvailableMachineTypes() ([]st ``` -Obviously, these API calls implement the `CloudProvider` interface of the [CA](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloud_provider.go#L50): +Obviously, these API calls implement the `CloudProvider` and `NodeGroup` interfaces of the [CA](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloud_provider.go#L50): ```go type CloudProvider interface { @@ -80,8 +80,6 @@ type CloudProvider interface { Pricing() (PricingModel, errors.AutoscalerError) - GetAvailableMachineTypes() ([]string, error) - NewNodeGroup(machineType string, labels map[string]string, systemLabels map[string]string, taints []apiv1.Taint, extraResources map[string]resource.Quantity) (NodeGroup, error) @@ -97,31 +95,537 @@ type CloudProvider interface { ... } + +type NodeGroup interface { + MaxSize() int + + MinSize() int + + TargetSize() (int, error) + + IncreaseSize(delta int) error + + DeleteNodes([]*apiv1.Node) error + + DecreaseTargetSize(delta int) error + + Id() string + + Nodes() ([]Instance, error) + + TemplateNodeInfo() (*schedulerframework.NodeInfo, error) + + Exist() bool + + Create() (NodeGroup, error) + + Delete() error + + Autoprovisioned() bool +} + ``` +In the following I detail the main operations of the CloudProvider GRPC service: + +```protobuf +syntax = "proto3"; + +package clusterautoscaler.cloudprovider.v1; + +import "google/protobuf/descriptor.proto"; +import "google/protobuf/timestamp.proto"; +import "google/protobuf/wrappers.proto"; + +import "k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto"; +import "k8s.io/apimachinery/pkg/api/resource/generated.proto"; +import "k8s.io/api/core/v1/generated.proto" + +option go_package = "clusterautoscaler.cloudprovider"; + +service CloudProvider { + // CloudProvider specific RPC functions + rpc Name(NameRequest) + returns (NameResponse) {} + + rpc NodeGroups(NodeGroupsRequest) + returns (GetNameResponse) {} + + rpc NodeGroupForNode(NodeGroupForNodeRequest) + returns (NodeGroupForNodeResponse) {} + + rpc PricingNodePrice(PricingNodePriceRequest) + returns (PricingNodePriceResponse) {} + + rpc PricingPodPrice(PricingPodPriceRequest) + returns (PricingPodPriceResponse) + + rpc NewNodeGroup(NewNodeGroupRequest) + returns (NewNodeGroupResponse) {} + + rpc GetResourceLimiter(GetResourceLimiterRequest) + returns (GetResourceLimiterResponse) {} + + rpc GPULabel(GPULabelRequest) + returns (GPULabelResponse) {} + + rpc GetAvailableGPUTypes(GetAvailableGPUTypesRequest) + returns (GetAvailableGPUTypesResponse) {} + + rpc Cleanup(CleanupRequest) + returns (CleanupResponse) {} + + rpc Refresh(RefreshRequest) + returns (RefreshResponse) {} + + // NodeGroup specific RPC functions + rpc NodeGroupTargetSize(NodeGroupTargetSizeRequest) + returns (NodeGroupTargetSizeResponse) {} + + rpc NodeGroupIncreaseSize(NodeGroupIncreaseSizeRequest) + returns (NodeGroupIncreaseSizeResponse) {} + + rpc NodeGroupDeleteNodes(NodeGroupDeleteNodesRequest) + returns (NodeGroupDeleteNodesResponse) {} + + rpc NodeGroupDecreaseTargetSize(NodeGroupDecreaseTargetSizeRequest) + returns (NodeGroupDecreaseTargetSizeResponse) {} + + rpc NodeGroupNodes(NodeGroupNodesRequest) + returns (NodeGroupNodesResponse) {} + + rpc NodeGroupDelete(NodeGroupDeleteRequest) + returns (NodeGroupDeleteResponse) {} + + rpc NodeGroupCreate(NodeGroupCreateRequest) + returns (NodeGroupCreateResponse) {} + + rpc NodeGroupTemplateNodeInfo(NodeGroupDTemplateNodeInfoRequest) + returns (NodeGroupTemplateNodeInfoResponse) {} +} +``` + +Note that, the rest of message used in these calls are detailed in the [Appendix](#appendix) section. + In order to talk to the custom cloud provider server, this new cloud provider has to be registered -when bootstrapping the CA. +when bootstrapping the CA. Consequently, the CA needs to expose new flags to specify the cloud provider and all the required properties to reach the remote gRPC server. A new flag, named `--cloud-provider-url=https://local.svc.io/mycloudprovider/server`, determines the URL to reach the custom provider implementation. -In addition this approach reuses the existing flag that defines the name of the cloud provider `--cloud-provider=mycloudprovider` -https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider. +In addition this approach reuses the existing flag that defines the name of the cloud provider using a pre-defined value `--cloud-provider=external` +https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider. `external` defines the usage +of an external cloud provider whose interface is handled by a remote gRPC service. -To connect the CA core with this new remote cloud provider, this approach needs to implement a new generic cloud provider +To connect the CA core with this new external cloud provider, this approach needs to implement a new generic cloud provider as part of the CA core code. -This new provider, named `CustomCloudProvider` simply makes gRPC calls to the remote functions exposed by the custom cloud provider server. In other words, it forwards the calls and handle the errors analogously how done in other existing providers. +This new provider, named `ExternalCloudProvider`, makes gRPC calls to the remote functions exposed by the external cloud provider server. In other words, it forwards the calls and handle the errors analogously how done in other existing providers. -Obviously, this new apprach needs to use TLS to ensure a secure communication between CA and this CA provider server. -The flags need to be defined [TODO]. +Obviously, this new approach needs to use TLS to ensure a secure communication between CA and this CA provider server. +Additional flags should be added to specify the path where to find the certificate to establish the communication to +the external cloud provider server. +This certificate would be mounted when deploying the cluster-autoscaler on the cluster. ## User Stories ### Story1 When using CA only a reduced list of cloud providers are supported, if users want to use their own private cloud provider (e.g. Openstack, OpenNebula,...), they need to implement its cloud provider interface for that environment. -This locked design limits the extensibility of CA and goes against certain native Kubernetes primitives. +This design limits the extensibility of CA and goes against certain native Kubernetes primitives. + +This approach aims to make CA pluggable allowing to create experimental providers without needing to change the base CA repository. This new experimental provider only needs to implement the gRPC defined operations. + + +## Appendix + +In the following we detail all the message used in the defined actions of the `CloudProvider` GRPC Service server. + +### NodeGroups + +NodeGroups stores all node groups configured for this cloud provider. + +```protobuf + +message NodeGroupsRequest { + // Intentionally empty. +} + +message NodeGroupsResponse { + // All the node groups that the cloud provider service supports. This + // field is OPTIONAL. + repeated NodeGroup nodeGroups = 1; +} +``` + +### NodeGroupForNode + +NodeGroupForNode returns the node group for the given node, nil if the node should not be processed by cluster autoscaler, or non-nil error if such occurred. Must be implemented. + +```protobuf +message NodeGroupForNodeRequest { + // Node group for the given node + k8s.io.api.core.v1.Node node = 1; +} + +message NodeGroupForNodeResponse { + // The node group for the given node. + repeated NodeGroup nodeGroup = 1; +} +``` + +### GetResourceLimiter + +GetResourceLimiter types store struct containing limits (max, min) for resources (cores, memory etc.). + +```protobuf +message GetResourceLimiterRequest { + // Intentionally empty. +} + +message GetResourceLimiterResponse { + // All the machine types that the cloud provider service supports. This + // field is OPTIONAL. + repeated string machineTypes = 1; +} +``` + + +### GetAvailableGPUTypes + +GetAvailableGPUTypes handles all available GPU types cloud provider supports. + +```protobuf +message GetAvailableGPUTypesRequest { + // Intentionally empty. +} + +message GetAvailableGPUTypesResponse { + // GPU types passed in as opaque key-value pairs. + map gpuTypes = 1; +} +``` + +### GPULabel + +GPULabel stores the label added to nodes with GPU resource. + +```protobuf +message GPULabelRequest { + // Intentionally empty. +} + +message GPULabelResponse { + // Label value of the GPU + string label = 1; +} +``` + +### PricingNodePrice + +NodePrice handles an operation that returns a price of running the given node for a given period of time. + +```protobuf +message PricingNodePriceRequest { + k8s.io.api.core.v1.Node node = 1;, + + k8s.io.apimachinery.pkg.apis.meta.v1.Time startTime = 2; + + k8s.io.apimachinery.pkg.apis.meta.v1.Time endTime = 3; +} + +message PricingNodePriceResponse { + // Price of the theoretical minimum price of running a pod for a given period + float64 price = 1; +} +``` + +### PricingPodPrice + +PodPrice handles an operation that returns a theoretical minimum price of running a pod for a given period of time on a perfectly matching machine. + +```protobuf +message PricingPodPriceRequest { + k8s.io.api.core.v1.Pod pod = 1;, + + k8s.io.apimachinery.pkg.apis.meta.v1.Time startTime = 2; + + k8s.io.apimachinery.pkg.apis.meta.v1.Time endTime = 3; +} + +message PricingPodPriceResponse { + // Price of the theoretical minimum price of running a pod for a given period + float64 price = 1; +} +``` + +### NewNodeGroup + +NewNodeGroup builds a theoretical node group based on the node definition provided. The node group is not automatically created on the cloud provider side. This action returns the created node group. + +```protobuf +message NewNodeGroupRequest { + // Machine type for the node group. + string machineType = 1; + + // Labels of the node group. + map labels = 2; + + // System Labels of the node group. + map systemLabels = 3; + + // Taints of the node group + repeated k8s.io.api.core.v1.Taint taints = 4; -This approach aims to make CA plugable, so any user can implement the logic of their own provider -following a pre-defined gRPC interface. + // ExtraResources of the node group + map minLimits = 1; + + // Contains the maximum limits for resources (cores, memory etc.). + map maxLimits = 2; +} +``` + +### Refresh + +Refresh is called before every main loop and can be used to dynamically update cloud provider state. +In particular the list of node groups returned by NodeGroups can change as a result of this action. + +```protobuf +message RefreshRequest { + // Intentionally empty. +} + +message RefreshResponse { +// Intentionally empty. +} +``` + +### Cleanup + +Cleanup cleans up open resources before the cloud provider is destroyed, i.e. go routines etc. + +```protobuf +message CleanupRequest { + // Intentionally empty. +} + +message CleanupResponse { +// Intentionally empty. +} +``` + +### Name + +Name stores the name of the cloud provider. + +```protobuf +message NameRequest { + // Intentionally empty. +} + +message NameResponse { + // Name of the node group + string name = 1; +} +``` + +### NodeGroup + +```protobuf +message NodeGroup { + // Name of the node group on the cloud provider + string name = 1; + + // ID of the node group on the cloud provider + string id = 1; + + // MinSize of the node group on the cloud provider + int32 minSize = 2; + + // MaxSize of the node group on the cloud provider + int32 maxSize = 3; + + // Exist reports if the node group really exists on the cloud provider + bool exist = 4; + + // Autoprovisioned returns true if the node group is autoprovisioned + bool autoProvisioned = 5; + + // Debug returns a string containing all information regarding this node group. + string debug = 6; +} +``` + +### NodeGroupTargetSize + +NodeGroupTargetSize returns the current target size of the node group. + +```protobuf +message NodeGroupTargetSizeRequest { + // ID of the group node on the cloud provider + string id = 1; +} + +message NodeGroupTargetSizeResponse { + // TargetSize + int32 targetSize = 1; +} +``` + +### NodeGroupIncreaseSize + +NodeGroupIncreaseSize describes the message type of an action that increases the size of the node group. + +```protobuf +message NodeGroupIncreaseSizeRequest { + int32 delta = 1; + + // ID of the group node on the cloud provider + string id = 2; +} + +message NodeGroupIncreaseSizeResponse { + // Intentionally empty. +} +``` + +### NodeGroupDeleteNodes + +NodeGroupDeleteNodes deletes nodes from this node group. + +```protobuf +message NodeGroupDeleteNodesRequest { + repeated k8s.io.api.core.v1.Node nodes = 1; + + // ID of the group node on the cloud provider + string id = 2; +} + +message NodeGroupDeleteNodesResponse { + // Intentionally empty. +} +``` + +### NodeGroupDecreaseTargetSize + +NodeGroupDecreaseTargetSize decreases the target size of the node group. + +```protobuf +message NodeGroupDecreaseTargetSizeRequest { + int32 delta = 1; + + // ID of the group node on the cloud provider + string id = 2; +} + +message NodeGroupDecreaseTargetSizeResponse { + // Intentionally empty. +} +``` + +### NodeGroupNodes + +NodeGroupNodes describes the message type of an action that returns a list of all nodes that belong to this node group. + +```protobuf +message NodeGroupNodesRequest { + // ID of the group node on the cloud provider + string id = 1; +} + +message NodeGroupNodesResponse { + repeated Instance instances = 1; +} + +message Instance { + // Id is instance id. + string id = 1; + // Status represents status of node. (Optional) + InstanceStatus status = 2; +} + +// InstanceStatus represents instance status. +message InstanceStatus { + // State tells if instance is running, being created or being deleted + int state = 1; + // ErrorInfo is not nil if there is error condition related to instance. + InstanceErrorInfo errorInfo = 2; +} + +// InstanceErrorInfo provides information about error condition on instance +message InstanceErrorInfo { + // ErrorCode is cloud-provider specific error code for error condition + string errorCode = 1; + // ErrorMessage is human readable description of error condition + string errorMessage = 2; + // InstanceErrorClass defines class of error condition + int instanceErrorClass = 3; +} +``` + +### NodeGroupDelete + +NodeGroupDelete deletes the node group on the cloud provider side. + +```protobuf +message NodeGroupDeleteRequest { + // ID of the group node on the cloud provider + string id = 1; +} + +message NodeGroupDeleteResponse { + // Intentionally empty. +} +``` + +### NodeGroupCreate + +NodeGroupCreate creates the node group on the cloud provider side. + +```protobuf +message NodeGroupCreateRequest { + // NodeGroup to be created on the cloud provider + NodeGroup nodeGroup = 1; +} + +message NodeGroupCreateResponse { + // NodeGroup that was created on the cloud provider + NodeGroup nodeGroup = 1; +} +``` + +### NodeGroupTemplateNodeInfo + +TemplateNodeInfo returns a NodeInfo structure of an empty (as if just started) node. + +```protobuf +message NodeGroupTemplateNodeInfoRequest { + // ID of the group node on the cloud provider + string id = 1; +} + +message NodeGroupTemplateNodeInfoResponse { + // nodeInfo extracted of the template node on the cloud provider + NodeInfo nodeInfo = 1; +} +``` From 0b0524643255be38cb82176bc7c3b36b452a716b Mon Sep 17 00:00:00 2001 From: Hector Fernandez Date: Wed, 2 Dec 2020 19:50:49 +0100 Subject: [PATCH 4/5] chore: add authors Signed-off-by: Hector Fernandez --- .../images/external-provider-grpc.jpg | Bin 0 -> 39488 bytes .../proposals/plugable-provider-grpc.md | 51 +++++------------- 2 files changed, 12 insertions(+), 39 deletions(-) create mode 100644 cluster-autoscaler/proposals/images/external-provider-grpc.jpg diff --git a/cluster-autoscaler/proposals/images/external-provider-grpc.jpg b/cluster-autoscaler/proposals/images/external-provider-grpc.jpg new file mode 100644 index 0000000000000000000000000000000000000000..1d9889d3a68df17528f7e0052a2f655c5a665acc GIT binary patch literal 39488 zcmeFZ2UL^IwlEy56hT5Sf|5cnLI_QYKqv{#gaiylL202EQ9-cMI{`umfrJu@^o~lE zUP3PxkS;cwf~Y_5J=gbm);aH8>wfos-}nFPzw@k>XEM*s>@s`L%%0gZKc{}a0kEQU zk-7jnIsky~@B#ds1!x0~9{D5u@q6?z96NFBk3fIo#PJgh^o)!Q^b8D)r%s<@WMW}r zU||-qdzRj%EZ8QnEIa#Kc54D^mO23=11v3 zfFnSZ!9RR^YB##`W`y&AUJ)ERJ#Ov76BXo?1-7dHd`YKx43%9em^S$oUnORn;}M zV+&w8c{5wD8#i;c@DGRy<1n3TkJ=#G54~Fvrq^@Y=RY075;(+x?)Yz*0074iVVyWh zN6&EhSr2#!`pEHPrx+QJ0*?M0vWKVuj~(X_LZ4uhjpyWoYhp~Xwq6g273|y~Ihc1& z>&T~*5P2=1gi6}zJXitu@KM#z8NjKdhcp9^0s$I;+S_*R!u$na-D=pUKb(~+>9Uv2 zSwwrq$KN0y9qM?xcvReL@{Fl2r4$827xw=VkO4(g_#L>#v!nK}5a%;#6%OC7j=nZO z5505@*P9ufZHg=Yq+q?MopgaoJ-^uejcZePI2Hgj7r)6BXk_-}`Sh)L7VL)Pep>0( z2bCW|&`SUvtKVu=HZ8AOcdog6qOUnD=^`DccY&?r*tqinbFv@nl5yKm5q114zw{;u zcb2E7sL(t^bCIk-0q%7n08zge{jC@f>3KFbz3B#_cRAuIrwH?SM8Vs23pvllT0V~4 znGiz9)3!@}`h<2u*N)b*VSkOAR1lCsx99B)5@GXn_DX=kxGBSXJYL7>+XLNfxa*~||go$jz>GB0Dw+`ZPbru0*SOo@4 zRRVOH|5q9c@ZTaAze^gL)Aj=8+Q9C!f~RFRbmI4aDefQdBaJjJ-M+cvngGW-z7aGI z+Y_2mqD+Cj4i9<0tCz{|0a?*0`9+Jm=e>B+k;Z zf;m}-si>sxiC}~I;W?793SH>sFszoPh+*`1Q~eB2n!aU-PCq1!>E3VT{I17gDq{<^ zKBCRHi@d*@55O*3=h-%2RYi6Kq20}$s~^%=_jy@AqH6D#U6UKSzuc@8WGMOrfTeRXp*j?YKcas=QkVel(}Vj4 z0nS_sX{U7LLdIhR-p!ha_olhYQ|IooSnCf6y;d$i z*12rFcDX9>$=b)_6t=?O8v~GZ@%Nl`ocO0#Y&}X-!Xx%U7VEa=Tw@UBoK?p}qT6`oCVf9KZ(5VQ zPnMYiDD_>OH{q;_PUt)?V1dS+6^wm>`OtRZ?1^{G*Z?Dc+0&lIAE2TU2dP4K)v0T# zwGMtIQOn)*vjveQ+Vg{Q=Ao$dS=2^PnOt5PmW}fKd=lmSN?ZAIa}9#GosBJ{dh>Qa zEs=r}kOrLOpC>eZb~#^k^OH6(7`00-T86V%DTpfv#CNA!Va2UNt}?Teb={lt7%fu4 zFc?OE6T>X>vCx%5t1%ahYuLd-!^|K!PI5kZ?d~`b2EU)u-e)s))AG!0bdgc5O9)&{ z9Ha&nkpiQLS<3_=pg8J3rZdpwX* z#2v5)QpHbny9(tR_RnlRB3=~;^r&;CH>son#l!yJrl$Xl@E<9l5|JAv_@|TGU&i|x zu)hl&;5QZjBWFa(YlP+d*(0YnN`g?izi8E{tOWk!MBtCS|55lKHvLE3`SaWVu<1`$ z{6~@hESuzAI`2qD2jyLpC@#Wj^*~*hLu#~X{$g`g!YNJ-1y&kbL=DwV5DdD}jCBjD zh4mF#>pIIdK$IJbAbq%q+q-tl34I_?MCrTRoT{gpxOS`=o)&-V)VJ7e$-~NfoEF@% zOdF#errc_@3)=Dd3a$@n`1tJPbwVLO!KkD^C1Y@YL_j%EK#!fpuYAz&hjX}Nxh`s* zp2ZK|w3E;qQC0!;DUdGNGS#_sftgY7C%}N2SLo~;HGpz3O#GdSWs=1~N9!{IkuwEL zH6BL(#h}{?Z)RH>16y=kx-#5Xag#yumh<1=8Y|vVdDo_2`e_Pa z!8er40io+dsP%0m=TjE>x3{#*njsOdneQKUbdgq@8FM4nEy=tWd-Az}zfr(<9%A-Y zX*BvJOXDY^M1TT-IkTkxiwn9I^*Zrcb^fzI0Q&v`@Xhb=6mC|&eOanIEpZp~gO>Ia zz~|MJGkJwg$Kz47(#zp@PZ&UT?KXhvnZDSYHT?fooFGj1bd1WR2qHO){!m_}2!tHh zvjI1e$~w-U02Bs=UaBIRNX0vW&c6-jDi)D!{7{4HbRQK*@f2 z(SJ!>Tc63NJy8AzT-!4*zhY?2o3C`{_6W;0j4sEFAeHr=%G?vGFd+OuuSW zijN~-N=D39TqHQ`BlJuIb%`js-KQITZt%}0J^3f!gA#x)#oZPdq~$XcrBtXMhbN}K z1)nDx+JizqCZcG_H6i-}AlLIj+0Z*f>{cyq^Q)KZ#Ce?TI-fBnPs>;LM#`7EudO4& zGm_(V6?Ns33KeB9AubD&OKq=P;FPuSsF_L8titGUPDe4n$rZ?Ksf?kiCbQSS5FyF& z$#$Qcx$UO#B}D(ik&0AfYf}Hx$Z5HHhf4|J!wNkNLojVA2?H#eSG2#jPv)tBt8ZET z7&GC@-1NAP%p>wZ9hY}9pX9W{7$ueBCOH-`*o8k;+x3JiE)f^C=v}AsMH~se@Tm!z zHnpc)oj?3i-{(r{U`sF2aPHko4q86{vw8iW0)6IBP2gHsn7;7!Czbi|dGxgK!JEAGms>e;V5UmXw(eBnVXdUgw(F$W;Ha3>sw1mK6qBV*GkZrl-*HeB7RuR` zyPs{x)_2PKeR4sAn9Txs3wyqKcbD(=#HgFsvw69kh=*y&+#A#@;SX5AU+YB^sf`ht1IzlN3&{fsd~c1Njip)?#)xj! z>x5;jE(&(AyQ!g0Ra0mrKMzNS@=lj8X^7qW^dh4y`m+6TxoECr#tY+dOPNd3KY033 zZ-OuWy%+v_lUWgLX^-FT}2}c{Z=YD?*aKrVv3cmv;7?* zG(wApz?wy;XIXyQxf zGi5a$D7cFLQp1@vPEYKsD}e$hwR%b#YdhW8DiL~8-HQ!>B;z`zm$hpEvK+yd#*>N7 z)M~Ori6ObFUia79!Jfm%&W{)HYB14B>95RR=%wR?PC?P)3!M0F zg0lrN(Ky0a`lkFK{0pC{v)ceWNPdHtm)DocY1c$8?2#4Y)(0blFFhYwJuf!GhVMS_ z3i!-;0C{sU-iePm#&*hWhc7wdt22@0c_?7C$Br&_T9u@+m*0eHrc9w?~rOG+{;CiQ# z_+ODMIGtnT=M)}pN!{D(vm+Wyb`*4UvjvdD=dX{^6#zU@+VcC zUylqh(fwm#hI>r)xg{S<9n~q+G+i`QOtPVnrX(`wYb^FcUO#HQ9D&dqQ>&botabWY zZgu)Y)XwX2)_bbg#%{AtGvipBPQd22bujz%#SAYcxIAjUtL^w4!2!+K8_gtNyX@at zUp|IE_n$Cz!Rb4GHYG+Ud>7lafr_@W{hH!@0*kfF?x{qqZf^Ike(hy?>O>U4lKNQKCdf4v4>JV$)D&B#(uG+D^nDm(=GZ? zd)$OB-Y)oo!zer3luLB4B9MBg^!~40M+)xag>0l`Wu-#u=kmPx3L2-Fy&7~E;3mH1 z;8XjN5kYxnUISFv|tY}ubtiXjyz3kft6zYh%^K9U(_G9a!1!ko4#~q%#urD zmc2yt>Ly_;of=#*GX`r8&?>Fy8pqs{*h0Uv)oPWkBj7$mD+A?!QJBsy_{zcy@0$W!m^g5P`NZ)!4o+cD2isbGcMumf_1BLYBfrQer*5mZpI;t1Ebo6m#3 zNeS_UApg}0jIwX5K*Ptm18roK5|tc+#sT+!$uSeXzKwNIq{?)a;5v%AqaE0SWUz$=};}y~sCVrr1X{@yj>k(Z8i-JIFI zY2zaDz4EBbjeRHBExLA0Tp9+S>5 zJ50suLR5QDeR})I8L`<`=uvr6lg;IekR9K&^xbfJ*!u%C~;rMj8RH^XTSOeyT(RXAgB*pKT6WRwk zY|#f#UQQ2}?MJ9?Ia#_jR_C;VERed0TyTGtbXoV0@^7D>fIz%U)$bh1!8y7~kb_OR zX`ziX6g%M>Rpw1+@fCt@4c>QBlH`xwJz-wS#FKTjHv!$3DVBG6*(&uZs%Ml|m#o1- zQ=z4`SwD-nLbm1mazOS(W5TeyWu;(eN!m=tkD0hfTBo*D2ToSrxi0XiX9Fs0>%OBc z0DhFw-Z$$m)Dv(GX{ZC0taq5{?(U}D(M#niSjUj^EXJHMu`rlwfB(G`e3Plwi}|u? zylG*@@GT#QVu{Zqeh(Y;(l6I$vz)2Ir-YTI?NP^YIJb9C>#kUei?G9)XPXoTgF`5g z#1@!Db_H8UquB^WcpgeHb}T0%*~W>OM3OU1byqIY?R#%jY?pnYA$HuL+nhl|Dt|g6 z9NpfoqT5-as8~Iz%`}jKf`0O~6zR{_ z+!A7*K4yxf!fpT85UlR53NjmS7cCA;0twVLBDUY?$SKu?#Ct-qnsX8b~C%}@@aV|yBpE2rU) zGp>4bfAyJ=R#C87(>Tj>U=T#qlv-hShLA=a0=3QpeZ8ac8B?xJvV7Bx&Jn4!8%zFs z2F>YH+_h7yegna$d-`=1+|zRNZ)6oqOqlr6j5b)JN2=2NjbQEKD<3L*b62NU^-o&M zLGqlZ@&F!qV^b7_+R`swMJfMvL!RZrf{Mf!L5xEV_fw#>lS-35F>Q z&jX2;GsX`#TvJw9_YsJW&W2(ggwj?BB2$seSv9N@)PJzRZJD?lRuz2rOOo$~t|ZNT zHO+ZNrcM$KbUf||MIX^z*-o$0F*WAq{m#ecMVfgOULkN=!}O@-q%!m-n{ieiV@WTf z*r`ZcUv!z;r0#O3Dna30b1HN)ub}@*SaG&wjS{NHry;XBemuf&Nv@CGfmq$Hq`MhD zMu6fatG)Xx)e@%9W++^7rhc8<-!G-61%+C2gtxY}wFw~!A(*wouV#aGJ6nJ&;yXFa*trEa{yw0Jq-t_&T3wDo60IX{f>tWUG?A z0RVUv=AV{wH6*moXr+4{BNs*`CP;R7?}unyJV_~ebJ<$7NWQD4ZF@o1SQ|-CjlfAu zzz_{zuddPT4POTH zuYeD5X4$d*Mh?fd#AFKBm1)C$MPb5+5^AA*SfnD#fHscJw#xTC{g<;tDzt|d{JY^G zdHNVrp4!QcQ;tKdPud)!SK}hc{yG4F4C&&(k;kOyD9jP`gmK5t-uGNsCUV47!9=sY zLoB#dHw=R$((LSsm1<8Pm)sW(>)(#IJeyJ_4XthXT<4%rl|4MxUyC>IAxw$6i5>g4 z8rYSh^=cd&+=xJ#%fIp?=@99+!04=nFuWr~Nq^-rq_=75V*UPnqLH&BY(zIaHGb7O zb&X`_w~pUf1@h#Ss=Q-?E}yYg(sDQl`6%#F*Bc5R}glTbw4xgB(fDkQ@@@{gH4;obj4?BRh?c> zq==5?{n058x%S^X{?+x9+klkX^oKzJKtl9y?JUWC^jn_Vt&LM>hj^akwt8wb!Y1!N z*F5+f$&rnqtO~|6t^W1?6KhA&KTd1@TrSfTNOK|0sX;-i=+6IU7s4+2cDpmQq+gP;QRRqA?r#yuxd(#4$}l z1o}`OKgw+~zWsJGCasD#)9Ye3zI2?Qfgga5NWMspW>_|muddQ!t&wp` z+;wn35INZt?p#Q{sENvlrK}c;5v0yv8bV09X|qK|BW6x*>N@CTg>(6_cAMgsTy%4G z*Hp^fL=ot_cvdeLg0vtWm;#Dj!=dcl)oCJ-9pJ8+^q`fA1&C6$W+0lGo)ohcJ2^lhQIL?@`65(cu*HhRkj z!REYMSCDtz0VDp@hmKngru1U|ZH8NSFmLLbQm*QlUy)ww#+KJ$$&AB&+uNJ@`%53B zJ+x!RJ<6^uEN}lCFx}D0*dGD+dc(gN4xf?{#J;v_=^R+~S~A+{o3nZqOwyq96`i?> z{s|b0-9u{-(&&PJAXcBTOGo z{?#>>5a08iPH_LiL+RfU)9O2WzcJD6!r^|#%8P@-?CJqP+vk5rB=i?%`0JmzG5-Yy z$(@JMBQunBG}Wl`4g}hW+_cP;omc>UInBd4*C!{~xM%-WN$~vI*yXrsWupxr%xxny zn|R3KO*>+f z0e`g@&Q>@b{XEBtRXS?>Y`JW`LG4*?Yk#$N<5Eyeu< z>_7kf>I;wkFZYFv<990N-8TX!ya!&)e9JZ)lk82)sE@$tclZMt7os6~;4rN|C=i$x z63&-URSBqk49ZZsW-)W^ZcJ2(TU&mVu=$w=yIyOBpn0b$om$M~RtA^=DzZftEI^{v)Qwh9|WBlcZO#5Z-{;hleW$r%uCLzRD zc%<4>%m0e`#&+GxDGc4MuR>}v61)DETOneAq#S?sN`<5~z)p==N^_?0ym|6>yvSRA zoPeI82s=IDM&og`k7$kEtlibr@%0TZol1v0TAij-Hpz+NXQ<9RNeI!fu`xO*gXZ3h z3zTs8I~zMvCzPRZU7K9ooTmT!)YnK+q`G&(XBNWR)psUQN}=-GPj(8rq%|Z2j!*ji z1l)ivK1zyt_^$aUppkb|GhJ)`!b;(ncRWJIzp|duPV+b4AAQSx_P2b!FEo8$-Ftaz z@7zF+XXe(Jdc{MIw_!desGk6f0NW1b0?NHoby=^y%d65&L*H_;A!kZ%a9gi@A08r} zn&D=6t)~4yApXC@DDAH$vcsu@Q40ZjW3a%4pMbBA6oT7s4PDyrqBlJ`53)584dIHN zv9EX^8&+w~eY(o|8r@noNg`r%ndlr?qnNv;FI6Qc*<)wvU>~2HNpd*H&|mV{N+cCL zW~ExWEwaK0Mq5OEDz{enc4;R*`GOn>X~^-HV=-eUgj1**jvh|SJehHYWS@04OFbi8&GXLa=je6I!PP6z zV~N^EjRzm^Z}jegujz@G#FoWsfK03ew-0Q)6DCi#Xz+|JC1z?W1&dDKI8oA5NJEWZ z$|TNC!N7t~^>%JqsC}CFio1Jm%8$+e&6k3=EQqXnwY9HZE!{uL-swH%dwmkT-&892 z)g?f{=Ks%m_Hl`he4jQnI;DBXC!G4w!+oG3jX~LpOU-Xg>@QFFogIJa^rhYM&Eh?C z-@Mn%9wBh)n8d=n%+?YE7o_D(+z)o)k&daWWbT!SZsjJtb;tz#&o;aSx~d%fXT6YF8)rF9yuxzZkw z1~SIkO@i=*R-+BTF=WS|gDn7X%2I&K?aODMvyW8)e;@rf@Inpe_pM(Q-03#z2&H`- zE3p3bmmaQp4vI(q((t>VGyf_gkahTd+MJh#7}6~@+)2h&l=mYZ1fo(`5k@;;F#Ia7 z#aLR=79FVDt=LQ4EqOXDHWGjDc_@_k?gC`%aM?vG{#&;3Hxk6hUyt@)heaZ1WWFmZ zta9)=&~jW69vITc1a>)DApq>ZR<@N)UDatnXdHC#O^(>P{abStLVs2py zYrC7g`!A7G^%%Dz#_zfP(5$9<@Sm=JRvip53J3;&QP2D$sqg)(*0*=5i;1y)9-GV* z)7w*ll<#EGBQA?#XB)OZHeA#GMe=F>9diizuI0TrH}UkY_V+UHg)XwW?oU=7%L~1E zLIKB4B51GT9lz09;syGlpE2DvtzC+_3u<}NIq^ICZg@Cp#~9l%TqnFZ2!-k&4Y#Q32K5Q05|R(Lz%$OP~Cr>;G|zUL#BPs?NhV_ii|reZT)` z?=8QY$cPy4QJPTTFKN!h0ppMq>Gio?5;eMd;9_uYyQNTFI*PesKDuDM#PSWNk&q2Q7EFFV=B+lr0weEk-8gCHWc_pY79u5 zXIVIpZU*(gPGRG_WAw<&lfs)oewA9X)h7kJrDx-k{M7bXqeai4c-kwvg@;v`x_LX! zh~;5V8C9b-X*qNLBh5+YRcTR~Advi)Zlye=(eL`FA_KfXEzswh%gjV$2~|{qQeoOi zy%tt5^T-WBZm%|~4a(s|B*=B|V(|8rT8`n5f}$Y#(wxaXj8Jlhri9KFKgVht7RxfO zR|7Ixj2|+43aEGvs2(`#QDAR~3pbF>tNi_H@9^8TVH#Yu)<}HVxG)q*B}U4YUP|n* zOl}Qa?Fi@vJ^_~-Xnne&NFr3@UY}Vb* zSV(+C4b)%Neen7F-x#Y`ZWnmzBv5Q7rKm?5EIK3pgv6<^Qy!eIsco9TkVmZP1lgVl zJsV=x+NrwUJ?ATNqZ;D^j|T!-3g;PVrli~+ePe^ON-27{^Q^GF{(uRa4a?77A2lnD zm7G0?5S3c`L_839cZ=P?Er$%vk4qFfhD2cwliYx;-vR$PS^fvN-N%e0tqf~^SF32R zGh|UqQqZ+X=E7R}>XzK0pMdh319mFO1vUSH#jZa`miDO7CLU(NWw$NsKv7sX*urK) z(oRKnZ+V^2i{}-9*Y3QOu+*twHy)S{#~^jz3j50__Gho0+co&|-mmMr19g30Yh)VL zj|(YFq;2aoF17mKh{x^_pJ@7gYBkYHeCmd+pTV&f^wLpyU30`xX3fv;rony!)`Xg0 z_lEfBPc=oGna1}%Oh*#Ln!Aob#Zb{HAvQ0)YT`wsg$WppA1|+~Ykc&rQ+I9hz9-*W z#`Cuf<@PRGTGYPG=w4dFcE4RL;4xfpmT5o#&JjCIGtP?1@>kKx!aG=AYxFP9Q|-#l zg(xH@FxKTZHNe(cS5jgceDEFPiJiJg(5jj3(&Lu>is>RvLU(CyKb0Dr79x~dusWTJo6DlKPX1)?w{HQL^Vm_z{02@#CB-@*nA#4 zsZ21Xz~(b?E>OfUUy!$fpR`WL;LltVmNT$<`xQu@6q71pQk-=MJn!s@$5=E~9ls;= zq&i@tD7C|o+oFL}RGqspCCp2lK!&%UY)gv?cjDzwhh#oPI-R+W5eErBOD#%Ux;C^j z7v$sWb!ap6Z7>U&&oweMymYiM0WV=rSfsTY2ss$F*XR1K2zc%&>COx=^EOmc>~fb) zwzs@eAzH**T}Imn2IN8ri1LJpp#wJk*%;bKZIQ_RH<4FrK=ByI=i6L^g|oB zuJ^KxBDDI2-g8X27?zpPsoeGAR17BZM|D!f-B?}K{n3LEQl@-$KF5g2^<-^@UQYV`*HCTwwdy3L83hx2l|w1@|9C3=R#W!As>$(ZAD4d& z9e>ev+}A^7DuTKczXDI75gB#G6G)xqK{bpABZC7>|JajbC zjLbPaw3TLg(TK98d`gTqydnA(H&z${9c!Yh=D8SEqKi}Ege^-k4bBSrgSEBiI|8Ry z9zVeiN);0j`(N^XqQpAEXbk+u?p1x-0A~1eNmqM&b7hE|iDzT@F=r_Qc(F|a*>A^A zq9`oA=%fXH&8XvnrryR+65jHv`^N0;;&Tu%jeSN1u&UPkJ&2|*`* zd>xKY$_1nS921IWu5pWj#I)mA<+N@vkB$!P-rEf{^@dy0z^cg`%FKbqh~5g$SISHUDVrNm#p1yvWUaW5rU z+;#LWUe8(_!!AicZNJ`Tw}P;xqlvHOiTrNlQdL^mP}=kqSnj2$t8wh3Dt2cFoFdnxBDOrm9)>c@JdO7=fe0ErMIpFB27UwLXB(}U1~i$$l+ zGL%rsHy2xHk#ix!%2ifWm2(+#0huTxnQFwyQoM+M_vRD`_GR+x%l!A-y{kQM3y~SF zzRhVp5YO|UD#T0^y4TVTyh!O_q_YKNrV0jk1*QVCkZO0qbYAG%w(HQS=87j)c#3;x zn|}p1vy@{Xl_p$mv%Sqgro5m}pv3c49nggPdf4xa_3xW|<{oZCb1M`}5ITr?LD_JV zI!5n?jGEI>7Iv}Qw6_i>MUWU98WBW8Z)qyMmOHxu+F#+i{V?#`k3}rU%*+15G%X|U zqe9_W$Tf;rSx2R@N+|7-L627fefN-@6j(B>`|f*T3F%|ZBXOv;Z|{NrYjs6pNt))` z7la%byt!>3leP#5QzLS$(O80?U_5tbT3AOO_XE^KVNk>8WNy~tLUH~a(1QeB!`oBt z0`=gKKo%jR%Gt#vux53hcB<;u;UV))E~wccU6T>F%VUtxn^Ep_R`FgXtrUneRw1W2 z{m?D&C%{F|VApY65FXFzP`st4N8JS>S)Z_#+|xBG?Z|I-IqhqJh6hbB>gl3h#@Mr% zEWCoQQ?&+^<=au3B4%h_2_unsuK46qr5$HHG)2u4KV5o3mEzoE4a1|d(kegePsc)? zfs6a>IjlXaysl1J4p}+GkefNJeO3Hm*Z0jKD+naD$ppwGaE{8Glw;p686Te>7wJjM zpU}lWp130>IiQndvVW_mgwX+D#R5`4y+<&DkjY&bny~ zgHBZzFvuD(-2Lpa;qv->`5Q#(3xh9E;=zmhfWo*<4=d%CFMoJr}LaXE~(C zNuVu86;^%`tg!{9UU<;TnOArrR3m@=j%@U&ATTP}w4|g@Pk&;eJ6*VY zt9!sN6&N=nu6iuPceqG=-C@xHirukSG2Hi+ip+cCAlnlL=gT&fblKPK6+ow^IthmF zEjZOE=x{xKW7W9S7?saZpKuu;ubIe|i`lf#)K4EBH7m|~Eo6e-cB#y(cj4mV-+Hvn%BeqRcEGFn|t%2~biYu5aS zN7dn3%YQ%Y#=cPLX)M->NFN?rx(AnBKW++@+ype<3oU*96UKSJl0kt>OsI8-HbT#4^7`* zmackVEN=Yx@n&5f%vt%Zz!dvt5J?<|!0WTVhA=jj{rKYIf96-#RDfSb$`Ss>2j;%7 zU%mIxCLXt*ymzOjh5g(KZyS1#Uw>KjId55i|1hy__{g=u&H9cR#wTJSD^6Svc@8Rm zoeeFScL|bbr1zLY94FrNOH3&WtKc$p9n1O5%K*zN3?K5c+&hPRdFxA)b+Mk{gp}Ei#4AjLi)&po@4#HDT!vzE9KF#SSS z52I3@qna~03(&4rGOv*R*Ox{KK@=D7tH2_$wh{Cy6rcPxK793Lj9CNBDnVN?36VGY zxG@%WO)Z8t2@;q7zFMUtWUNXxKHSK>Gx8xG2b=0?mXtKSnQp9v#P~wE?7wvfH45R) z({_IXBF($5^GVVRGWSLqF*S#zCi040(xi%^Z7c8Zgr&dLB_T$2`#>LxoF_8-2zBa@T3(`$hYJ1ymH?Jz`7v*umZu(_^&JBZ-+Y*}OX-WJ~^P1W&v_qu6L3CYN~0O2g|n@i`qj&ZsPF zwy?`#X44@y)3!hfu)&G7`ll{Knr`u;G2r5Yr~#ba(kGsvh-S7uX(0-23g2&4Pw4>yPJmaNJ8@4X@_GS9FfsRQox{_%vG8N-hkBKbI}j2VSZ4jWCnN$f*U|_Nk*tgW(gs!wycWLn&&xv~w1q zVxRb1v22N9mDtxRiATb0G5MX6>H>t~&tf>e0B63|Lmw)W0cR(Im1Hi9pfL0}Yp5V{ zy{yQHoai`US1Cny?W!+*Ht7$lrjE;$zEc7DX=A+W_o6k9Vl@q*G(w`b=#3fpVx0&I z=A)-mJ8SI670-pzV2fB$yBv|sDFx;0@Xvj9W6@S5sYS@hn4GxyBkz9lfViI3H8cb3 zXQ8PTexw+IAX1r!SZYbN{zjckOD~c6PDcrNnezDjxCf#IRIRHB_-N-*5s0G!XE%fThdQ=RB zCh?}m=g=d8ybgD3!=CxM|44pwyPEv1XiNTUrW0Ln#PYtCQlr_&D|=ny;^JD1mv;Ip zo&8@`$P)@VTkBYR*+QYc#)+C~*e{Cl$%|&rLx(0|xaC{_oU)Sj$KFdr6k_;s4Uvd& zA}z^Il`5AOev`Kj#3F9Vk+JOHVo{00#Y6Q6kg8#9Axn6Z<&WVDw3)w2O2E;KBxT1Qjc>T z-lW&($|7sAd+(H9ti7}9G+w+>DJ~dyKgN#$SCqSvJ$hHl7+CickgJUxk|$y_CR9=2 zB(%sY|LXi)@$OZGHSO^mcPr-Xh-^0TwZzh%G0CRjEHw-Rlf2kVYk{mTbo zq+~d|hH6yQPr$dcwNUqyH{iY05^~n|TvMm%cts6XVgn|f@-#{7x#DS_Z%B-s6tuDB z)L0E~mM(I}c)+uxX2N@}Tj zd*I}rxGuC@gWDA*ID@RHmBSI_$OW%{cz}4}y?>(W%oUb_0 zCuurwiEQGFFA<_awA;JkyrjXd>#JuT7mC4EAM0SvmDzQ}%f9$-B9vbW^vtuaru=^Y zh3bSOFd$W=f+nqL?jJfYYRo_1JD9RyCvh-iS|qXovRE%6FW(!o zH7mLB>i+n$?NpOMyVZF zg^?RKBI9tca=xP2L%w=f+Dn7nl+!^bZIM7mr8NNae? zt5)p| zC(TALdEq^-_QT>d)htsM=+`V~Tt$OM~(^#{7StrU2C$D}9^@_lfV;xv}? zQDVrvqo;VJ&wY);tJO>?>3L67O^9D552-y+I`y?WLI<pfOq9xiW_&W zCpo8yzSaz4CG;pT$dT}o)r_9!Y{a9tPMG`i)wH=b~CRL!5zX?)}!)Hxr zUDGdRCgzE_MMy(%iXmICwKEQ*aq`sRxzE~6uJrD(O-+Y8 z4t~)Of4@ab!p+Ut&X=Mf{#6GlQQ%LRyu-RBTRD)_Df1?`(fdH6br|CpU=?6D<3)G! zR3c;uLm($0Nz`GkXoUCcKTwMB&K2?q2^Z#r;;U zvOwVnF`%Jq?rW}D_r6Jn-$q=%zIFWUM$mkmv~8weJA6$br<8_ zA~JdFHh_*p>z}X(w)hV1>_P&x^#6=u3cu-LzQwd}3;pQ-*mmy@()$nhhPL0`8@&I? zAe^#XIcv?!@3L$wkU10;qFb>gE4v}`Iqt>Z4ZhgF8GMgE9yO{t^EqaNVNc5dNGV*2 zx@BY1%XV2q9{>W-S>jx{K zfK&;fNFa1jLPv@~C<(nPASw_7p?9Sy7D_@V)KCRNFGA=jqV!%uuhOK6s5B83e|d}O z+wT4DTKBHI?)rWCgA6CLXXebAGqY#*v!5-McXf7%m@=S`-X?Q;wML7&#AO`pZ zpqM#YZi~S@$6Uqt!X~yRxg!1QpA#IGcN}>b1xipW@{+T5wo=D(eS}Ok1uUZzA=uhvIsdBoXvh4Egye599fFtYFc_wn0Ob_k$6?O{^=s-^qh{rA)@Sk@St?CG`J&jO~@?+Sl~ zd%pT-JiUaK3-8q*5c+VT>Mv~veFv5rKkfLkNic=PZ-DV{fNr}6So&)0ldt|?uh}g2 zZZv)abZzr_Ok~&=VjQ>(r+s0a{CxVa`<>X>h*UN49z>#tG0X%=7b@D`g$O3l4VtzC z4Y4XG9rL}chlWg%bl0G8j*ff>eM38q_bdG!xiD3moUk|?ZaML5hgDj<=^jO_jBc(4 zmyrBJ6Ci#l(M^x`b$ly=bB*6OL}nu{-e@zZ*y@tIF~zPPodH8#EDndrgyo@kWr8eQ z^o|JnFbMW6Ro7D3A#7c&E-T!<`fPGJ{(Rfe7A8OwmzQmvOoOD#2-wZKx%v<>+c=;Wj-Y3kU{^{{s&93(v36Ko zIoijvC7TQn#^PynJR$Um$?xtr6%cV;xm7_F>edJUtZOVPs%pYrd<{qEnN$%g{~T% zme!Y?G-l5~h-qr7rJ!60adi;GUkI-nsw_qZEXL@Ek*q}`3pwklDOovN6kmDlR{3N* z?E))V&mQFtwipYzXl-kXeJBc#w1!QHj?1>ve}9URV%=po`4d!nK+^Plc@k`BmuQl~ zFq=Xg$mGrBeKw7WMXfW`fhLyA&M7fxWn55PQr&>q!InuLtJ_aEraG5_1M`j)pGXG5 z-SuO$+N(6Xcf0ijq6T%^&8(M*3-%beVH^`8Q|uxHB3I^F>}YhU`0aW~QL-rY(Dh!1 zbni2(`eyL3ZUT(AyS+*#$1-k8-mUD5{ynAEP%}_9znzCU@5hf~2`VZDD7z$4gPAMu zy)u)rMh-?p1S4LIHWp+tPEtZyt}b&8e2tla3_w5-p+`)w9()~0d%AmsW;yKYRC`s+ zPk5noYlQ5!9Od4K*KV0GMrRWp==KwnUfYBJ(EEjDmH)rslrErlj3@i`y*`b5E-e?M zw+12)ZgREs&jxHidosT2+{>IzawbU$Y)Vz{bD=0a}Gb7NP~ub*DlRDI|P zrUlXxin?xnr8wAR*XFcr8<@E1WIa#Yhi|efH|%yeS<4*|aYSlfLh$an?A@gs7Y}sgHwF&4SDWRyq|IdF5`JvNd_J@qcD^)HP;Ke8D#57U0Ar}nu zm2PlfO%e;+@>I>5+zS0j`8ovVr#VqrY^`f>@&bs{`Ze{CnfGc_v5PwVy0)rXl4g4s zr%*A0ZrFZcZCUc+^Mxac!*-lk83d?Wo{FhF!^A;#koj)>lr|JBdE{0+Aj3%~x;`!> zG^M?YN|i*1=6RV?DnOOzEvuvtOR`}IwFM+BFyMjNemUAYSR7&y3Id7q673+M4jE|5 zaC|h$$}$;)*q{Pzl>?jkx*s!@i{g}dr{H0`?kmSYFw81=nib zRM&?d}2HybQEyLdc+Exc^Vg*M5HP|$x+uSA>XVFvn2HQKj%~djd@eA8aeFYkV z7~Whr9DbWPq@nv7Z!brVaEF?3NWI?5xzBr|nu+fl;MqKB|Bx&44d6-j$C3dJv;U{E zB$jDJgj}YO1V!cu3thYvp1an5_LX6$Da!0Uwb;BqGRF|1U4@v}+%|-3KUmZtg80&g zZWJ2%p@xQy)9T_3u?qvkjO6cLTMSv_IF{mK!U`|07xO8MN#(z!Y6>4E7zI*XC@G^b zo-qkjq+oE{=}E#E^amxF!3dq^7_@!`=}Zud8V8MuaLo2ScoIt98Di$X6c;eksaT1f zL_6ARuI_fX_Z&YR$CMLijm2yU=QtKs0a(C4+txyW!5k6UK|Vrbxh(lhd1f0}m=g|~ z7jDNz%Og4|M+fTOB@eyju<);WfE{AZ{~2|TU93*+uMcGB3QG3;^ondeZd3KO(6Yxx z3$+f8oauYFUwB#jMVD6h%GW_8M!Op?Urq;}4!2sev>MY{;E1Stj-(x*+G|JHvy0N7 zZ}bPAoH>zvXjbFY(CcH~c#$52X!nHZ zGFml$^1$jFz~&o3q~I&Je~x~`4(Xk?NRPSv^wc-N#s2K^pcVPiP2#-}`9!ysb85aU z>9HAG->-qjCIj`&uCozi*7W6>OxPLR#x1lg$2CeftL8x+*)39mK-z2yy|?Q~R3D@p zu{b)0TdS^VFOicp{TP4eIJ?o82Ax*bfr{aY+}yI&TDlM!>U&cej{@CTKqokyiY66t zESOeewr$~7R3;}^n(~gh<%y5|TXn0=(;dg$-cD_*38~0z#|6z7Ff>SE<`3%7dI?CFcp~#zg-E6K~@DPrb_8_esa?;i| zWdu&*h5-T0UNACAe4S>Tx6?w>-h|z*+jPz%v!uxe7ua*+6!0mR@O3~=Ffq+t;vpTYtr3{!)tMfI7_^=>(tA`&J$tq z)6uv^)su<c}x#&m1fr**Ytw_B~t(vmLp@6k^RJk6O zRFNJnQiLG_^v+WOlw;2t=mMq(-bE?(<`z}xGgRR;HR<#nYmfu0&2-zLFt!dG9La$S z@bs4Ss`#^NnTE%2Smft>5=1rAtW`B1zrs9t6cD0Ar6!RLc5IYcKxsE>@rk3NfhdQ> z@;%IxLOZv+&Urra#rLxYO*H4{7Wl7?No1Pchsr|IG!Vg>$>Md^nW|5uHpc;4K?oIG z8uW_vH-JmQr)QhOPYQYH*WB_V%p-a_0;B;kVgIg#TuXA71;sbo8zy@Y^? z;n7>7;#6Cd(dD=_8t@kIwf6c3i2JIDiOeqCHDGC#p5GX04}pjhjr+t@DxiRgBH8O) z&s+6B41Ca1R#xuZxtivU2#|Ov4$OzkJp>8Gh_dxJNHtb%UQ^CpslKzE1lh%=1RICZ zBVl?D_1^%sRchU|(d%1KiEd>Jv$I!yhz6BZM_tbqtPE)+dlrBbD380;NiU%kfs)ML z*jTtLbdCh^;+Ad^i_=_;eh1HmtR#|;S=-9d&=He7E4@46=&;C-;ih0^m0V*N0rKdx z@<(b-hvMVn-drBhpNV^e$K90|hTgG zu)I&%IIb%4-ENKeZ-C=933Uc}`#u3x&kPbZI0hIQ`k1WO*Bk}6^^19K!gWN~XmQ~k zpn3~wMjpAX8`red(RqaU4+jy=KLhbw9PPuroPh6>bUv10@Ox}hy(K6UaeEDu=jX6P zbbH?TI-UqV1D|ICZ?jud`me){Q@MnAu;^HmMTA_4%GS%0r}wmYK9AX7 zw<>q(3n2Q}-@p9j=zW<{=JhGjIkEJ27R9a+Jx%iTu!CK|yZ6+9H-{O6en=l}cp-K4 zM@E)I>08v_0RLEz!OsNV?8_xuFXmc36-CnF)=bVppjF&Mu)FHsd>m)sd;(>&sacO6 zmqrh|#&an~Y|GmQA4h9lSt2rCW7CLIn0ax|=Epu@dAiqo#n;uaS-Gw%)lQ4KLWoi< zpP$>1MEXWhRU2lUn7eh7X}2I%rlh`NN+XUzSvQxqWtBA9$GL1Od6=U0t8@zEZ`_GQ z*7!fg!VuS&Y+!bmTm*%=IeOUxjUU;<^5GrAz%-~@DHJJc`yrhQYj32f-k-+Rf+;Wdf;b0dYq6r|m9kY*5<1@rVAO*7vfJDjWRR;gug4=hIix~tf=}J{ zc8nXyM0HJj?tM<xB01dmDrMeSQ;*kV4sNf6I7gI@?c+MBsG445*w}-LnT_joGIZkZCn!4jYVPLeh zd1P)c^m58HyjM+R_oX{W(rT!oX4ia(46|VLEqt^S6w;YUf+4SA;N}LwIf(N9FGKOu z=+fbHW{|La1f$VxjuTz>ba{iNzE&$pa&%CT4Qf!eG~MqjNycw`Pz-Ji ziF#AM7U7L=N)d2rzmc;bAoVcRhl4w|xS3VvL<~t4?7ATQg4~(XlX0L#NR}`=O<8in zkzS85WymJWRePr65duAkjZNPRHHnW-gFiNE>6jU6vy{a?iMh`62x}zS6Pz``c!2hp|Wfg`qcKd}hipDqGJTk~sXxTv?V;iyKM>z5)bffqxWPe_EHFJ4`Q~}g9_WxQkFgeq zX)+R@&sDjC?;(-hKVt6u$ii%1CDPMnAKJwg-&}v30%B2rHcfKQQUey8G!&KTN~zt8 zbq5KCZE|*#i38B!JNM%auZ}3&uO=4ULviNWM#$b;irNATbL{qVO(<3obsh6#wABji zaLZ*`!{^^T#e6<)k%O&0H&sh4aI=uo&Ob?YMq6rrk|?$oqJh$@njtOyPgJC{RY*Lk z$1;xqpOi%g30Sa$<3~PXRW9Dkcz{F+_e+A#Y_m4J&!lq5Ksl0_BdqPkvdxu6&YL?p zK6SRaeb1aX^OlXSS8r78u&ygQiA7Ib(~bxYBcTl_KzcyIaMK&E03mg9Ro}3p%en&) zzH^;*5V(`Sbb?}A3E55!A%HgsJ5RB!|45k&QT)hb0Ot^utdB3|!smr^n?y&i0|5$; z0f46FfuSO6y;G*G2K}`3E6mo4jhZ(s1kgd&YDxXX?yyPlc~etq6e4A7K(HJDpr-$s zh~fsmL9pA!K$Ge?Ih?6pLopG{3rGa`k?%Tm;tjBPXm^Vuy(0-!_eJ) zZ;Lvb$}bh^nc-g3{p(YHLW4sN(D6_9?i6M)LuEn80W`s*!f5#Gp%B}u}DCGpxw1|tB^(!uYU1cXr3@m%~soPQv^BfT2qeJJ>2LW-8 zkApyKG{YLsw*0;Plrm2;@dh;*w+@#`P3yN%p+MT@B!&uHu8oaVApvx7SOMGpSJNXK zceL+mN~oTcDplngeOp-<;=O`SI+5Z%h6&-`DDE(zG>C5JdH7@V)sW`RloZC_V+h(;rPZh0so@Iuefp@5^N)|`j?(cOF>YhF+EiDc7T`dRuGXd)JZh(#h^ zEk)zflzWZWUNHzV(CWs1Q0}rgpYXE0xk`*9_Bia%G+=skirwfVnsfsjF>}03GnvzSg1{THNlei$IY};eVuk%4p?4n zLfmo>J!YxpHEEVRZYa7*t!Pjg1x5s58y#L8Mu zm&Losm7MEt_w)nLJs%;WpRDfE8V#2MXFsVt>8XY4XjO{#${BFa|2#-aLcer7r0(B&JJcmrnph{wWy z@r6;^2JGB{cbh!(9`<9b`B5k#tkBt0qJrHOI=j>lelTE#?Eg#%vq76t#delc+m~Vk z!&or?ACc9;F{l?k+<=d_%lLU=k2J_39o6P z)gzsTFjI_BG!LY`B5oMc*)X@}tJbRJ()u;3ps?oY%vbd;5)4T5x_dFBY{GD%UB0^n zC2Qce7E8jxc;0YNG~%S~3pe=@tzyH*MQE+BU18%0_e|Y8#meKFDhf^QkHA=*?Sx+^ zJq8c8ySj@ZjD3l!x=s&ERS_%~!$x09eWCkqlrG19FJCD|t9j>?VBYL}e!k&uNNDtxuJj1Q%J?*l$CMidZLct=o(a9|L}aqiaVD5p zG})M)@Mj|9@LY{CTu(SfRJN);Uf7csvPSnmC=1&nKhq3X|;*%)dDf6h8h|8PGFc z%_A~@j7>YHAk60PQO=A(6oC6*er}B>X{~nO&KO;;7 z5u!EiLX^C<;buw;NVl;>b&q0GKJj8wKsosgq_5@Yzr*-w4i~E}Ro)&7&8ang?dHs} z|8n#wa-Qj)w3PDi@ z`nEt#0>!Z@F*XJnW{X%K29hT;>|P@WvqiW{E7wcu_1Nh_$Tt9?Jt4^O+L@IPZ0#s6 zr7qBOQQAIziLlk=tZ#q>qylfiJjdD+vCAH}eH>mu$c^f@?%z-%e8I|HDjXW|&I>oC zH11o6aG)1~j89W1ewnO8!&T9HA&a7OYF5|K`1eOn{`kr5bjWn?+g}za)3O$UiYA`> zXA~~97f)@B{xai6A&fU}+yAnpScM2(>Xs*XYxAL+qC$QybHeol z>nfUdd$nfj54=pbc;jBQq-8GoWa8X7;%ZN?VSTu(hRlEtHUtUpmZZ#u6^h}Lg7Ho8 zPET#u((Mj3a?fFQkUED2EFJG;htgZYk|xc~E|H8MZInKt;$o zcH-stH2i`MkCsTRVS0>=loYh6>?Q_npPm;p_@o6Qih_|Q`6R>tFb`C(g00>y2^1Tv zej(Lk#_hb>i78Qgmv;0BX@UkWLQT?op(;bPXaDi`wQo8UH)ndD(}(A=cu_lw!UtH{ znu~*8VMxS9vMDYx_{^(UMaS2cZB{$|grct_FoL(DiQ9G`nVi&n8@E$o26SfoP4cA(wr4@j7k#rO0ZL?6nY4}&#$aj@^MaiH#~}JKc*{id|EsOQ#!2G3JgR zZLNRQT;E@}V-?FtFOocHuy$sV(6(rh?~vFlNumcZP%|IEgjm`3cL?ZOj^8>nAFO z$wS1dOhiR}>b;~SEtUvtTcpGG^jK2>Tx`b(du9V8rQQ$Gex>XCgO1GUpRD@c5=G`=L#V?F#jHRJTrPI@+X? z!}CI7wScC%jtz>^^jhOmKZ}mmtz<-IMt8>++=Gg8X$TQ$-JP$T^%$@u^s#Y-$I|{& z%I?s_rzy4>Z=T%8oi=<0)@xYJlz?FC>>O{waH2*|?*lAVJ$K)gxXf(c%~cN^ddx)j zv{;93XBZNgGfxd2z%dbGV4&6V1Hf8VdMasRY;Rjr5J!x{UF_sYkXl|}$> zY{dGT`1U1l#(1vEKYjWFK%RN#`!?_JJN!+F|C2J#dAciysW`O>nS?1X_6(X`n|t)1 zCp54#x0QB}`C(Z+N0*kRPtSLo z3Ad+53Ik>fppyDRI-~vl^WRS>L{4_f%w8irkBG+sf#+4#mNDM|KlZs-=S0+?9t;w2v# z^S=}C#0({5dT_-gj$N?6Yfnbvn)N7y8jUuLsQX1NK939k1^`LE?4Z_Iz4mI8(~t87 z^K%}E)%}BsgsN53`I!!mO=ax9vE&!@tCL*P7qhMW+r!^S0%WxjZDe07Sv`tfoe@!= zI6CeMupU62AN|yf^S>1nUT0iWP56hpXy|bwFZ>-jGa*8VvTF0^qYEJBCGaXH`_$~tf|47kl8+FWG}^ASHpjRiKDUQn$rieHNh%#fQ& zuxQ)IQ=&qtDY2QCz98#361PZ(4^PYMYOyPq=~a)gCnsy{D*70joSg9r18_NY~0FOEsx0{GRKvIa>x z2Tl%edx)DK864=7EuZY#rOCv`;rlDF*i6#e0fDgh1N;mqDA7)IP*GSg*NPqVA(!o^ ziOVMUUhaTEf@uFM0Cr|uDRu6iJSpRAig`mQ)G@56TzPw4*80Cr>1eCg%LLAQc+iOM z`{=1M{=g-syu0DlfxS5?j-4a*3q*FGO}+w;gpY3{^~+W>=&YxdFm8*Ofl?~cZ^ zEZ%!sHdXrcrL)zmMyH0Al93t8HDPkfyIoXn3g%-Q*UV?#nyyEVdR>k`dx!P>0X^w} zS?S+-{wMn9K;TUEIT%D{Iwi6kK~}!+PZl}f>h|m?rO7;Z*g(;ZIQG*M7k}%{R4J6v{Xm; zJU@yKig@JOq_2C!)syaKDH;{+>a6e$+{(^bAMZ&z~n*WY@sp;~bJ~c+Wsg4@Gnfm@5?k1Om&{kwgSYYMWrHJ0;R%Om> zpAr*y>G#^LihntTzAX;{vb1$Kx0R;1Dt>&BGZAs8&Fk2g3^1BFhSJ+W#|RKVxB8 zy{ZLsz4o(e#^~;`yW|U%8p+P&JS@pEShhLC`_NOQwC8cZ`VFihT=8JDy{o_s9N&|- zo#-4?tRk#*e^-uCH}wyu{y%c=iS_-7GW`2%b#m8O>BnpJYmDqGj9CxqiJJ{J*$q;b zQPdIhPQATPQVBA2Dv}T5p#@e{EUvUD)2P@AWKo3y0cjKM7~g7>)~aSA-ry|%My%TR z#s>_gAW|8Tl0yZepZQctxqsxvn|m{NM^N{6>ldh;C;RGpF3yZTv*I2+Mc0zXef%ET zEze(N8%VN|8`W=@r;+jR>}M`jG&Y;e-@4k*bkb~hYolD_W92($`@@n>X$q?RSvu%f z;e?-rXa6nyz3czA4wEW5B%=ofHNIC6CUWbqjD1emY_23bN5h)WeSE10IT%oXPAH|k zOK?-JKEp4FN|@7T11FB+7wiV^7ZB2{0v}1GH zgrS~BcaR_Qq6%n-&|)3MOEv=#a23 z%369lL>#<*b-7$eny^@6I0#WnsSG?`>^@vDF2_hZ?0ed4*SoIX7B&%6{Cs_8@6HlL z8lO$>W^pPX+%y1N%{EEj17d9LB{2@X6;R92#ozf*hf-I6<*N!-FE1va!;GhzfI!NZ#bs+< z3Us3swl08ddKjmw5V{+@c|=5$%5mFz>3`?ynEl^xt0cP@Kf+eg z5yw!!aEWe-Jg?>Ybrj6%>}eeh$DYfVl2!6mlKn)OBN$NG-fLn~+0owZqGaQn#-|<+ zUy7dGGPfh_vR_%ee)3BGg{1tHwh2c)ZR#N?Ur$TwZHw&EX+!wxoquAP=jerE+hpv;`jsa*Qd6I>^Yq>7De z#WL}6Byh#4TYuAs%ysE0wDsr%2vBh|B*+qS&ZUfZ5Y;%K?+zgi=~GcDVZPU&b_Rs& zX;>dAeV~C3DGu7Lwa(FU+g0TArEh?!51VE?4euIWu*H9M z{`~GejSaBkmD%0#p7*ZFT07O#DX&g+>D36V??s*;mu_xbZ#qzJ>u?_V26#?7BXj#r zR4xTMc-QfxjpWN#N6C>_UhaOZLaMV*%I9T$5zF-Kg#Cm=GMY(vQ=e6*xtf9c`$r7G-NUf-?Yw8uqU6>H-GXxH}qFjL|bR~7Av?dt|1ve=*r?G79+`mBNI zS^ASwGzHw1npIydo#1F5>j-ZC_{?dD%j1+WF_ku^foEnX;u}i6SqOTz@ojpDZJ!a4 z+)8mk($({by%imcs>KmF(Y~e_L(5odpi#EuB-*CrOOgCwE*RAlT|sMP?tRQZebdM1!owUSvY4^ZTI!F~tQI01Wn5@1^yQg< zRvO@$nWca8F8_4tpJUDf{=_Npoqg`N04LvFn`qXH0{-L$_*3bBoBS^N3vt~acnr=Q z`#Tal0N}{en{3A{o`2R%&tDcRQV}K+Xs~}Po%9XRuAFsCB%gs_{NiAfeVradQ?}7I{Pu)} z_*dT08TX53?DS^!y3>pPtt?}ENz-dV?>G-c2Cp#t9yk6#H57DWWGzCox09`Fq^)`6 zouQ;R`-h+^dG2bTjHm720AU{eM{gC(aMVRPI^`{PuV-}EJNRjlB1hgPd7ClYTg}_w zE?|8Yx!0xzkf1D{XU+e2asTeS{|^UBkAt7)j`Q$om37twkEsshd7<7cL3Y2sKIAx5~UCV6Anv>H5+Ohp&dh%6a^G+P&82(9a9}$hDhiCu#0@ z-42Q03_`i@;E6kmVlE3vC!~qMX0Lw}r(n4TA|TC-i~9u`2#v-nk*Mdn?gLJeFh+0i zdz@pv8TlsSj2btz>9B*Fd*0XUSi{E+k^%&%va6B@uS4RC892a6VME?XqzwTwv9h;5 z-8b8vzHuLLqE7y@r2C#I&*#VXSqtU26&^wS>i__j!z>Rz7GH1&CdM>XT`H+(v=LTg-Mls)kYoErl6DrfJ)Zzd zkjNFAXLhwt0y+>l2bSEsL`9{pnzNrvuKfax`PvTvjHURf@L~LE`b(U`Gyw2ZmY)Pkp*#=s{3Q7A!vFJbImb6^V9vj>Jub8PdaCvv z6F;oy1$u65h7rkbG?YzOW%ebRR~gk&6_sWUPBTeEB9VaO!+*Qd|94$v|2F>r07> (@hectorj2f) + + ## Motivation CA is released as a bundle which includes a hardcoded list of supported cloud providers. @@ -29,46 +35,13 @@ There are couple of examples of pluggable designs using Go SDKs that would guide to support custom providers as plugins. This approach is inspired based on [Hashicorp go-plugin](https://github.com/hashicorp/go-plugin) and [Grafana Go SDK for plugins](https://github.com/grafana/grafana-plugin-sdk-go). -There are two alternatives on how to plug custom cloud providers: - -* **Option1:** Install the plugin as a binary that would be mounted into the CA container -and invoked by CA server. -In this option CA server launches each provider plugin as a subprocess and communicates with it over gRPC. - -* **Option2:** A custom cloud provider server is deployed along side CA and both communicates via gRPC with TLS/SSL. - -Regardless of the chosen option, both solutions have to expose a common gRPC API -with the following operations: - -```go -type clusterAutoscalerCustomProviderServer struct { - ... -} - -func (s *clusterAutoscalerCustomProviderServer) NodeGroups(ctx context.Context) ([]pb.NodeGroup, error) { - ... -} -... +The proposed solution will deploy the external grpc provider along side CA and both communicates via gRPC with TLS/SSL. The external provider would be part of a separate deployment, and so we should deploy it independently, as shown in the ![Figure](./images/external-provider-grpc.jpg). -func (s *clusterAutoscalerCustomProviderServer) NodeGroupForNode(ctx context.Context, *apiv1.Node) (*pb.NodeGroup, error) { - ... -} -... +This approach exposes a common gRPC API server. -func (s *clusterAutoscalerCustomProviderServer) Refresh() error { - ... -} -... -func (s *clusterAutoscalerCustomProviderServer) GetAvailableMachineTypes() ([]string, error) { - ... -} - -... - -``` -Obviously, these API calls implement the `CloudProvider` and `NodeGroup` interfaces of the [CA](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloud_provider.go#L50): +The API calls would be defined by the operations exposed by the `CloudProvider` and `NodeGroup` interfaces in [CA](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/cloud_provider.go#L50): ```go type CloudProvider interface { @@ -214,13 +187,13 @@ to reach the remote gRPC server. A new flag, named `--cloud-provider-url=https://local.svc.io/mycloudprovider/server`, determines the URL to reach the custom provider implementation. -In addition this approach reuses the existing flag that defines the name of the cloud provider using a pre-defined value `--cloud-provider=external` -https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider. `external` defines the usage +In addition this approach reuses the existing flag that defines the name of the cloud provider using a pre-defined value `--cloud-provider=externalgrpc` +https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/cloudprovider. `externalgrpc` defines the usage of an external cloud provider whose interface is handled by a remote gRPC service. To connect the CA core with this new external cloud provider, this approach needs to implement a new generic cloud provider as part of the CA core code. -This new provider, named `ExternalCloudProvider`, makes gRPC calls to the remote functions exposed by the external cloud provider server. In other words, it forwards the calls and handle the errors analogously how done in other existing providers. +This new provider, named `ExternalGrpcCloudProvider`, makes gRPC calls to the remote functions exposed by the external cloud provider server. In other words, it forwards the calls and handle the errors analogously how done in other existing providers. Obviously, this new approach needs to use TLS to ensure a secure communication between CA and this CA provider server. Additional flags should be added to specify the path where to find the certificate to establish the communication to From 68c984472acce69cba89d96d724d25b3c78fc4a0 Mon Sep 17 00:00:00 2001 From: Hector Fernandez Date: Sat, 2 Jan 2021 12:51:13 +0100 Subject: [PATCH 5/5] docs: polished the rpc operations Signed-off-by: Hector Fernandez --- .../proposals/plugable-provider-grpc.md | 212 ++++++------------ 1 file changed, 68 insertions(+), 144 deletions(-) diff --git a/cluster-autoscaler/proposals/plugable-provider-grpc.md b/cluster-autoscaler/proposals/plugable-provider-grpc.md index 75d684482a69..0758cdacab0f 100644 --- a/cluster-autoscaler/proposals/plugable-provider-grpc.md +++ b/cluster-autoscaler/proposals/plugable-provider-grpc.md @@ -21,13 +21,14 @@ In particular users need to follow these steps to support a custom private cloud This is a concern that has been raised in the past [PR953](https://github.com/kubernetes/autoscaler/issues/953) and [PR1060](https://github.com/kubernetes/autoscaler/issues/1060). -Therefore a new implemetation should be added to CA in order to extend it without breaking any backwards compatibility or +Therefore a new implementation should be added to CA in order to extend it without breaking any backwards compatibility or the current cloud provider implementations. ## Goals * Support custom cloud provider implementations without changing the current `CloudProvider` interface. * Make CA extendable, so users do not need to fork the CA repository. +* Add a mock gRPC provider to test this new functionality. ## Proposal @@ -114,12 +115,10 @@ import "k8s.io/apimachinery/pkg/apis/meta/v1/generated.proto"; import "k8s.io/apimachinery/pkg/api/resource/generated.proto"; import "k8s.io/api/core/v1/generated.proto" -option go_package = "clusterautoscaler.cloudprovider"; +option go_package = "v1"; service CloudProvider { // CloudProvider specific RPC functions - rpc Name(NameRequest) - returns (NameResponse) {} rpc NodeGroups(NodeGroupsRequest) returns (GetNameResponse) {} @@ -127,18 +126,12 @@ service CloudProvider { rpc NodeGroupForNode(NodeGroupForNodeRequest) returns (NodeGroupForNodeResponse) {} - rpc PricingNodePrice(PricingNodePriceRequest) + rpc PricingNodePrice(PricingNodePriceRequest) // Optional returns (PricingNodePriceResponse) {} - rpc PricingPodPrice(PricingPodPriceRequest) + rpc PricingPodPrice(PricingPodPriceRequest) // Optional returns (PricingPodPriceResponse) - rpc NewNodeGroup(NewNodeGroupRequest) - returns (NewNodeGroupResponse) {} - - rpc GetResourceLimiter(GetResourceLimiterRequest) - returns (GetResourceLimiterResponse) {} - rpc GPULabel(GPULabelRequest) returns (GPULabelResponse) {} @@ -165,13 +158,7 @@ service CloudProvider { returns (NodeGroupDecreaseTargetSizeResponse) {} rpc NodeGroupNodes(NodeGroupNodesRequest) - returns (NodeGroupNodesResponse) {} - - rpc NodeGroupDelete(NodeGroupDeleteRequest) - returns (NodeGroupDeleteResponse) {} - - rpc NodeGroupCreate(NodeGroupCreateRequest) - returns (NodeGroupCreateResponse) {} + returns (NodeGroupNodesResponse) {} rpc NodeGroupTemplateNodeInfo(NodeGroupDTemplateNodeInfoRequest) returns (NodeGroupTemplateNodeInfoResponse) {} @@ -180,6 +167,12 @@ service CloudProvider { Note that, the rest of message used in these calls are detailed in the [Appendix](#appendix) section. +Among all the operations, the CA calls in many places the function `NodeGroupForNode`. +As a consequence this operation might impact the overall performance of the CA when calling it via a remote external cloud provider. +A solution is to cache the rpc responses of this operation to avoid causing a performance degradation. +Another proposed alternative could be to move the entire logic of this function to the CA source code. +Initially this proposal assumes the rpc responses for this operation are cached to avoid any performance degradation. + In order to talk to the custom cloud provider server, this new cloud provider has to be registered when bootstrapping the CA. Consequently, the CA needs to expose new flags to specify the cloud provider and all the required properties @@ -231,39 +224,57 @@ message NodeGroupsResponse { } ``` +### ExternalGrpcNode + +ExternalGrpcNode is a custom type. This object defines the minimum required properties of a given Kubernetes node for a node group. +This new type reduces the amount of data transferred in all the operations instead of +sending the whole `k8s.io.api.core.v1.Node` rpc message. + +```protobuf +message ExternalGrpcNode{ + // ID of the node assigned by the cloud provider in the format: :// + // +optional + optional string providerID = 1; + + // Name of the node assigned by the cloud provider + optional string name = 2; + + // labels is a map of {key,value} pairs with the node's labels. + map labels = 3; + + + // If specified, the node's annotations. + map annotations = 4; +} +``` + +Initially, we defined a list of 4 properties, but this list could increase during the implementation phase. + ### NodeGroupForNode NodeGroupForNode returns the node group for the given node, nil if the node should not be processed by cluster autoscaler, or non-nil error if such occurred. Must be implemented. +**IMPORTANT:** Please note, this operation is extensively used by CA and can cause some performance degradations on large clusters. +The initial proposal assumes the rpc responses are cached to offload the performance impact of constantly calling this function. + ```protobuf message NodeGroupForNodeRequest { // Node group for the given node - k8s.io.api.core.v1.Node node = 1; + ExternalGrpcNode node = 1; } message NodeGroupForNodeResponse { // The node group for the given node. repeated NodeGroup nodeGroup = 1; } -``` - -### GetResourceLimiter -GetResourceLimiter types store struct containing limits (max, min) for resources (cores, memory etc.). - -```protobuf -message GetResourceLimiterRequest { - // Intentionally empty. +message NodeGroupForNodeRequest { + // Node group for the given node + ExternalGrpcNode node = 1; } -message GetResourceLimiterResponse { - // All the machine types that the cloud provider service supports. This - // field is OPTIONAL. - repeated string machineTypes = 1; -} ``` - ### GetAvailableGPUTypes GetAvailableGPUTypes handles all available GPU types cloud provider supports. @@ -275,7 +286,7 @@ message GetAvailableGPUTypesRequest { message GetAvailableGPUTypesResponse { // GPU types passed in as opaque key-value pairs. - map gpuTypes = 1; + map gpuTypes = 1; } ``` @@ -297,10 +308,11 @@ message GPULabelResponse { ### PricingNodePrice NodePrice handles an operation that returns a price of running the given node for a given period of time. +PricingNodePrice is an optional operation that is not implemented by all the providers. ```protobuf message PricingNodePriceRequest { - k8s.io.api.core.v1.Node node = 1;, + ExternalGrpcNode node = 1;, k8s.io.apimachinery.pkg.apis.meta.v1.Time startTime = 2; @@ -316,6 +328,8 @@ message PricingNodePriceResponse { ### PricingPodPrice PodPrice handles an operation that returns a theoretical minimum price of running a pod for a given period of time on a perfectly matching machine. +PricingPodPrice is an optional operation that is not implemented by all the providers. + ```protobuf message PricingPodPriceRequest { @@ -332,52 +346,6 @@ message PricingPodPriceResponse { } ``` -### NewNodeGroup - -NewNodeGroup builds a theoretical node group based on the node definition provided. The node group is not automatically created on the cloud provider side. This action returns the created node group. - -```protobuf -message NewNodeGroupRequest { - // Machine type for the node group. - string machineType = 1; - - // Labels of the node group. - map labels = 2; - - // System Labels of the node group. - map systemLabels = 3; - - // Taints of the node group - repeated k8s.io.api.core.v1.Taint taints = 4; - - // ExtraResources of the node group - map minLimits = 1; - - // Contains the maximum limits for resources (cores, memory etc.). - map maxLimits = 2; -} -``` - ### Refresh Refresh is called before every main loop and can be used to dynamically update cloud provider state. @@ -407,28 +375,10 @@ message CleanupResponse { } ``` -### Name - -Name stores the name of the cloud provider. - -```protobuf -message NameRequest { - // Intentionally empty. -} - -message NameResponse { - // Name of the node group - string name = 1; -} -``` - ### NodeGroup ```protobuf message NodeGroup { - // Name of the node group on the cloud provider - string name = 1; - // ID of the node group on the cloud provider string id = 1; @@ -438,14 +388,8 @@ message NodeGroup { // MaxSize of the node group on the cloud provider int32 maxSize = 3; - // Exist reports if the node group really exists on the cloud provider - bool exist = 4; - - // Autoprovisioned returns true if the node group is autoprovisioned - bool autoProvisioned = 5; - // Debug returns a string containing all information regarding this node group. - string debug = 6; + string debug = 4; } ``` @@ -488,7 +432,7 @@ NodeGroupDeleteNodes deletes nodes from this node group. ```protobuf message NodeGroupDeleteNodesRequest { - repeated k8s.io.api.core.v1.Node nodes = 1; + repeated ExternalGrpcNode nodes = 1; // ID of the group node on the cloud provider string id = 2; @@ -539,8 +483,15 @@ message Instance { // InstanceStatus represents instance status. message InstanceStatus { - // State tells if instance is running, being created or being deleted - int state = 1; + // InstanceState tells if instance is running, being created or being deleted + enum InstanceState { + // InstanceRunning means instance is running + InstanceRunning = 1 + // InstanceCreating means instance is being created + InstanceCreating = 2 + // InstanceDeleting means instance is being deleted + InstanceDeleting = 3; + } // ErrorInfo is not nil if there is error condition related to instance. InstanceErrorInfo errorInfo = 2; } @@ -556,40 +507,13 @@ message InstanceErrorInfo { } ``` -### NodeGroupDelete - -NodeGroupDelete deletes the node group on the cloud provider side. - -```protobuf -message NodeGroupDeleteRequest { - // ID of the group node on the cloud provider - string id = 1; -} - -message NodeGroupDeleteResponse { - // Intentionally empty. -} -``` - -### NodeGroupCreate - -NodeGroupCreate creates the node group on the cloud provider side. - -```protobuf -message NodeGroupCreateRequest { - // NodeGroup to be created on the cloud provider - NodeGroup nodeGroup = 1; -} - -message NodeGroupCreateResponse { - // NodeGroup that was created on the cloud provider - NodeGroup nodeGroup = 1; -} -``` - ### NodeGroupTemplateNodeInfo -TemplateNodeInfo returns a NodeInfo structure of an empty (as if just started) node. +TemplateNodeInfo returns a NodeInfo as a structure of an empty (as if just started) node. +The definition of a generic `NodeInfo` for each potential provider is a pretty complex approach and does not cover all the scenarios. +For the sake of simplicity, the `nodeInfo` is defined as a Kubernetes `k8s.io.api.core.v1.Node` type +where the system could still extract certain info about the node. + ```protobuf message NodeGroupTemplateNodeInfoRequest { @@ -598,7 +522,7 @@ message NodeGroupTemplateNodeInfoRequest { } message NodeGroupTemplateNodeInfoResponse { - // nodeInfo extracted of the template node on the cloud provider - NodeInfo nodeInfo = 1; + // nodeInfo extracted data from the cloud provider node using a primitive Kubernetes Node type. + k8s.io.api.core.v1.Node nodeInfo = 1; } ```