Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the EGDS protocol #10023

Closed
wants to merge 11 commits into from
Closed

Implement the EGDS protocol #10023

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
077afdc
Implement the EGDS protocol
gengleilei Dec 26, 2019
b309be2
Add missing dependencies
gengleilei Feb 13, 2020
2d9446c
Fix formatting error
gengleilei Feb 13, 2020
dfdfa1c
Fix formatting error
gengleilei Feb 13, 2020
71d337d
Optimize the code
gengleilei Feb 17, 2020
f22ade3
EGDS messages are compatible with V2/V3 versions
gengleilei Feb 20, 2020
8e64ac9
Support for reload health hosting mechanisms
gengleilei Feb 21, 2020
2b838de
Adjust some comments
gengleilei Feb 21, 2020
1e0b06b
Format the code and adjust some comments
gengleilei Feb 25, 2020
51180a1
Fix spelling errors
gengleilei Feb 25, 2020
d37fd08
Add egds to the spelling dictionary
gengleilei Feb 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions api/envoy/api/v2/egds.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
syntax = "proto3";

package envoy.api.v2;

import "envoy/api/v2/core/config_source.proto";
import "envoy/api/v2/discovery.proto";
import "envoy/api/v2/endpoint/endpoint_components.proto";

import "google/api/annotations.proto";
import "google/protobuf/duration.proto";
import "google/protobuf/wrappers.proto";

import "envoy/annotations/resource.proto";
import "udpa/annotations/migrate.proto";
import "validate/validate.proto";

option java_package = "io.envoyproxy.envoy.api.v2";
option java_outer_classname = "EgdsProto";
option java_multiple_files = true;
option java_generic_services = true;
option (udpa.annotations.file_migrate).move_to_package = "envoy.service.endpoint.v3";

// [#protodoc-title: EGDS]
// Endpoint discovery :ref:`architecture overview <arch_overview_service_discovery_types_egds>`

service EndpointGroupDiscoveryService {
option (envoy.annotations.resource).type = "envoy.api.v2.EndpointGroup";

// The resource_names field in DiscoveryRequest specifies a list of clusters
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: /s/clusters/endpoint groups

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

// to subscribe to updates for.
rpc StreamEndpointGroups(stream DiscoveryRequest) returns (stream DiscoveryResponse) {
}

rpc DeltaEndpointGroups(stream DeltaDiscoveryRequest) returns (stream DeltaDiscoveryResponse) {
}

rpc FetchEndpointGroups(DiscoveryRequest) returns (DiscoveryResponse) {
option (google.api.http).post = "/v2/discovery:endpoint_groups";
option (google.api.http).body = "*";
}
}

// [#not-implemented-hide:] Not configuration. Workaround c++ protobuf issue with importing
// services: https://github.com/google/protobuf/issues/4221 and protoxform to upgrade the file.
message EgdsDummy {
}
27 changes: 27 additions & 0 deletions api/envoy/api/v2/endpoint.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ syntax = "proto3";

package envoy.api.v2;

import "envoy/api/v2/core/config_source.proto";
import "envoy/api/v2/endpoint/endpoint_components.proto";
import "envoy/type/percent.proto";

Expand Down Expand Up @@ -114,4 +115,30 @@ message ClusterLoadAssignment {

// Load balancing policy settings.
Policy policy = 4;

// References endpoint groups by name.
// Both endpoints and EGDS references will be used when present
repeated Egds endpoint_groups = 6;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can a ClusterLoadAssignment send both Endpoints and EndpointGroups? Or either of them? If either of them we should probably use oneof ? I am assuming this to work similar to how Rds and route_config work for Http Connection Manager.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it does impact cluster warming and Envoy init. For EDS type of cluster, a cluster is considered warmed, if it gets at least one Endpoint Response. With this change, we should consider a cluster is fully warmed, only after receiving responses to all endpoint groups?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A ClusterLoadAssignment does not recommend sending Endpoints and EndpointGroups together, and while the code implementation can be done, it can cause confusion when applied.

If EGDS is used, I think it is necessary to initialize the cluster after receiving the response from all the endpoint groups. Otherwise, how should the envoy handle the initialization state of the cluster ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since named_points is there, we probably should not move to union { endpoints, named_endpoints, endpoint_groups}.
That says we need extra document/comment here to describe the two fields

}

// New xDS payload. The size of endpoints is fully arbitrary.
// Also the EGDS requests to Pilot can be merged into one to boost efficiency
message EndpointGroup {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can you correct the comment above and explain what EndpointGroup is?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, it's been modified.

string name = 1 [(validate.rules).string = {min_bytes: 1}];

// List of endpoints to load balance to.
repeated endpoint.LocalityLbEndpoints endpoints = 2;

// Map of named endpoints that can be referenced in LocalityLbEndpoints.
// [#not-implemented-hide:]
map<string, endpoint.Endpoint> named_endpoints = 3;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This field is not currently in use, but I think it should be retained because the endpoint group just groups the endpoints and the original definition should not be changed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO it's a great chance to declare endpoints and named_endpoints as union.
protobuf doesn't support a field both union and repeated but it's viable to embed endpoints in another message StructEndpoints, and make

  union {
    StructEndpoints                          endpoints = 1
    map<string, endpoint.Endpoint> named_endpoints = 2;
  }
`
WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you mean https://developers.google.com/protocol-buffers/docs/proto3#oneof

}

// The EGDS request configuration.
message Egds {
// Configuration for the source of EGDS updates for this Cluster.
core.ConfigSource config_source = 1;

// the endpoint group resource name.
string endpoint_group_name = 2;
}
31 changes: 31 additions & 0 deletions api/envoy/config/endpoint/v3/endpoint.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,4 +122,35 @@ message ClusterLoadAssignment {

// Load balancing policy settings.
Policy policy = 4;

// References endpoint groups by name.
// Both endpoints and EGDS references will be used when present
repeated Egds endpoint_groups = 6;
}

// New xDS payload. The size of endpoints is fully arbitrary.
// Also the EGDS requests to Pilot can be merged into one to boost efficiency
message EndpointGroup {
option (udpa.annotations.versioning).previous_message_type = "envoy.api.v2.EndpointGroup";

string name = 1 [(validate.rules).string = {min_bytes: 1}];

// List of endpoints to load balance to.
repeated LocalityLbEndpoints endpoints = 2;

// Map of named endpoints that can be referenced in LocalityLbEndpoints.
// [#not-implemented-hide:]
map<string, Endpoint> named_endpoints = 3;
}

// The EGDS request configuration.
message Egds {
option (udpa.annotations.versioning).previous_message_type =
"envoy.api.v2.ClusterLoadAssignment.EgdsConfig";

// Configuration for the source of EGDS updates for this Cluster.
core.v3.ConfigSource config_source = 1;

// the endpoint group resource name.
string endpoint_group_name = 2;
}
1 change: 1 addition & 0 deletions docs/root/intro/version_history.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ Version history
restart Envoy. The behavior will not switch until the connection pools are recreated. The new
circuit breaker behavior is described :ref:`here <arch_overview_circuit_break>`.
* upstream: changed load distribution algorithm when all priorities enter :ref:`panic mode<arch_overview_load_balancing_panic_threshold>`.
* upstream: added support for the EGDS protocol

1.13.0 (January 20, 2020)
=========================
Expand Down
46 changes: 46 additions & 0 deletions generated_api_shadow/envoy/api/v2/egds.proto
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

27 changes: 27 additions & 0 deletions generated_api_shadow/envoy/api/v2/endpoint.proto
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

32 changes: 32 additions & 0 deletions generated_api_shadow/envoy/config/endpoint/v3/endpoint.proto
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions source/common/config/resources.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ class TypeUrlValues {
const std::string ScopedRouteConfiguration{
"type.googleapis.com/envoy.api.v2.ScopedRouteConfiguration"};
const std::string Runtime{"type.googleapis.com/envoy.service.discovery.v2.Runtime"};
const std::string EndpointGroup{"type.googleapis.com/envoy.api.v2.EndpointGroup"};
};

using TypeUrl = ConstSingleton<TypeUrlValues>;
Expand Down
89 changes: 89 additions & 0 deletions source/common/upstream/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -342,6 +342,9 @@ envoy_cc_library(
hdrs = ["eds.h"],
deps = [
":cluster_factory_lib",
":egds_api_impl_lib",
":egds_cluster_mapper_lib",
":endpoint_groups_manager_lib",
":upstream_includes",
"//include/envoy/config:grpc_mux_interface",
"//include/envoy/config:subscription_factory_interface",
Expand Down Expand Up @@ -568,3 +571,89 @@ envoy_cc_library(
"@envoy_api//envoy/config/cluster/v3:pkg_cc_proto",
],
)

envoy_cc_library(
name = "endpoint_group_monitor_interface",
hdrs = ["endpoint_group_monitor.h"],
deps = [
"//include/envoy/upstream:upstream_interface",
"@envoy_api//envoy/config/endpoint/v3:pkg_cc_proto",
],
)

envoy_cc_library(
name = "egds_cluster_mapper_lib",
srcs = ["egds_cluster_mapper.cc"],
hdrs = ["egds_cluster_mapper.h"],
deps = [
":endpoint_group_monitor_interface",
":upstream_includes",
"//include/envoy/local_info:local_info_interface",
"//include/envoy/upstream:upstream_interface",
"//source/common/common:minimal_logger_lib",
"//source/common/network:resolver_lib",
"@envoy_api//envoy/config/endpoint/v3:pkg_cc_proto",
],
)

envoy_cc_library(
name = "endpoint_groups_manager_interface",
hdrs = ["endpoint_groups_manager.h"],
deps = [
"//include/envoy/upstream:upstream_interface",
"@envoy_api//envoy/config/endpoint/v3:pkg_cc_proto",
],
)

envoy_cc_library(
name = "endpoint_groups_manager_lib",
srcs = ["endpoint_groups_manager_impl.cc"],
hdrs = ["endpoint_groups_manager_impl.h"],
deps = [
":endpoint_group_lib",
":endpoint_group_monitor_interface",
":endpoint_groups_manager_interface",
"//include/envoy/upstream:upstream_interface",
"@envoy_api//envoy/config/endpoint/v3:pkg_cc_proto",
],
)

envoy_cc_library(
name = "endpoint_group_lib",
srcs = ["endpoint_group.cc"],
hdrs = ["endpoint_group.h"],
deps = [
":endpoint_group_monitor_interface",
"//include/envoy/upstream:upstream_interface",
"@envoy_api//envoy/config/endpoint/v3:pkg_cc_proto",
],
)

envoy_cc_library(
name = "egds_api_impl_lib",
srcs = ["egds_api_impl.cc"],
hdrs = ["egds_api_impl.h"],
deps = [
":egds_cluster_mapper_lib",
":endpoint_group_monitor_interface",
":endpoint_groups_manager_lib",
":upstream_includes",
"//include/envoy/config:grpc_mux_interface",
"//include/envoy/config:subscription_factory_interface",
"//include/envoy/config:subscription_interface",
"//include/envoy/stats:stats_interface",
"//include/envoy/upstream:locality_lib",
"//include/envoy/upstream:upstream_interface",
"//source/common/common:minimal_logger_lib",
"//source/common/config:metadata_lib",
"//source/common/config:subscription_factory_lib",
"//source/common/config:utility_lib",
"//source/common/grpc:common_lib",
"//source/common/network:address_lib",
"//source/common/network:utility_lib",
"//source/common/protobuf:utility_lib",
"@envoy_api//envoy/config/core/v3:pkg_cc_proto",
"@envoy_api//envoy/config/endpoint/v3:pkg_cc_proto",
"@envoy_api//envoy/service/discovery/v3:pkg_cc_proto",
],
)
Loading