[meta] Add support for proxies in ECS

[webmat]

We'd like to have support for proxies in ECS. We think the following could fall into this:

• load balancers (self-managed or services)
• caching proxies, both self-managed (e.g. Varnish) or CDNs
• level 4 load balancers

More details to come...

Past issues on proxies

#158, #387, #504, #544, #549, #626, sub-discussion in #51 (comment)

IP Forwarding

The network.forwarded_ip field as currently designed and documented has issues. They may be addressed separately, or could be part of this work on proxies.

Past issues around XFF or network.forwarded_ip: #523, #771, #874, #880, #895

[leehinman]

Would this just be limited to proxies or would any device in the middle of a network connection (not source or destination) be applicable? Things like NAT device, Router, Firewall, Switch, Wireless Access Point, etc?

[ebeahan]

I see proxy applying to any proxy-like devices like load balancers, API gateways, CDNs, web proxies, caching appliances, etc. A device that receives requests from the original source (device acts as a server), perhaps does some operations, and then passes the request upstream to an origin destination (device now acts as client).

If we use that criteria, I see layer two and three networking devices like routers, switches, or wireless APs excluded. While a NAT gateway are rewriting source and/or destination IPs and ports, I'd still not consider it a proxy. Depending on the feature set used on the firewall, we can get into some fuzzier area: URL filtering, SSL/TLS interception, DLP.

[webmat]

My thinking also reflects Eric's.

@leehinman do you see things that are missing, for the additional devices you mention?

[leehinman]

I'm OK with the proxy definition and scope.

I just thought some of the verbal discussions included a more generic definition (device in middle of a network connection) and wanted to see if those were in or if we were just focusing on proxies.

[dainperkins]

I don't think just "in the middle" qualifies, the difference being plain nat will usually just rewrite ips, ports (and maybe some app layer ip/port references, iirc seq/acks are unchanged) proxies will terminate a session and start a new one on behalf of the client.

Thinking it thru I think we'll need (assuming proxy as observer for global truth):

network fields

• proxy ip/vip (e.g. destination.ip from original client's perspective)
• snat ip (e.g. source "nat" ip, could be NAT, could also be a vip used as source for observer managed connection)
• proxied server ip (e.g. target system proxy/lb sends connection to)
• maybe an indicator for "nat / proxy" to minimally drive related.ip[s], related.community_id processing for e.g. ingest processing?

observer fields

• vip pool name/id/ip pool (e.g. load balancer log might reference)
• nat pool name/id/ip pool (e.g. load balancer log might reference)
• snat pool name/id/ip pool (e.g. load balancer log might reference)

/d

[dainperkins]

I went thru the f5 docs from #895 and put the following together for a possible extension of observer (currently the definition certainly works for proxies, load balancers, etc.). Many of the existing "observers" are also acting on the traffic at one layer or more so it seems to me to be reasonable (end of the day, observer is something in between source and dest..).

API gateway, CASB, and CDN might be different animals - but for the "legacy" proxies, load balancers, adcs, etc. I think observer is definitely the place. (x-forwarded-for, being very specifically an http header, would be neither mutually inclusive nor mutually exclusive to these fields.. that is - if its there, use it...)

Nat (source nat as well as vip from a external frame of reference) is covered with source/client/destination/server.nat, but we'll need to extend the observer fields to add in specifics related to the environment. I'm not 100% sure on some of the applicability of the names across other devices (need to see e.g. netscaler, pulse, etc.)

Syntax	Description
observer.pool.name	name of the pool (f5: VIRTUAL_SNATPOOL_NAME or server pool name)
observer.pool.ip[]	associated IPs / ranges
observer.pool.type	snat, nat, server, etc.
observer.vip.name	Name of the Load balanced service (f5: VIRTUAL_NAME)
observer.vip.ip	Observer VIP used for load balancing connections (f5: VIRTUAL_IP)
observer.vip.port	Observer Port used for load balancing connections (f5: VIRTUAL_PORT)
observer.service.class	class profile - may need to add fields (class.name, class.(setting1, setting2, etc)
related.community_id	needed to stitch information between src / observer / server frames of reference
observer.latency.request	time it took the observer to handle the request
observer.latency.response	time it took the observer to handle the response

F5 WAM_X_WA_INFO - docs say "An entry that specifies a diagnostic string (X-WA-Info header) used by BIG-IP acceleration to process the request." could be added to http.headers[] or possible normalized to an observer header or session field? Probably needs assessment against other vendors.

Updated: consolidated nat/server pools into a single object -> observer.pool with an additional "type" field (snat, nat, server, etc)

[ltflb-bgdi]

Is there any progress on this? I am working on the traefik integration and would like to have generic fields as well.

[VannTen]

I encountered the same issue while trying to see how I could map the logs of ingress-nginx (Kubernetes ingress controller) to ECS. I expect any implementation of Ingress (or its future replacement, Gateway API) will have that kind of information.