Mapping Cloudflare Logs onto ECS

[aleph-zero]

I'm working on mapping Cloudflare logs onto ECS. The official Cloudflare documentation on the fields and their descriptions:

• Fields: https://developers.cloudflare.com/logs/logpull-api/requesting-logs/#fields
• Description: https://developers.cloudflare.com/logs/reference/glossary/

My first pass is in this Excel sheet: Cloudflare-ECS.xlsx

@MikePaquette and @webmat can you help me by sanity checking what I've done so far?

[MikePaquette]

@aleph-zero can you confirm that you are using the logpull API and that a single event may contain both the client<->edge and edge<->origin parts of the transaction, if both exist?

[MikePaquette]

I think the ECS model looks like this:

[Camilia]

@aleph-zero can you confirm that you are using the logpull API and that a single event may contain both the client<->edge and edge<->origin parts of the transaction, if both exist?

Hi there, Yes, single event can contain both. Currently there are more than 60 fields in Cloudflare Enterprise Logs and tracks both communication between the Origin and Cloudflare and also between the Client and Cloudflare. So, all fields are being populated.

I have attached example of log file in json with all the latest fields included. Please notice there are three new fields which are nested fields (array):
FirewallMatchesActions
FirewallMatchesSources
FirewallMatchesRuleIDs

20190802_20190802T152500Z_20190802T153000Z_ef64f613.txt

Please let me know if you have any questions.

Many thanks,
Kamilla

[Camilia]

I think the ECS model looks like this:

I have attached the latest Schema with all the current fields and it's type. Please let me know if you have any questions.
Many thanks.

schema.txt

[webmat]

@aleph-zero Your mapping is looking good so far.

One small thing you can add is a mapping for ASN. It's not in an official ECS release yet. But in the next release, it will be client.as.number, so you can start using that directly.

The rest of the fields don't have a direct mapping in ECS at this time. You and @Camilia providing these details will be useful for us, when we work on adding more explicit support for firewalls and proxies in the future, so thank you both!

In the meantime, all of the unmapped fields should be tracked in a custom field.

[Randy-312]

I"d be curious how this goes, as I need to do similar exercises for other log formats.
Different product, same need, for the Layer 7 API Gateway logs, although not nearly as many fields as your 60.

---

CA has documented.](https://community.broadcom.com/communities/community-home/librarydocuments/viewdocument?DocumentKey=af4d855f-307d-4c9d-8e8d-5f90b3cf007e)

Here are their fields

[aleph-zero]

@Randy-312 Thanks for your interest. Would you please open a new issue and repost your above? Thanks!

[aleph-zero]

@webmat Here's the latest Excel with the Cloudflare mappings. I added a few questions in bold red. Can you take a look and answer? Thanks!

Cloudflare-ECS.xlsx

[webmat]

Since we're going away from observer for Cloudflare, I wouldn't use observer to document the vendor. Instead, perhaps we can set event.dataset to "cloudflare.log"?

I've also thought about the Ray ID: since they identify an HTTP request uniquely, perhaps you could map it to event.id.

I think we may have misunderstood what @Camilia said on the call, wrt ClientRequestHost. This field should be mapped to url.domain, not source.address.

I missed EdgeStartTimestamp and EdgeEndTimestamp in our review. They both have a good mapping in ECS, respectively event.start and event.end.

One very important point about source.address: I recommend copying source.ip to it verbatim. Even if the information is already in source.ip, the idea is that across all sources of web access logs (other than this one), users need a field that is guaranteed to always contain a value -- regardless of if it's an IP, a hostname or a unix socket. This is important when looking at the long tail of weirdness. Then of course whenever the user knows they want to deal with IP addresses, they can rely on source.ip being validated and indexed as an actual ip address.

[MikePaquette]

Since we're going away from observer for Cloudflare, I wouldn't user observer to document the vendor. Instead, perhaps we can set event.dataset to "cloudflare.log"?

Sorry, I missed this - why are we not using ECS observer.* fields? It would seem that observer.ip, observer.type:"proxy" , observer.hostname would all be useful fields to have.

Also, In this case, I think the destination.* field(s) and source.* fields should be replicated to the server.* and client.* fields respectively.

[aleph-zero]

@webmat Can you respond to @MikePaquette 's points about observer.* fields? I don't know which is correct and I need to nail this down.

[webmat]

I understood observer.* to be more of a device operating inside one's network, such as a network monitoring tool or a firewall at the edge. So Cloudflare being a massive distributed service didn't seem to fit for observer.

But since all traffic for a given service is passing through Cloudflare, I suppose we can indeed use observer.*.

It's important that both ends of the network connection remain the "real" client IP and the origin server, however. So don't touch those, other than copying to client/server in addition to source/destination, as @MikePaquette suggested.

So I'm thinking:

• EdgeServerIP => observer.ip
• hard code observer.vendor to "cloudflare"
• Keep event.dataset as "cloudflare.log"
• hard code observer.type to "proxy", as Mike suggests

I haven't seen any other obvious mappings for observer, including observer.hostname.

[webmat]

@aleph-zero Feel free to close the issue when the integration comes out :-)

This is good work. Eager to see that get out there!

[ghost]

I would like this too, any update?

[webmat]

@davidgagnegarmin I think it's available already, but not sure where :-)

@aleph-zero who needs to be contacted for this? Elastic or Cloudflare?

[webmat]

We created meta-issue #938 to add proxy support in ECS. I'm sure some custom fields could end up having first party support via #938

[ebeahan]

Appears the Cloudflare mappings are maintained by @aleph-zero here: https://github.com/cloudflare/cloudflare-elastic/tree/master/conf

[legoguy1000]

I'm currently trying to work elastic/beats#2666 to make a filebeat module for cloudflare. Will definitely use this to guide me. If anyone can provide more sample logs for testing, that would be great. Also there appear to be 3 datasets of logs; http, spectrum, firewall. If we can get all 3 that would be great and if there is a way to identify which logs are which.

[legoguy1000]

Initial Cloudflare package was added in elastic/integrations#984

[ebeahan]

Considering this effort complete with all the mapping done as part of creating the Cloudflare integration package.