Probe gopacket capture status

[ghost]

Saw that integration of gopacket based packet sniffer was attempted earlier but abandoned/disabled/experimental as of now. A few queries:

1. What is the current support for packet sniffing in probe? Is there a way to try this out on master?

2. If this works/ever worked, were the packet relayed back to app? Was there a way to extract/fwd packets from probe to app?

The ability to capture packets optionally (i.e. only if user wants, in interest of performance) is very essential to us and were looking at adding this support to probe. If someone can help me try out current state of the art, more than happy to help in dev/testing of this feature.

[2opremio]

You mean #317 ? This was never merged and there are no plans to work on this at the moment. The only current use of gopacket in Weave Scope is in the DNS Sniffer, which is still very much in use. Most importantly, sniffing is normally a means, not a goal. What metrics are important to you? If sniffing by itself is the goal (for diagnostics, for instance) there was also an attempt at a wireshark control at #1428 but was never merged since it was a proof of concept and required productization. @alban made an initial attempt at reviving it a couple of months ago, but never materialized. Please describe why this is important to you, providing details about your use case.

[2opremio]

Dupe of #1633

[ghost]

Most importantly, sniffing is normally a means, not a goal. What metrics
are important to you?

We want to use captured packets for deeper analysis from latency point of view. An APM use case. Scope gives us coarse level metrics but what matters most to us is the interaction latency between services and I think all this data is already there in packets. We did try using tools like packetbeat etc but I believe those tools do more than what we need where they are deployed i.e. heavy on resources. An ideal solution would be the ability to turn on/off packet capture (no deeper inspection of packets at the source) on demand, dump those packets in a time searies database and analyze them in deeper details.

If this feature is not on roadmap, no worries. If someone can confirm that the packet capture code worked well, I will start hacking it on my branch and see how it comes along. I really dont care about controls, just the packet capture and shipper code. Was that in good shape? Do point me to the appropriate commit and I will take a look.

[2opremio]

what matters most to us is the interaction latency between services and I think all this data is already there in packets

Not directly, it's hard to obtain that information from raw traffic . Out of curiosity, how are you planning to do this? Would you show the latency metrics in Scope too? Would it be displayed in a separate system?

An integration with a distributed tracing system like https://github.com/uber/jaeger (which we may do in the future) would be a much simpler way to obtain the latency metrics (and, most importantly, a breakdown of the services involved on each request) but of course, that requires instrumenting the applications.

If this feature is not on roadmap, no worries.

Getting latency is in the roadmap #1631 but not a high priority right now. For HTTP latency metrics, Scope already has an eBPF plugin https://github.com/weaveworks-plugins/scope-http-statistics

If someone can confirm that the packet capture code worked well

It worked, but it was considered a failed experiment and we removed it from the code. We discarded it due to performance problems, see #466

So, I don't want to completely discourage you from doing this, but you will have a small chance of getting it merged unless you thoroughly demonstrate it won't have a big performance impact and that it can be selectively enabled (this will most probably require implementing UI controls). A design document we can comment on before you jump into developing it would be highly recommended.

Do point me to the appropriate commit and I will take a look.

#317 (which is highly outdated, the internal data-structures have changed quite a bit since then)