Load test HTTP request count does not come close to events received by Honeycomb #209
Assignees
Milestone
Comments
|
Interestingly, as I've tested each release, we only ever received 10 events. This is the same settings as the ab command's |
robbkidd
changed the title
JamieDanielson
added a commit
that referenced
this issue
Oct 3, 2023
## Which problem is this PR solving? In service to #209, add more meta telemetry for easier data validation - #209 ## Short description of the changes - Add `meta.seqack` to events emitted by the agent. The SEQ corresponds to ACK of the HTTP request, and ACK corresponds to SEQ of the HTTP response. - Add `meta.request.packet_count`, and `meta.response.packet_count` to events emitted by the agent. Including these in telemetry may make it easier to validate proper matching of request/response pairs. ## How to verify that this has the expected result Events in Honeycomb contain these fields --------- Co-authored-by: Robb Kidd <[email protected]> Co-authored-by: Mike Goldsmith <[email protected]>
|
I can look into this. |
robbkidd
added a commit
that referenced
this issue
Oct 31, 2023
<!-- Thank you for contributing to the project! 💜 Please see our [OSS process document](https://github.com/honeycombio/home/blob/main/honeycomb-oss-lifecycle-and-practices.md#) to get an idea of how we operate. --> ## Which problem is this PR solving? - #209 ## Short description of the changes During the debugging of suspected load testing issues, the following bugs were found and fixed: - Switch from `sync.Map` to regular map w/ mutex 3b8a008 - Ensure stats accessed concurrently are explicitly atomic fe2ebd3 - Remove unused `sync.Waitgroup` from TCP stream factory 2e039ad - Add packet counts for the foundMatch path in GetOrStore() ## How to verify that this has the expected result On my local Linux machine, I compiled the main branch and the changes in this PR, and ran them concurrently while sending several thousand HTTP requests to a random netcat server. The results are identical: Main: ``` DBG TCP assembler stats IPdefrag=0 active_streams=0 conn_rejected_FSM=0 event_queue_length=500 goroutines=6 rejected_FSM=0 rejected_Options=0 source_dropped=0 source_if_dropped=0 source_received=4000 total_TCP_bytes=97500 total_streams=500 uptime_ms=25450 6:33PM DBG Stopping TCP assembler assembler_page_usage="pageCache: used: 0:" closed=500 ``` PR: ``` DBG TCP assembler stats IPdefrag=0 active_streams=0 conn_rejected_FSM=0 event_queue_length=500 goroutines=6 rejected_FSM=0 rejected_Options=0 source_dropped=0 source_if_dropped=0 source_received=4000 total_TCP_bytes=97500 total_streams=500 uptime_ms=20920 6:33PM DBG Stopping TCP assembler assembler_page_usage="pageCache: used: 0:" closed=500 ``` --------- Signed-off-by: Dan Bond <[email protected]> Co-authored-by: Robb Kidd <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Following up from above, I don't think this is something new. I've worked back through our releases and each of them drop some portion of traffic.
Observed using echoserver and via apache benchmark with following command:
ab -n 500 -c 10 -k http://localhost:80/echo_body\={agent_version}I've opened the following ticket to understand why / how much is not picked up.
I used echo server with 2 replicas and was events coming into Honeycomb:
However, the numbers do not match up with what locust things it's sent
After more fiddling, I can definitely see some traffic is being lost. I hit the end point manually once, wait then 5 more times.
This shows the one call, then only 2/5.
Originally posted by @MikeGoldsmith in #194 (comment)
The text was updated successfully, but these errors were encountered: