Nodes appear and disappear intermittently

[2opremio]

While testing #889 (b103f93) with the ECS demo, after letting it run for a few hours I see containers appearing a disappearing intermittently.

There are 3 httpserver containers and 3 dataproducer containers which should appear at the same time in the UI but they come and go erratically:

Here's the report, although I am not sure it will help here: https://gist.github.com/2opremio/b3b4f435b568fb836306

I don't see a particularly high CPU or memory consumption but the UI is sloppy and loading the topologies takes a considerable time when loading the UI.

The ECS demo uses very small AWS instances (t2.micro) but Scope works just fine after it's freshly spawned.

[2opremio]

Related: #869 #827 (and maybe #854)

[paulbellamy]

Possibly the probes on different hosts are being slow and missing their deadlines.

[tomwilkie]

Paul can't reproduce.

[tomwilkie]

@2opremio please try and reproduce.

[2opremio]

I can reproduce.

Possibly the probes on different hosts are being slow and missing their deadlines.

I think @paulbellamy is right.

Looking at the logs I find a lot of:

<probe> WARN: 2016/04/25 16:25:22.817941 Topology tagger took longer than 1s
<probe> WARN: 2016/04/25 16:25:37.092299 Endpoint reporter took longer than 1s
<probe> WARN: 2016/04/25 16:25:38.651785 Docker reporter took longer than 1s
<probe> WARN: 2016/04/25 16:25:42.024260 Topology tagger took longer than 1s
<probe> WARN: 2016/04/25 16:25:55.008294 Endpoint reporter took longer than 1s
<probe> WARN: 2016/04/25 16:25:58.244352 Topology tagger took longer than 1s
<probe> WARN: 2016/04/25 16:26:11.398667 Endpoint reporter took longer than 1s
<probe> WARN: 2016/04/25 16:21:15.602084 docker container: dropping stats.
<app> ERRO: 2016/04/25 16:21:16.178981 Error on websocket: websocket: close 1006 (abnormal closure): unexpected EOF

Full logs: logs.txt.gz

It seems that the AWS micro instances used for the demo are not powerful enough for Scope. They used to work fine, so there must have been a performance degradation.

[2opremio]

Yep, the CPU consumption of the probe/app is pretty high considering the small amount of containers and that there are only 3 hosts:

Probe profile: pprof.localhost:4041.samples.cpu.002.pb.gz

App profile: pprof.localhost:4040.samples.cpu.001.pb.gz

The garbage collector is dominating the CPU consumption.

Related: #1010

[2opremio]

So, the solution here is to:

• Improve performance (obviously)
• Maybe use larger instances for the ECS demos (I would like to avoid this if possible since larger instances are not covered by the free tier)
• Notify users in the UI when probes are not meeting their deadlines. I have created Notify users when probes are not meeting their deadlines #1379

[tomwilkie]

I think #1418 should help a lot towards CPU usage of the app. We could probably close this one.

[tomwilkie]

#1418 is in, so I think we can close this.

[2opremio]

Reopening since it still happens with the 0.15 candidate:

All the probes are dropping reports:

[ec2-user@ip-172-31-0-6 ~]$ docker logs --since="5m"  weavescope |& grep Dropping | tail -n 20 && date
<probe> ERRO: 2016/05/13 10:45:36.602905 Dropping report to 10.32.0.2:4040
<probe> ERRO: 2016/05/13 10:45:36.602948 Dropping report to 10.36.0.2:4040
<probe> ERRO: 2016/05/13 10:45:41.777678 Dropping report to 127.0.0.1:4040
<probe> ERRO: 2016/05/13 10:45:46.752431 Dropping report to 10.36.0.2:4040
<probe> ERRO: 2016/05/13 10:45:50.243380 Dropping report to 10.36.0.2:4040
<probe> ERRO: 2016/05/13 10:45:50.243410 Dropping report to 127.0.0.1:4040
<probe> ERRO: 2016/05/13 10:45:54.513633 Dropping report to 127.0.0.1:4040
<probe> ERRO: 2016/05/13 10:45:59.237180 Dropping report to 10.32.0.2:4040
<probe> ERRO: 2016/05/13 10:45:59.237238 Dropping report to 10.36.0.2:4040
<probe> ERRO: 2016/05/13 10:46:01.406421 Dropping report to 127.0.0.1:4040
<probe> ERRO: 2016/05/13 10:46:01.406500 Dropping report to 10.32.0.2:4040
<probe> ERRO: 2016/05/13 10:46:01.406560 Dropping report to 10.36.0.2:4040
<probe> ERRO: 2016/05/13 10:46:06.290269 Dropping report to 127.0.0.1:4040
<probe> ERRO: 2016/05/13 10:46:06.290325 Dropping report to 10.32.0.2:4040
<probe> ERRO: 2016/05/13 10:46:06.290377 Dropping report to 10.36.0.2:4040
<probe> ERRO: 2016/05/13 10:46:09.527878 Dropping report to 127.0.0.1:4040
<probe> ERRO: 2016/05/13 10:46:09.527907 Dropping report to 10.32.0.2:4040
<probe> ERRO: 2016/05/13 10:46:13.921125 Dropping report to 10.36.0.2:4040
<probe> ERRO: 2016/05/13 10:46:18.355134 Dropping report to 10.36.0.2:4040
<probe> ERRO: 2016/05/13 10:46:18.355182 Dropping report to 127.0.0.1:4040
Fri May 13 10:46:22 UTC 2016

The apps are consuming ~40% CPU in two of the nodes just like it did at the beginning of this issue.

Which means that #1418 doesn't seem to be helping.

App profile: pprof.localhost:4040.samples.cpu.001.pb.gz

[2opremio]

Related #1457

[2opremio]

This may already be fixed after the recent CPU-consumption improvements. Worth reviewing again.

[rade]

Let's close this; I don't see anything in this issue that points to causes other than cpu usage, which is already covered by numerous issues.