Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

app gets into weird state that prevents timeline from populating and syslog goes nuts from montage #26

Closed
alager opened this issue Aug 27, 2015 · 21 comments

Comments

@alager
Copy link

alager commented Aug 27, 2015

Here is a 1 second sample of syslog when this is happening:

Aug 26 17:28:18 myZoneMonitor zms[6011]: FAT [Unable to load monitor id 1 for streaming]
Aug 26 17:28:18 myZoneMonitor zms[6012]: FAT [Unable to load monitor id 2 for streaming]
Aug 26 17:28:18 myZoneMonitor zms[6010]: INF [Got signal 15 (Terminated), exiting]
Aug 26 17:28:18 myZoneMonitor zms[6009]: FAT [Unable to load monitor id 3 for streaming]
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12519 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12524 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12515 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12516 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12521 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12523 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor zms[6004]: INF [Got signal 15 (Terminated), exiting]
Aug 26 17:28:18 myZoneMonitor zms[6005]: INF [Got signal 15 (Terminated), exiting]
Aug 26 17:28:18 myZoneMonitor zms[6006]: FAT [Unable to load monitor id 3 for streaming]
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12529 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12526 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor zms[6010]: FAT [Unable to load monitor id 4 for streaming]
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12517 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12522 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor zms[6005]: FAT [Unable to load monitor id 4 for streaming]
Aug 26 17:28:18 myZoneMonitor zms[6004]: FAT [Unable to load monitor id 2 for streaming]
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12520 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12527 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12525 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12528 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor zms[6014]: FAT [Unable to load monitor id 2 for streaming]
Aug 26 17:28:18 myZoneMonitor zms[6013]: FAT [Unable to load monitor id 1 for streaming]
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12531 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12533 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12530 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12532 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor zms[6015]: FAT [Unable to load monitor id 3 for streaming]
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12534 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12535 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor zms[6016]: FAT [Unable to load monitor id 4 for streaming]
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12536 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12537 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor zms[6018]: FAT [Unable to load monitor id 4 for streaming]
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12538 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12542 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor zms[6020]: FAT [Unable to load monitor id 1 for streaming]
Aug 26 17:28:18 myZoneMonitor zms[6019]: FAT [Unable to load monitor id 2 for streaming]
Aug 26 17:28:18 myZoneMonitor zms[6022]: FAT [Unable to load monitor id 2 for streaming]
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12539 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12540 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12545 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12548 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12541 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
Aug 26 17:28:18 myZoneMonitor mysqld: 150826 17:28:18 [Warning] Aborted connection 12546 to db: 'zm' user: 'zmuser' host: 'localhost' (Unknown error)
@pliablepixels
Copy link
Member

The problem description:
Chrome does not allow more than 6 open TCP connections. When viewing monitor feeds in montage, I am using the "single image" mode of ZMS instead of "open stream" mode. This allows me to keep opening and closing connections at will and lets me display more than 6 monitors at the same time. I do a new image grab every second. The problem is that these requests are authenticated and each hit logs an INF record that user XXX logged in. Unlike in Aaron's case, I am not getting a "Aborted connection" - Im getting INF logs about authentication, but I suspect the core issue is the same.

Right now I don't have an easy way to solve this, but solve we must..... will think.....

@pliablepixels
Copy link
Member

Okay , note to self: I think I can fix it but I need to use ZM's auth token and not user=&pass=

@pliablepixels
Copy link
Member

@alager actually this issue does not seem to be the issue I thought it was. According to your log report, mysqld is throwing warnings --> it also shows failed zms. Are your montage view feeds working?

@alager
Copy link
Author

alager commented Aug 27, 2015

It appears that they stop after some event on the app. Like maybe the phone going to sleep?
They do show an image, it just isn't updating.

@alager
Copy link
Author

alager commented Aug 27, 2015

It's in that mode again...The montage on the phone isn't loading any images. Also, while it's doing that, the web console has difficulty displaying the monitors and the montage there doesn't work at the same time. If I go over to the logs panel (to get it to stop pulling data) then the web console starts working again.
If I let the app montage continue to "load" then the web console eventually returns this:
Unable to connect to ZM db.SQLSTATE[HY000] [1040] Too many connections
max_connections is 100 in my.cnf.

@alager
Copy link
Author

alager commented Aug 27, 2015

restarting the app returns the syslog messaging to normal and I can run montage on both the web console and the app at the same time.

@pliablepixels
Copy link
Member

So I just updated my version of ZM to the latest version. I am just not seeing mysqld warnings or errors. I somehow feel for some reason you have an issue with mysql's configuration. I'll dig more, but I've checked on my DB and I've asked some other testers on Android and they don't seem to be seeing this.

What they are however seeing is while in montage view or monitor live view mode, a log entry INF [Authenticated user 'admin'] every second - I am aware of that part and am going to fix it (eventually)

@pliablepixels
Copy link
Member

try this: don't go to montage view. Go to some other view and wait for your logs to stabilize. Then go to "Monitor" view and click on "Live View" - do you see your mysqld complaining?

@alager
Copy link
Author

alager commented Aug 27, 2015

After the app has been sitting a while, with the phone idle(screen shuts down), then starting it up:
Sitting on one monitor, the first few lines come in with an error, then nothing else for a while. The monitor was fully functional the whole time:

Aug 27 14:52:04 myZoneMonitor zma_m7[11421]: INF [Bullet-958c: 52000 - Analysing at 4.98 fps]
Aug 27 14:52:08 myZoneMonitor nph-zms[16504]: ERR [Can't insert log entry: MySQL server has gone away]
Aug 27 14:52:08 myZoneMonitor nph-zms[16504]: INF [Got signal 15 (Terminated), exiting]
Aug 27 14:52:08 myZoneMonitor zms[16503]: INF [Got signal 15 (Terminated), exiting]
Aug 27 14:53:25 myZoneMonitor zmc_m5[11403]: INF [Dome-92a9: 53000 - Capturing at 5.00 fps]

I have constant motion events happening and one camera recording, so the DB should have plenty of "traffic" to keep it awake. I don't see these types of messages now that it's up. I can go to any of the other live views and no messages. I can also go to the live view on the web console, no error messages.

@alager
Copy link
Author

alager commented Aug 27, 2015

my db setting for wait_timeout was set to 600. I put it back to the default of 28800, we'll see if that makes things better.

@alager
Copy link
Author

alager commented Aug 27, 2015

With montage working, my syslogs look like this:

Aug 27 15:14:53 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 433]
Aug 27 15:14:53 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2633]
Aug 27 15:14:53 myZoneMonitor zmc_m5[17704]: WAR [Packet in sequence, gap 543]
Aug 27 15:14:53 myZoneMonitor zmc_m5[17704]: WAR [Discarding frame 2686]
Aug 27 15:14:53 myZoneMonitor zmc_m8[17745]: WAR [Packet in sequence, gap 712]
Aug 27 15:14:53 myZoneMonitor zmc_m8[17745]: WAR [Discarding frame 1778]
Aug 27 15:14:56 myZoneMonitor zmc_m5[17704]: WAR [Packet in sequence, gap 424]
Aug 27 15:14:56 myZoneMonitor zmc_m5[17704]: WAR [Discarding frame 2700]
Aug 27 15:14:56 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 779]
Aug 27 15:14:56 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2647]
Aug 27 15:14:56 myZoneMonitor zmc_m7[17729]: WAR [Packet in sequence, gap 414]
Aug 27 15:14:56 myZoneMonitor zmc_m7[17729]: WAR [Discarding frame 2667]
Aug 27 15:14:58 myZoneMonitor zmc_m8[17745]: WAR [Packet in sequence, gap 1102]
Aug 27 15:14:58 myZoneMonitor zmc_m8[17745]: WAR [Discarding frame 1793]
Aug 27 15:15:02 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 354]
Aug 27 15:15:02 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2675]
Aug 27 15:15:09 myZoneMonitor zmc_m7[17729]: WAR [Packet in sequence, gap 260]
Aug 27 15:15:09 myZoneMonitor zmc_m7[17729]: WAR [Discarding frame 2733]
Aug 27 15:15:11 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 148]
Aug 27 15:15:11 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2719]
Aug 27 15:15:16 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 612]
Aug 27 15:15:16 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2742]
Aug 27 15:15:19 myZoneMonitor zmc_m7[17729]: WAR [Packet in sequence, gap 475]
Aug 27 15:15:19 myZoneMonitor zmc_m7[17729]: WAR [Discarding frame 2783]
Aug 27 15:15:22 myZoneMonitor zmc_m8[17745]: WAR [Packet in sequence, gap 189]
Aug 27 15:15:22 myZoneMonitor zmc_m8[17745]: WAR [Discarding frame 1879]
Aug 27 15:15:22 myZoneMonitor zmc_m8[17745]: WAR [Packet in sequence, gap 900]
Aug 27 15:15:22 myZoneMonitor zmc_m8[17745]: WAR [Discarding frame 1881]
Aug 27 15:15:26 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 401]
Aug 27 15:15:27 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2793]
Aug 27 15:15:29 myZoneMonitor zmc_m7[17729]: WAR [Packet in sequence, gap 317]

@pliablepixels
Copy link
Member

Thanks. These logs are related to ZM capturing rtsp frames. They should
also occur if you are viewing the ZM web montage.
So no more dB errors?

On Thursday, August 27, 2015, alager [email protected] wrote:

With montage working, my syslogs look like this:

Aug 27 15:14:53 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 433]
Aug 27 15:14:53 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2633]
Aug 27 15:14:53 myZoneMonitor zmc_m5[17704]: WAR [Packet in sequence, gap 543]
Aug 27 15:14:53 myZoneMonitor zmc_m5[17704]: WAR [Discarding frame 2686]
Aug 27 15:14:53 myZoneMonitor zmc_m8[17745]: WAR [Packet in sequence, gap 712]
Aug 27 15:14:53 myZoneMonitor zmc_m8[17745]: WAR [Discarding frame 1778]
Aug 27 15:14:56 myZoneMonitor zmc_m5[17704]: WAR [Packet in sequence, gap 424]
Aug 27 15:14:56 myZoneMonitor zmc_m5[17704]: WAR [Discarding frame 2700]
Aug 27 15:14:56 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 779]
Aug 27 15:14:56 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2647]
Aug 27 15:14:56 myZoneMonitor zmc_m7[17729]: WAR [Packet in sequence, gap 414]
Aug 27 15:14:56 myZoneMonitor zmc_m7[17729]: WAR [Discarding frame 2667]
Aug 27 15:14:58 myZoneMonitor zmc_m8[17745]: WAR [Packet in sequence, gap 1102]
Aug 27 15:14:58 myZoneMonitor zmc_m8[17745]: WAR [Discarding frame 1793]
Aug 27 15:15:02 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 354]
Aug 27 15:15:02 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2675]
Aug 27 15:15:09 myZoneMonitor zmc_m7[17729]: WAR [Packet in sequence, gap 260]
Aug 27 15:15:09 myZoneMonitor zmc_m7[17729]: WAR [Discarding frame 2733]
Aug 27 15:15:11 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 148]
Aug 27 15:15:11 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2719]
Aug 27 15:15:16 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 612]
Aug 27 15:15:16 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2742]
Aug 27 15:15:19 myZoneMonitor zmc_m7[17729]: WAR [Packet in sequence, gap 475]
Aug 27 15:15:19 myZoneMonitor zmc_m7[17729]: WAR [Discarding frame 2783]
Aug 27 15:15:22 myZoneMonitor zmc_m8[17745]: WAR [Packet in sequence, gap 189]
Aug 27 15:15:22 myZoneMonitor zmc_m8[17745]: WAR [Discarding frame 1879]
Aug 27 15:15:22 myZoneMonitor zmc_m8[17745]: WAR [Packet in sequence, gap 900]
Aug 27 15:15:22 myZoneMonitor zmc_m8[17745]: WAR [Discarding frame 1881]
Aug 27 15:15:26 myZoneMonitor zmc_m6[17713]: WAR [Packet in sequence, gap 401]
Aug 27 15:15:27 myZoneMonitor zmc_m6[17713]: WAR [Discarding frame 2793]
Aug 27 15:15:29 myZoneMonitor zmc_m7[17729]: WAR [Packet in sequence, gap 317]


Reply to this email directly or view it on GitHub
https://github.com/pliablepixels/zmNinja/issues/26#issuecomment-135571086
.

@alager
Copy link
Author

alager commented Aug 28, 2015

No I still get them. I'm trying to narrow it down. I know that if I start the app and go to the montage, there are no errors. But if I lock the phone (without exiting the app) and then try it some time later, then I get the errors.
That's why I thought it might be the 600 second timeout, but that didn't make a difference.

I won't be able to try any more until Monday.

pliablepixels referenced this issue Aug 28, 2015
…d Monitor view does not hit the ZM DB and generate a million records. Solves the log pings of https://github.com/pliablepixels/zmNinja/issues/26
@pliablepixels
Copy link
Member

I've released version 0.73 - may take a few hours for you to see it. Once your app upgrades, please let me know if it resolves your issues.

@alager
Copy link
Author

alager commented Aug 31, 2015

syslog behaves much better with 0.73. The issue I was seeing with the
Can't insert log entry: MySQL server has gone away
appears to be related to 100% cpu usage. My system is a little dual core fanless PC. Running my 4 cameras it runs around 80% cpu. ZM is able to run and I can view its montage, but when I was using zmNinja, it was causing "too many" connections per second, and the little CPU couldn't keep up.

Here is a cpu chart from nmon. The CPU is at 100% is when zmNinja is in montage (getting syslog errors too). When it is below 100% zmN is on the monitor panel, and ZM is both on the console and in montage view.
image

An interesting test, I exited zmN, then restarted it, and went to montage view, and things are well behaved; both CPU and syslog!
image

For a good 10+ minutes I let zmN sit in montage mode, and there were no issues. I locked my phone screen, then turned it back on, still no issue. I locked it and waited about 3-4 minutes, turned it back on and blamo! High CPU and syslog errors.

@pliablepixels
Copy link
Member

Thanks - some questions:

a) Are you saying your CPU runs at 80% even without viewing montage views of Zomeminder?
b) Can you please try this when you get time -- please keep a terminal open and check how many "zms" or "nph-zms" processes are running when your spikes happen

@alager
Copy link
Author

alager commented Aug 31, 2015

a) yes. I just modified path_swap to use a ram disk (/var/run) things seem to be better, now at 65% typical.
b) with 2 cameras configured for motion detect., there are 2 zmc, 2 zma, 0 zms.
image

running zmN montage( the step up):
image

left zmN in montage and locked phone. waited 3 minutes, unlocked it, and bad behavior:
image

@pliablepixels
Copy link
Member

Unfortunately I am just not able to replicate your sleep-wait 3-4 mins-awake-syslog/CPU issue

I just ran the latest client version on Android - Samsung S3
My ZM is running on a low end e-machines 140 with 8GB ram, dual-core
I have 4 HD cameras @ 1280x960 32bit color depth + 1 @ 640x480 24 bit

On an average, my CPU load is 18%. When you add a montage view of zmNinja, the CPU load goes to 36% when all cameras are enabled. When I use ZM's own Montage view the CPU goes to 22% (zmNinja takes more CPU because it uses a single snapshot mode - this ensures that zmNinja can show more than 6 monitors - default chrome limit or 15 monitors - default FF limit).

I tried powering off the screen 3 times, waited for 3-4 minutes each time and monitored top as well as syslog - both of them were normal - each time I turned back on, CPU went up to 33-36% and syslog said "user admin successfully logged in" -- thats it.

@alager
Copy link
Author

alager commented Aug 31, 2015

how are the cameras configured? motion detect, monitor, record?

On Mon, Aug 31, 2015 at 3:59 PM, Pliable Pixels [email protected]
wrote:

Unfortunately I am just not able to replicate your sleep-wait 3-4
mins-awake-syslog/CPU issue

I just ran the latest client version on Android - Samsung S3
My ZM is running on a low end e-machines 140 with 8GB ram, dual-core
I have 4 HD cameras @ 1280x960 32bit color depth + 1 @ 640x480 24 bit

On an average, my CPU load is 18%. When you add a montage view of zmNinja,
the CPU load goes to 36% when all cameras are enabled. When I use ZM's own
Montage view the CPU goes to 22% (zmNinja takes more CPU because it uses a
single snapshot mode - this ensures that zmNinja can show more than 6
monitors - default chrome limit or 15 monitors - default FF limit).

I tried powering off the screen 3 times, waited for 3-4 minutes each time
and monitored top as well as syslog - both of them were normal - each time
I turned back on, CPU went up to 33-36% and syslog said "user admin
successfully logged in" -- thats it.


Reply to this email directly or view it on GitHub
https://github.com/pliablepixels/zmNinja/issues/26#issuecomment-136522847
.

@pliablepixels
Copy link
Member

All modect. It is very important to lower FPS though on a low end machine - I've changed in-camera FPS of all of them to 3 (not in ZM - you need to do it in camera)

@pliablepixels
Copy link
Member

closing. I assume this is resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants