increased CPU and/or memory usage over time

[segordon]

(Edited by @pliablepixels - this issue affects devices as well, not just desktops)

Platform & OS Version
Linux x64. Arch. Kernel 4.14.r1028.g1122

The version of the app you are reporting:
1.2.503D

Device details:
Lenovo D30 workstation. 2x Xeon E5-2660, 32gb ECC RAM

What is the nature of your issue
BUG

Details
Increased CPU usage over time. Main process slowly creeps from 1%-3% to 100% over 50 hour span (could be shorter, gathering data now regarding how aggressively it eats CPU.)

UI becomes sluggish/slow to respond with relation to resources taken. Requires non-graceful forced exit at its' worst.

Let me know what I can provide to help. I'm going to upgrade to 1.2.504 to see if the issue is resolved, but I didn't read anything regarding this problem in the CHANGELOG...

Thanks!

[pliablepixels]

@segordon - do you compile from source? I have a patch to test. If not, I'll upload a binary.

[segordon]

Have been using the binary releases, but can compile to facilitate trouble-shooting.

The problem persists in 1.2.504. UI lag accompanied by abnormal CPU usage @ the 18hr mark.

Will now compile from source and test the new commits.

[segordon]

Getting software versions lined up for compilation is a pain in the neck on my platform -- a binary test release would be much appreciated.

If I don't hear back in a few hours i'll spin a *buntu VM and tackle compilation again.

(what ever happened to a nice make file? I feel so old nowadays ... Kidding ;) )

[pliablepixels]

I really have no idea where the issue is. There are two options:
a) Something my app is doing - I can try and fix that
b) This memory/cpu problem is with electron (the app wrapper I am using). I can't fix that

I assume the issue only is in the montage screen? Can you confirm?

Here is a test build:
https://drive.google.com/file/d/1d2L4MRulToh2ddmnpa8l3kIKhH8IlopC/view?usp=sharing

What I'm doing is reloading the montage screen every 1 hr - please see if that helps.

[segordon]

I have the test build running -- i'll let you know how it's going in a few hours.

I assume the issue only is in the montage screen? Can you confirm?

I've only had the problem develop in montage mode (since the app sits in that mode 95% of the time), but I cannot confirm whether or not it would eat CPU if it were sitting in a different mode. Can try that next.

What I'm doing is reloading the montage screen every 1 hr - please see if that helps.

Is this the same type of reload that occurs when one changes modes from the hamburger menu (say from system status/settings to montage/events) ? If so , I have done that manually when I felt the UI start getting sluggish with little to no effect on symptoms.

As for the electron stuff -- yeah I understand. If electron is eating CPU cycles that fast for us then I can't imagine the havoc it's causing elsewhere -- at least if the problem is on that side it'll probably be hunted down pretty fast.

Anyway, I have it running. I'll let you know how it goes and report back.

[segordon]

Looking good so far.

Caught the montage reload; CPU usage on the main thread dropped from ~7-9% back down to 2-5%. The rendering thread sits consistently at about 15-30% regardless, but doesn't seem to inflate from there.

No UI sluggishness yet, either. Will continue testing.

[pliablepixels]

Thanks for the report - I'll keep my fingers crossed. There are other open issues that I think are affected by this core issue of memory build up- I think they are all related in some way. If this works out I'll close out all of them and only keep one open. #526 #441

[segordon]

I haven't yet seen the memory usage inflate with my particular problem, but I'll keep an extra close watch this session to see if I notice it change.

I'll test with 1.2.504 too to see if I can trigger the sluggish UI problem again and take a look at the memory usage. I'm fairly sure that memory usage was tame when I saw the CPU pegged, but I don't have any logs saved.

Not by any means saying the problems are unrelated, just being specific in the interest of helping to track this down.

[segordon]

New test build ran smoothly all night. No noticeable resource inflation. Great!

Reverted back to 1.2.504.

The primary zmNinja process is sane, with no unusual CPU or memory inflation.

'--type=renderer' process, forked from process above, pegged at 80-100% usage after 4 hours of montage view. Sluggish UI. Memory use slightly elevated, from an initial 670m reserved at startup to now about 830m reserved at 4 hours of use.

The montage reset in the new test build seems to keep the renderer process subdued in the short-term. Haven't tested long enough to rely on it solving the issue completely.

[pliablepixels]

Thanks for the reports - I'll wait for confirmation from you before I can close this out. I'll keep this running on my android phone too tonight.

[segordon]

Test build still looks good. No unusual resource allocation.

Montage reset timer appears to have worked -- I'll keep an eye on it, but the former build would've ate itself many times over by now.

I'd say this hack is a success -- at least until the real cause is found. I think it might be a good idea to revisit this later on -- either when electron and the supporting software matures a bit longer, or we get more eyes on zmNinja. Non-conditional watchdog timers are usually not that great of a way to deal with these sort of gremlins; and if this is ends up being an issue with one of the underlying libraries we might wake up one day with this problem disappeared for us.

Thanks for all your effort!

[pliablepixels]

No problem. I can confirm my android phone survived an overnight montage onslaught as well. I actually don't think its an electron issue.

Its either my app, or Angular/Ionic framework itself, because this happens on phones too.

[pliablepixels]

@knnniggett recently reported this 'force reload' causes issues in Android 4.4 (crosswalk build). I could blame him for being the only one that uses 4.4 ;-) but I guess its time to revisit. Still keeping this closed - I'll reopen when my spirits are raring to dive into this

[knight-of-ni]

Now, now, my odroid is open and free. I only noticed the issue with android 4.4 on my way up to android 7. Sounds like a good idea to wait to look into this, however. If someone else reports it, we will know exactly what to look at.

Since I am likely going to forget what the error was, I'll post it here:

D/TEST    ( 5778): cdvfile://localhost/files/zmNinjaLog.txt: 68                                                                                                      
I/chromium( 5778): [INFO:CONSOLE(6)] "Mar 6, 2018 11:55 AM INFO zmAutologin successfully logged into Zoneminder", source: file:///android_asset/www/lib/filelogger/d)
I/chromium( 5778): [INFO:CONSOLE(6)] "Mar 6, 2018 11:55 AM DEBUG auth-success emit:Successful", source: file:///android_asset/www/lib/filelogger/dist/filelogger.min)
I/chromium( 5778): [INFO:CONSOLE(6)] "Mar 6, 2018 11:55 AM DEBUG REAUTH", source: file:///android_asset/www/lib/filelogger/dist/filelogger.min.js (6)                
I/chromium( 5778): [INFO:CONSOLE(6)] "Mar 6, 2018 11:55 AM INFO  Calling window.stop()", source: file:///android_asset/www/lib/filelogger/dist/filelogger.min.js (6) 
I/chromium( 5778): [INFO:CONSOLE(6)] "Mar 6, 2018 11:55 AM DEBUG DataModel: Getting auth from https://xxxxxxxxxxxxxxxx/zm/index.php?view=watch&mid=13&connkey=un)
I/chromium( 5778): [INFO:CONSOLE(6)] "Mar 6, 2018 11:55 AM DEBUG Storing login time as Tue Mar 06 2018 11:55:32 GMT-0600", source: file:///android_asset/www/lib/fil)
I/chromium( 5778): [INFO:CONSOLE(6)] "Mar 6, 2018 11:55 AM INFO DataModel: Extracted a stream authentication key of: df2b1cc2f088a18ccaeae4ba47cce89e", source: file)
I/chromium( 5778): [INFO:CONSOLE(6)] "Mar 6, 2018 11:55 AM INFO Stream authentication construction: &auth=df2b1cc2f088a18ccaeae4ba47cce89e", source: file:///android)
D/TEST    ( 5778): cdvfile://localhost/files/zmNinjaLog.txt: 74                                                                                                      
D/TEST    ( 5778): cdvfile://localhost/files/zmNinjaLog.txt: 56                                                                                                      
D/TEST    ( 5778): cdvfile://localhost/files/zmNinjaLog.txt: 34                                                                                                      
D/TEST    ( 5778): cdvfile://localhost/files/zmNinjaLog.txt: 49                                                                                                      
D/TEST    ( 5778): cdvfile://localhost/files/zmNinjaLog.txt: 146                                                                                                     
D/TEST    ( 5778): cdvfile://localhost/files/zmNinjaLog.txt: 83                                                                                                      
D/TEST    ( 5778): cdvfile://localhost/files/zmNinjaLog.txt: 112                                                                                                     
D/TEST    ( 5778): cdvfile://localhost/files/zmNinjaLog.txt: 101                                                                                                     
I/chromium( 5778): [INFO:CONSOLE(6)] "Mar 6, 2018 11:55 AM INFO Reloading view to keep memory in check...", source: file:///android_asset/www/lib/filelogger/dist/fi)
D/XWalkCordovaUiClient( 5778): onPageLoadStarted(file:///android_asset/www/index.html#/app/montage)                                                                  
D/CordovaWebViewImpl( 5778): onPageDidNavigate(file:///android_asset/www/index.html#/app/montage)                                                                    
D/JsMessageQueue( 5778): Set native->JS mode to null                                                                                                                 
W/cr.BindingManager( 5778): Cannot call determinedVisibility() - never saw a connection for the pid: 5778                                                            
E/chromium( 5778): [ERROR:runtime_javascript_dialog_manager.cc(118)] Not implemented reached in virtual void xwalk::RuntimeJavaScriptDialogManager::CancelActiveAndP)
I/chromium( 5778): [INFO:CONSOLE(8)] "The key "viewport-fit" is not recognized and ignored.", source: file:///android_asset/www/index.html#/app/montage (8)          
D/JsMessageQueue( 5778): Set native->JS mode to EvalBridgeMode                                                                                                       
I/chromium( 5778): [INFO:CONSOLE(106)] "Uncaught SyntaxError: Block-scoped declarations (let, const, function, class) not yet supported outside strict mode", source)
D/XWalkCordovaUiClient( 5778): onPageLoadStopped(file:///android_asset/www/index.html#/app/montage)                                                                  
D/CordovaWebViewImpl( 5778): onPageFinished(file:///android_asset/www/index.html#/app/montage)                                                                       
I/chromium( 5778): [INFO:CONSOLE(165)] "******* PLATFORM READY ****", source: file:///android_asset/www/index.html (165)                                             
I/App     ( 5778): WARNING: Back Button Default Behavior will be overridden.  The backbutton event will be fired!                                                    
I/chromium( 5778): [INFO:CONSOLE(26799)] "[native transition] enabling plugin", source: file:///android_asset/www/lib/ionic/js/ionic.bundle.js (26799)               
I/chromium( 5778): [INFO:CONSOLE(26799)] "[native transition] disabling ionic transitions", source: file:///android_asset/www/lib/ionic/js/ionic.bundle.js (26799)   
I/chromium( 5778): [INFO:CONSOLE(6)] "2018-03-06T17:55:38.281Z INFO Device is ready", source: file:///android_asset/www/lib/filelogger/dist/filelogger.min.js (6)    
I/chromium( 5778): [INFO:CONSOLE(6)] "2018-03-06T17:55:38.284Z INFO You are running on android", source: file:///android_asset/www/lib/filelogger/dist/filelogger.mi)
I/chromium( 5778): [INFO:CONSOLE(26799)] "[native transition] $stateChangeStart", source: file:///android_asset/www/lib/ionic/js/ionic.bundle.js (26799)             
I/chromium( 5778): [INFO:CONSOLE(26799)] "[native transition]", source: file:///android_asset/www/lib/ionic/js/ionic.bundle.js (26799)                      

[segordon]

@pliablepixels @knnniggett what issues were experienced with 4.4? I have a 24 inch tablet that I use as a wall-mounted ZM head that runs Android 4.4.2 that began experiencing API connection failure messages on force-reload. The only way to reconnect to the ZM server is to kill the process and reload it completely. Is this similar to what you're seeing?

I haven't done much to diagnose it yet since it's easy enough to reload the app, and I hardly use the tablet (and i've been lazy) -- but if we're seeing something similar I can provide logs.

(I band-aided it by sitting the tablets' browser on the image stream for the camera that mattered; but i'd prefer to use zmninja again on that head)

[knight-of-ni]

Sounds like you've noticed the same issue.

After 60 minutes of runtime, zmninja re-authenticates with the zerver.
On my test machine, using either android 7 or Armbian (Ubuntu Xenial) + zmninja desktop, this process causes the screen to briefly flash black to gray then back to normal.

However, when I loaded Android 4.4.4 on my test machine or ran the test on my Galaxy Pro tablet w/ Android 4.4.2, the screen would flash but then zmninja would go to the to API failed, see the FAQ, screen. I have to completely exit zmninja and restart it. It's like it silently gives up. No errors are visible in the zmninja log or the web server log.

The logs you see in my previous post are from running logcat at a rooted command prompt.

Since it sounds like you are using zmninja for the same purpose, I'd be curious to learn what hardware you are using. Do they really make a 24" tablet, or is this something you have built?

[pliablepixels]

grudglingly reopened two people in the world are still using 4.4 :-))
(actually according to android stats, 12% still use 4.4)

[pliablepixels]

@segordon yep exactly what Andy reported. The offending code is this line. Looks like the chrome version bundled with the crosswalk library doesn't like how this code reloads the page and it goes bonkers.

[segordon]

@knnniggett https://www.nabitablet.com/nabi-big-tab-hd24 . They are a re-brand of some other Chinese group. They come loaded with a bunch of kid stuff, but they are rootable, and make a nice wall-mountable surveillance head. The resolution isn't that hot, but they can be had for pretty cheap.

That said : I wouldn't expect updates for it, and i'd keep it on the private network that we all keep around for shady overseas network devices. (we DO do that, right?!)

[knight-of-ni]

https://www.nabitablet.com/nabi-big-tab-hd24 . They are a re-brand of some other Chinese group. They come loaded with a bunch of kid stuff, but they are rootable, and make a nice wall-mountable surveillance head. The resolution isn't that hot, but they can be had for pretty cheap.

Wow, it's even 1080p, and to think I was going to couple an odroidxu4 to one of these instead:
https://www.newegg.com/Product/Product.aspx?Item=N82E16824016320R&cm_re=planar_touchscreen-_-24-016-320R-_-Product

Talk about paying more for less.

[segordon]

Wow, it's even 1080p, and to think I was going to couple an odroidxu4 to one of these instead:
https://www.newegg.com/Product/Product.aspx?Item=N82E16824016320R&cm_re=planar_touchscreen-_-24-016-320R-_-Product

That's almost exactly what I was going to do initially. The big tab is a nice solution if you just want a surveillance head to run zmNinja on, but the odroid will always offer a lot more flexibility for future projects.

I just needed a screen for my elderly parents to look at a surveillance camera on, and it worked beautifully for that. As a tinkerer i'd probably go for the odroid otherwise.

[pliablepixels]

@segordon I have an APK that seems to be working with Xwalk (android 4.4). Would you like to give it a try? If so, please shoot me an email (pliablepixels gmail) - give it the usual soak test - overnight, and if it also continues to help with CPU like the original version did

[pliablepixels]

closing, please follow #598