[Update Request] Mono to 5.16 (current stable) - fixes Radarr/Sonarr memory leaks

[jasonla]

Sonarr and Radarr packages are experiencing memory leaks, and I believe it was trace to Mono being the culprit.

Radarr Issue: Radarr/Radarr#1580
Sonarr Issue: Sonarr/Sonarr#2296

Mono Issue that addresses the memory leak in version 5.14: mono/mono#7356

Can we please upgrade to Mono 5.16?

[piejanssens]

5.16 stable is out - I suggest to update your issues's title to "[Update Request] Mono 5.16 (current stable) - fixes Radarr/Sonarr memory leaks"

[jasonla]

5.16 stable is out - I suggest to update your issues's title to "[Update Request] Mono 5.16 (current stable) - fixes Radarr/Sonarr memory leaks"

Request updated. Thanks for the heads up.

[piejanssens]

I just checked the changelogs of mono and given we are currently on 5.8, updating mono will resolve another major and long outsdanding issue for Radarr and Sonarr.

From mono 5.12 release notes:
HttpWebRequest async handling has been rewritten. This resolves many long-standing and hard to reproduce issues involving requests cancellation or timeout. As HttpWebRequest is used as the underlying implementation by other types like HttpClient this should improve the reliability of a broad range of types.

[ymartin59]

@m4tt075 Hello. Do you have timeslot to update Mono package?

[m4tt075]

@ymartin59 Happy to look at it in the next couple of days. Will put it on my list. But I think there is some backlog in terms of open PRs which deserved attention first...

[m4tt075]

I've just tried to update to latest release but ran into a regression, which occurs with cross-compilations. See Mono Issue 9951 for details. I'm hesitant to trade one set of problems for another. To those of you, who have had memory leak issues with the current package: How bad is it?

[jasonla]

@m4tt075 It's pretty bad. I'm on a smaller NAS with 1GB of RAM. If I don't restart Sonarr or Radarr once a week, the entire system is unusable. A hard reboot is required. If you check Radarr thread I linked, you'll see others have up to 4 GB of memory used by mono. Some have 2GB. So looks like it various depending on how much RAM your system has, which sounds right because the runtime will adjust itself to system RAM capacity.

[m4tt075]

@jasonla OK, I don't like to work around the regressed 5.16, but will try the 5.14 version instead. It should contain the fix already. Which NAS / platform are you running? Would like you to test it, when I'm done.

[jasonla]

@m4tt075 I'm on DSM 6, and using a Synology DS214play, which I believe is the Evansport platform. Yes, I'm willing to test it.

[m4tt075]

OK chaps. DSM 6.1 and 6.2 test builds available here:
https://github.com/m4tt075/spksrc/releases/tag/mono-5.14-1
Please give it a try and report back whether this fixes your issue.

Of note: After the update the 88f6281 architecture does not compile the btls library anymore. I could simply exclude it, but not sure about the consequences. Does anyone know whether it is required, can be replaced or simply be omitted?

[jasonla]

@m4tt075 I’ll give it a try after I get home from work. If install works, it will probably take a few days of watching memory usage before I can give you any meaningful info.

Would it be ok for me to publicize this test build on Reddit? Just so we have people testing on different platforms and not just my 214play?

Edit: Installation went fine. Waiting and watching memory now.

[m4tt075]

@jasonla Thanks for testing. Yes, please feel free to link those builds via Reddit, if you haven't done that already. The more testers the better. If possible, please also re-post the question I've raised, as it affects ARM5 users, if we don't find a way forward for that platform...

[piejanssens]

Working great (tested Armada 370). Random HTTP socket issues seem to be resolved.

[Faltimore]

@Matt075 just a question. would this update fix issues if radarr is in a docker container? and which one for Synology DS918+

Cheers

[jasonla]

@Faltimore If it’s in Docker, then the Dockerfile controls which version of mono is installed.

Edit: And it also depends which docker repo you're pulling from. If it's the common linuxserver.io repo that a lot of people recommend (no idea if it's legit or not, I just see it recommended), then no this won't help your issues at all. This package update is only going to help if you're running the Radarr/Sonnar packages directly on DSM, not inside a Docker container.

I tracked down which version of mono the linuxserver.io Radarr images are built on, and it's 5.10.1.20. If you want the docker images updated, you need to talk to the owner/maintainer of those images.

Cheers.

[m4tt075]

Thanks for stepping in, @jasonla! 👍
Can you share some insights on your memory usage already or is it still too early to say?

[jasonla]

I've noticed that the memory leak is much improved when using 5.14. However, it started climbing again today, but in very small increments, and not nearly as dramatic as it was under 5.8. I'm tempted to call this at least an improvement, but can we wait a couple more days to see if it gets worse?

[m4tt075]

Sure. No prob. It also seems there have only been 11 downloads of the test packages so far. Certainly not enough yet...

[jasonla]

Reporting back. So far, so good. I've noticed that memory will get released, and it doesn't reach the levels I was seeing before.

[jasonla]

Welp, Radarr has an internal upgrade mechanism, and after upgrading, I can't seem to run it anymore. I'm checking to see if it's related to the upgrade script being screwed up, or if Radarr is incompatible with this 5.14 build, or my DSM is screwed up.

[piejanssens]

FYI - I did the update and it still works fine here. So it's not related to the mono test pkg.

[jasonla]

Thanks for the feedback @piejanssens! I did a little digging, and this is what I found.

1. Something was wrong with my install of Radarr. So I uninstalled it and tried to reinstall (and forgot to backup my config/database for Radarr. Oops, but that's not important.)

2. I have the test mono 5.14 package installed. When I tried to install Radarr via the SynoCommunity package using the built in package manager UI, it installs fine, but will not start.

3. There's a reason it doesn't start: Radarr/Sonarr/Jackett,etc were all using a version of NLog.dll that was not compatible mono 5.10 or above. See this thread: Systemd service won't start after mono update Radarr/Radarr#2585

The error with NLog.dll has been fixed in subsequent versions of Radarr. The problem is, the version of Radarr that is installed by default in the SynoCommunity packages relies on mono 5.8, and users will experience the same start up issues I had if they install 5.14, and then try to install Radarr. Users can't use the self-updater built into Radarr since they can't get it to start.

Based on the thread I linked, I believe Sonarr and Jackett would experience this issue too if mono is at version 5.14 (or anything above 5.10).

If we want to release the 5.14 package, it would mean the you'd have to update the Sonarr/Radarr/Jackett packages that are installed by default in SynoCommunity packages. I don't know how much work is involved in that.

Edit: Further reference links:

https://forums.sonarr.tv/t/mono-5-10-update-broke-sonarr/17800/19
https://www.reddit.com/r/sonarr/comments/80ylur/new_sonarr_version_v2005183_and_mono_510_guide_if/

[m4tt075]

Thanks for looking into this and posting all the references, @jasonla. Very helpful and much appreciated. I just find it very difficult to advance from here: From what I've experienced in the last couple of weeks / months, it is very difficult to find any half-way recent stable mono version that does not create some kind of problem or regression. What I also observed is that the number of test package installations of my mono 5.14 is still very low (14 as of now). This usually indicates, that only few people actually have issues with our current package. I'd therefore struggle to push any update until all relevant mono, sonarr, radarr and jackett issues are fixed upstream.

@ymartin59, @Safihre (if you are still around) : Any advice / further insights from your side?

[ymartin59]

@m4tt075 I really agree with you... it make sense to "group" these applications deliveries because of publish process efforts. Hurry is a high risk for urgent support requests because of application disfunctions, and produce much more hurry then.

[ERR1R]

When compiling jackett from source it also seems it is using the wrong dll so it wont work on mono 5.8
Mono 5.16 does work..

Jackett/Jackett#4232

[gotson]

Recent versions of Jackett are broken with the current Mono 5.8. Seems fixed with the latest 5.18 though.

Jackett/Jackett#4369

[defaultsecurity]

@m4tt075
First of all. Good job! Everything seems to work. Here are my impressions of the first 30 minutes.

Mono:
Installed and started under a minute.
Jackett:
Installed and started under a minute. Successfully updated with restart to latest (0.11.46.0) version. Seems to be functioning normally.
Radarr:
Installed and started under a minute. Restored config.xml and nzbdrone.db via winscp and restarted successfully. Seems to be functioning normally.

I will continue testing and report back in a few days. Is there anything in particular you want me to watch out for?

So far so good. Again, congratulations and sincere thank you for your hard work.

[m4tt075]

@bolhaskutya Great. Many thanks for testing. Could you (or anybody else) please do me a favor and test Sonarr as well? I've just updated the package ("Sonarr-19") and the link above. Needed to rebase, so would like to make sure it (still) works.

[defaultsecurity]

@m4tt075
Sure thing, glad to help. Sonarr also installed and ran under a minute. Everything seems to be working fine at first glance. All the menus are responsive and load their respective pages. The log doesn't show any unusual entries. I don't use Sonarr, so I can't test it as readily and thoroughly as Radarr, but it seems to be good to go!

I'll report back again on all packages in a few days.

[m4tt075]

@bolhaskutya Excellent. Many thanks.

[defaultsecurity]

@m4tt075
I have found a minor bug. Jackett doesn't start if Mono is installed on another volume, due to the way Mono's path is called. This does NOT happen with Radarr/Sonarr.

Jackett's log:
"env: /volume2/@appstore/jackett/../mono/bin/mono: No such file or directory"
The way I see it, Jackett assumes that Mono is in its parent directory (relative path)

I'm no expert, but it can possibly be fixed following Radarr/Sonarr's example in this file:

jackett_x64-6.1_0.11.43-8.spk/scripts/service-setup (Line 34)
Change this
MONO_PATH="${SYNOPKG_PKGDEST}/../mono/bin"
To this
MONO_PATH="/usr/local/mono/bin"

I don't know if this is valid for other architectures or platforms.

[m4tt075]

Wow, that's a corner case really and never worked. Anyways, you are right and your proposal should fix the problem. @ymartin59 I'm torn and tired of producing more and more test packages. If you agree, I'd propose to merge and publish first and fixing this later. Or we just change it "at risk". Please advise...

[ymartin59]

I am publishing already... For MONO_PATH, I have preference for DSM-way: /var/packages/mono/target/bin

[defaultsecurity]

Awesome. Just reporting in after 4-5 days of usage. Jackett, Radarr, Sonarr are working great, their logs are clean, communication between Jackett and other packages is perfect. Thumbs up!

[m4tt075]

@ymartin59 Yes, fine as well. We just have to hardcode the links and not absolute paths to fix this corner case, but we'll keep it for the next round. And again, many thanks for building and publishing. I couldn't have done it with my equipment...
@bolhaskutya Thanks for the confirmation. I'm glad we finally found a package combination that seems to work...

[ymartin59]

@m4tt075 This PATH issue is a minor trouble as there is a simple work-around: install both packages on same volume.
I have all packages ready and most have been uploaded in spkrepo - may you please check to confirm none is lacking?
But mono packages are just too large and fails with timeout - I have CPUs and disks but my upload bandwidth is too low and so operations timed out. I already tried to tune nginx/uwsgi previously but not perfect yet - if unlucky in worst cases, I may take benefits of a better internet link this week-end.

[m4tt075]

@ymartin59 Yes. Many thanks. Just checked and it seems all the "mother" packages are complete for 6.1 platforms. Not sure there is still a need for 5.2 builds. Maybe not or only on demand...

W.r.t. your mono "troubles": I actually find it reassuring that I'm not the only one lacking "horsepower" in one way or the other. Good thing is, we can team up: I have 12Mbit/s upload rate on offer. If you fancy a visit to Germany and you bring your machine, more than welcome... :-)

[ymartin59]

@m4tt075 That is my idea. I will publish 5.2 only on demand.
Have you noticed each mono package is 140 MiB large? 20 MiB more than previous version... Isn't there a way to get it lighter?

[defaultsecurity]

I see that the new packages have appeared on the synocommunity.com, but in my synology's package center I only see the old versions (Jackett 0.10.589-7, Radarr 20180303-6) Is that normal?

[m4tt075]

Yes, they have been uploaded but not yet activated. Mono is still missing because of the problems @ymartin59 mentioned above. All packages need to be activated at once.

[defaultsecurity]

Great! Thanks for the info. I installed the latest Jackett and Radarr manually by downloading from synocommunity.com, they both work flawlessly with Mono previously linked in this thread.

[Jayjo4]

I don't know if this is the correct place form my issue below, but after searching for a while I don't see other options.

My DS112+ worked fine until the last update of mono. At this moment Radarr and Sonarr are not working. I've tried several attemps of uninstall / install. Ended on this page and tried the latest packages (mono_88f6281-6.1_5.18.0.240-12 / nzbdrone_88f6281-6.1_20190315-15 and radarr_88f6281-6.1_20190304-8). The result is the same. Neither Sonarr or Radarr starts. The radarr.log contains the following error:
Any idea??

Native stacktrace:

    /usr/local/mono/bin/mono() [0xf1574]
    /usr/local/mono/bin/mono() [0xb30d8]
    /lib/libc.so.6(__default_rt_sa_restorer_v2+0) [0x40179370]
    /lib/libc.so.6(gsignal+0x40) [0x40177fa0]
    /lib/libc.so.6(abort+0x1b0) [0x4017bea8]
    /usr/local/mono/bin/mono() [0x27f53c]
    /usr/local/mono/bin/mono() [0x279fcc]
    /usr/local/mono/bin/mono() [0x294fdc]
    /usr/local/mono/bin/mono(monoeg_assertion_message+0x20) [0x29522c]
    /usr/local/mono/bin/mono() [0x7db1c]
    /usr/local/mono/bin/mono() [0xf7bf8]
    /usr/local/mono/bin/mono() [0xf8488]
    /usr/local/mono/bin/mono() [0x38e50]
    /usr/local/mono/bin/mono() [0xb67ec]
    /usr/local/mono/bin/mono() [0xb6e78]

Pkilling 0x41d56450 from 0x400217c0
Entering thread summarizer pause from 0x400217c0
Finished thread summarizer pause from 0x400217c0.

Waiting for dumping threads to resume

Debug info from gdb:

mono_gdb_render_native_backtraces not supported on this platform, unable to find gdb or lldb

Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.

[m4tt075]

@Jayjo4 Not the right place. Please open a new issue. However, before doing this, please do the following:

1. ssh into your NAS as "admin"
2. gain root priviledge via "sudo su -" (using the admin password again)
3. Execute "synogear install" and wait
4. Reproduce the error above and then post the (hopefully extended) output in the new issue.

[stormtrooper298]

Memory leak here was at almost 7GB of ram used. Randomly discovered this. Updated mono to the latest version and not Radarr won't start.

[m4tt075]

@stormtrooper298 What NAS architecture and firmware do you run?

[Damien13150]

Hello, any updates available for DS118 NAS ?

[mreid-tt]

@jasonla, what's the current status of this issue? is it now considered closed?

[jasonla]

@houndtt I don't think I can speak about it anymore. I haven't tested this change since moving to a Syno box with Docker support. If someone else feels comfortable with confirming that the fix worked, then by all means, please close it.

[publicarray]

Since we have mono versions 5.8,x 5.18.x and 5.20.x I assume this can be closed