Sync fails: "too many open files" (original matter addressed but bulk upload introduced another w/ similar symptoms)

[gcala]

Hi, sync seems broken on my laptop. It fails with a series of "Too many open files" messages in the client. It stops for a moment and then restarts the whole process from beginning.

I don't know if this error is related to desktop client or machine where it runs.

Any idea? Thanks

Client configuration

Client version: 2.5.1 git (rev. b37cbe)

Operating system: Arch Linux

OS language: Italian

Qt version used by client package (Linux only, see also Settings dialog): 5.12.0

Client package (From Nextcloud or distro) (Linux only): community/nextcloud-client 2.5.1-2

Server configuration

Operating system: Arch Linux

Web server: apache 2.4.37

Database: mariadb 10.1.37

PHP version: 7.3.0

Nextcloud version: 15.0.2

Logs

1. Client logfile: https://cuteworks.it/nextcloud/s/KzWbigWWwNjRk2n

2. Web server error log: no errors

3. Server logfile: no errors

[benjamin-leiding]

I have the same issue on my Linux system and it seems to be only triggered when I process larger numbers of very small files, e.g., small pictures or git-repositories. Client 2.3.3 works, so it has to be a problem that was introduced with version 2.5.x. I also tried the dev-branch for the nextcloud-client -> same issue.

Client configuration

Client version: 2.5.1
Operating system: Xubuntu 18.04 LTS
OS language: English

Server configuration
Managed Nextcloud (Owncube.com) - Version: 15.0.0

Logs
Web server error log: to my knowledge -> no errors
Server logfile: to my knowledge -> no errors
Client logfile:
20190109_1148_owncloud.log.0.gz
20190109_1200_owncloud.log.0.gz
20190109_1200_owncloud.log.1.gz
20190109_1212_owncloud.log.0.gz
20190109_1213_owncloud.log.0.gz

Let me know whether you need more info for debugging purposes!

[GieltjE]

Could it be this crashes on Windows machines? When we try to sync large numbers of new files (2k+) it just crashes after a while.

[benjamin-leiding]

Yes, I was also able to reproduce the same results on a virtual machine with Win 7 and the 2.5.1 client for Windows. Unfortunately, I cannot provide logs at the moment. But the Linux logs should already help the dev team once they take care of this issue.

@GieltjE Could you maybe also upload logfiles to make it easier for the dev team to fix our issue?

[GieltjE]

The client log just logs sync complete, a friend of mine will hopefully post the windows logs soon.

We also noticed that it happens on lots of small files, when the files get bigger it doesn't crash every time.

@bleidingGOE, the windows logbook should provide the most information

[dennmuel]

Confirming the "too many files" error on Linux Mint. However the client does not restart but instead either dies without notification or causes the operating system to freeze.
As a workaround one can disable sync for the causing folders, so that at least all others get synced.

[benjamin-leiding]

It has been 2 month since @gcala opened the issue - the dev team did not even respond despite us providing several logs and precise descriptions. Is nextcloud still actively developed?

[GieltjE]

According to github there are enough commits in the server code, but the desktop client just seems to be getting updated translations....
We can't even force the clients to update to new versions resulting in major parts of the userbase running outdated crap....

[camilasan]

Hi All,

Note that Github isn't a support portal. We provide support to customers only. Github is our developer platform. If you're not a customer you can ask the community for help on help.nextcloud.com.

We of course try to deal with bugs as quick as we can, but it can take time as we have lots of other things on our plate. You are welcome to contribute to this free and open source product, to submit patches that fix bugs for example. We try to respond quickly to pull request from contributors.

Have a great day!
Camila

[lbdroid]

I ran into this issue today after a distro upgrade. The version I was running prior to the upgrade was 57bc791 . I am running that version again, and it is not affected by this REGRESSION.

If anybody cares enough about this problem to work on it, I'd suggest bisecting with that commit as a starting point.

@camilasan : your response comes off as hostile and rude. Not a particularly good way to encourage people to help in this project. I mean after all, how dare people want to ASSIST the project by reporting and discussing bugs in lieu of supporting it financially.

[jmjos]

Hello!

I also encountered the problem. It's not a bug within nextcloud, rather a system feature limiting the number of open files. E.g. see https://easyengine.io/tutorials/linux/increase-open-files-limit/ for a fix.

The issue can be closed.

[GieltjE]

It's not a Linux only problem, we only use Windows clients.

[benjamin-leiding]

I already tried this on my ubuntu 18.04 LTS - did not work. Moreover, the problem also occurs on Windows.

…

On 27 June 2019 10:00:14 CEST, JM Joseph ***@***.***> wrote: Hello! I also encountered the problem. It's not a bug within nextcloud, rather a system feature limiting the number of open files. E.g. see [https://easyengine.io/tutorials/linux/increase-open-files-limit/](https://easyengine.io/tutorials/linux/increase-open-files-limit/) for a fix. The issue can be closed. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #1035 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[wioxjk]

Experiencing the same problem with 2.6.1 on the following distros:

Ubuntu 18.04, 19.10
Ubuntu Budgie 18.04, 19.10
Trisquel 8, 9
Debian 9, 10
Parabola
Windows 8, 8.1, 10
Linux Mint 18, 19
OpenSuSE Tumbleweed

sadly, changing ulimit does not work.

The problem does not occur on 2.3.1

Photo and videofolder is 900 GB

[todoneunl]

+1 on Ubuntu 19.10.

I'm trying to sync a 25GB photo folder. 'Too many open files', 100% CPU and application freeze. I tried it with or without hidden files.

Seems like a huge issue and I'm perplexed that this is not taken seriously and still not fixed.

[wioxjk]

Seems like a huge issue and I'm perplexed that this is not taken seriously and still not fixed.

I agree, I've asked in the official Nextcloud channels but I have not even gotten a reply there.

[wioxjk]

Still no attention from the developers?
I am strongly considering stop using Nextcloud both for private use and corporate, this issue is starting to become a plague for both usecases.

[claell]

I added some appropriate labels. Hopefully some developer can take a look at this, soon.

Just to get the impact correctly: This is completely preventing the concerning files to be ever synced and does not slowly disappear after more and more files got synced, correct?

[benjamin-leiding]

Thanks for adding the labels!
Regarding your question: No, it does not slowly disappear - I left it running for several days without any progress.

[claell]

Thanks for clarifying @bleidingGOE. That make the issue rather severe in cases where it appears.

[SolarDesalination]

It's a bit embarrassing that Nextcloud is more focused on developing sub-standard apps that there already exist viable open source solutions for instead of making sure that foundational pieces such as folder synchronization is working properly. This bug has been open for over a year. I won't be recommending this software in the future to corporate clients.

[lbdroid]

@mpass3 : in all fairness (and I have no ties to nextcloud from a development perspective, just as a user), desktop synchronization really isn't a primary feature of nextcloud, and in many/most cases, the concept of synchronization is too confusing to actually use safely/reliably. I.e., change a file in both local and remote filesystem... Which version wins? The fact that there is no INTUITIVE answer to this means that the concept is flawed. In fact, this very problem is the reason why tools like git exist, and you would never hand that tool to a corporate paper pusher.

For corporate use, what you really want is to mount the nextcloud data into the local filesystem. You can use WebDAV for that. Let the end user decide on what files to overwrite.

[benjamin-leiding]

@lbdroid : I absolutely disagree. Even for large corporations, a client synchronization feature is essential. The issue of "which file should I keep" has been solved by Dropbox, GDrive, Syncthing, and so many other projects.

Moreover, based on your suggestion I tried to use my Nextcloud on Ubuntu 18.04 LTS with different WebDAV approaches -> They are horribly slow. No way to use this productively. Once you start googling the "slow WebDAV" issue they recommend using the client, which leads us back to this issue here.

[claell]

Can we please keep the discussion here on topic? Feel free to continue in the forum, you may post a link to a that here, so the others know where to continue. The topic you are talking about is definately worth disputing. But this is the issue tracker of nextcloud and this is a specific issue, so your comments are rather unrelated.

[benjamin-leiding]

Again, I disagree. The relevance of an issue is determined by how many users it affects and whether it can easily be solved by using a different approach. Not clarifying the gravity of the issue is why it has not been addressed in more than a year.

[carforge]

Happening on Version 3.4.2 (Ubuntu) Sync Client on Nextcloud Hub 3 (25.0.1) on Ubuntu 22.04.1 LTS

[mrbc42]

Also happening with 3.6.2 app-image on Debian 11.5

[seventhsite]

3.4.2 Ubuntu 22.04.1
I did it myself - copied the files to the server and occ scan.
It's just a semi-automatic application.

[Veehem]

Also happening to me when reinstalling a new Nextcloud :(
client 3.4.2 - Xubuntu 22.04 - Nextcloud 25.0.2

[suaudeau]

I solved this problem by adding the following line to the config.php file of the Nextcloud server :
'bulkupload.enabled' => false,
I have experienced this problem on several servers with different configuration and Nextcloud version 25.0.2.
The function is an optimization for small files, but the server still work pretty fast.

[GieltjE]

Still getting the "An exception occurred while executing a query: SQLSTATE[40001]: Serialisation failure 1213 Deadlock found when trying to get lock; try restarting transation)" when uploading thousands of tiny files at once.

But at least it functions now :)

[mrbc42]

Still getting the "An exception occurred while executing a query: SQLSTATE[40001]: Serialisation failure 1213 Deadlock found when trying to get lock; try restarting transation)" when uploading thousands of tiny files at once.

But at least it functions now :)

I went backwards using all versions and found that 3.3.6 is the version that will work fast and without errors when you have 1000's of files to upload initially. After seeding the initial files, you can switch back to the latest version and it will run fine. I am convinced that the nextcloud team does not test new client versions with an initial upload of 1000's of files since there are so many reports about this problem over the last couple of years with no fix. Using nextcloud with a steadily increasing number of files is no problem so they don't actually witness the issue in order to fix it. For me, I had a problem with 72k text, image and pdf files until I installed the version above.

[avatar4d]

I'm also being bitten by this issue with NC client Version 3.4.2 on Pop!_OS 22.04 LTS with kernel pop-6.2.6-76060206-generic. NC Server is Hub 3 (25.0.5) running on Debian 11 via Docker.

@suaudeau's suggestion appears to have subdued the error and permitted the files to sync, but I also saw the serialization failures noted by @GieltjE (mariadb is the backend).

What seems weird to me is that the client was showing a too many files error when the open files didn't seem to come close to the limit (on both client and server).

[ChaosBlades]

I solved this problem by adding the following line to the config.php file of the Nextcloud server : 'bulkupload.enabled' => false, I have experienced this problem on several servers with different configuration and Nextcloud version 25.0.2. The function is an optimization for small files, but the server still work pretty fast.

This also solved the issue for me.
Just did a clean reinstall of NextCloud and was trying to re-upload everything back to the server via the desktop app on Linux and I ran into this error for the first time. Never had this issue before and I have done this several times.

[trivialfis]

The bulkupload seems to be problematic. After disabling it, not only the too-many open files issue is workarounded, nextcloud is also correctly synchronizing files that were mysteriously ignored.

[herrdeh]

I solved this problem by adding the following line to the config.php file of the Nextcloud server :
'bulkupload.enabled' => false,

I cannot find this statement in my configs. Would you @suaudeau be so kind and mention the precise location?

Thanks a lot in advance, Wolf

[suaudeau]

I cannot find this statement in my configs. Would you @suaudeau be so kind and mention the precise location?

You can add in in file : <nextcloud_dir>/config/config.php the line anywhere in the array :

<?php
$CONFIG = array (
...
  'bulkupload.enabled' => false,
...
);

[herrdeh]

Thanks for dummy-proof clarification - much required here...(;
Solution works for me.

Cheers,
Wolf

[evdcush]

I cannot find this statement in my configs. Would you @suaudeau be so kind and mention the precise location?

You can add in in file : <nextcloud_dir>/config/config.php the line anywhere in the array :

<?php
$CONFIG = array (
...
  'bulkupload.enabled' => false,
...
);

I don't see a <nextcloud_dir>/config/config.php in my nexcloud_dir or in ~/.config/Nextcloud/.

However, I do see a ~/.config/Nextcloud/nextcloud.cfg.
Can we specify this option there?

If not, are we supposed to add a config.php file to a synchronized directory?

[suaudeau]

I don't see a <nextcloud_dir>/config/config.php in my nexcloud_dir or in ~/.config/Nextcloud/.

However, I do see a ~/.config/Nextcloud/nextcloud.cfg.
Can we specify this option there?

If not, are we supposed to add a config.php file to a synchronized directory?

~/.config/Nextcloud/nextcloud.cfg is the configuration file of the client.

This problem has to be solved on the server. Are you searching <nextcloud_dir>/config/config.php on your server?

[PhilippVerpoort]

This issue is still persistent in NC client version 3.4.2.

Setting bulkupload.enabled to false (as described above) solves the problem.

Nextcloud version 3.4.2-1ubuntu1
Using Qt 5.15.12, built against Qt 5.15.2
Using Qt platform plugin 'xcb'
Using 'OpenSSL 3.0.2 15 Mar 2022'
Running on KDE neon 6.0, x86_64

[PhilippSchlesinger]

This issue is still persistent in NC client version 3.4.2.

@PhilippVerpoort This version is more than three years old, the recent version is 3.12.3.

[JulienFS]

This issue is still persistent in NC client version 3.4.2.

@PhilippVerpoort This version is more than three years old, the recent version is 3.12.3.

Thank you for pointing this out. It seems that Ubuntu Jammy (22.04 LTS) is shipping an outdated version. Ubuntu users should consider moving to the Appimage from the official website instead. It seems that the package is still maintained as mantic ships 3.10 and noble 3.11 (source https://packages.ubuntu.com/search?keywords=nextcloud-desktop). Maybe it's time to move to the next LTS...

[joshtrichards]

Quick summary because unfortunately what happens with these long lived issues is that multiple similar but different situations end up getting collapsed together which creates confusion:

• Original issue was likely covered by Free IconJob after use #1899 (within 2.6.x branch only) and Free IconJob after use #2477 (in >=v3.1.0)
• Later, when bulk upload was introduced (in v3.4.0), it seems to have brought out an issue with similar symptoms. This is the current matter this issue seems to have grown to cover.

Ideally the original would have been closed out and later unrelated reports in their own issue, but since there's already so much history here I'll leave this open for now. I will, however, tag it to go along with our collection of bulk upload matters.