How do I handle this error: "too many open files in system"

[josh-m-sharpe]

+------------------------------------------------------------------------------+

| (5) cache-push                                                               |
+------------------------------------------------------------------------------+
| id: cache-push                                                               |
| version: 2.2.0                                                               |
| collection: https://github.com/bitrise-io/bitrise-steplib.git                |
| toolkit: go                                                                  |
| time: 2019-09-20T12:29:56Z                                                   |
+------------------------------------------------------------------------------+
|                                                                              |
Config:
- Paths: ./vendor/bundle -> Gemfile.lock
./node_modules -> yarn.lock
- IgnoredPaths: 
- CacheAPIURL: [REDACTED]
- FingerprintMethodID: file-content-hash
- CompressArchive: false
- DebugMode: false
- StackID: 
Cleaning paths
Done in 8.034978765s
Checking previous cache status
Previous cache info found at: /tmp/cache-info.json
Done in 2m16.542042163s
Checking for file changes
Previous cache is invalid, new cache will be generated:
0 files needs to be removed
0 files has changed
116815 files added
File changes found in 58.662739ms
Generating cache archive
Failed to populate archive: failed to lstat(/Users/vagrant/git/node_modules/appium-support/node_modules/core-js/fn/math/isubh.js), error: lstat /Users/vagrant/git/node_modules/appium-support/node_modules/core-js/fn/math/isubh.js: too many open files in system
WARN[12:33:35] Step (cache-push) failed, but was marked as skippable 
|                                                                              |
+---+---------------------------------------------------------------+----------+
| ! | cache-push (exit code: 1)                                     | 3.7 min  |
+---+---------------------------------------------------------------+----------+
| Issue tracker: https://github.com/bitrise-steplib/steps-cache-push/issues    |
| Source: https://github.com/bitrise-steplib/steps-cache-push.git              |
+---+---------------------------------------------------------------+----------+

[josh-m-sharpe]

Output from ulimit:

+ ulimit
unlimited

[bitce]

@josh-m-sharpe can you please include a build URL too?

[josh-m-sharpe]

Apologies, I have asked this same question on intercom - Andrew responded to me there.

The failing build in question: https://app.bitrise.io/build/46a6fb95a940bd50#?tab=log

[josh-m-sharpe]

For what it's worth, I see this post, and the answer, but would argue that the response is invalid:
https://discuss.bitrise.io/t/too-many-open-files-in-system-from-cache-push/9114

On CircleCI and Bitrise, my yarn install takes roughly 3 minutes.

On CircleCI, the cache restore takes 50 seconds.

So there's definitely some room for optimization.

[santiagofm]

@bitce @josh-m-sharpe Any update on this? we are experiencing the same issue: https://app.bitrise.io/build/d7b9552977bbe2e4#?tab=log

[josh-m-sharpe]

I have another email thread going with Andrew at bitrise. He says they’re investigating.

…

On Tue, Oct 8, 2019 at 9:39 AM pera ***@***.***> wrote: @bitce <https://github.com/bitce> @josh-m-sharpe <https://github.com/josh-m-sharpe> Any update on this? we are experiencing the same issue: https://app.bitrise.io/build/d7b9552977bbe2e4#?tab=log — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30?email_source=notifications&email_token=AAAJUMIOY6YTY4D4PJ4KZITQNSEQ5A5CNFSM4IYWRQPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAUGKIY#issuecomment-539518243>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAJUMJDOXBGEWLBDG4O3DTQNSEQ5ANCNFSM4IYWRQPA> .

[bitce]

Hi guys! This should be fixed since #32 so in the latest 2.2.1 step version. Can you let us know what you're experiencing? :)

[josh-m-sharpe]

It seems like I'm now able to cache ./node_modules but unfortunately, I don't think the change is worthwhile. There still seem to be some shortcomings of cache-push. Here are two builds, from the same branch/commit. So they have identical yarn.lock files. 632: https://app.bitrise.io/build/bd2b912489795274#?tab=log 633: https://app.bitrise.io/build/a776dd9c3b72879a#?tab=log 632: cache-pull: 13.92 seconds yarn install: 3.6 minutes cache-push: 7.7 minutes This seems expected. Long yarn install times and long cache push times. 633 was run immediately following 632: cache-pull: 1.1 minutes yarn install: 6.5 seconds cache-push: 3 minutes Why is cache-push still 3 minutes when it (should be) pushing the exact same data that was set up in 632? I would have expected that 633's cache-push would have recognized that the cache already existed and finished very quickly.

…

On Tue, Oct 15, 2019 at 7:55 AM Bence Kubik ***@***.***> wrote: Hi guys! This should be fixed since #32 <#32> so in the latest 2.2.1 step version. Can you let us know what you're experiencing? :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30?email_source=notifications&email_token=AAAJUMMXYM355WZSWGRW6ITQOWVSVA5CNFSM4IYWRQPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBIO7YI#issuecomment-542175201>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAJUMOBJ7F3C6QLY3OAKWDQOWVSVANCNFSM4IYWRQPA> .

[ghost]

Hey @josh-m-sharpe , thanks for sharing! The step is working as intended. In build 633, the JSON parser takes 2 min 40 seconds to parse your previous cache's manifest to see if any additional uploads need to be performed. It then determines there are no changes and completes in a few hundred milliseconds.

[josh-m-sharpe]

@ahvth-bitrise I'm a little confused. I gotta say, I assumed that the reason the "Cache paths" input was structured the way it is:

...was because the step was doing something with the file on the right-hand side. But it sounds like this algorithm does something with every single file in the cache? That seems like huge waste of time and resources to me.

Why not just md5 the lock file? That's what I was assuming it was doing. md5 takes all of zero seconds:

time md5 yarn.lock
MD5 (yarn.lock) = 25efdb164f36da36ee85427c8494f665

real    0m0.020s
user    0m0.003s
sys     0m0.005s

[ghost]

Hm, ideally the update indicator file would be the only file that gets checked in this case, yes. One issue that comes to mind looking at your specific example is that perhaps the relative paths are tripping up the step here. Could you try switching those out with the absolute paths of your update indicator files?

If that fails, I will make an investigation task for our tooling team to explore why the hashing took so long in your case.

FYI, there is also a second method here that you can make use of which only checks the date modified to designate files for upload. Could you please try setting the file-mod-time Fingerprint Method in the cache-push step's settings?

[josh-m-sharpe]

I have no interest in caching based on the timestamp of the file - that will just produce false broken caches. MD5 should really be the way to go here.

[ghost]

Understood. Could you please try with absolute file paths for the update indicator files and let me know if it works for you? If not, please send me the build URL here or in our Intercom conversation and I will initiate an investigation.

[josh-m-sharpe]

Closing this as the "too many open files in system" is was resolved. Thanks!

[lpusok]

Hi @josh-m-sharpe,
In the latest release (https://github.com/bitrise-steplib/steps-cache-push/releases/tag/2.2.2) speeds are improved when using an indicator file.