High CPU usage (99%) by NuGet.Server

[karthickramasamy08]

Hi,

We are configured our own NuGet feed using NuGet Server.
We have hosted nearly 20 sub applications in our server. each application is a NuGet feed.
We are faced NuGet server cache issue recently but as per below posts latest patch updated, the issue resolved. after that no issues occurred.

#5230

but recently we are facing our application taking 99% CPU usage. and below error logging continuously in event viewer. can you please suggests what is the reason for this issue.

[joelverhagen]

@karthickramasamy08, thanks for reporting this. It sounds like there are two issues going on:

1. Intermittent UnathorizedAccessException.
2. Extremely high CPU utilization. This has been mentioned before by @NagarajMasub.

Do you notice any relationship between these two events?

Do all of the feeds have the same packages or do they have different packages and different number of packages? Could you provide some numbers per sub-application (feed):

• How many packages are on each feed?
• What is the total size of the packages directory on each feed?
• What is the size of the cache.bin file on each feed?

You said you have 20 sub-applications. Are they all running as different processes?

Could you look through your event viewer and verify that all of the UnauthorizedAccessException have the same stack trace?

[karthickramasamy08]

Hi @joelverhagen ,

Please find the below reply for each queries mentioned in the last comment .

Do all of the feeds have the same packages or do they have different packages and different number of packages?
Yes, We have maintained the different number of packages in different feeds.
For Ex: http://nuget.feed.com/packages/aspnet
means all the aspnet related packages are available in single folder.

Could you provide some numbers per sub-application (feed):

• How many packages are on each feed? Each feed link has approximately minimum of 150 packages to maximum of 11,500 packages(All versions included) & also minimum 200zip files for website direct downloads.
• What is the total size of the packages directory on each feed? The size of each feed link has more than 25GB for some application. For less packages, size should be less than 7Gb.
• What is the size of the cache.bin file on each feed? Based on package size on each sub application, cache-bin file size was different. The size for each feed has less than 1Mb to more than 13Mb.

You said you have 20 sub-applications. Are they all running as different processes?
At first, we have maintained the each application should run at different processes(Application pool), but now,, all the sub applications have run as single processes(Same Application pool).

Could you look through your event viewer and verify that all of the UnauthorizedAccessException have the same stack trace?
Yes,we have ensured from our side, all exception error having same stack trace.

[karthickramasamy08]

Hi @shishirx34 ,

Any update on this

[skofman1]

@karthickramasamy08 , as far as I understand, you are running multiple NuGet.Server applications on a single VM.
From looking at the code that throws the exception:
https://github.com/NuGet/NuGet.Server/blob/master/src/NuGet.Server.Core/Infrastructure/ServerPackageCache.cs#L263

I see that there's a ReaderWriterLockSlim that protects the write operation from being performed by multiple threads simultaneously. However, if multiple processes try to write to the same file, this lock will be useless, and the write operation will be susceptible to collisions.
Please let me know if you agree with this analysis.

[karthickramasamy08]

Hi @skofman1,

Thanks for the update.

Yes, we have already set the write permission for all the sub application folder. Now, we have accessed the NuGet packages in the Visual Studio and also we didn't caught the UnAuthorizedAccessException in the Event viewer.

But, we are still faced the high CPU percentage issue in our NuGet Server. Already we have logged this CPU issue in this link(#5230). Please check and update your concerns.

[skofman1]

@karthickramasamy08 , @joelverhagen and I have a theory that the cause of the 99% CPU is the FileSystemWatcher. It updates the package cache appropriately on file system changes.
You can turn it off by setting this setting to "false": https://github.com/NuGet/NuGet.Server/blob/master/src/NuGet.Server/Web.config#L71

Please try this out, and let us know if it resolves the issue.

[MathanKumarHJ]

@skofman1 ,

Thanks for sharing details,

We have tried this case, but, still the CPU usage taken 99%. do we need to do anything further? or Could you please check and provide the correct fix for this server performance issue? We have faced production level issue due to this.

Note: I posted this message behalf of @karthickramasamy08

Thanks,
Mathan Kumar H J

[MathanKumarHJ]

@skofman1/ @joelverhagen ,

Could you please update on this? is any other possibility to fix this issue?

Thanks,
Mathan Kumar H J

[skofman1]

@MathanKumarHJ , @karthickramasamy08 , since you are running NuGet.Server in a huge scale, it's hard for us to replicate your setup and investigate your scenario. All we can do from here is look at the code and make guesses.
FileSystemWatcher that you turned off by setting
https://github.com/NuGet/NuGet.Server/blob/master/src/NuGet.Server/Web.config#L71 to 'false' is the only background process in NuGet.Server, so we need to look deeper to find the issue, but for this we need more information that only you can provide.
To investigate the root cause, please take the https://github.com/NuGet/NuGet.server code from here, build in "Debug" mode and deploy to your system. Run a CPU profiler to determine which method causes the 99% CPU. Send us the details, and we will take it from there.

[skofman1]

@karthickramasamy08 , any updates?

[karthickramasamy08]

Hi @skofman1,

We can’t do this in our NuGet production server.

After Install NuGet.Server version 3.0.3-jver-big-tmp-21027 from the following package source:
https://dotnet.myget.org/F/nuget-build/api/v3/index.json
Packages are assembled in folder structure. Root folder package will not remove after folder structure.

Example:

Root folder name – Package Path
We have publish below .nupkg file in Root folder (Package Path).
Exp1.Wpf15.4.0.25.nupkg
Exp2.Wpf15.4.0.25.nupkg

Exp1.Wpf14.2.0.12.nupkg
Exp2.Wpf14.2.0.12.nupkg

While accessing our feed link this .nupkg packages are folder structure like below.

Root folder	Sub folder	Inner folder	File name 1	File name 2
Package Path	WPf	15.4.0.25	Exp1.Wpf15.4.0.25.nupkg	Exp2.Wpf15.4.0.25.nupkg
Package Path	Wpf	14.2.0.12	Exp1.Wpf14.2.0.12.nupkg	Exp2.Wpf14.2.0.12.nupkg

Question:
If we have removed .nupkg file from root folder. All the packages properly display in visual studio ( from sub folder).

1. Why nuget file not removed from root folder? After creating folder structure.
2. Is this root folder (.nupkg) file taking CPU 99% ?Any other option to reduce CPU.
   Also, can you please check this.
   “500 internal server” problem instead of return the “No Packages found” message in NuGet server hosted application. Home#6681

Thanks in Advance,
Karthick R

[joelverhagen]

After Install NuGet.Server version 3.0.3-jver-big-tmp-21027

Note that you shouldn't need to use this private build anymore. This fix has gone into the main branch and is available in NuGet.Server 3.1.0.

Why nuget file not removed from root folder? After creating folder structure.

The package can be left in the root folder for a variety of reasons. For example, if the package already exists or if the package file is invalid in some way the package won't be moved.

While accessing our feed link this .nupkg packages are folder structure like below.

Under the package path, the sub folder should be the full package ID. Wpf seems to only be a prefix of the ID. I would expect four subfolders (assuming these are also the IDs, based off of the file names provided above {ID}.{Version}.nupkg):

• Exp1.Wpf15
• Exp2.Wpf15
• Exp1.Wpf14
• Exp2.Wpf14

We can’t do this in our NuGet production server.
Is this root folder (.nupkg) file taking CPU 99% ?Any other option to reduce CPU.

We have been unable to reproduce the high CPU utilization on our side so it is very hard for us to guess what's causing the problem on your servers.

Could you take some time to attempt to reproduce the issue in an isolated environment so you can run some CPU profiling tools? For example, you could spin up a dedicated server isolated from your production environment and push the same set of packages that are in your production environment.

If this is too much effort, you could try a minimally intrusive profiling technique like PerfView. For example, I was able to find the following CPU flame graph from the IIS process (iisexpress.exe, in my case):

If you could provide an .etl.zip captured during your high CPU utilization, it will help us diagnose the issue.

[joelverhagen]

Please feel free to re-open this issue if you are still blocked and you can provide the requested performance analysis.