Look at tuning options / scripts provided by user

[apatrida]

Look at incorporating as examples, or documentation these scripts provided by user @magicdude4eva

https://gist.github.com/magicdude4eva/3b5fec150fbcaafdc34c

[apatrida]

For the GC tuning, would be good to know what JDK version they are targeted at, assuming JDK 7. How did that compare to G1 without tuning parameters? In the example the heap is 3G, surprised that it has a lot of GC pause or timeout that would need tuned outside of using G1.

[apatrida]

Including a pointer to the GIST in the startup/shutdown section for now, until i can circle back

[magicdude4eva]

The Lucene guys (https://wiki.apache.org/lucene-java/JavaBugs) do not recommend G1:

"Do not, under any circumstances, run Lucene with the G1 garbage collector. Lucene's test suite fails with the G1 garbage collector on a regular basis, including bugs that cause index corruption."

I will take on production load next week - our index is only 1.5GB big and the JVM tuning was based on JDK1.8. Only thing I did was remove incompatible options. I don't feel comfortable using G1 based on the Lucene feedback.

[apatrida]

It looks to be only a 32bit problem for GC1.

[magicdude4eva]

Saw that now too. I think I will try out G1 on JDK8 - for the time being I will use Shawn's baseline settings (unless you have better suggestions) - https://wiki.apache.org/solr/ShawnHeisey

[apatrida]

No, that's a good start

[magicdude4eva]

I updated the GIST with G1 tuning options and have been running it in a non-production environment for 4 days. The JVM settings are based on two indices (2GB and 400MB) running on:

• CentOS 7 with 4 cores and 9GB RAM
• JDK 8 64bit build 1.8.0_51-b16

Using GCViewer and JClarity, heap usage is good without any major pauses. Going to update to solr_undertow 1.4 and then switch into production.

[magicdude4eva]

For what it's worth, we could not get the same performance out of Solr running on 5.3.x. We had to revert back to Solr 5.2.1 which outperforms 5.3.x. Over the next week we will move to Solr 5.4.0 and see if this makes a difference.

Software stack is still CentOS7, 4 cores, 10GB RAM. Using JDK 8 64bit 1.8.0_60 with G1GC. Solr heap is set at 3GB with 250ms pause time. This configuration works perfectly fine on 5.2.1.

On 5.3.x Solr runs fine for a number of days and eventually performance degrades. There are no system issues (swapping, IO, spiking CPU) or Java issues (no full GCs). It looks like Solr is just locking up processing requests.

[stephlag]

Have you tried more recent releases (5.5.2 or even 6.1) ?