Total thread count in testng parallel tests with dataproviders

[ajmera25]

TestNG Version

7.0.0-beta3

Expected behavior

Need to have Global thread count so that we can control total thread when we have x number of thread-count and x number of data provider thread count.

Actual behavior

Let say,If we have 10 thread count and 10 data provider thread count then we will end up running 100 thread.

I am not reporting this is as a bug. Created this issue so that it will kick off the discussion around this and if it can be considered as a feature request.
https://groups.google.com/forum/#!topic/testng-users/BKfSgHoAChU

Is the issue reproducible on runner?

Yes

[krmahadevan]

ping @cbeust @juherr Your thoughts on this ?

[juherr]

Yes, it is something we can do.

[enivid]

Seems like you could just modify the Invoker class and change this code:

// For invocationCount > 1 and threadPoolSize > 1 run this method in its own pool thread.
if (testMethod.getInvocationCount() > 1 && testMethod.getThreadPoolSize() > 1) {
return invokePooledTestMethods(testMethod, suite, parameters, groupMethods, testContext);
}

Add a new conditional for shared thread pool so that data provider tests would also go through this code and share the main threadpool.

[krmahadevan]

@enivid would you be willing to submit a pull request with this change ?

[enivid]

I looked at this a bit more and I don't see an easy fix. The TestRunner class is creating the ExecutorService with the thread limit. You'd now have to pass a reference to the ExecutorService down to the TestInvoker so it can use that ExecutorService instead of creating its own executor when running parallel dataproviders or tests with multiple invocation counts.

[enivid]

Maybe instead of refactoring everything, the easiest thing to do is just make a new BlockingQueue. The BlockingQueue will have a fixed size to how many tests you want running. Now testng can create as many threads as it wants but its still going to limit how many tests run at a time. You could add this into TestNg itself or just make your own ITestListener. You can use the start and finish methods to add and remove from your BlockingQueue.

[kool79]

Can we setup higher priority for that?
TestNG gots high popularity because of 2 things:

• We can run tests in parallel in few clicks
• Beautiful and simple implementation for DDT out-of-the box with @dataProvider methods
  But when those features are used together we still have issues like huge thread count in some cases. It is VERY critical for webdriver tests where each test either consumes many system resources (when we use local executions, without grid hub) or occupies whole execution queue on grid-hub.

[xuanzhaopeng]

Hey @krmahadevan , may I ask is it still on the roadmap? since it's very hard to control the count of threads when I enable method parallel and dataprovider parallel. Better we could have a Max number all threads.

[krmahadevan]

@xuanzhaopeng Yes this is on the radar. But yet to get to it due to bandwidth issues. Would you be willing to help raise a PR for this ?

[Dinhbaon]

@krmahadevan Is this something that is still going to be worked on??

[krmahadevan]

@Dinhbaon - There is plan to get this sorted out. But I haven't been able to get to this yet. Would you like to raise a PR ?

[avishkargautam77]

@krmahadevan @ajmera25 @juherr Is this resolved? If Yes can anyone explain this a bit how can we use it? I am also stuck with a similar problem where I need global thread-count.

[krmahadevan]

@avishkargautam77 Please refer to https://testng.org/#_controlling_threadpool_usage

[avishkargautam77]

@krmahadevan For example, when I use use-global-thread-count=true and set thread-count=2, with a testng.xml containing two tests—one using a DataProvider with two datasets and the other without any DataProvider—three threads are being initialized. Could you explain why this happens, and is there any way to restrict the thread usage to exactly 2?

[krmahadevan]

Pls share the following:

• test cases that are being executed ( need to see a simple trimmed down version )
• the suite XML file being used.
• Testng version being used

[krmahadevan]

Also please go through the documentation.

There is a flag called useGlobalThreadPool which ensures TestNG uses one thread pool for data driven and non data driven tests ( by default its two different pools )

There is another flag called shareThreadPoolForDataProviders which ensures that all data driven tests in your suite share the same thread pool.

[avishkargautam77]

@krmahadevan
Suite XML:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.1.dtd">
<suite name="XML -Job Suite" thread-count="2" parallel="tests" share-thread-pool-for-data-providers="true">
	<listeners>
		<listener class-name="com.qa.ExtentReportListener.ExtentReporterNG" />
	</listeners>
		<test name="Appointment Test Cases-done 1">
			<classes>
				<class name="com.crm.qa.testcases.AppointmentPageTest">
					<methods>
						<include name="addAppointment" />
					</methods>
				</class>
			</classes>
		</test>
		<test name="All Assigned Candidate Page Test Case-11">
			<classes>
				<class name="com.crm.qa.testcases.AllAssignedCandidatesPageTest">
					<methods>
						<include name="bulkEmailToCandidateInAllAssignedCandidateListView" />
					</methods>
				</class>
			</classes>
		</test>
</suite>

Testcases Being Executed:

@Test(dataProvider = "getAppointmentDetails")
public void addAppointment(String relatedTo) {
    // ... (rest of the test code)
}

@Test
public void bulkEmailToCandidateInAllAssignedCandidateListView() {
    // ... (test code)
}

@DataProvider(parallel = true)
public Object[][] getAppointmentDetails() {
    return new Object[][] { { "Data1" }, { "Data2" } };
}

Testng Version: 7.9.0
I tried using shareThreadPoolForDataProviders=true as well, but the same thing is still happening.

[krmahadevan]

Please include the xml here instead of the google drive link. Also please try using the latest released version 7.10.2

[avishkargautam77]

@krmahadevan Let me try.

[avishkargautam77]

Please include the xml here instead of the google drive link. Also please try using the latest released version 7.10.2

Tried with 7.10.2 as well, this still persists.

[krmahadevan]

I am not sure I understand what the problem is.

You have specified this in your suite file

<suite name="XML -Job Suite" thread-count="2" parallel="tests" share-thread-pool-for-data-providers="true">

What this means is as below:

• I would like to execute all <test> tags in parallel.
• I would like to set the thread pool size of normal test methods to 2 and I would like to set the thread pool size of data driven tests to 10 (You haven't specified any value, so we are going to be defaulting to 10)
• You want to use one common thread pool for all data driven tests in your entire suite and the size of the thread pool is 10.

So with these in place, you should basically see that at any given point in time irrespective of how many ever data driven tests exist in your suite file, ONLY 10 will run in parallel.

I am quoting the documentation once again for your reference

-shareThreadPoolForDataProviders	true,false	Indicates if TestNG should use a global shared thread-pool (at suite level) for running data driven tests. TestNG will consider the value set for the configuration parameter -dataproviderthreadcount as the size of the thread pool.
-useGlobalThreadPool	true,false	Indicates if TestNG should use a global shared thread-pool (at suite level) for running regular and data driven tests. TestNG will consider the value set for the configuration parameter -threadcount as the size of the thread pool.

[avishkargautam77]

@krmahadevan , Let me try to make this more clear.
<suite name="XML -Job Suite" thread-count="2" parallel="tests" use-global-thread-pool="true">
What I expect:
Since I’ve set thread-count="2", I expect only 2 threads to run at any given time, even when executing tests that use a DataProvider to run in parallel.

What’s actually happening:
TestNG is initializing more than 2 threads when executing the above suite, which exceeds the limit I intended.

[krmahadevan]

@avishkargautam77 - When you enable parallel=tests I think this feature is getting disabled because <test> can still contain parallelism within it. So you should perhaps try with a suite file that looks like below:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.1.dtd">
<suite name="XML -Job Suite"
       verbose="2"
       thread-count="3"
       parallel="methods"
       use-global-thread-pool="true">
    <test name="combined">
        <classes>
            <class name="com.rationaleemotions.DataDrivenTestCase"/>
            <class name="com.rationaleemotions.NormalTestCase"/>
        </classes>

    </test>
</suite>

[krmahadevan]

Also please note that this feature is not immune to issues especially when there are data driven tests involved because of the current design of TestNG (TestNG tries to submit additional data driven test tasks from within a data driven test after it figures out that it is dealing with a data driven test, whereas TestNG should first expand a data driven test into n tests and then submit them and implementing this will take time since a lot of internal design would need to be changed).