Deprecate packaging "eclipse-test-plugin"

[kwin]

It seems that there is no longer a reason to use the packaging eclipse-test-plugin as this is under the hood using almost the same configuration as eclipse-plugin but some features are just not available there (compare with #1491). If there is not a reason I currently fail to see I would rather recommend to deprecate the packaging eclipse-test-plugin, i.e. emit some warning when this is used.
It seems that also the documentation at https://wiki.eclipse.org/Tycho/Packaging_Types#eclipse-test-plugin is heavily outdated as meanwhile both ITs and unit tests are supported with regular eclipse-plugin/
In the long term (i.e. with Tycho 4.0) all the code related to eclipse-test-plugin can finally be removed. WDYT?

This proposal is not about getting rid of dedicated eclipse-plugin modules just containing ITs, just about using the same packaging as the bundle code to be tested.

[laeubi]

The only think to consider when converting those is, that eclipse-plugin uses the **/*Test pattern for JUnit and the **/*IT pattern for Plugin Tests / Integration Tests. So if one want to convert 1:1 then this pattern needs to be configured differently.

About the wiki, I think we should migrate the Information to the site-docs or to the GH-Wiki I'm not aware of anyone maintaining that wiki anymore...

There is even

tycho/tycho-bundles/org.eclipse.tycho.embedder.shared/src/main/java/org/eclipse/tycho/PackagingType.java

Lines 22 to 27 in d8fc0dc

	public static final String TYPE_ECLIPSE_PLUGIN = ArtifactType.TYPE_ECLIPSE_PLUGIN;
	public static final String TYPE_ECLIPSE_TEST_PLUGIN = "eclipse-test-plugin";
	public static final String TYPE_ECLIPSE_FEATURE = ArtifactType.TYPE_ECLIPSE_FEATURE;
	public static final String TYPE_ECLIPSE_REPOSITORY = "eclipse-repository";
	public static final String TYPE_ECLIPSE_TARGET_DEFINITION = "eclipse-target-definition";
	public static final String TYPE_P2_IU = ArtifactType.TYPE_INSTALLABLE_UNIT;

maybe we can convert this to javadoc and then use this in the site-docs?

[kwin]

I agree that adding a documentation of the packagings to https://github.com/eclipse-tycho/tycho/tree/master/src/site/markdown and then link from https://tycho.eclipseprojects.io/doc/3.0.0/ would make sense (and afterwards deprecate the wiki page and just add a link towards the Maven site page)
Thanks for the hint regarding the different configuration of tycho-surefire-plugin depending on the packaging. Shouldn't this be documented in https://tycho.eclipseprojects.io/doc/3.0.0/tycho-surefire-plugin/integration-test-mojo.html, though?

[kwin]

I meanwhile found the difference here in the lifecycle bindings:

tycho/tycho-maven-plugin/src/main/resources/META-INF/plexus/components.xml

Line 50 in d8fc0dc

org.eclipse.tycho:tycho-surefire-plugin:${project.version}:integration-test

vs

tycho/tycho-maven-plugin/src/main/resources/META-INF/plexus/components.xml

Line 108 in d8fc0dc

org.eclipse.tycho:tycho-surefire-plugin:${project.version}:test

Very confusing. Also the name tycho-surefire-plugin is a bit unfortunate as this plugin is all about integration tests. IMHO tycho-failsafe-plugin would have been a much better name.

[mickaelistria]

It seems that there is no longer a reason to use the packaging eclipse-test-plugin

That doesn't seem true. eclipse-test-plugin starts by default the tycho-surefire-plugin against all *Test classes in the bundle. This is extremely convenient and widely for about all Eclipse projects I'm working on; and I believe it cannot be made to work with a plain eclipse-plugin as other convention for test detection and execution is used.
Therefore, we cannot just deprecate/remove it; we need first to have upstream projects adopting "better" approaches to not (silently) break them by not running their tests.
So we need to provide much assistance to downstream consumers here. I imagine an example block of configuraiton for tycho-surefire-plugin to place in the parent pom.xml can do the trick, can't it?

[laeubi]

You need to configure the following mojo:

https://tycho.eclipseprojects.io/doc/3.0.0/tycho-surefire-plugin/integration-test-mojo.html#includes

please note that the documentation about default value seems wrong because it is inherited from the abstract class, the default pattern is: "**/PluginTest*.class", "**/*IT.class"

One needs to configure the default life-cycle configuration, I think this is a bit strange, therefore it might be not a good idea to have this added to the default-life-cycle and I will create a issue to remove this with a note for 3.0.0 users how to enable it like it is done for the maven-failsafe-plugin here:

https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html

we can then even think about using the default test pattern here, so users have the following choices:

1. Only configure "plain" JUnit tests
2. Only configure Plugin Tests
3. Configure both, but use different patterns (of whatever choice seems suitable)

[kwin]

@laeubi eclipse-test-plugin uses another mojo for executing ITs (https://tycho.eclipseprojects.io/doc/3.0.0/tycho-surefire-plugin/test-mojo.html) which really confused me. I tried to clarify this in #1493

[laeubi]

Why is this confusing? These are different packaging types so they can use different mojos and have slightly different behavior:

1. test-mojo is the "old" traditional eclipse-plugin-test packaging mojo that behaves like a unit-test but actually executes at the integration-test phase
2. integration-test-mojo is meant to be used with eclipse-plugin testing and behaves like the maven fail-safe plugin as documented here: https://tycho.eclipseprojects.io/doc/3.0.0/tycho-surefire-plugin/integration-test-mojo.html

[kwin]

that behaves like a unit-test but actually executes at the integration-test phase

That part is really confusing for people familiar with surefire/failsafe. Also it is actually executed inside Eclipse so for me rather an IT than a Unit test.

[laeubi]

that behaves like a unit-test but actually executes at the integration-test phase

That part is really confusing for people familiar with surefire/failsafe.

Well thats why I introduced a more "maven like" workflow and always advocate to not use eclipse-plugin-test at all ;-)
But we can't change this without breaking users hard...

Also it is actually executed inside Eclipse so for me rather an IT than a Unit test.

It does not execute inside Eclipse, it executes inside an OSGi framework (what can run Eclipse if desired).

[mickaelistria]

Maven is kind of going against the "treat tests as 1st class citizen" rule usually; while Eclipse has embraced it from the beginning, making test artifacts real artifacts; which allows to install them reliably and to run them in more configurations without requiring more than Eclipse itself. This approach may not suit "libraries" but it's often much more convenient for Eclipse plugins.
"eclipse-test-plugin" do serve this approach. It's indeed not recommended, and even less enforced to use them, but I don't think this should be any deprecated until we have an equally good or better way to serve the pattern of test plugins in Tycho.
If no-one is willing to invest in developing (or maybe just documenting) such a good/better approach for existing test plugins without eclipse-test-plugin, then this packaging type is here to stay.

[laeubi]

Maven is kind of going against the "treat tests as 1st class citizen" rule usually;

Can you explain this? Maven even has dedicated scopes for tests what "Eclipse" has a long time not ..

while Eclipse has embraced it from the beginning, making test artifacts real artifacts;

Please don't confuse PDEs inability of handling test-only code (that should not be included in the final artifact) and thus forcing us using hacks like test-fragments with something making "real" artifacts, a fragment (and eclipse-plugin can be a fragment too) is nothing "different" if it contains test-code or not ...

which allows to install them reliably and to run them in more configurations without requiring more than Eclipse itself.

Looking at the complex and fragile setup of those test I hardly can agree with "without requiring more than Eclipse itself"

This approach may not suit "libraries" but it's often much more convenient for Eclipse plugins.

We obviously have different understandings about "convenient" here ;-)

"eclipse-test-plugin" do serve this approach.

As mentioned "eclipse-test-plugin" is simply a workaround for the workaround PDE introduced, one won't need a special packaging type for this kind of stuff, because a "test-fragment" is nothing special, it just contains an additional mojo-execution what could have been added by pom configuration already without the need of introduce a new packaging type.

It's indeed not recommended, and even less enforced to use them, but I don't think this should be any deprecated until we have an equally good or better way to serve the pattern of test plugins in Tycho. If no-one is willing to invest in developing (or maybe just documenting) such a good/better approach for existing test plugins without eclipse-test-plugin, then this packaging type is here to stay.

I think the most easy way would be if maven would support wildcards in its file activation policy (tycho might provide a custom ProfileActivator) then one could simply use:

<activation>
    <file>
        <exists>**/*.test</exists>
    </file>
</activation>

to do anything "special" for project named .test