Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide @Rule alternative to SpringJUnit4ClassRunner [SPR-7731] #12387

Closed
spring-projects-issues opened this issue Nov 10, 2010 · 41 comments
Closed
Assignees
Labels
has: votes-jira Issues migrated from JIRA with more than 10 votes at the time of import in: test Issues in the test module type: enhancement A general enhancement
Milestone

Comments

@spring-projects-issues
Copy link
Collaborator

spring-projects-issues commented Nov 10, 2010

Dave Syer opened SPR-7731 and commented

Overview

Sometimes (always?) it would be nice to have a @Rule implementation that did the job of the SpringJUnit4ClassRunner.

The main motivation for this is to be able to use other runners (e.g. @Parameterized, @Categories, Mockito, etc.).

There might even be a case for deprecating the runner in favor of a @Rule?


Further Resources


Attachments:

Issue Links:

25 votes, 30 watchers

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

There's a push for this from the JUnit end, where David Saff has suggested a move to @Rule for Spring test, response to my desire for more DRYness: https://github.com/KentBeck/junit/issues/100#issue/100/comment/779203.

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

Sam.

I'm looking into this and will hopefully get to prototype it on Thursday.

Let me know if you make a start. I suspect you may find some limitations to the @Rule/@ClassRule implementation, but this patch of mine may be what we need: nealeu/junit@da6c5e5

I'd be interested on feedback on whether this looks useful for the @Rule related JIRAs on Spring Test

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

Here's a proof of concept based on a reasonable patch for ComplexRule support in JUnit 4.9.

To use, you'll need the two commits from this branch: https://github.com/nealeu/junit/commits/complex-rules.

i.e. checkout the complex-rules branch from [email protected]:nealeu/junit.git

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

Sam & Dave

If you can find an elegant way to avoid the need for ComplexRule, then I'll buy you a beer!

Failing that, I suggest you checkout my JUnit git branch and use the attached file as a starting point, such that you can give some yay-nay feedback in time to get some agreement on what support is needed at the JUnit end.

As noted in the javadoc of my attachment, I think this would be most elegant if JUnit also allowed @Rule(class) on annotations (i.e. both instantiates the rule for us, and looks on annotations for meta-annotations).

I'll park it there for now.

@spring-projects-issues
Copy link
Collaborator Author

Dave Syer commented

I'll have a look at your junit code. Any chance of an executive summary of what it does and why it's important for Spring test context? Is it just because a Spring test context manager has a role at the class and the method level? I imagine you could hack around that with a static instance of a normal @Rule in your test case?

Note that you can submit pull requests to Spring via https://github.com/cbeams/spring-framework.

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

Pull requests... that's a bit 21st century isn't it :p Didn't know, so thanks. Forkin marvellous as some might say.

Static @Rule. Possibly, but that's not your only issue. The other is that you need the java.lang.reflect.Method reference, which is the second commit in my junit fork.

With just that enhancement to JUnit's Description class, your job becomes a lot easier, as the static @Rule can call Description.getMethod() and test for null to determine if it's the @ClassRule or method @Rule that's being evaluated.

Introducing ComplexRule does a couple of things (as you'll see at nealeu/junit@da6c5e5#diff-0):

  • it has 2 methods one which is called for @ClassRule statements and one which is called for @Rule
  • it adds Class<?> clazz to the applyClassRule() version and Object target to the applyMethodRule() version

Actually I didn't need the Class param, as that's avail as Description.getTestClass().

So, in summary, it looks like minimal requirements at JUnit end are:

  • add Object target to TestRule.apply()
  • add getMethod() to Description

I could rustle up an alternative JUnit patch later if that looks about right.

Dave, I've given you commit rights on my junit fork if you want to hack on what I've already done.

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

Take 2: Just pushed to branch enhanced-rule what I think is sufficient for above.

See https://github.com/nealeu/junit/commits/enhanced-rules

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

Tweaked to work with lower impact enhanced-rule proposal for JUnit instead of introducing ComplexRule.

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

There's a related JUnit issue, https://github.com/KentBeck/junit/issues/#issue/32, which looks for JUnit to manage the instantiation of Rules. I've commented on how I think this can go forwards and requested feedback before diving in with another proactive patch. For replacing features currently requiring Runners I'm suggesting that we're really looking at a broader interceptor model, which would be a good foundation for both Rules and Runner replacements.

@spring-projects-issues
Copy link
Collaborator Author

Dave Syer commented

I think an interceptor model would be good for JUnit rules - I commented to that effect on one of their issues (not sure if it was that one). If a rule had the ability to modify the chain it was involved in by returning a decorated version it would open up a lot of useful possibilities (e.g. switching off the @Before/@After which is a much requested feature). Good luck with getting that into JUnit.

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

Looks like modifying the chain could be on the cards:

From https://github.com/KentBeck/junit/issues/199:

The interfaces proposed might change, or be combined. We also might consider making BlockBuildRule an abstract class, so that future extension points could be added without breaking extenders.

This change would also introduce a base RuleChainer for BlockJUnit4ClassRunner, which would return Rules representing the basic built-in wrapper chain (which would need to be re-expressed as rules):

[ExpectingExceptions, PotentialTimeout, WithBefores, WithAfters, WithRules]

And for ParentRunner:

[WithBeforeClasses, WithAfterClasses, WithClassRules]

This would allow BlockBuildRules to insert, re-order, delete, etc. these built-in wrappers.

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Example of a limited, partial solution in the wild: SpringIntegration Rule from the Thucydides project.

@spring-projects-issues
Copy link
Collaborator Author

The Alchemist commented

Any updates on this?

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

The Alchemist, this issue is currently assigned to the 3.3 backlog and will be reassigned to a concrete release pending available resources and demand from the community.

Regards,

Sam

@spring-projects-issues
Copy link
Collaborator Author

spring-projects-issues commented Jan 26, 2013

Sam Brannen commented

Pull request submitted by Philippe Marschall in conjunction with #14850:

#222

@spring-projects-issues
Copy link
Collaborator Author

Neale Upstone commented

Just noticed that I've broken links here to my github account. I changed account name from nealeoc to nealeu - edit the links and they'll work, if you need to get at them.

@spring-projects-issues
Copy link
Collaborator Author

Vasiliy Gagin commented

I'm very interested in this being implemented. I've being discussing this with JUnit guys, asking them to provide better integration points, no luck yet.
They have some valid concerns. In general they like to support tests which are not based on Java Method. I've used Cucumber once, so i see the value. I guess that is why their current poster child TestRule gets Description as a parameter instead of FrameworkMathod as was in case of MethodRule.
Current TestContext has a reference to java.lang.reflect.Method. It seems unnecessary to restrict Spring decorated tests to only Java method based.
Would there be any will to change TestContext to remove reference to a Method and use something like array of Annotations or Description instead? It seems that Method is used for annotations only anyway.
This will allow for @Rule based implementation of TestRule?

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Would there be any will to change TestContext to remove reference to a Method and use something like array of Annotations or Description instead?

No, a change of this magnitude would not be possible since it would break backwards compatibility.

Furthermore, JUnit's Description appears to be too limited for our purposes. Similarly, relying on JUnit to provide Spring with the annotations we need (e.g., via an array of annotations) would never be acceptable: Spring has very different goals with regard to the annotation and meta-annotation programming model that we provide users of Spring. Thus Spring must always be in control of the annotation and meta-annotation look-up algorithms. See AnnotationUtils, AnnotatedElementUtils, and MetaAnnotationUtils for primary examples.

@spring-projects-issues
Copy link
Collaborator Author

Eric Rizzo commented

Pull request is more than 2 years old; what's the status? What's holding back getting this issue addressed?

@spring-projects-issues
Copy link
Collaborator Author

Elnur Abdurrakhimov commented

I want/need this too. It's a shame it hasn't been implemented yet. Each developer/team has to come up with their own hacks to solve the same common problem.

@spring-projects-issues
Copy link
Collaborator Author

spring-projects-issues commented May 11, 2015

Sam Brannen commented

FYI: development work for this issue is tracked in my #12387 feature branch.

Feel free to follow along or try it out for yourself.

Note, however, that this is a work in progress (based off the proof of concept from Philippe Marschall).

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Update

The work on this issue is now (hopefully) feature complete.

The following is an example of how the new SpringClassRule and SpringMethodRule can be used in an integration test.

@ContextConfiguration
public class BaseAppCtxRuleTests {

	@Configuration
	static class Config {

		@Bean
		public String foo() {
			return "foo";
		}
	}


	@ClassRule
	public static final SpringClassRule SPRING_CLASS_RULE = new SpringClassRule();

	@Rule
	public final SpringMethodRule springMethodRule = new SpringMethodRule(this);

	@Autowired
	String foo;


	@Test
	public void test() {
		assertEquals("foo", foo);
	}

}

I would appreciate feedback within the next few days since we plan to complete development for Spring Framework 4.2 RC1 next week.

So, thanks in advance for any input you can provide!

Sam

@spring-projects-issues
Copy link
Collaborator Author

The Alchemist commented

@Sam Brannen: Thanks, Sam! Is there a SNAPSHOT release on a Maven repo somewhere we can try?

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Hi The Alchemist,

There's not yet a snapshot in a public Maven repository, but I'm attaching a locally built spring-test-4.2.0.BUILD-SNAPSHOT.jar that you can download and use with recent snapshot jars from the rest of core Spring Framework.

Regards,

Sam

p.s. I've just pushed my feature branch to the official spring-framework GitHub repository, and that's building now: https://build.spring.io/browse/SPR-PUB3-1

So hopefully (if all works well), there should be a publicly available snapshot soon.

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Hi The Alchemist,

You should now be able to use the following to try out the snapshot.

  • Repository: http://repo.spring.io/libs-snapshot-local/
  • Group ID: org.springframework
  • Artifact ID: spring-test
  • Version: 4.2.0.SPR-7731-SNAPSHOT

Let me know how it goes!

@spring-projects-issues
Copy link
Collaborator Author

Dave Syer commented

Looks awesome and seems to work with Spring Boot. Do we have any guice with the JUnit devs (you must know how to communicate with them by now) over the 2 rules thing? Can you explain a bit what the problem is?

I don't like the names particularly ("ClassRule" and "Rule" are duplicated). Maybe they could be more descriptive, like other @Rules (e.g. ExpectedException)? ApplicationContextCache for the class rule and AutowiredHandler for the method rule, perhaps (guessing their actual function)?

I also wondered if the ClassRule could live in an interface so it doesn't have to be in every test class (I tried it and it didn't work but I don't know if there's something about the implementation of the MethodRule that might need to change to support that).

@spring-projects-issues
Copy link
Collaborator Author

Mark Michaelis commented

At least with JUnit 4.12 it is possible to annotate a static field with both, @Rule and @ClassRule like in this example:

@ClassRule
@Rule
public static TestRule temporaryFolder = new TemporaryFolder();

For details see:

Unfortunately without changing JUnit we still require two rules as we need a non-static field to refer to this. The only reason for this: Inside a TestRule.apply(Statement,Description) you have no access to this (Description has no handle to it, although the runner creating the description would be able to provide this information).

See:

Derived from the discussion mentioned above the following might have worked:

@ClassRule
@Rule
public static CombinedTestRuleMethodRule THE_RULE = new CombinedTestRuleMethodRule();

where CombinedTestRuleMethodRule implements both, TestRule and MethodRule (MethodRules know the object they run on).

Unfortunately org.junit.internal.runners.rules.RuleMemberValidator.MemberMustBeNonStaticOrAlsoClassRule clearly states:

// We disallow:
//  - static MethodRule members
//  - static @Rule annotated members
//    - UNLESS they're also @ClassRule annotated
// Note that MethodRule cannot be annotated with @ClassRule

Thus to get rid of the two rules above we would require issue #351 to be fixed -- possibly by just removing this validation for MethodRules.

I will add a comment to the issue with reference to this issue.

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Hi Dave,

Looks awesome and seems to work with Spring Boot.

Great. Thanks for trying it out!

I'll now respond to your points in separate comments.

Cheers,

Sam

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Can you explain a bit what the problem is?

Yes, I can do that. ;)

The Spring TestContext Framework cannot work properly unless all of the lifecycle callbacks in TestContextManager are supported.

Core API of TestContextManager:

  • beforeTestClass()
  • prepareTestInstance(Object testInstance)
  • beforeTestMethod(Object testInstance, Method testMethod)
  • afterTestMethod(Object testInstance, Method testMethod, Throwable exception)
  • afterTestClass()

Spring must therefore be able to invoke beforeTestClass() and afterTestClass() as class-level callbacks and all other methods as instance/method-level callbacks.

JUnit rules can implement either TestRule or MethodRule but not both.

There are several restrictions to using JUnit rules (see JUnit's RuleMemberValidator for details):

  1. If a TestRule is static and annotated with @ClassRule, it will be used to handle class-level callbacks.
  2. If a TestRule is static and annotated with @ClassRule and @Rule, it will be used to handle both class-level and method-level callbacks, but... it will not be able to access the test instance.
  3. If a TestRule is not static and annotated only with @Rule, it will be used to handle method-level callbacks.
  4. If a MethodRule is not static and annotated with @Rule, it will be used to handle method-level callbacks.
  5. A MethodRule cannot be annotated with @ClassRule.
  6. If a rule implements both TestRule and MethodRule (which is possible from a Java perspective), JUnit will only consider it to be a TestRule.

In summary, the above restrictions simply make it impossible for Spring to provide a single rule. :(

The state of affairs may potentially (hopefully!) change with JUnit 5, but that likely will not happen until 2016.

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

I don't like the names particularly ("ClassRule" and "Rule" are duplicated). Maybe they could be more descriptive, like other @Rules (e.g. ExpectedException)? ApplicationContextCache for the class rule and AutowiredHandler for the method rule, perhaps (guessing their actual function)?

SpringClassRule performs the following:

  • instantiates and owns the TestContextManager
  • supports @IfProfileValue at the class-level
  • invokes testContextManager.beforeTestClass()
  • invokes testContextManager.afterTestClass()

SpringMethodRule performs the following:

  • supports @IfProfileValue at the method-level
  • invokes testContextManager.prepareTestInstance()
  • invokes testContextManager.beforeTestMethod()
  • invokes testContextManager.afterTestMethod()
  • supports Spring's @Repeat
  • supports Spring's @Timed

As I mentioned earlier, the responsibilities are split between the class-level and instance/method-level callbacks.

Purely descriptive alternatives could include something like SpringClassLevelCallbacks and SpringMethodLevelCallbacks, but I don't think that's any better than the status quo. The problem is that these rules are not like other rules which have a single purpose. These Spring rules potentially serve many purposes depending on which TestExecutionListeners are registered.

So, if you or anyone else can come up with better names, I'd be glad to hear them! :)

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

I also wondered if the ClassRule could live in an interface so it doesn't have to be in every test class (I tried it and it didn't work but I don't know if there's something about the implementation of the MethodRule that might need to change to support that).

It's unfortunately not possible to declare it on an interface. JUnit's infrastructure for finding annotations is very limited in comparison to Spring's AnnotationUtils and AnnotatedElementUtils. As a consequence, rules annotated with @ClassRule must be public, static, declared in a public class, and of type TestRule. Again, consult RuleMemberValidator for details.

But you shouldn't have any problem declaring the SpringClassRule in an abstract or concrete base class. The BaseAppCtxRuleTests and Subclass1AppCtxRuleTests in my commit test exactly that.

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Mark Michaelis, thank you for the thorough analysis and numerous links!

And... it appears we were composing similar answers simultaneously. ;)

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Hi everybody,

I'm going to hold off on pushing my branch to master for a few more days.

That will give more of you a chance to try out the snapshot (see comments above) and provide feedback.

Cheers,

Sam

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

The Alchemist, did you have a chance to try out the new rule support yet?

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

FYI: I have committed some changes with bug fixes and complete Javadoc. These are included in latest 4.2.0.SPR-7731-SNAPSHOT that was just published on http://repo.spring.io/libs-snapshot-local/.

Feel free to give it a try! :)

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Implemented as described in GitHub commit d1b1c4f:

Introduce JUnit Rule alternative to SpringJUnit4ClassRunner

Since Spring Framework 2.5, support for integrating the Spring TestContext Framework (TCF) into JUnit 4 based tests has been provided via the SpringJUnit4ClassRunner, but this approach precludes the ability for tests to be run with alternative runners like JUnit's Parameterized or third-party runners such as the MockitoJUnitRunner.

This commit remedies this situation by introducing @ClassRule and @Rule based alternatives to the SpringJUnit4ClassRunner. These rules are independent of any Runner and can therefore be combined with alternative runners.

Due to the limitations of JUnit's implementation of rules, as of JUnit 4.12 it is currently impossible to create a single rule that can be applied both at the class level and at the method level (with access to the test instance). Consequently, this commit introduces the following two rules that must be used together.

  • SpringClassRule: a JUnit TestRule that provides the class-level functionality of the TCF to JUnit-based tests
  • SpringMethodRule: a JUnit MethodRule that provides the instance-level and method-level functionality of the TCF to JUnit-based tests

In addition, this commit also introduces the following new JUnit Statements for use with rules:

  • RunPrepareTestInstanceCallbacks
  • ProfileValueChecker

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

FYI: minor changes in latest commit:

Introduce TestContextManager cache in SpringClassRule

In order to simplify configuration of the SpringMethodRule and to ensure
that the correct TestContextManager is always retrieved for the
currently executing test class, this commit introduces a static
TestContextManager cache in SpringClassRule.

In addition, since it is not foreseen that SpringClassRule and
SpringMethodRule should be able to be subclassed, their internal methods
are now private instead of protected.

Consequently, the required configuration now looks like this:

@ClassRule
public static final SpringClassRule SPRING_CLASS_RULE = new SpringClassRule();

@Rule
public final SpringMethodRule springMethodRule = new SpringMethodRule();

The only change from the user's perspective is that this no longer has to be supplied to the SpringMethodRule constructor.

Regards,

Sam

@spring-projects-issues
Copy link
Collaborator Author

Thomas Darimont commented

Just stumbled upon this issue, this is great stuff Sam!

I just wondered whether it would be possible to extend this PoC to add support for specifying test case specific configuration classes which comes in handy If you want to be able to test different configurations with just minor differences.
I gave this a quick spin a while ago: https://gist.github.com/thomasdarimont/9bf439fa3c48f110b49e

One could specify an additional config class to use via (a new) @Use annotation on method level:

@ContextConfiguration(loader = AnnotationConfigContextLoader.class)
public class SpringJunit4RunnerRuleTests {
 
	@ClassRule public static SpringContext springContext = new SpringContext();
 
	@Autowired Foo foo;
 
	@Test
	public void test1_should_use_ConcreteFoo() {
 
		Assert.assertNotNull(foo);
		Assert.assertTrue(foo instanceof ConcreteFoo);
	}
 
	@Test
	@Use(SpecialConfig.class)
	public void test2_should_use_SpecialFoo() {
 
		Assert.assertNotNull(foo);
		Assert.assertTrue(foo instanceof SpecialFoo);
	}
 
	@Test
	public void test3_should_use_ConcreteFoo() {
 
		Assert.assertNotNull(foo);
		Assert.assertTrue(foo instanceof ConcreteFoo);
	}
 
	static interface Foo {}
 
	static class ConcreteFoo implements Foo {};
 
	static class SpecialFoo implements Foo {}
 
	@Configuration
	static class Config {
 
		@Bean
		public Foo foo() {
			return new ConcreteFoo();
		}
	}
 
	// Not visible for the test directly
	public static class Nested {
 
		@Configuration
		public static class SpecialConfig extends Config {
 
			@Bean
			public Foo foo() {
				return new SpecialFoo();
			}
		}
	}
}

@spring-projects-issues
Copy link
Collaborator Author

Dave Syer commented

I like that (I think I suggested using @ContextConfiguration at method level somewhere else). It would be really useful to be able to override the odd bean for a single method and not have to @DirtiesContext for the main, parent context.

@spring-projects-issues
Copy link
Collaborator Author

Sam Brannen commented

Thanks, Thomas!

Glad you like the Rule support. :)

Regarding your proposal, we already have something similar in SPR-12031. So could you please post your comments to that JIRA issue?

Cheers,

Sam

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
has: votes-jira Issues migrated from JIRA with more than 10 votes at the time of import in: test Issues in the test module type: enhancement A general enhancement
Projects
None yet
Development

No branches or pull requests

2 participants