Add test.result(testName) to get results

[wmertens]

🚀 Feature Proposal

This is a more concrete version of #5823, with a working prototype. I'm opening it to increase visibility; if this stil counts as a duplicate I apologize, please close but please consider this option.

Motivation

It is hard to write tests for an expensive operation that requires many steps. It would be great if individual steps could be written as tests and a dependency on other tests expressed.

Example

I'd like to propose this API, getting a return value from test and friends which can be called to get the test result; as a side effect it can await Promises:

// this test is synchronous
test('gets config', () => {
  const config = readConfig()
  expect(config).toHaveProperty('enabled', true)
  return config
})

// this test is asynchronous
test('can create', async () => {
  const config = test.result('gets config')  // sync
  const thing = await makeThing()
  expect(thing).toBeTruthy()
  return thing
})

test('can doFoo', async () => {
  const thing = await test.result('can create') // async
  await thing.doFoo()
  expect(thing.didFoo).toBeTruthy()
})

test('can doBar', async () => {
  const thing = await test.result('can create')
  await thing.doBar()
  expect(thing.didBar).toBeTruthy()
})

Now, if you only run the 'can doBar' test, it will actually run 'gets config' and then 'can create' first, and error out if those fail.

Pitch

It's a very intuitive interface fitting right in with the rest of the API. It makes it easy to express dependencies, and if you don't use it, there's no change.

Prototype

I put this in a file and import it at the beginning of a test file. It works great, but it won't detect dependency loops, and if you only run a single test, then the tests it runs for dependencies won't count as the tests that ran. To fix those, it would need integration into Jest.

// https://github.com/facebook/jest/issues/5823
class TestError extends Error {
	constructor(title, error) {
		super(title)
		if (error instanceof TestError) {
			// A dependency threw this
			this.message = error.deep
				? error.message
				: `Dependency "${error.title}" failed`
			this.stack = null
			this.deep = true
		} else {
			this.message = `test "${title}" failed: ${error.message}`
			this.stack = error.stack
			this.title = title
		}
	}
}
const origTest = global.test
const origDescribe = global.describe
const runners = {}
let prefix = ''
global.describe = (name, fn) => {
	const prev = prefix
	prefix = `${prefix}${name} | `
	origDescribe(name, fn)
	prefix = prev
}

global.test = (title, fn) => {
	const name = `${prefix}${title}`
	let alreadyRan = false
	let result = undefined
	let fnError = undefined
	const handleError = (error, saveError) => {
		const newError = new TestError(name, error)
		if (saveError) fnError = newError
		throw newError
	}

	const runFnOnce = () => {
		if (alreadyRan) {
			if (fnError) throw fnError
			else return result
		}
		alreadyRan = true
		try {
			result = fn()
		} catch (err) {
			// synchronous error
			handleError(err, true)
		}
		// async error
		if (result?.catch) result = result.catch(err => handleError(err, false))
		return result
	}
	runners[name] = runFnOnce
	origTest(title, runFnOnce)
	return runFnOnce
}
// add .each etc
Object.assign(test, origTest)
test.result = title => {
	if (runners[title]) return runners[title]()
	else {
		const msg = `test.result(title): unknown test "${title}". Known:\n${Object.keys(
			runners
		).join('\n')}`
		throw new Error(msg)
	}
}

[wmertens]

@SimenB if I were to make a PR that implements this, would it have a chance of being merged?

[SimenB]

Hi!

I think marking a dependency inside of another test would mean we'd still have to run it, then bail with a special error. Some API where the dependency is marked as part of the test declaration is needed, I think...

Also, how would this work with concurrent tests? Still experimental, but still

[wmertens]

Hi @SimenB :)

the dependency is handled the way you describe and it should work correctly with concurrent tests. Here's my latest version:

// https://github.com/facebook/jest/issues/5823
class TestError extends Error {
	constructor(title, error) {
		super(title)
		if (error instanceof TestError) {
			// A dependency threw this
			this.message = error.deep
				? error.message
				: `Dependency "${error.title}" failed`
			this.stack = ' ' // truthy empty error
			this.deep = true
		} else {
			this.message = `test "${title}" failed: ${error.message}`
			this.stack = error.stack
			this.title = title
		}
	}
}
const origTest = global.test
const origDescribe = global.describe
const runners = {}
let prefix = ''
global.describe = (name, fn) => {
	const prev = prefix
	// Note, using async local storage, this could be used automatically
	prefix = `${prefix}${name} | `
	origDescribe(name, fn)
	prefix = prev
}

global.test = (title, fn) => {
	const name = `${prefix}${title}`
	let alreadyRan = false
	let result = undefined
	let fnError = undefined
	const handleError = (error, saveError) => {
		const newError = new TestError(name, error)
		if (saveError) fnError = newError
		throw newError
	}

	const runFnOnce = () => {
		if (alreadyRan) {
			if (fnError) throw fnError
			else return result
		}
		alreadyRan = true
		try {
			result = fn()
		} catch (err) {
			// synchronous error
			handleError(err, true)
		}
		// async error
		if (result?.catch) result = result.catch(err => handleError(err, false))
		return result
	}
	runners[name] = runFnOnce
	origTest(title, runFnOnce)
	return runFnOnce
}
// add .each etc
Object.assign(test, origTest)
test.result = title => {
	if (runners[title]) return runners[title]()
	else {
		const msg = `test.result(title): unknown test "${title}". Known:\n${Object.keys(
			runners
		).join('\n')}`
		throw new Error(msg)
	}
}

and a types.d.ts file to match:

declare namespace jest {
	interface It {
		result: (testName: string) => any
	}
}

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[wmertens]

Still interested in this

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[wmertens]

@SimenB any chance of this?

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.