Mock time in tests #1986
Comments
|
A collection of resources I've found as part of my research on the topic:
|
|
To help asses whether mocking time solved some related issues, here's a list of tests known to fail because of our current usage of time:
|
|
I'd say that, In general, if we can mock time nicely, all of these tests and others could be written in both a much cleaner and a more deterministic manner. Right now we've made a bunch of compromises with their quality to get something working, but if time is deterministic, we won't care about CI performance differences all that much and should be able to write much better tests as a result. |
|
#537 was closed, but I decided to see which our our tests are slow with https://github.com/gotestyourself/gotestsum#finding-and-skipping-slow-tests And we have a lot 😞 Skimming that list, I think that once we have capabilities to mock time in tests, for a lot of thse we can both refactor them to be more resilient (less "flaky"), but also reduce the time they actually take to execute. Right now they take a bunch of time because we are running actual executors or load tests and we leave generous margins to get our expected results, since we don't know exactly how long certain actions will take. |
|
In the process of working with @olegbespalov on #1719 we had the opportunity to experiment and learn a lot about time mocking. After discussing our results and findings with @na--, we've reached the agreement that we should not pursue our effort towards mocking time. Instead, we should invest some time into refactoring our codebase components to be less dependent on time. Whenever making an assertion about some code execution based on time, we are making assumptions on scheduling (both from the OS and Go perspective), and on performance. Asserting that a given operation takes a given amount of time with a set acceptance delta, asserts whether the program, (and the OS and hardware running it) were able to perform it in said time. A problem we experienced in that context is that while those kinds of tests would pass on a beefy setup, they were much more likely to behave in a flaky way on shared/virtualized hardware with minimal, unreliable, specs, such as CI's. // lorem_test.go
func TestLorem(t *testing.T) {
...
go func() {
// bad practice, since we have no control/guaranty from the go scheduler when this go-routine will be executed
now := time.Now()
assert.WithinDuration(t,
startTime.Add(expectedTime),
time.Now(),
time.Millisecond*12,
"%d expectedTime %s", current, expectedTime,
)
}
}Our understanding of most tests involving time assertions was they really wanted to verify some behavior over time, rather than timings and durations themselves. Given an executor's configuration, they would assert that its behavior is corresponding to our expectation; based on when certain actions happen. The conclusion of our research and experimentation were: while this expectation and behavior does need to be asserted to validate k6 behaves appropriately, we believe another strategy should be used to achieve it. How to do that without relying on duration-based assertions is one of the ongoing objectives we set out for ourselves. While experimenting with time mocking, specifically with the clock library, we've also found out that time mocking can lead to somewhat unpredictable behaviors. When addressable, we judged these workarounds would lead to introducing logic and code in the tests, which would effectively add more complexity than a plain redesign. One of the main issues we've encountered first hand is that, in the context of using a mocked clock, while Finally, and linked to the previous finding, scheduling is neither predictable, nor controllable. Scanning through clock's documentation, and the mocking time in Go article, one will notice they recommend using Nonetheless, we still intend to make flaky tests reliable again. To do so, we have decided on a few axes moving forward. We want to redesign components which led to time-based assertions in the first place. Moving forward, we need to keep asserting performance still. It allows us having certainties about the behavior of the program. It also allows us to provide our users guarantees. Thus, we will drive an effort towards establishing such a performance testing strategy. One that's more reliable than what we have now. |
Our tests, especially the ones for executors, are somewhat flaky and incomplete. Because we rely on real timers, we can't test them as well as we should... 😞
The solution is to somehow mock time. For example, instead of using functions from the
timepackage directly, we can instead rely on an interface. For real-world code, that interface will be implemented by the sametimepackage, so nothing would change, but for test we'd be able to manually control the time "flow"...This was previously mentioned in #1357 (comment) and #1386 (comment), but it probably deserves its own issue, so I created this.
Specific issues that this will likely make easier: #1715, #1719, #1974
The text was updated successfully, but these errors were encountered: