[question] Getting passing tests in spec addon but failing in browser console. Does storybook render twice or catch story errors?

[swyxio]

Describe the bug

i have a weird bug with react testing library and the storybook specs addon (its no longer maintained but we were dumb enough to use it) and i have a weird symptom of passing tests in the addon but failing tests inside the browser console

repro repo: https://github.com/sw-yx/repro-react-storybook-specs-addon-bug
live demo of bug: https://repro-react-storybook-specs-addon-bug.netlify.com/

i spent way too much time trying to replicate this problem in a standalone repo and i think i have a handle on it, but i cant get this error out of the way:

does storybook somehow render twice? or does it catch async story errors somehow? trying to puzzle out how to even track down this bug - it could be storybook, it could be the specs addon, it could be react-testing-library, i have no idea :(

[shilman]

cc @ndelangen

[swyxio]

offtopic - also wondering if you'd like to take on maintenance of storybook-addon-specs. its pretty much the only thing you recommend for interaction testing, and it is unmaintained. i probably dont need to sell you guys on how its probably a good idea to have a good interaction testing story together with storybook. the question is who takes on the maintenance 😂 (i can't, im only on this project another 1 month). fwiw we have about 600+ tests written inside storybook tests and no idea how to move them to CSF if we ever end up moving to that.

[shilman]

@sw-yx i'll be trying to build out a bunch of maintainers this year, and will keep addon-specs in mind. it's come up a couple times in support, but otherwise doesn't seem to be particularly widely-used.

agree about interaction testing support, but i'm guessing the documentation needs to be updated more than anything. with CSF the sky's the limit on interop with your favorite testing tools. that said, i don't have a recommendation off the top of my head. 😅

[swyxio]

yeah. obviously no pressure, you dont owe us anything. You could even wipe out that page and just ask for someone to step up with a new take on it. a simple blessed way to colocate stories and tests would be nice and i think CSF is perfect for it.

anyway, back to the main question of this issue - how come i have a failing test in the browser console but not in the addon? just looking for a hint as to how storybook might work internally, i saw some mentions of double rendering or maybe there's some sort of catch-and-retry strategy going on? i made that repro so hopefully its easy to take a quick look for someone with knowledge of internals.

[ndelangen]

@sw-yx I'm looking at repro-react-storybook-specs-addon-bug.netlify.com but I'm seeing this:

I'm not seeing a specs panel at all. Nor do I see any errors in the console.
https://repro-react-storybook-specs-addon-bug.netlify.com/?path=/story/test--button

I'll add looking into this on my todo-list, thanks for the reproduction repo.

[swyxio]

wow, you're right - it disappeared in production?! i think i did have it while in dev. even weirder :(

Fwiw, I've put this work on hold now, tho it has been a concern for us upgrading to 5.3 since it uses the deprecated onStory api. someday we'll have to separate our stories from our specs. Unless you come across other big users of this addon, I dont want to put unnecessary maintainence burden on you guys.

[ndelangen]

@sw-yx it's on the wish-list to revamp the connection between testing + storybook.

Both on the e2e & unit test front, I have a lot of amazing things in mind. but not really the time to work on it, right now.

[swyxio]

yeah. i'm gonna go ahead and close this for now. but hey, when you start it back up, lmk!