Don't continue if the storybook throws an error

[tmeasday]

What

In order to avoid roadbumps we decided to ignore logged errors when evaluating your story list and getting a list of stories (sometimes people's code logs errors but it's not actually an issue).

We should re-evaluate this behaviour or add a flag to opt-into it.

Why

Often errors are logged when evaluating story files (if there's a JS bug in one of the files). What this means (e.g this build at this url) is that we end up getting a truncated list of stories (all the ones that evaluated before the error was thrown).

Chromatic does not make you accepted deleted stories (which is a tricky problem actually) and this leads to a bad time.

However, some users have stuff logged to their error console as a matter of course and can't necessarily remove it easily. So we still need a way to opt out.

How

1. Let's change the default behaviour to cancel the build if there's something logged to the error console.

2. Let's add a flag --allow-console-errors that reverts to the current behaviour

3. In the case of 1., let's say something like: "Your Storybook logged the following to the error console while loading. Errors loading your Storybook can lead to Chromatic getting your list of stories wrong and can cause a variety of problems. If you want to continue even with the errors, pass the --allow-console-errors flag to the chromatic command, however we do not recommend it."

[ndelangen]

This is another potential gain from tri-config and declarative stories definition, we could isolate the runtime of each story file being added.

[zol]

This just bit a customer, see here: https://app.intercom.io/a/apps/zj7sn9j1/inbox/inbox/all/conversations/25290402621

We should include this in our next package release and inform them.

[tmeasday]

@ndelangen this is not working as expected on next. I need to set LOG_LEVEL=verbose to see any messages at all about errors now. Try

LOG_LEVEL=verbose yarn chromatic -a <code> -u https://59c59bd0183bd100364e1d57-mkwmrssyja.chromatic.com/iframe.html

vs

yarn chromatic -a <code> -u https://59c59bd0183bd100364e1d57-mkwmrssyja.chromatic.com/iframe.html

[tmeasday]

Also --allow-console-errors doesn't appear to work.

[ndelangen]

Also --allow-console-errors doesn't appear to work.

Maybe I'm forgetful but I don't remember us having this argument. I'm not seeing any reference to it in the entire repo.

[ndelangen]

LOG_LEVEL=verbose yarn chromatic

should now set the log-level inside the storybook-build bin as well in #86

[tmeasday]

Maybe I'm forgetful but I don't remember us having this argument. I'm not seeing any reference to it in the entire repo.

It's at the top of this ticket -- we were going to add this flag to allow continuing the build even if the storybook outputs errors.

---

The current behaviour isn't right btw -- what I posted here is a bug.