Skip to content

Make lowering failure tests less fragile #448

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 12, 2020

Conversation

nathanwhit
Copy link
Member

The current lowering failure tests check that the error message resulting from the lowering attempt matches a given string exactly. This is fine in most cases, but tests which expect parsing errors are incredibly fragile in that the contents of their error messages change whenever the lexer or parser is modified.

This PR makes a small change so that lowering failure tests which produce a parsing error only have to provide the expected error kind (such as "UnrecognizedToken", etc.) instead of a full error message, which makes the tests less fragile to parser/lexer changes.

It's not exactly a beautiful fix, but it at least alleviates the immediate annoyance of tests frequently breaking.

@nathanwhit nathanwhit force-pushed the lowering-test-cleanup branch from f5757ef to 436e2da Compare May 10, 2020 16:25
@AzureMarker
Copy link
Member

The test macro allows you to only give the prefix of the output you expect (ex. "Unique") by using assert_same:

fn assert_same(result: &str, expected: &str) {

We could do the same thing for lowering tests, and then we wouldn't need the special parse_error syntax and it would work for non-parse errors (assuming they exist). It should also be backwards-compatible with the existing tests.

lowering_error! {
    // ...
    error_msg {
        "parse error: UnrecognizedToken"
    }
}

@jackh726
Copy link
Member

I agree with @Mcat12 here: we should just check whether the message starts with the expected text. That assert_same function can just be moved into test_util

@nathanwhit nathanwhit force-pushed the lowering-test-cleanup branch from 436e2da to 881af29 Compare May 10, 2020 21:35
@nathanwhit
Copy link
Member Author

Sounds reasonable to me. I've updated the PR to make that change.

@jackh726 jackh726 merged commit 3e9c250 into rust-lang:master May 12, 2020
@nathanwhit nathanwhit deleted the lowering-test-cleanup branch May 25, 2020 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants