Needs more tests

[xd009642]

So tarpaulin is in need of some more tests. Currently I'm relying on a small number of tests and doing informal runs of projects to test things but as more features get added on this becomes harder to do.

There are a few areas I'd like to tackle, if anyone chooses to help I'm willing to help mentor you through any issues and generally help out.

Integration tests

• Test tarpaulin on run throughs of more of the test projects
• Assert against lines that should be hit
• Try with more variations of config options
• Create examples that represent coverage requirements derived from Inaccurate coverage estimates #63

Unit tests
This one is a bit trickier based on the reliance of ptrace or processing a binary in tarpaulin.

• More unit tests for the source_analysis module would be useful.
• Look to test report generation - particularly cobertura reports
• Look for more testable areas that don't rely on external files or ptrace

Test projects for all Rust expression types
List taken from syn

• Box
• Array
• Call - adequately covered by other tests
• MethodCall
• Tuple
• Binary
• Unary
• Lit - covered by any constants in other tests
• Cast
• Type
• Let
• If
• While
• ForLoop
• Loop
• Match
• Closure
• Unsafe
• Block
• Assign - <expr> = <expr> form so way more to test than I've done but covered by a lot of others
• AssignOp
• Field
• Index
• Range
• Path
• Reference
• Break
• Continue
• Return
• Macro
• Struct
• Repeat
• Paren
• Group
• Try
• Async
• TryBlock
• Yield

Bonus challenge
Related to the issue with unit tests. This is potentially a challenging issue and might be ultimately more trouble than it's worth but is at least worth an investigation. It might even result in the genesis of a design that enables future non-Linux based development as I imagine it will result in a coverage tracing trait of some form.

• Look into stubbing solutions to test tarpaulin with a simulated ptrace

[xd009642]

So for those that are interested I've pushed a new remote branch issue-54 to demonstrate my thoughts on testing for regression coverage in covering rust expressions. This has a project that should get 100% coverage and tests it gets this. If it doesn't it's because it's trying to cover additional instructions that aren't hit or the wrong instructions and missing them either showing that source_analysis needs updating or test_loader needs updating

[xd009642]

Examples of how to write these tests are in tests/mod.rs and tests/data projects if anyone fancies helping out 😄

[camchenry]

Is this list up-to-date? I'd like to have a try at adding another test, specifically the loop functionality, since that seems like an important one to test.

[xd009642]

It should be look in tests/data they tend to be named after what they test

[camchenry]

Okay! I've been looking around in there already. How can I help with adding coverage for another keyword or anything?

…

On Fri, Oct 4, 2019, 4:54 PM xd009642 ***@***.***> wrote: It should be look in tests/data they tend to be named after what they test — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#54?email_source=notifications&email_token=AALRVQDD6YMUIPTPH5T37UDQM6UO5A5CNFSM4D6YXHN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAM3PDY#issuecomment-538556303>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALRVQHREPEXSIBDIKP342DQM6UO5ANCNFSM4D6YXHNQ> .

[xd009642]

pick a language construct from the list that hasn't been done (I think the names were taken from expression types in syn so you can use that for reference). Then write a small project that uses it in a number of different ways that should get 100% coverage.

Once that's done you can add it to the projects that get run by adding a named test for it in tests/mod.rs

[xd009642]

I'm just going to close this issue, we've got a relatively good language coverage via the different users and current tests (34 small test projects) and most coverage issues appear in larger crates.

Instead as a strategy I'm moving to a more directed projects that recreate issues users observe.