Infection mutates boolean values in annotations

[delolmo]

Hi there, everyone.

Within my test suite, I am running infection/infection in a project for test mutation. Although it is clear that some mutants sometimes are harmless and are not covered by tests, there is an issue that I would like to raise to see what solutions you people have come up with.

When annotating entities through PHP annotations, infection mutates true/false values - regular behavior, no surprise here. For example,

#[ORM\Column(type: 'boolean',  options: ['default' => true))]
protected bool $autorenewable = true,

Gets mutated to:

#[ORM\Column(type: 'boolean',  options: ['default' => false]))]
protected bool $flag = true,

I do check for the default value of $flag in the phpunit tests, but not for the ORM mapping annotation. As a result, I get an uncovered mutant here. This particular example can be solved by deleting the default options and relying on the PHP default option for the object itself - not the best solution however imo.

A second example would be unique constraints, such as:

#[ORM\Column(type: 'string', unique: true)]
protected string $email,

Unique constraints, once again, would generate an uncovered mutant unless you regenerate the database every time you run your phpunit tests - which to me, seems overkill.

However, in this case, the mutant can get easily covered by adding the UniqueConstraint annotation instead:

#[ORM\UniqueConstraint(columns: ['email'])]

With nullable values, however, it becomes harder because there is no such #[ORM\NullableColumns] equivalent annotation in doctrine/orm:

#[ORM\Column(type: 'text', nullable: true)]
protected string|null $picture = null,

Again, this generates an uncovered mutant when infection mutates the code to:

#[ORM\Column(type: 'text', nullable: false)]
protected string|null $avatar = null,

Considering that all these annotations could very well be written separately in an XML file (or using any other acceptable mapping types), it doesn't seem logical that infection should show these mutations as "uncovered".

My question is:

How do you people solve this issue? Any ideas on how to prevent these mutants from being generated?

[SenseException]

It's a pretty interesting case for catching mutants when it comes to ORM. They indicate the configuration that creates the schema of the database, tables and columns instead of being the usual runtime code. The tests that will be mutated are usually testing the features and functionality of the application's code and because the attributes in question are connected to the created database, you definitely won't come far with unit tests.

If you have tests that are using a database, you would usually test your implementation doing write and read access to the database. The thing with an application is that you wouldn't necessarily let data be written into a database without validation, that could prevent you from reaching the INSERT or UPDATE parts with now considered "faulty" data that doesn't fit to the mutated db table schema. If some validators exist that are using the current configuration of the ORM attributes, the mutants would break the test and catch a mutant before it reaches the db. In other cases you might have to create tests that make writing into the database fail, because the mutation created e.g. a nullable: false but the data that's tried to be written in the original test contains a null value. Or tests that try to write the same data twice to trigger a unique constraint error. That involves the recreation of db schemas as you already mentioned.

I also saw some tests that were just checking a generated create table query, but this would also have the side effect that every third party change, that influences the SQL query creation, can break your test too while your application would still work fine with it. Such a test would also just exist for the mutants and not for the coverage of the application's feature tests. I don't see many advantages in this.

Although it is clear that some mutants sometimes are harmless and are not covered by tests

That's also something to consider. If any tests of your app against the db don't break on a mutation, then how important are the configurations like default values or nullable? Can they even fail when you have already a default value in PHP? Should they be ignored by infections's configuration or maybe keep them and expect a min-msi lower than 100%?

However, in this case, the mutant can get easily covered by adding the UniqueConstraint annotation instead

It actually doesn't create a mutant, because there's probably no mutator that can work with UniqueConstraint arguments. It can be a way to handle it though if it's not important enough to be mutation-tested.

unless you regenerate the database every time you run your phpunit tests - which to me, seems overkill.

In one simple data transformation and CRUD project I used ORM entities only without custom SQL or DQL. I configured an in-memory SQLite database for the test environment that improved the runtime speed by far compared to MySQL, but it's still possible that there might be some behavior differences that can break code in production that worked well in with tests, so it's nothing I can suggest for every situation.
There are some other databases that support in-memory like e.g. MySQL does, but I don't have any experience about how and if it can be used in Doctrine ORM with all of them. AFAIK does PostgreSQL have no in memory solution.