command + backend/local: -refresh-only and drift detection

[apparentlymart]

This is a light revamp of our plan output to make use of Terraform core's new ability to report both the previous run state and the refreshed state, allowing us to explicitly report changes made outside of Terraform.

Because whether a plan has "changes" or not is no longer such a straightforward matter, this now merges views.Operation.Plan with views.Operation.PlanNoChanges to produce a single function that knows how to report all of the various permutations. This was also an opportunity to fill some holes in our previous logic which caused it to produce some confusing messages, including a new tailored message for when terraform destroy detects that nothing needs to be destroyed.

This also allows users to request the refresh-only planning mode using a new -refresh-only command line option. In that case, Terraform only performs drift detection, and so applying a refresh-only plan only involves writing a new state snapshot, without changing any real infrastructure objects. That is therefore equivalent to terraform refresh except that it gives an opportunity to review what Terraform detected before creating a new state snapshot.

[apparentlymart]

While working on this PR I tripped over a few little hurdles in other parts of the code, but they are not really directly related to the task at hand here so I've split them off into separate PRs in the hope that they'll be easier to review and merge that way, while I finish up the remaining work on this PR.

Assuming we merge all of those, I'll rebase them out of this PR before I take it out of draft status.

[apparentlymart]

I think this is ready to review now!

Along with reviewing the implementation, if anyone is willing I'd appreciate to have someone try this from an end-user perspective and try some different permutations to see how the planning output behaves. There's a bunch of different combinations of interesting situations to report now, some of which interact with one another, so while I think it's doing something reasonable in all cases I wouldn't be surprised to learn that there are some less common combinations that I didn't consider.

[apparentlymart]

While testing @alisdair spotted that we weren't properly respecting refresh-only mode for the destroy actions generated for "deposed" objects in the state.

This is unfortunately a specific aspect of a more general problem that we've historically not been handling deposed objects consistently with "orphaned" current objects, even though they are conceptually the same idea just in two different circumstances. Consequently, the fix for that ended up being the fix for that general problem, and so I've added a new commit here which is quite out of scope for what this PR was setting out to do but unfortunately -refresh-only mode is built on the assumption that we'll handle these things consistently, so I believe it's necessary to fold this in to make sure the result will behave correctly in all cases with deposed objects.

The commit message for my extra commit contains some additional context:

core: Treat deposed objects the same as orphaned current objects

In many ways a deposed object is equivalent to an orphaned current object
in that the only action we can take with it is to destroy it. However, we
do still need to take some preparation steps in both cases: first, we must
ensure we track the upgraded version of the existing object so that we'll
be able to successfully render our plan, and secondly we must refresh the
existing object to make sure it still exists in the remote system.

We were previously doing these extra steps for orphan objects but not for
deposed ones, which meant that the behavior for deposed objects would be
subtly different and violate the invariants our callers expect in order
to display a plan. This also created the risk that a deposed object
already deleted in the remote system would become "stuck" because
Terraform would still plan to destroy it, which might cause the provider
to return an error when it tries to delete an already-absent object.

This also makes the deposed object planning take into account the
"skipPlanChanges" flag, which is important to get a correct result in
the "refresh only" planning mode.

It's a shame that we have almost identical code handling both the orphan
and deposed situations, but they differ in that the latter must call
different functions to interact with the deposed rather than the current
objects in the state. Perhaps a later change can improve on this with some
more refactoring, but this commit is already a little more disruptive than
I'd like and so I'm intentionally deferring that for another day.

The main upshot of this for folks not using -refresh-only mode is that Terraform will now (arguably more correctly) refresh deposed objects to make sure that they still exist before planning to destroy them. This matches our handling of orphaned objects where we allow for the fact that the user might've destroyed them already outside of Terraform, and if we were to plan a destroy action in that case it would probably fail when the provider tries to delete a non-existent object. The new behavior is therefore arguably more correct but still different than before, creating a new opportunity to encounter an error during planning.

[alisdair]

This is great! 👏

I've tested the branch out locally with various configurations, although probably not comprehensively. The UI was very readable throughout, and the drift detection in particular is really clear.

I have one inline test concern (with the JSON output golden reference), and a rambling musing about the changes to the golden reference, but otherwise ✅ from me.

[alisdair]

This new message surprised me at first, but I guess it's just a consequence of the plan rendering being restructured. My first concern was that it would make integration more difficult, but fortunately the changes object includes an operation field that I had forgotten about, so I think this is just fine.

I believe that it also won't affect integrations which use a saved plan, which was my main concern anyway.

[apparentlymart]

Yes, this new message is the JSON equivalent of a new behavior in the human apply output where Terraform repeats the plan that it's about to apply, because subjectively I found it jarring to have Terraform just immediately start doing something without explaining what it was doing, particularly in the case where there are no actions at all and thus updating the state was the only side-effect, where the output in that case made it seem like Terraform had done nothing at all.

While introducing this into the JSON output wasn't my original intent, once I figured out what I'd done and why it was happening it retroactively seemed justified, because a hypothetical custom UI around this output could for example choose to summarize that only 5 of 6 planned additions happened, for example. (Admittedly that would also be possible by parsing the detailed plan, but that's also true for the apply summary and so it seemed like making this information conveniently available was the intent.)

[apparentlymart]

Backported to v0.15 as:

• 14749dd
• 8354ce8
• 8c4ea4d
• 64a4d6d

[szEvEz]

Is there an option/flag to get rid of the output that was introduced with this?

While this output is useful for some use cases, I do not want to have a second plan output with changes outside of terraform which I decided to ignore via the lifecycle -> ignore_changes

For instance

# module.dqwwdq.default has been changed
  ~ resource "aws_db_instance" "default" {
        id                                    = "dwqdqw"
      ~ latest_restorable_time                = "2021-05-21T12:20:54Z" -> "2021-05-21T13:00:55Z"
        name                                  = "qwdqdq"
        tags                                  = {}   
        # (45 unchanged attributes hidden)

        # (1 unchanged block hidden)                                                                                                         
    }

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.