feat: stream output for custom workflows #2261

ascandella · 2022-05-17T06:57:58Z

Fixes #2054

I noticed that in #1937 which originally added this feature (e.g. https://github.com/runatlantis/atlantis/pull/1937/files#diff-edf527ba8643ff7bfca5f560491ea7055af472f5d6f3bbda127f1776b63d4b06L179) that the documentation around setting up terragrunt for custom workflows removed the -no-color option.

I'm not sure if this was by mistake, but to allow colorization in the in-browser terminal I've added ansi.Strip() for parsing all command output from the new runner abstraction. If this is out of scope I can back those changes out (but we should probably update the docs to re-add the -no-color flags to terragrunt workflows, otherwise the output is mangled in e.g. GitHub comments).

Tested by updating my atlantis instance to a build of this branch and removing the -no-color flags from my workflow.

I'm using the following server configuration:

repos:
  - id: "/.*/"
    workflow: terragrunt
    pre_workflow_hooks:
      - run: >
          terragrunt-atlantis-config generate
          --output atlantis.yaml
          --autoplan --automerge
workflows:
  terragrunt:
    plan:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt plan -out=$PLANFILE
      - run: terragrunt show -json $PLANFILE > $SHOWFILE
    apply:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt apply $PLANFILE

Output as usual in GitHub comments (I have --enable-diff-markdown-format on):

server/core/runtime/run_step_runner.go

jamengual · 2022-05-17T16:42:01Z

you could check : #1751 and #1325 or ask the contributors.

ascandella · 2022-05-17T19:40:46Z

server/core/runtime/models/shell_command_runner.go

@@ -0,0 +1,155 @@
+package models


I don't think this is a particularly good place for this, but noticed it was where the other exec functionality was put. Not totally sure on the naming convention of this project, but it couldn't be in runtime because of import cycles with the terraform package.

ascandella · 2022-05-17T19:41:08Z

server/core/runtime/models/shell_command_runner.go

+// Callers can use the input channel to pass stdin input to the command.
+// If any error is passed on the out channel, there will be no
+// further output (so callers are free to exit).
+func (s *ShellCommandRunner) RunCommandAsync(ctx command.ProjectContext) (chan<- string, <-chan Line) {


This code was extracted as-is from terraform_client.go

ascandella · 2022-05-17T19:44:56Z

you could check : #1751 and #1325 or ask the contributors.

@jamengual can you clarify on the "waiting-for-response" label? Is that with regards to the ANSI stripping, or something else?

jamengual · 2022-05-17T19:46:40Z

you could check : #1751 and #1325 or ask the contributors.

@jamengual can you clarify on the "waiting-for-response" label? Is that with regards to the ANSI stripping, or something else?

usually PRs do not get updated same day and linger for quite a while, it is a way to track when we are waiting for something from the contributor, nothing special

ascandella · 2022-05-24T14:19:07Z

@ascandella need any help with this? Very interested in this feature!

Nope, this is ready to go on my end, just waiting for review.

mustafa89 · 2022-05-31T09:22:24Z

Thanks, waiting eagerly for this. Any chances for a review @jamengual ?

adutchak · 2022-06-01T11:00:07Z

We're looking forward for having this feature too :)

SudoSpartanDan · 2022-06-07T13:27:35Z

@ascandella I think they fixed some build bugs in the upstream, can you fetch from upstream so we can get green checkmarks on this PR?

ascandella · 2022-06-07T13:58:16Z

@SudoSpartanDan thanks for the heads up. I've merged the latest master into this branch

frank-bee · 2022-06-14T17:50:26Z

I tried to work with the docker image produced by this PR plus terragrunt. Somehow I always went into lock problems but not really reproducible.
Maybe an upstream Problem?anyone else faced similar issues?

jamengual · 2022-06-14T18:10:09Z

I'm interesting in merging this to the prerelease but @frank-bee comment makes me wonder @ascandella is there anything you could assist here? @frank-bee can you describe your atlantis config and atlantis.yaml?

ascandella · 2022-06-14T18:12:11Z

I've been running this branch on my install for about a month now without issues. I haven't rebuilt/redeployed since I merged master in last week though. I'll try that now.

@frank-bee very curious what you mean by "lock problems". Could you provide any more detail/logs/observed behavior?

frank-bee · 2022-06-15T07:34:02Z

Hi @jamengual and @ascandella ,

here some more info...

"lock problems"
error message: The default workspace at path teams/aibse is currently locked by another command that is running for this pull request. Wait until the previous command is complete and try again.

atlantis repo config

version: 3
automerge: true
delete_source_branch_on_merge: true
projects:
  - name: aibse
    dir: teams/aibse
    workflow: terragrunt
    terraform_version: v1.2.2
    autoplan:
      when_modified: ["*.tf", "*.hcl"]
      enabled: true
  - name: platform
    dir: teams/platform
    workflow: terragrunt
    terraform_version: v1.2.2
    autoplan:
      when_modified: ["*.tf", "*.hcl"]
      enabled: true
workflows:
  terragrunt:
    plan:
      steps:
        - env:
            name: TERRAGRUNT_TFPATH
            command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
        - run: terragrunt plan -out=$PLANFILE
        - run: terragrunt show -json $PLANFILE > $SHOWFILE
    apply:
      steps:
        - env:
            name: TERRAGRUNT_TFPATH
            command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
        - run: terragrunt apply $PLANFILE

atlantis server config

# repos lists the config for specific repos.
repos:
  # id can either be an exact repo ID or a regex.
  # If using a regex, it must start and end with a slash.
  # Repo ID's are of the form {VCS hostname}/{org}/{repo name}, ex.
  # github.com/runatlantis/atlantis.
  - id: /.*/
    # branch is an regex matching pull requests by base branch
    # (the branch the pull request is getting merged into).
    # By default, all branches are matched
    branch: main
    # apply_requirements sets the Apply Requirements for all repos that match.
    apply_requirements: [mergeable, undiverged]
    # workflow sets the workflow for all repos that match.
    # This workflow must be defined in the workflows section.
    workflow: default
    # allowed_overrides specifies which keys can be overridden by this repo in
    # its atlantis.yaml file.
    allowed_overrides: [apply_requirements, workflow, delete_source_branch_on_merge]
    # allowed_workflows specifies which workflows the repos that match
    # are allowed to select.
    allowed_workflows: [default]
    # allow_custom_workflows defines whether this repo can define its own
    # workflows. If false (default), the repo can only use server-side defined
    # workflows.
    allow_custom_workflows: true
    # delete_source_branch_on_merge defines whether the source branch would be deleted on merge
    # If false (default), the source branch won't be deleted on merge
    delete_source_branch_on_merge: false
    # pre_workflow_hooks defines arbitrary list of scripts to execute before workflow execution.
    #pre_workflow_hooks:
    #  - run: my-pre-workflow-hook-command arg1
    # post_workflow_hooks defines arbitrary list of scripts to execute after workflow execution.
    #post_workflow_hooks:
    #  - run: my-post-workflow-hook-command arg1
# workflows lists server-side custom workflows
workflows:
  default:
    plan:
      steps:
        - init
        - plan
    apply:
      steps: [apply]

endriu0 · 2022-06-15T08:27:45Z

From my own experience with Atlantis / Terragrunt I run into a lot of issues with autoinit.

Due to this I went with below setup. Granted I also had to account for workspaces being used so that could be part of the reason.

                steps:
                  - env:
                      name: TERRAGRUNT_TFPATH
                      command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
                  - env:
                      name: TERRAGRUNT_AUTO_INIT
                      value: false
                  - run: terragrunt init -input=false -no-color
                  - run: if [[ $WORKSPACE != "default" ]]; then terragrunt workspace select -no-color $WORKSPACE; fi
                  - run: terragrunt validate -no-color
                  - run: terragrunt plan -no-color -out=$PLANFILE

ascandella · 2022-06-15T14:46:35Z

@frank-bee thanks for the info. Out of curiosity, are you saying that these problems happen only when running Atlantis from this branch?

Just trying to rule things out, since it's not really clear to me how the changes here could affect the locking behavior.

mustafa89 · 2022-06-16T20:41:28Z

@frank-bee if there is something kindly update here. This PR missed the pre release and we would very much like to have it.

frank-bee · 2022-06-17T08:07:23Z

@ascandella and @mustafa89 I have no updates here. Also cannot reproduce it at the moment because I switched to the released atlantis and do not work with terragrunt anymore.
If nobody else faced thesee kind of problems, please merge and close this PR!

SudoSpartanDan · 2022-06-17T14:26:16Z

@jamengual can we get this into the next release? Our team would love to start using this feature

jamengual · 2022-06-17T20:26:44Z

@jamengual can we get this into the next release? Our team would love to start using this feature

I'm not the only one that needs to review the code, so we need to wait for someone else to review it.

SudoSpartanDan · 2022-06-21T21:40:41Z

@jamengual who else can review? This PR's been sitting for a bit and I know folks have been waiting for this feature.

mustafa89 · 2022-06-29T21:33:00Z

I was testing this in the pre-release and it seems the same problem @frank-bee has raised happened to us. I saw multiple MRs where one of the workspaces started showing the same

The default workspace at path ... is currently locked by another command that is running for this pull request. Wait until the previous command is complete and try again.

to the extent I had to kill the processes in the atlantis pod.

I've verified it and it's definitely a bug. This happens in specific scenarios only when Terraform has to wait for an input as far as I've seen.

To reproduce:

Declare a variable without a value, so Terraform waits for the input. On 0.19.4 terraform simply bypasses this prompt and exits and releases the command lock.

variable "test_variable" {
  description = "Testing"
  type        = string
}

With this change you can see the plan commands stuck when you do a ps aux.

ascandella · 2022-06-29T21:46:35Z

@mustafa89 thanks for the details. Looking at the built-in terraform client, it seems to append -input=false to plan and apply commands.

Does it work for you if you add the same flags to your workflow? For example:

workflows:
  terragrunt:
    plan:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt plan -input=false -no-color -out=$PLANFILE
      - run: terragrunt show -no-color -json $PLANFILE > $SHOWFILE
    apply:
      steps:
      - env:
          name: TERRAGRUNT_TFPATH
          command: 'echo "terraform${ATLANTIS_TERRAFORM_VERSION}"'
      - run: terragrunt apply -input=false $PLANFILE -no-color

If so, then this is probably more of a documentation fix than something that can be done in code, since we're running arbitrary shell commands (with the run: config) and probably shouldn't be mangling those without the user's knowledge.

mustafa89 · 2022-06-29T22:20:13Z

Good point, I guess -input=false should be added regardless when working with Atlantis. I just added it and as expected, all good now. Thanks for the help 👍

With runatlantis#2261, we now open a stdin pipe like the built-in terraform client does. Doing so apparently caused some user's `plan`s to hang when they had variables requiring inputs, since `terraform` saw that stdin was open and prompted for a value.

ascandella · 2022-06-29T22:25:01Z

Cool, glad that worked! Opened #2350 to update the terragrunt workflow docs

With #2261, we now open a stdin pipe like the built-in terraform client does. Doing so apparently caused some user's `plan`s to hang when they had variables requiring inputs, since `terraform` saw that stdin was open and prompted for a value.

* Start threading job output to RunStepRunner * Strip ANSI * Fix lint * Use waitgroup to avoid test flakiness * Move waitgroup higher * Add ANSI test and use strings.Builder * Fix lint * Use errors.Wrap per style guide * Create ShellCommandRunner to encapsulate streaming * WIP: shell command runner * Update signatures to propagate error finding version * Fix log output * Fix error checking * Fix accidental whitespace stripping * Remove unused struct field * Fix error checking in terraform client * Add unit tests to verify command output handler was called * Remove err from async interface * Remove duplicative log now that shell command runner does it * Hide output in stream for env/multienv * Add comment explaining goroutines * Use printf for better macOS compatibility

With runatlantis#2261, we now open a stdin pipe like the built-in terraform client does. Doing so apparently caused some user's `plan`s to hang when they had variables requiring inputs, since `terraform` saw that stdin was open and prompted for a value.

ascandella added 2 commits May 16, 2022 23:29

Start threading job output to RunStepRunner

f7cd127

Strip ANSI

0200c62

ascandella commented May 17, 2022

View reviewed changes

server/core/runtime/run_step_runner.go Outdated Show resolved Hide resolved

ascandella added 5 commits May 17, 2022 00:07

Fix lint

22388bc

Use waitgroup to avoid test flakiness

efb38a8

Move waitgroup higher

831a8f6

Add ANSI test and use strings.Builder

589b43c

Fix lint

51cfc39

ascandella changed the title ~~Stream output for custom workflows~~ feat: stream output for custom workflows May 17, 2022

Use errors.Wrap per style guide

bdae433

ascandella marked this pull request as ready for review May 17, 2022 16:37

ascandella requested a review from a team as a code owner May 17, 2022 16:37

jamengual added feature New functionality/enhancement waiting-on-response Waiting for a response from the user terragrunt labels May 17, 2022

ascandella mentioned this pull request May 17, 2022

Terraform log streaming for custom workflows #2054

Closed

ascandella added 9 commits May 17, 2022 11:33

Create ShellCommandRunner to encapsulate streaming

f1fda9d

WIP: shell command runner

a6567b5

Update signatures to propagate error finding version

ad6a941

Fix log output

4e894c4

Fix error checking

d87edd8

Merge branch 'master' into stream-all-output

2b5e5e9

Fix accidental whitespace stripping

c9bc5df

Remove unused struct field

427c5de

Fix error checking in terraform client

7fe7e3e

ascandella commented May 17, 2022

View reviewed changes

jamengual removed the waiting-on-response Waiting for a response from the user label May 17, 2022

jamengual added the waiting-on-review Waiting for a review from a maintainer label May 31, 2022

Merge branch 'master' into stream-all-output

45aacf3

jamengual merged commit ff1094f into runatlantis:master Jun 22, 2022

ascandella mentioned this pull request Jun 29, 2022

Update terragrunt workflow documentation #2350

Merged

tweeks-reify mentioned this pull request Jul 15, 2022

CHANGED - update streaming-logs doc to mention terragrunt support #2395

Merged

nitrocode added this to the v0.19.5 milestone Mar 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: stream output for custom workflows #2261

feat: stream output for custom workflows #2261

ascandella commented May 17, 2022 •

edited

Loading

jamengual commented May 17, 2022

ascandella May 17, 2022

ascandella May 17, 2022

ascandella commented May 17, 2022

jamengual commented May 17, 2022

ascandella commented May 24, 2022

mustafa89 commented May 31, 2022 •

edited

Loading

adutchak commented Jun 1, 2022

SudoSpartanDan commented Jun 7, 2022

ascandella commented Jun 7, 2022

frank-bee commented Jun 14, 2022

jamengual commented Jun 14, 2022

ascandella commented Jun 14, 2022

frank-bee commented Jun 15, 2022

endriu0 commented Jun 15, 2022

ascandella commented Jun 15, 2022

mustafa89 commented Jun 16, 2022

frank-bee commented Jun 17, 2022

SudoSpartanDan commented Jun 17, 2022

jamengual commented Jun 17, 2022

SudoSpartanDan commented Jun 21, 2022

mustafa89 commented Jun 29, 2022 •

edited

Loading

ascandella commented Jun 29, 2022

mustafa89 commented Jun 29, 2022

ascandella commented Jun 29, 2022

feat: stream output for custom workflows #2261

feat: stream output for custom workflows #2261

Conversation

ascandella commented May 17, 2022 • edited Loading

jamengual commented May 17, 2022

ascandella May 17, 2022

Choose a reason for hiding this comment

ascandella May 17, 2022

Choose a reason for hiding this comment

ascandella commented May 17, 2022

jamengual commented May 17, 2022

ascandella commented May 24, 2022

mustafa89 commented May 31, 2022 • edited Loading

adutchak commented Jun 1, 2022

SudoSpartanDan commented Jun 7, 2022

ascandella commented Jun 7, 2022

frank-bee commented Jun 14, 2022

jamengual commented Jun 14, 2022

ascandella commented Jun 14, 2022

frank-bee commented Jun 15, 2022

endriu0 commented Jun 15, 2022

ascandella commented Jun 15, 2022

mustafa89 commented Jun 16, 2022

frank-bee commented Jun 17, 2022

SudoSpartanDan commented Jun 17, 2022

jamengual commented Jun 17, 2022

SudoSpartanDan commented Jun 21, 2022

mustafa89 commented Jun 29, 2022 • edited Loading

ascandella commented Jun 29, 2022

mustafa89 commented Jun 29, 2022

ascandella commented Jun 29, 2022

ascandella commented May 17, 2022 •

edited

Loading

mustafa89 commented May 31, 2022 •

edited

Loading

mustafa89 commented Jun 29, 2022 •

edited

Loading