[Custom policy checks] Custom policy checks formatting broken

[endriu0]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Custom policy checks formatting always breaks on first policy. Instead of showing the nice heading :

Policy Set: Custom

We get :

Show Output

```diff
#### Policy Set: `Custom`
```diff

Attached screenshots. Looks like the summary block is interfering with the diff.

Reproduction Steps

Custom policy checks outputting anything always throw this on first policy. Second policy looks ok but first one probably due to being hidden behind the Show Output breaks format.

Logs

{"level":"info","ts":"2024-03-12T14:27:18.971Z","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"echo \\\"Required test output below:\\\" && conftest test -o table --policy /atlantis-data/library/general/mypolicy/mypolicy.rego  --namespace terraform.ourpolicy $SHOWFILE\" in \"/atlantis-data/repos/redacted"","json":{"repo":"redacted","pull":"1","duration":0.049932837}}

{"level":"error","ts":"2024-03-12T14:27:19.136Z","caller":"events/project_command_runner.go:529","msg":"Required test output below:\n+---------+------------------------------------------------------------------------------------+----------------+---------+\n| RESULT  |                                        FILE                                        |   NAMESPACE    | MESSAGE |\n+---------+------------------------------------------------------------------------------------+----------------+---------+\n| success | redacted/workspace.json | terraform.ourpolicy | SUCCESS |\n| success | /atlantis-data/repos/redacted/workspace.json | terraform.ourpolicy | SUCCESS |\n| success | /atlantis-data/repos/redacted/workspace.json | terraform.ourpolicy | SUCCESS |\n+---------+------------------------------------------------------------------------------------+----------------+---------+\n\nAnother test output below:\n| UPDATE? | NAME | CONSTRAINT | VERSION | LATEST MATCHING | LATEST |\n|---------|------|------------|---------|-----------------|--------|\n+---------+-----------------+-------------------+---------+\n| RESULT  |      FILE       |     NAMESPACE     | MESSAGE |\n+---------+-----------------+-------------------+---------+\n| success | version.json | tf_version | SUCCESS |\n| success | version.json | tf_version | SUCCESS |\n+---------+-----------------+-------------------+---------+\n","json":{"repo":"redacted","pull":"1"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandRunner).doPolicyCheck\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/project_command_runner.go:529\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.(*DefaultProjectCommandRunner).PolicyCheck\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/project_command_runner.go:240\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:74\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).PolicyCheck\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:42\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.runProjectCmds\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/project_command_pool_executor.go:48\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.(*PolicyCheckCommandRunner).Run\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/policy_check_command_runner.go:65\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.(*PlanCommandRunner).run\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/plan_command_runner.go:290\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/plan_command_runner.go:306\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/command_runner.go:365"}

{"level":"info","ts":"2024-03-12T14:27:19.136Z","caller":"events/instrumented_project_command_runner.go:88","msg":"policy_check success. output available at: https://redacted","json":{"repo":"redacted","pull":"1"}}

Environment details

• Atlantis version: 0.27.2
• Deployment method: helm
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: Reproducible on each version since the custom policy check released.
• Atlantis flags: --enable-policy-checks

Additional Context

Atlantis server-side policy check conf:

        policy_check:
          steps:
            - run: echo "Required test output below:" && conftest test -o table --policy /atlantis-data/library/general/mypolicy/mypolicy.rego  --namespace terraform.ourpolicy $SHOWFILE
            - run: echo "nAnother test output below:" &&  version check $DIR && conftest test -o table --policy /atlantis-data/library/general/tf_version/version.rego --namespace tf_version version.json

Another issue showing the same problem :
#4243

[kumaresh0]

@endriu0 does policy approval work properly for you?
because i have encountered this issue #4243 ( anyone able to approve the policy failures ) and for me it shows unable to unmarshal conftest output just wonder if you least able to see the table format result for a couple of tests. Could you please share your server config for reference?

[endriu0]

Sorry @kumaresh0 totally missed that. so I just tested this and you're right. I was able to approve with a user that's not on the list of approvers. As for the unable to unmarshal error I still use conftest for my policies so my config wont change much unfortunately - I just add the policies the old fashioned way of adding them at the workflow level.