[Bug] unable to unmarshal conftest output or Anyone able to approve Atlantis policy failures

[kumaresh0]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

1. Anyone ( non policy owners ) able to approve Atlantis policy failures ( seems major bug )
   I am using below server config with custom_policy_check: true & policy_check: true

2. if I use custom_policy_check: false & policy_check: true i get the unable to unmarshal conftest output error

But based on this documentation https://www.runatlantis.io/docs/policy-checking.html#step-2-define-the-policy-configuration we tried the config as i mentioned below

Reproduction Steps

Nothing special just used Alpine image and added the below server-side workflow and repo side workflow then triggered policy failure, if anyone comments the atlantis approve_policies policy failures are fixed with approval

Logs

Issue screen shot

Environment details

If not already included, please provide the following:

• Atlantis version: v0.27.1
• Deployment method: ecs
• If not running the latest Atlantis version have you tried to reproduce this issue on the latest version: Yes ( it' already latest release )
• Atlantis flags: No custom flags ( using default flags with repo details )
  ECS environment variables

  container_definition_environment = [
    {
      name  = "ATLANTIS_CHECKOUT_STRATEGY"
      value = var.checkout_strategy
    },
    {
      name  = "ATLANTIS_ALLOW_COMMANDS"
      value = var.atlantis_allow_commands
    },
    {
      name  = "ATLANTIS_ALLOW_REPO_CONFIG"
      value = var.allow_repo_config
    },
    {
      name  = "ATLANTIS_RESOURCE_BUCKET_NAME"
      value = data.aws_s3_bucket.atlantis_bucket.id

    },
    {
      name  = "ATLANTIS_LOG_LEVEL"
      value = var.atlantis_log_level
    },
    {
      name  = "ATLANTIS_ASSUME_ROLE_NAME"
      value = var.atlantis_assume_role_name
    },
    {
      name  = "ATLANTIS_PORT"
      value = var.atlantis_port
    },
    {
      name  = "ATLANTIS_CONFIG"
      value = var.atlantis_config
    },
    {
      name  = "ATLANTIS_ATLANTIS_URL"
      value = local.atlantis_url
    },
    {
      name  = "ATLANTIS_WRITE_GIT_CREDS"
      value = true
    },
    {
      name  = "ATLANTIS_GH_USER"
      value = var.atlantis_github_user
    },
    {
      name  = "ATLANTIS_REPO_ALLOWLIST"
      value = join(",", var.atlantis_repo_allowlist)
    },
    {
      name  = "ATLANTIS_REPO_CONFIG"
      value = var.atlantis_repo_config
    },
    {
      name  = "ATLANTIS_ENABLE_POLICY_CHECKS"
      value = var.atlantis_enable_policy_checks
    },
    {
      name  = "ATLANTIS_HIDE_PREV_PLAN_COMMENTS"
      value = var.atlantis_hide_prev_plan_comments
    },
    {
      name  = "ATLANTIS_DISABLE_APPLY"
      value = var.atlantis_disable_apply
    },
    {
      name  = "ATLANTIS_DEFAULT_TF_VERSION"
      value = var.atlantis_default_terraform_version
    },
    {
      name  = "ATLANTIS_POLICY_DIRECTORY"
      value = var.atlantis_policy_directory
    },
    {
      name  = "ATLANTIS_POLICY_REPO_NAME"
      value = var.atlantis_policy_repo_name
    },
    {
      name  = "ATLANTIS_POLICY_REPO_REF"
      value = var.atlantis_policy_repo_ref
    }
  ]

Atlantis server-side config file:

repos:
- id: "/.*/"
  branch: "/master/"
  pre_workflow_hooks:
    - description: Pull the Atlantis policies.
  post_workflow_hooks:
    - run: python3 /home/atlantis/review.py 
  allow_custom_workflows: false
  custom_policy_check: true
  policy_check: true
  allowed_overrides: [workflow, apply_requirements, delete_source_branch_on_merge]
  apply_requirements: [approved, mergeable]
  import_requirements: [approved, mergeable]

metrics:
  prometheus:
    endpoint: /metrics

policies:
    conftest_version: 0.45.0
    owners:
      users:
        - user1
        - user2
        - user3
    policy_sets:
        - name: audit_policy
          path: /home/atlantis/policy/security
          source: local
          approve_count: 2
          owners:
            users:
              - securityuser1
        - name: comman_policy
          path: /home/atlantis/policy/regula
          source: local

workflows:
  default:
    plan:
      steps:
      - init:
          extra_args: [ "-reconfigure"]
      - run: TF_WORKSPACE=$WORKSPACE tflint --config /home/atlantis/.tflint.hcl
      - plan
      - run: terraform$ATLANTIS_TERRAFORM_VERSION show -no-color -json $PLANFILE > tfplan.json
    apply:
      steps:
      - apply
    policy_check: &policy_check
      steps:
      - run: |
          aws sts get-caller-identity --output json | jq '{"aws": .}' | jq '{"external": .}' > external-data.json
      - run: conftest test tfplan.json --namespace main --namespace security -o table --policy /home/atlantis/policy -d external-data.json
    import:
      steps:
      - import

Repo atlantis.yaml file:

version: 3
automerge: true
delete_source_branch_on_merge: true

projects:
- name: test-case/default-workflow
  dir: test-case/default-workflow
  workspace: default
  autoplan:
    when_modified: ["*.tf", "../modules/**.tf", "Terrafile"]
    enabled: true
  workflow: default

Our Atlantis is deployed in ECS fargate with ghcr.io/runatlantis/atlantis:v0.27-alpine

Additional Context

• Policy Check Error: unable to unmarshal conftest output #4308
• Custom policy checks with non-JSON output produce errors #3682
• [Custom policy checks] Custom policy checks formatting broken #4349
• Anyone can approve custom policy exceptions #4962

[kumaresh0]

similar issuer reported here : #4308

[kumaresh0]

Adding some update on this:

From console, i can see policy test results

example

Resources: {"aws_security_group.inline_invalid_security_group"}

4 tests, 2 passed, 0 warnings, 2 failures, 0 exceptions

But in PR it show unable to unmarshal conftest output and anyone able to approve the policy failures

[Mezage]

I hope this issue is getting traction, our team really needs to upgrade Atlantis but we cant since custom policies were introduced, this policy approval bug has been an issue. Anyone can approve policies, so policy set approvers doesnt even matter. :/

[nitrocode]

We're running into this issue too. We might need a reproducible example here.

{"level":"error","ts":"2024-10-30T18:36:33.314Z","caller":"events/instrumented_project_command_runner.go:78","msg":"Error running policy_check operation: unable to unmarshal conftest output","json":{"repo":"snip/snip","pull":"2985"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:78\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).PolicyCheck\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:42\ngithub.jparrowsec.cn/runatlantis/atlantis/server/events.runProjectCmdsParallel.func1\n\tgithub.jparrowsec.cn/runatlantis/atlantis/server/events/project_command_pool_executor.go:29"}

[nitrocode]

cc @bgalkows @pseudomorph if you folks have seen this in your environments

[pseudomorph]

I have not had any issues.

I'm fairly confident that the issue lies in the conftest command. The policy check logic expects json output from conftest, and it's being set to table.

Try changing -o table to -o json.

I believe someone also introduced a flag which allows for custom policy commands to bypass this logic, if that's what's desired. I cannot find it at the moment though..

[pseudomorph]

Oh apologies, I see that it was specified above -- custom_policy_check.

So, my previous comment should address issue 2 (unmarshaling error).

I'm not 100%, but I imagine issue 1 (anyone can approve) is likely due to bypassing the granular policy logic when custom_policy_check: true is set.

[pseudomorph]

Can you go into a bit more detail about the anyone can approve issue? Is the "anyone" a user who is explicitly listed or is a member of a listed team in your config?

[bgalkows]

Wow, I'm glad you tagged me in this bug. The issue comes from defaulting all custom policy checks into an undefined policy set called "Custom", which as a result has no user approval restrictions.

I spun up a PR with the fix: #5331

[bgalkows]

My company is using Atlantis in "maintenance mode" as we shift to another actuation tool so we don't pull new releases and use an internal fork.

This bug was noticed a long time ago and a coworker implemented the fix - I wish we'd thought to contribute it here too