Scan only PR commits for Gitleaks instead of whole codebase (#2504)

* quick build * quick build * quick build TEST_KEYWORDS=repository_gitleaks * quick build TEST_KEYWORDS=repository_gitleaks * @DariuszPorowski quick build TEST_KEYWORDS=repository_gitleaks * quick build TEST_KEYWORDS=repository_gitleaks * quick build TEST_KEYWORDS=repository_gitleaks * quick build TEST_KEYWORDS=repository_gitleaks quick build TEST_KEYWORDS=repository_gitleaks * remove casefold * changelog * quick build TEST_KEYWORDS=repository_gitleaks quick build TEST_KEYWORDS=repository_gitleaks * quick build TEST_KEYWORDS=repository_gitleaks * update gitleaks descriptor * update config.get * docs build * fix utils_reporter conflict * typo * style utils_reporter * update utils to new standard * build --------- Co-authored-by: nvuillam <[email protected]>
oxsecurity · May 13, 2023 · 7ff24de · 7ff24de
1 parent 5ff5f1e
commit 7ff24de
Show file tree

Hide file tree

Showing 11 changed files with 267 additions and 49 deletions.
diff --git a/.cspell.json b/.cspell.json
@@ -1246,8 +1246,10 @@
         "sonarqube",
         "soossssoso",
         "sososhhhyyso",
+        "sourcecommitid",
         "sourcefilepath",
         "sourceroot",
+        "sourceversion",
         "splitchar",
         "sprintf",
         "sqlfluff",
@@ -1299,6 +1301,8 @@
         "tableofcontents",
         "tabularx",
         "tagname",
+        "targetbranch",
+        "targetbranchname",
         "tcort",
         "tekton",
         "tektonlintrc",
@@ -1442,4 +1446,4 @@
         "\u0636clojure",
         "\u077bcbad"
     ]
-}
+}
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -18,6 +18,7 @@ Note: Can be used with `oxsecurity/megalinter@beta` in your GitHub Action mega-l
   - Web site: [my-devops-lab.com](https://www.my-devops-lab.com/tools)
 
 - Linter enhancements & fixes
+  - Gitleaks: add support to scan PR commits only on PRs when `VALIDATE_ALL_CODEBASE` is set to `false`, by @DariuszPorowski [#2504](https://github.com/oxsecurity/megalinter/pull/2504)
   - Ensure ESLint actually runs in project mode rather than silently doing nothing, by @Kurt-von-Laven [#2455](https://github.com/oxsecurity/megalinter/pull/2455).
   - Prevent jscpd to create output folder if the repo isn't writable. Fixes [#2108](https://github.com/oxsecurity/megalinter/issues/2108)
   - Fix corrective .cspell.json file generated from cspell output

diff --git a/README.md b/README.md
@@ -525,6 +525,9 @@ Add the following job in your `azure-pipelines.yaml` file
     pool:
       vmImage: ubuntu-latest
     steps:
+      # Checkout repo
+      - checkout: self
+
       # Pull MegaLinter docker image
       - script: docker pull oxsecurity/megalinter:v6
         displayName: Pull MegaLinter
@@ -533,7 +536,6 @@ Add the following job in your `azure-pipelines.yaml` file
       - script: |
           docker run -v $(System.DefaultWorkingDirectory):/tmp/lint \
             --env-file <(env | grep -e SYSTEM_ -e BUILD_ -e TF_ -e AGENT_) \
-            -e CI=true \
             -e SYSTEM_ACCESSTOKEN=$(System.AccessToken) \
             -e GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \
             oxsecurity/megalinter:v6
@@ -755,7 +757,7 @@ See [mega-linter-runner installation instructions](https://megalinter.io/mega-li
 Example
 
 ```shell
-npx mega-linter-runner --flavor salesforce -e 'ENABLE=,DOCKERFILE,MARKDOWN,YAML' -e 'SHOW_ELAPSED_TIME=true'
+npx mega-linter-runner --flavor salesforce -e "'ENABLE=DOCKERFILE,MARKDOWN,YAML'" -e 'SHOW_ELAPSED_TIME=true'
 ```
 
 Note: You can also use such command line in your custom CI/CD pipelines

diff --git a/docs/descriptors/repository_gitleaks.md b/docs/descriptors/repository_gitleaks.md
@@ -5,17 +5,60 @@ description: How to use gitleaks (configure, ignore files, ignore errors, help &
 <!-- markdownlint-disable MD033 MD041 -->
 <!-- @generated by .automation/build.py, please don't update manually -->
 # gitleaks
-[![GitHub stars](https://img.shields.io/github/stars/zricethezav/gitleaks?cacheSeconds=3600)](https://github.com/zricethezav/gitleaks) ![sarif](https://shields.io/badge/-SARIF-orange) [![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/zricethezav/gitleaks?sort=semver)](https://github.com/zricethezav/gitleaks/releases) [![GitHub last commit](https://img.shields.io/github/last-commit/zricethezav/gitleaks)](https://github.com/zricethezav/gitleaks/commits) [![GitHub commit activity](https://img.shields.io/github/commit-activity/y/zricethezav/gitleaks)](https://github.com/zricethezav/gitleaks/graphs/commit-activity/) [![GitHub contributors](https://img.shields.io/github/contributors/zricethezav/gitleaks)](https://github.com/zricethezav/gitleaks/graphs/contributors/)
+[![GitHub stars](https://img.shields.io/github/stars/gitleaks/gitleaks?cacheSeconds=3600)](https://github.com/gitleaks/gitleaks) ![sarif](https://shields.io/badge/-SARIF-orange) [![GitHub release (latest SemVer)](https://img.shields.io/github/v/release/gitleaks/gitleaks?sort=semver)](https://github.com/gitleaks/gitleaks/releases) [![GitHub last commit](https://img.shields.io/github/last-commit/gitleaks/gitleaks)](https://github.com/gitleaks/gitleaks/commits) [![GitHub commit activity](https://img.shields.io/github/commit-activity/y/gitleaks/gitleaks)](https://github.com/gitleaks/gitleaks/graphs/commit-activity/) [![GitHub contributors](https://img.shields.io/github/contributors/gitleaks/gitleaks)](https://github.com/gitleaks/gitleaks/graphs/contributors/)
+
+## Scan only Pull Request commits
+
+`VALIDATE_ALL_CODEBASE: false` doesn't make gitleaks analyze only updated files. To analyze only commits on Pull Request, set `VALIDATE_ALL_CODEBASE: false` together with `REPOSITORY_GITLEAKS_PR_COMMITS_SCAN: true` (you have to specify it explicitly), but only works for selected platforms: GitHub Actions, Azure Pipelines, GitLab Pipelines\* (Merge Requests and External Pull Requests)
+
+- \* Only GitLab self-managed and GitLab SaaS (Premium and Ultimate) are supported (limitation due to GitLab itself) and [Merge result pipelines](https://docs.gitlab.com/ee/ci/pipelines/merged_results_pipelines.html#enable-merged-results-pipelines){target=_blank} feature has to be enabled.
+- If MegaLinter with the gitleaks runs on PR on the not listed platform above, then the analysis is performed on the whole repository - default gitleaks behavior (checked-out commits, depends on fetch-depth configuration).
+  - You can still scan only PR commits in your CI/CD platform by setting MegaLinter envs: `PULL_REQUEST=true`\*, `REPOSITORY_GITLEAKS_PR_COMMITS_SCAN: true`, `REPOSITORY_GITLEAKS_PR_SOURCE_SHA` with last commit sha from your PR and `REPOSITORY_GITLEAKS_PR_TARGET_SHA` commit sha from your target branch (for example, `main` if you do PR to main branch). Example on how to get source commit sha `git rev-list -n 1 refs/remotes/origin/<source_branch>` and target commit sha `git rev-parse refs/remotes/origin/<target_branch>`
+    - \* `PULL_REQUEST` environment variable must be set to `true` only on Pull Requests, so you must calculate the value in your pipeline and pass the outcome.
+- PR commits scan feature, if applicable, will override your `--log-opts` argument if you used it in the `REPOSITORY_GITLEAKS_ARGUMENTS`.
+
+### Repository checkout on Pull Requests
+
+To scan only PR commits, the [shallow fetch](https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---depthltdepthgt){target=_blank} for a repository checkout has to be 0. Below is an example configuration for supported platforms:
+
+#### GitHub Actions
+
+```yml
+- uses: actions/checkout@v3
+  with:
+    fetch-depth: 0
+```
+
+#### Azure Pipelines
+
+```yml
+- checkout: self
+  fetchDepth: 0
+```
+
+#### GitLab Pipelines
+
+```yml
+variables:
+  GIT_DEPTH: 0
+```
+
+#### Git
+
+```shell
+git fetch --depth=0
+```
 
 ## gitleaks documentation
 
 - Version in MegaLinter: **8.16.3**
-- Visit [Official Web Site](https://github.com/zricethezav/gitleaks#readme){target=_blank}
-- See [How to configure gitleaks rules](https://github.com/zricethezav/gitleaks#configuration){target=_blank}
+- Visit [Official Web Site](https://github.com/gitleaks/gitleaks#readme){target=_blank}
+- See [How to configure gitleaks rules](https://github.com/gitleaks/gitleaks#configuration){target=_blank}
   - If custom `.gitleaks.toml` config file isn't found, [.gitleaks.toml](https://github.com/oxsecurity/megalinter/tree/main/TEMPLATES/.gitleaks.toml){target=_blank} will be used
-- See [How to ignore files and directories with gitleaks](https://github.com/zricethezav/gitleaks#configuration){target=_blank}
+- See [How to disable gitleaks rules in files](https://github.com/gitleaks/gitleaks#gitleaksallow){target=_blank}
+- See [How to ignore files and directories with gitleaks](https://github.com/gitleaks/gitleaks#gitleaksignore){target=_blank}
 
-[![gitleaks - GitHub](https://gh-card.dev/repos/zricethezav/gitleaks.svg?fullname=)](https://github.com/zricethezav/gitleaks){target=_blank}
+[![gitleaks - GitHub](https://gh-card.dev/repos/gitleaks/gitleaks.svg?fullname=)](https://github.com/gitleaks/gitleaks){target=_blank}
 
 ## Configuration in MegaLinter
 
@@ -24,6 +67,9 @@ description: How to use gitleaks (configure, ignore files, ignore errors, help &
 
 | Variable                                        | Description                                                                         | Default value                                   |
 |-------------------------------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------|
+| REPOSITORY_GITLEAKS_PR_COMMITS_SCAN             | Scan only PR commits on Pull Request                                                | `false`                                         |
+| REPOSITORY_GITLEAKS_PR_SOURCE_SHA               | Pull Request source commit sha (N/A to supported scenarios)                         | ``                                              |
+| REPOSITORY_GITLEAKS_PR_TARGET_SHA               | Pull Request target commit sha (N/A to supported scenarios)                         | ``                                              |
 | REPOSITORY_GITLEAKS_ARGUMENTS                   | User custom arguments to add in linter CLI call<br/>Ex: `-s --foo "bar"`            |                                                 |
 | REPOSITORY_GITLEAKS_PRE_COMMANDS                | List of bash commands to run before the linter                                      | None                                            |
 | REPOSITORY_GITLEAKS_POST_COMMANDS               | List of bash commands to run after the linter                                       | None                                            |
@@ -74,11 +120,11 @@ gitleaks is called once on the whole project directory (`project` CLI lint mode)
 ### Example calls
 
 ```shell
-gitleaks detect --no-git --verbose --source .
+gitleaks detect --redact --no-git --verbose --source .
 ```
 
 ```shell
-gitleaks detect -c .gitleaks.toml --no-git --verbose --source .
+gitleaks detect -c .gitleaks.toml --redact --no-git --verbose --source .
 ```
 
 

diff --git a/docs/flavors/security.md b/docs/flavors/security.md
@@ -55,7 +55,7 @@ Optimized for security
 | <img src="https://github.com/oxsecurity/megalinter/raw/main/docs/assets/icons/default.ico" alt="" height="32px" class="megalinter-icon"></a> <!-- linter-icon --> | [**REPOSITORY**](https://megalinter.io/beta/descriptors/repository/) | [**checkov**](https://megalinter.io/beta/descriptors/repository_checkov/)<br/>[_REPOSITORY_CHECKOV_](https://megalinter.io/beta/descriptors/repository_checkov/)             |                                    [![GitHub stars](https://img.shields.io/github/stars/bridgecrewio/checkov?cacheSeconds=3600)](https://github.com/bridgecrewio/checkov) ![sarif](https://shields.io/badge/-SARIF-orange)                                     |
 | <img src="https://github.com/oxsecurity/megalinter/raw/main/docs/assets/icons/default.ico" alt="" height="32px" class="megalinter-icon"></a> <!-- linter-icon --> | [**REPOSITORY**](https://megalinter.io/beta/descriptors/repository/) | [**devskim**](https://megalinter.io/beta/descriptors/repository_devskim/)<br/>[_REPOSITORY_DEVSKIM_](https://megalinter.io/beta/descriptors/repository_devskim/)             | ![downgraded version](https://shields.io/badge/-downgraded%20version-orange) [![GitHub stars](https://img.shields.io/github/stars/microsoft/DevSkim?cacheSeconds=3600)](https://github.com/microsoft/DevSkim) ![sarif](https://shields.io/badge/-SARIF-orange) |
 | <img src="https://github.com/oxsecurity/megalinter/raw/main/docs/assets/icons/default.ico" alt="" height="32px" class="megalinter-icon"></a> <!-- linter-icon --> | [**REPOSITORY**](https://megalinter.io/beta/descriptors/repository/) | [**dustilock**](https://megalinter.io/beta/descriptors/repository_dustilock/)<br/>[_REPOSITORY_DUSTILOCK_](https://megalinter.io/beta/descriptors/repository_dustilock/)     |                                     [![GitHub stars](https://img.shields.io/github/stars/Checkmarx/dustilock?cacheSeconds=3600)](https://github.com/Checkmarx/dustilock) ![sarif](https://shields.io/badge/-SARIF-orange)                                      |
-| <img src="https://github.com/oxsecurity/megalinter/raw/main/docs/assets/icons/default.ico" alt="" height="32px" class="megalinter-icon"></a> <!-- linter-icon --> | [**REPOSITORY**](https://megalinter.io/beta/descriptors/repository/) | [**gitleaks**](https://megalinter.io/beta/descriptors/repository_gitleaks/)<br/>[_REPOSITORY_GITLEAKS_](https://megalinter.io/beta/descriptors/repository_gitleaks/)         |                                    [![GitHub stars](https://img.shields.io/github/stars/zricethezav/gitleaks?cacheSeconds=3600)](https://github.com/zricethezav/gitleaks) ![sarif](https://shields.io/badge/-SARIF-orange)                                     |
+| <img src="https://github.com/oxsecurity/megalinter/raw/main/docs/assets/icons/default.ico" alt="" height="32px" class="megalinter-icon"></a> <!-- linter-icon --> | [**REPOSITORY**](https://megalinter.io/beta/descriptors/repository/) | [**gitleaks**](https://megalinter.io/beta/descriptors/repository_gitleaks/)<br/>[_REPOSITORY_GITLEAKS_](https://megalinter.io/beta/descriptors/repository_gitleaks/)         |                                       [![GitHub stars](https://img.shields.io/github/stars/gitleaks/gitleaks?cacheSeconds=3600)](https://github.com/gitleaks/gitleaks) ![sarif](https://shields.io/badge/-SARIF-orange)                                        |
 | <img src="https://github.com/oxsecurity/megalinter/raw/main/docs/assets/icons/default.ico" alt="" height="32px" class="megalinter-icon"></a> <!-- linter-icon --> | [**REPOSITORY**](https://megalinter.io/beta/descriptors/repository/) | [**secretlint**](https://megalinter.io/beta/descriptors/repository_secretlint/)<br/>[_REPOSITORY_SECRETLINT_](https://megalinter.io/beta/descriptors/repository_secretlint/) |                                   [![GitHub stars](https://img.shields.io/github/stars/secretlint/secretlint?cacheSeconds=3600)](https://github.com/secretlint/secretlint) ![sarif](https://shields.io/badge/-SARIF-orange)                                    |
 | <img src="https://github.com/oxsecurity/megalinter/raw/main/docs/assets/icons/default.ico" alt="" height="32px" class="megalinter-icon"></a> <!-- linter-icon --> | [**REPOSITORY**](https://megalinter.io/beta/descriptors/repository/) | [**semgrep**](https://megalinter.io/beta/descriptors/repository_semgrep/)<br/>[_REPOSITORY_SEMGREP_](https://megalinter.io/beta/descriptors/repository_semgrep/)             |                                    [![GitHub stars](https://img.shields.io/github/stars/returntocorp/semgrep?cacheSeconds=3600)](https://github.com/returntocorp/semgrep) ![sarif](https://shields.io/badge/-SARIF-orange)                                     |
 | <img src="https://github.com/oxsecurity/megalinter/raw/main/docs/assets/icons/default.ico" alt="" height="32px" class="megalinter-icon"></a> <!-- linter-icon --> | [**REPOSITORY**](https://megalinter.io/beta/descriptors/repository/) | [**syft**](https://megalinter.io/beta/descriptors/repository_syft/)<br/>[_REPOSITORY_SYFT_](https://megalinter.io/beta/descriptors/repository_syft/)                         |                                            [![GitHub stars](https://img.shields.io/github/stars/anchore/syft?cacheSeconds=3600)](https://github.com/anchore/syft) ![sarif](https://shields.io/badge/-SARIF-orange)                                             |

diff --git a/docs/installation.md b/docs/installation.md
@@ -184,6 +184,9 @@ Add the following job in your `azure-pipelines.yaml` file
     pool:
       vmImage: ubuntu-latest
     steps:
+      # Checkout repo
+      - checkout: self
+
       # Pull MegaLinter docker image
       - script: docker pull oxsecurity/megalinter:v6
         displayName: Pull MegaLinter
@@ -192,7 +195,6 @@ Add the following job in your `azure-pipelines.yaml` file
       - script: |
           docker run -v $(System.DefaultWorkingDirectory):/tmp/lint \
             --env-file <(env | grep -e SYSTEM_ -e BUILD_ -e TF_ -e AGENT_) \
-            -e CI=true \
             -e SYSTEM_ACCESSTOKEN=$(System.AccessToken) \
             -e GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \
             oxsecurity/megalinter:v6
@@ -414,10 +416,9 @@ See [mega-linter-runner installation instructions](https://megalinter.io/mega-li
 Example
 
 ```shell
-npx mega-linter-runner --flavor salesforce -e 'ENABLE=,DOCKERFILE,MARKDOWN,YAML' -e 'SHOW_ELAPSED_TIME=true'
+npx mega-linter-runner --flavor salesforce -e "'ENABLE=DOCKERFILE,MARKDOWN,YAML'" -e 'SHOW_ELAPSED_TIME=true'
 ```
 
 Note: You can also use such command line in your custom CI/CD pipelines
 
-
 <!-- installation-section-end -->
diff --git a/docs/reporters/AzureCommentReporter.md b/docs/reporters/AzureCommentReporter.md
@@ -23,7 +23,6 @@ Example:
       - script: |
           docker run -v $(System.DefaultWorkingDirectory):/tmp/lint \
             --env-file <(env | grep -e SYSTEM_ -e BUILD_ -e TF_ -e AGENT_) \
-            -e CI=true \
             -e SYSTEM_ACCESSTOKEN=$(System.AccessToken) \
             -e GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \
             oxsecurity/megalinter:v6