Merge 'main' into feature/multilingual

Signed-off-by: Jeffrey Martin <[email protected]>

.github/CODEOWNERS

@@ -0,0 +1 @@
+/.github/ @leondz @erickgalinkin @jmartin-tech

.github/ISSUE_TEMPLATE/bug_report.md

@@ -10,7 +10,7 @@ labels: "bug"
   Useful Links:
   - Wiki: https://docs.garak.ai/garak
 
-  Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
+  Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues
 -->
 
 ## Steps to reproduce

.github/ISSUE_TEMPLATE/documentation.md

@@ -11,7 +11,7 @@ labels: "documentation"
   - Wiki: https://docs.garak.ai/garak
   - Code reference: https://reference.garak.ai/
 
-  Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
+  Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues
 -->
 
 ## Summary

.github/ISSUE_TEMPLATE/feature_suggestion.md

@@ -10,7 +10,7 @@ labels: "enhancement"
   Useful Links:
   - Wiki: https://docs.garak.ai/garak
 
-  Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
+  Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues
 -->
 
 ## Summary

.github/ISSUE_TEMPLATE/plugin_suggestion.md

@@ -10,7 +10,7 @@ labels: "new-plugin"
   Useful Links:
   - Wiki: https://docs.garak.ai/garak
 
-  Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
+  Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues
 -->
 
 ## Summary

.github/ISSUE_TEMPLATE/question.md

@@ -11,7 +11,7 @@ labels: "question"
   - Wiki: https://docs.garak.ai/garak
   - Code reference: https://reference.garak.ai/
 
-  Before opening a new issue, please search existing issues https://github.com/leondz/garak/issues
+  Before opening a new issue, please search existing issues https://github.com/NVIDIA/garak/issues
 -->
 
 ## Summary

.github/PULL_REQUEST_TEMPLATE.md

@@ -20,7 +20,7 @@ List the steps needed to make sure this thing works
 - [ ] ...
 - [ ] **Verify** the thing does what it should
 - [ ] **Verify** the thing does not do what it should not
-- [ ] **Document** the thing and how it works ([Example](https://github.com/leondz/garak/blob/61ce5c4ae3caac08e0abd1d069d223d8a66104bd/garak/generators/rest.py#L24-L100))
+- [ ] **Document** the thing and how it works ([Example](https://github.com/NVIDIA/garak/blob/61ce5c4ae3caac08e0abd1d069d223d8a66104bd/garak/generators/rest.py#L24-L100))
 
 If you are opening a PR for a new plugin that targets a **specific** piece of hardware or requires a **complex or hard-to-find** testing environment, we recommend that you send us as much detail as possible.
 

.github/workflows/cla.yml

@@ -14,22 +14,22 @@ permissions:
 
 jobs:
   CLAAssistant:
-    if: github.repository_owner == 'leondz'
+    if: github.repository_owner == 'NVIDIA'
     runs-on: ubuntu-latest
     steps:
       - name: "CA & DCO Assistant"
         if: (github.event.comment.body == 'recheck' || github.event.comment.body == 'I have read the Contributor Agreement including DCO and I hereby sign the Contributor Agreement and DCO') || github.event_name == 'pull_request_target'
-        uses: contributor-assistant/[email protected]
+        uses: cla-assistant/[email protected]
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           # the below token should have repo scope and must be manually added by you in the repository's secret
           # This token is required only if you have configured to store the signatures in a remote repository/organization
           PERSONAL_ACCESS_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
         with:
           path-to-signatures: 'signatures/cla.json'
-          path-to-document: 'https://github.com/leondz/garak/blob/main/CA_DCO.md' # e.g. a CLA or a DCO document
+          path-to-document: 'https://github.com/NVIDIA/garak/blob/main/CA_DCO.md' # e.g. a CLA or a DCO document
           # branch should not be protected
-          branch: 'main'
+          branch: 'signatures'
           use-dco-flag: true
           allowlist: 
 

.github/workflows/labels.yml

@@ -26,7 +26,7 @@ on:
 
 jobs:
   handle-labels:
-    if: github.repository_owner == 'leondz'
+    if: github.repository_owner == 'NVIDIA'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/github-script@v7
@@ -81,7 +81,7 @@ jobs:
                     git push origin <BRANCH_NAME>
 
                     # Now browse to the following URL and create your pull request!
-                    # - https://github.com/leondz/garak/pulls
+                    # - https://github.com/NVIDIA/garak/pulls
                     \`\`\`
 
                     This helps protect the process, ensure users are aware of commits on the branch being considered for merge, allows for a location for more commits to be offered without mingling with other contributor changes and allows contributors to make progress while a PR is still being reviewed.
@@ -119,7 +119,7 @@ jobs:
 
                     This includes:
 
-                    - All of the item points within this [template](https://github.com/leondz/garak/blob/master/.github/ISSUE_TEMPLATE/bug_report.md)
+                    - All of the item points within this [template](https://github.com/NVIDIA/garak/blob/master/.github/ISSUE_TEMPLATE/bug_report.md)
                     - Screenshots showing the issues you're having
                     - Exact replication steps
 
@@ -131,7 +131,7 @@ jobs:
                   close: true,
                   comment: `
                     When creating an issue, please ensure that the default issue template has been updated with the required details:
-                    https://github.com/leondz/garak/issues/new/choose
+                    https://github.com/NVIDIA/garak/issues/new/choose
 
                     Closing this issue. If you believe this issue has been closed in error, please provide any relevant output and logs which may be useful in diagnosing the issue.
                   `

.github/workflows/maintain_cache.yml

@@ -19,7 +19,7 @@ permissions:
 
 jobs:
   build:
-    if: github.repository_owner == 'leondz'
+    if: github.repository_owner == 'NVIDIA'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3

.github/workflows/nightly.yml

@@ -0,0 +1,24 @@
+name: Nightly Testing
+
+on:
+  schedule:
+    - cron: "0 0 * * *"
+  workflow_dispatch:
+
+jobs:
+  linux:
+    name: Nightly Linux
+    if: github.repository_owner == 'NVIDIA'
+    uses: ./.github/workflows/test_linux.yml
+  windows:
+    name: Nightly Windows
+    if: github.repository_owner == 'NVIDIA'
+    uses: ./.github/workflows/test_windows.yml
+  macos:
+    name: Nightly MacOS
+    if: github.repository_owner == 'NVIDIA'
+    uses: ./.github/workflows/test_macos.yml
+  package_test:
+    name: Nightly Packaging
+    if: github.repository_owner == 'NVIDIA'
+    uses: ./.github/workflows/remote_package_install.yml

.github/workflows/remote_package_install.yml

@@ -0,0 +1,34 @@
+name: Garak pip - install from repo
+
+on:
+  push:
+    branches:
+      - 'main'
+  pull_request:
+  workflow_dispatch:
+  workflow_call:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10","3.12"]
+    steps:
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: pip install from repo
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install -U git+https://github.com/${GITHUB_REPOSITORY}.git@${GITHUB_SHA}
+      - name: Sanity Test
+        run: |
+          python -m garak --model_type test.Blank --probes test.Test
+          set +e
+          grep -E "(WARNING|ERROR|CRITICAL)" $HOME/.local/share/garak/garak.log
+          if [ $? != 1 ]; then
+            echo "Errors exist in the test log"
+            exit 1
+          fi

.github/workflows/test_linux.yml

@@ -6,21 +6,10 @@ on:
       - 'main'
   pull_request:
   workflow_dispatch:
+  workflow_call:
 
 jobs:
-  pre_job:
-    runs-on: ubuntu-latest
-
-    steps:
-      - id: skip_check
-        uses: fkirc/skip-duplicate-actions@v5
-        with:
-          concurrent_skipping: 'outdated_runs'
-          cancel_others: 'true'
-
   build:
-    needs: pre_job
-    if: needs.pre_job.outputs.should_skip != 'true'
     runs-on: ubuntu-latest
     strategy:
       matrix:

.github/workflows/test_macos.yml

@@ -6,6 +6,7 @@ on:
       - 'main'
   pull_request:
   workflow_dispatch:
+  workflow_call:
 
 jobs:
   build_macos:

.github/workflows/test_windows.yml

@@ -6,6 +6,7 @@ on:
       - 'main'
   pull_request:
   workflow_dispatch:
+  workflow_call:
 
 jobs:
   build_windows:

.gitignore

@@ -164,6 +164,7 @@ garak.*.jsonl
 garak.log
 hitlog.*.jsonl
 .vscode
+.history/
 garak_runs/
 runs/
 logs/

CONTRIBUTING.md

@@ -31,19 +31,21 @@ And if you like the project, but just don't have time to contribute, that's fine
 
 If you want to ask a question, good places to check first are the [garak quick start docs](https://docs.garak.ai) and, if its a coding question, the [garak reference](https://reference.garak.ai/).
 
-Before you ask a question, it is best to search for existing [Issues](https://github.com/leondz/garak/issues) that might help you. In case you have found a suitable issue and still need clarification, you can write your question in this issue. It is also advisable to search the internet for answers first. You can also often find helpful people on the garak [Discord](https://discord.gg/uVch4puUCs).
+Before you ask a question, it is best to search for existing [Issues](https://github.com/NVIDIA/garak/issues) that might help you. In case you have found a suitable issue and still need clarification, you can write your question in this issue. It is also advisable to search the internet for answers first. You can also often find helpful people on the garak [Discord](https://discord.gg/uVch4puUCs).
 
 
 If you then still feel the need to ask a question and need clarification, we recommend the following:
 
-- Open an [Issue](https://github.com/leondz/garak/issues/new).
+- Open an [Issue](https://github.com/NVIDIA/garak/issues/new).
 - Provide as much context as you can about what you're running into.
 - Provide project and platform versions (nodejs, npm, etc), depending on what seems relevant.
 
 We will then take care of the issue as soon as possible.
 
 ## I Want To Contribute
 
+Check out our [guide to contributing garak code](https://reference.garak.ai/en/latest/contributing.html). It's also important to make sure that your contribution is within scope of the project - it's so sad to not merge good code which does things outside of where `garak` is going. Check the [FAQ](FAQ.md) for info on what `garak` is and isn't. If you have any doubt, get in touch!
+
 ### Legal Notice <!-- omit in toc -->
 When contributing to this project, you must agree that you have authored 100% of the content, that you have the necessary rights to the content and that the content you contribute may be provided under the project license. All contributors are asked to sign the garak CA/DCO (contributor agreement/developer certificate of origin) when sending their first pull request. The process is automated by a bot that will comment on the pull request. We use the same DCO as the Linux Foundation requires its contributors to sign.
 
@@ -56,7 +58,7 @@ A good bug report shouldn't leave others needing to chase you up for more inform
 
 - Make sure that you are using the latest version.
 - Determine if your bug is really a bug and not an error on your side e.g. using incompatible environment components/versions (Make sure that you have read the [documentation](https://reference.garak.ai/). If you are looking for support, you might want to check [this section](#i-have-a-question)).
-- To see if other users have experienced (and potentially already solved) the same issue you are having, check if there is not already a bug report existing for your bug or error in the [bug tracker](https://github.com/leondz/garak/issues?q=label%3Abug).
+- To see if other users have experienced (and potentially already solved) the same issue you are having, check if there is not already a bug report existing for your bug or error in the [bug tracker](https://github.com/NVIDIA/garak/issues?q=label%3Abug).
 - Also make sure to search the internet (including Stack Overflow) to see if users outside of the GitHub community have discussed the issue.
 - Collect information about the bug:
   - Stack trace (Traceback)
@@ -73,7 +75,7 @@ You should never report security related issues, vulnerabilities or bugs includi
 
 We use GitHub issues to track bugs and errors. If you run into an issue with the project:
 
-- Open an [Issue](https://github.com/leondz/garak/issues/new). (Since we can't be sure at this point whether it is a bug or not, we ask you not to talk about a bug yet and not to label the issue.)
+- Open an [Issue](https://github.com/NVIDIA/garak/issues/new). (Since we can't be sure at this point whether it is a bug or not, we ask you not to talk about a bug yet and not to label the issue.)
 - Explain the behavior you would expect and the actual behavior.
 - Please provide as much context as possible and describe the *reproduction steps* that someone else can follow to recreate the issue on their own. This usually includes your code. For good bug reports you should isolate the problem and create a reduced test case.
 - Provide the information you collected in the previous section.
@@ -96,14 +98,14 @@ This section guides you through submitting an enhancement suggestion for garak,
 
 - Make sure that you are using the latest version.
 - Read the [documentation](https://reference.garak.ai/) carefully and find out if the functionality is already covered, maybe by an individual configuration.
-- Perform a [search](https://github.com/leondz/garak/issues) to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.
+- Perform a [search](https://github.com/NVIDIA/garak/issues) to see if the enhancement has already been suggested. If it has, add a comment to the existing issue instead of opening a new one.
 - Find out whether your idea fits with the scope and aims of the project. It's up to you to make a strong case to convince the project's developers of the merits of this feature. Keep in mind that we want features that will be useful to the majority of our users and not just a small subset. If you're just targeting a minority of users, consider writing an add-on/plugin library.
 - Check out our [guide for contributors](https://reference.garak.ai/en/latest/contributing.html), which includes our coding workflow and a [guide to constructing a plugin](https://reference.garak.ai/en/latest/contributing.generator.html).
 
 <!-- omit in toc -->
 #### How Do I Submit a Good Enhancement Suggestion?
 
-Enhancement suggestions are tracked as [GitHub issues](https://github.com/leondz/garak//issues).
+Enhancement suggestions are tracked as [GitHub issues](https://github.com/NVIDIA/garak/issues).
 
 - Use a **clear and descriptive title** for the issue to identify the suggestion.
 - Provide a **step-by-step description of the suggested enhancement** in as many details as possible.
@@ -141,4 +143,4 @@ Updating, improving and correcting the documentation
 
 <!-- omit in toc -->
 ## Attribution
-This guide is based on the **contributing-gen**. [Make your own](https://github.com/bttger/contributing-gen)!
+This guide is based on the **contributing-gen**. [Make your own](https://github.com/bttger/contributing-gen)!