Skip to content
This repository has been archived by the owner on Jun 9, 2024. It is now read-only.

added --test, consolidate files, reports working #83

Merged
merged 4 commits into from
Jul 11, 2023
Merged

Conversation

SilenNaihin
Copy link
Contributor

Background

Some quality of life for users and preparation for writing paper.

Changes

  • --test TestWriteFile just runs a single test specified such as TestWriteFile in that example
  • instead of having config.json, regression_tests.json, 'benchmarks.py', and reports/ all in seperate files, they are now under the agbenchmark folder. Moved local files (regression and config) to agbenchmark/, they are now just used for mocks. If you'd like to run a specific agent, navigate to a submodule and run it from there
  • Reports! Simple example:
{
    "command": "agbenchmark start --test TestWriteFile --mock",
    "completion_time": "2023-07-10-21:32",
    "time_elapsed": "0.29 seconds",
    "tests": {
        "TestWriteFile": {
            "difficulty": "basic",
            "dependencies": [],
            "test": "agbenchmark\\challenges\\interface\\write_file",
            "success": true
        }
    },
    "config": {
        "workspace": "autogpt/workspace/auto_gpt_workspace",
        "entry_path": "benchmarks.py",
        "home_path": "",
        "cutoff": 60
    }
}

PR Quality Checklist

  • I have run the following commands against my code to ensure it passes our linters:
    black .
    isort .
    mypy .
    autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports --ignore-pass-after-docstring --in-place agbenchmark

Copy link

@ai-maintainer ai-maintainer bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution! 

This pull request is too large for automated review.

@waynehamadi waynehamadi merged commit 8df8290 into master Jul 11, 2023
@waynehamadi waynehamadi deleted the feat/reports branch July 11, 2023 02:25
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants