Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhanced game loading and running #4

Merged
merged 3 commits into from
Jan 15, 2025

Conversation

Gnurro
Copy link

@Gnurro Gnurro commented Jan 14, 2025

Adds enhanced GameSpec and CLI running functionality. Allows passing of underspecified GameSpec-like JSON to cli.py run -g to run unifying game registry entries, and benchmark version lists to run game registry entries with matching benchmark values. Batch running by these methods also prevents reloading models for each game. (Would enhance upcoming v2 runs.)

Commits: Starting on GameSpec and CLI enhancement; Add GameSpec.unify() method; Add custom game registry with three entries for the wordle variants for testing; Change clemgame.select_game() to return list of GameSpecs; Add unifying GameSpec check to clemgame.select_game(); Change clemcore.run() to iterate over list of GameSpecs; Change cli.py run -i default argument to None; Change GameBenchmark.setup() method to use CLI instances_name if given, but use GameSpec instances value otherwise; Add explicit handling of underspecified GameSpecs; Add handling of benchmark version lists in JSON GameSpec in CLI input - currently not compatible with underspecified GameSpec unification

…; Add custom game registry with three entries for the wordle variants for testing; Change clemgame.select_game() to return list of GameSpecs; Add unifying GameSpec check to clemgame.select_game(); Change clemcore.run() to iterate over list of GameSpecs; Change cli.py run -i default argument to None; Change GameBenchmark.setup() method to use CLI instances_name if given, but use GameSpec instances value otherwise; Add explicit handling of underspecified GameSpecs; Add handling of benchmark version lists in JSON GameSpec in CLI input - currently not compatible with underspecified GameSpec unification
@Gnurro
Copy link
Author

Gnurro commented Jan 14, 2025

This includes handling of the optional instances game registry key to specify different instances files to be loaded for different game variants, allowing their results directories to be structured as in prior versions of clembench. This change is needed to properly run updated wordle with variants.

clemcore/clemgame/game_registry_custom.json Outdated Show resolved Hide resolved
clemcore/clemgame/__init__.py Show resolved Hide resolved
…y_custom.json; Update clemcore/clemgame/__init__.py select_game() function docstring
@AnneBeyer
Copy link
Owner

merges added functionality for

  • calling games and collections by passing a json-like object to cli.py,
  • specifying instances file names through game_registry.

@AnneBeyer AnneBeyer merged commit c46c89d into AnneBeyer:game_registry Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants