Roadmap for v1.0.0

[bauersimon]

Highlights 🌟

• Spring Boot 🥬 unit test support: evaluating how models generate tests for Spring
• Code Migration 🔁 task: asking models to migrate i.e. tests from JUnit 4 to JUnit 5

Changes 💡

• Development & Management 🛠️
  • Linters in development environment and CI Include linters in the development environment and CI #18 Install and run some Go linters #355 fix, Use all the rules of Go linter "revive" but exclude some that do not make sense and exit "1" on errors, always #359 Lint unused code in GitHub Actions #377 @zimmski
  • Bump golang.org/x/image from 0.11.0 to 0.18.0 Bump golang.org/x/image from 0.11.0 to 0.18.0 #340 #@dependabot
  • VSCode recommended extensions and debugging configuration VSCode recommended extensions and debugging configuration #361 @bauersimon
  • Enable Symflower linter by default in VSC Enable Symflower linter by default in VSC #362 @ruiAzevedo19
  • Script for tagging and releasing Automate revision string of the binary #344 Release script for tagging and updating the binary version #391 @bauersimon
  • Add Symflower License in GitHub Actions so the CI works again Add License in GitHub Actions so the CI works again #389 Upgrade Symflower to v45435 to include license changes #393 @zimmski
  • Clean up old container images Clean up container images on GitHub #394 fix, Do not clean up the "main" or "latest" container image tags because the point to the latest stable version #395 @zimmski
  • Tag container images of releases Build container images on tagging releases #396 @zimmski
• Documentation 📚
  • Add issue templates for bug and feature requests Add issue templates for bug and feature requests #338 Issue templates for bugs and feature requests #342 Overhaul bug and feature request issue templates #345 fix, Input form template does not allow special rendering #346 @bauersimon
  • Improve wording of templates and documentation Improve wording of templates and documentation #343 @bauersimon
  • Update symflower.com with DevQualityEval entries Update symflower.com with DevQualityEval entries #339 Fix typos in Ruby README paths and update to new deep dive #348 @zimmski
  • Roadmap issue template clarification on workflow and payment page Clarify roadmap/release process #388 @bauersimon
  • Roadmap issue template clarification on how to use the roadmap issue itself to create release notes Clarify order of tasks for a release #397
• Evaluation ⏱️
  • refactor, Clarify wording, logs, debugging regarding symflower fix Clarify wording, logs, debugging regarding symflower fix #352 @bauersimon
  • refactor, Simplify assessment cleanup for tests Simplify assessment cleanup for tests #353 @ruiAzevedo19
  • Cleanup logging and log to JSON for easier parsing JSON, structured logging #379 @bauersimon
  • fix, Solving Plain once (with or without template) is enough to not get disqualified fix, Solving Plain once (with or without template) is enough to not get disqualified #360 @bauersimon
• Models 🤖
  • none
• Reports & Metrics 🗒️
  • Write run count to CSV report Write run count to CSV report #337 @bauersimon
  • Remove category SVG from the report, because we are moving to a leaderboard Remove category SVG from the report, because we are moving to a leaderboard #356 @zimmski
  • Remove categorization and total score of reporting, and the "report" command Remove categorization and total score of reporting, and the "report" command #357 @zimmski
  • Remove scoring from within the evaluation and remove and lint unused functions Remove scoring from within the evaluation and remove and lint unused functions #369 @bauersimon
  • Remove assessment collection Remove assessment collection #378 @bauersimon
  • Max scores per case in repository Max scores per case in repository #390 Save maximum scores for cases and write CSV per case #392 @bauersimon
• Operating Systems 🖥️
  • none
• Tools 🧰
  • Script to convert all SVG files to PNG with Inkscape Script to convert all SVG files to PNG with Inkscape #349 @bauersimon
  • Script to find max coverage score from result data Script to find max coverage score from result data #380 @bauersimon
• Tasks 🔢
  • "Write-test" task with Symflower Smart Test Template as base "Write-test" task with Symflower Smart Test Template as base #350 "Write-test" task with Symflower Smart Test Template as base #351 Always apply "symflower fix" and refactor "write test" task and LLM prompting to prepare for templates #354 "Write-test" task with Symflower Smart Test Template as base #358 @bauersimon
  • Spring Boot "write-test" Spring Boot "write-test" #365 Support Spring #367 @bauersimon
  • Migration from JUnit4 to JUnit5 Migration from JUnit4 to JUnit5 #375 New task for code migration #376 @ruiAzevedo19

Details 🔍

• Take details from automatic release description of GitHub.

Tasks of this

❓ When should a release happen? Check the README!

• Do a full evaluation with the version
  • Exclude certain Openrouter models by default
    • nitro cause they are just faster
    • extended cause longer context windows don't matter for our tasks
    • free and auto cause these are just "aliases" for existing models
  • Exclude special-purpose models
    • Vision models
    • Roleplay and creative writing models
    • Classification models
    • Models with internet access (usually denoted by -online suffix)
    • Models with extended context windows (usually denoted by -1234K suffix)
  • Always prefer fine tuned (-instruct, -chat) models over a plain base model
• Tag version (tag can be moved in case important merges happen afterwards)
• For all issues of the current milestone, one by one, add them to the roadmap tasks (it is ok if a task has multiple issues) with the users that worked on it
  • Fixed bugs should always be sorted into respective relevant categories and not in a generic "Bugs" category!
• For all PRs of the current milestone, one by one, add them to the roadmap tasks (it is ok if a task has multiple issues) with the users that worked on it
  • Fixed bugs should always be sorted into respective relevant categories and not in a generic "Bugs" category!
• Search all issues for ...
  • Unassigned issues that are closed, and assign them someone
  • Issues without a milestone, and assign them a milestone
  • Issues without a label, and assign them at least one label
• Write the release notes @ https://github.com/symflower/eval-dev-quality/releases
  • Use the tasks that are already there for the release note outline
  • Add highlighted features based on the done tasks, sort by how many users would use the feature
• Do the release
  • With release notes
  • Set as latest release
• Prepare the next roadmap
  • Create a milestone for the next release
  • Create a new roadmap issue for the next release
    • Move all open tasks/TODOs from this roadmap issue to the next roadmap issue.
    • Move every comment of this roadmap issue as a TODO to the next roadmap issue. Mark when done with a 🚀 emoji.
• Blog post containing evaluation results, new features and learnings
  • Update README with blog post link and new header image
  • Update repository link with blog post link
  • https://github.com/symflower/eval-dev-quality/discussions
    • Remove the previous announcements
    • Add a "Deep dive: $blog-post-title" announcement for the blog post, unpin all others and pin this one
    • Add a "v$version: $summary-of-highlights" announcement for the release, unpin all others and pin this one
  • symflower.com
    • Update "latest DevQualityEval deep dive" mentions
    • Update DevQualityEval blog series lists with new entries
    • Update LLM blog series lists with new entries
  • Update payment process for supporting DevQualityEval
    • New Stripe payment link for this version
    • Update payment logic with new Google Drive folder of the evaluation
    • Update Payment link in this README
    • Update Payment link on symflower.com (except for the one deep dive that mentions exactly these results)
  • Create an issue in the company tracker for Markus to announce the new deep dive on Twitter and LinkedIn
• Close this issue
• Close the current milestone
• Announce release
• Eat cake 🎂