Evaluation for the paper #69
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e tool and the tutor tool
GirZ0n
approved these changes
Jul 27, 2021
vyahhi
added a commit
that referenced
this pull request
Sep 22, 2021
* wps-light support (#17) Add wps-light support * New flake8 plugins support (#18) Added support for flake8-broken-line, flake8-string-format, flake8-commas (they are WPS dependencies). Added support for cohesion. * Radon support (#19) Added new inspector: Radon. Added maintainability index. * Requirements upgrade (#21) Updated versions of dependencies Added django dictionary support Fixed tests * xlsx-run-tool (#24) Added xlsx_tool_run.py – script to run the tool on multiple code samples stored in xlsx file * Penalty system (#25) Add penalty system * Penalty coefficients update (#28) * Updated penalty coefficients; * Added new tests; * Fixed bug with _get_issue_class_to_influence, which caused the dictionary to get issues that should not be penalized. * Fixed two cases in test_categorize. There were two identical issues in the list of previously made issues. * Update version to 1.2.0 * Evaluation/code quality (#29) * Refactor the tool for evaluation: add multithreading, add a possibility to handle CSV arguments * Add script to filter solutions by language and drop duplicates * Add script to distribute grades from unique solutions to all solutions * Add script to find diffs between two graded dataframes * Add tests and descriptions * Rename output_format to format * Fix double quotes * Fix small issues * Fix small issues * Fix double quotes * Fix double quotes * Inspectors fix (#39) * Add origin class for maintainability index * Add WPS518 to ignore due to collision with C0200 by Pylint * New category (#38) * Added a new category INFO, in order not to take into account some issues in the evaluation. * Dataset labeling utility (#31) Added a utility for dataset marking * resolved merge conflicts * Fixed trailing commas * Delete create_directory function * Delete create_directory function * Update version * Graph plotter (#41) Added a script for plots * updated whitelist to resolve merge conflicts * Qodana simulation model (#40) * Add small dataset preparation block, train, and evaluation module to train and run Roberta-model for solving a multilabel classification task * Qodana stat (#42) Add a script to convert the data received by the Qodana into the format of the Hyperstyle tool for analysis and statistics gathering. * Delete Path from setup.py * Change exit code * Update whitelist * Inspector statistics (#48) * Calculate statistics for inspectors for four main categories of issues * Fix GitHub actions (#44) * Use the same environment as in the production dockerfile * Fix Dockerfiles * Install some dependencies for TeamCity * Update version in Github Actions * Reorganize dependencies and update dockerfile * History script (#47) * Add script that allows you to generate history based on issues from previous solutions * Fixed incorrect file name (#51) * Corrected the name of the dataset labeling script in the README * Distributed the issue categories (#53) * Distributed the issue categories * Qodana imitation model resources (#55) * Add Qodana imitation model resources * Small Github Actions config update (#56) * Update build.yml * Whitelist for flake8-spellcheck (#57) * Added whitelist * PMD update (#59) * Updated PMD to v6.36.0 * Small fix * Undo f4ab486 * Checkstyle update (#60) * Checkstyle update * Categorization update (#61) * Updated categorization for PMD and Checkstyle * New tests and documentation (#63) * Added tests for functions parsing PMD and Checkstyle output. * Documented methods related to PMD and Checkstyle. * Merge main into develop (#65) * Merge main into develop (#65) * Script for getting raw issues (#66) * Added Encoder and Decoder for 'raw' issues * Added test for RawIssueEncoder and RawIssueDecoder * Added __init__.py * Added get_raw_issues.py * Added __init__.py * Added to_safe_path flag * Fixed flake8 issue * Added test for get_raw_issues.py * Small fix * Fixed get_output_path test data * get_language -> get_language_version * Added ISSUE_TYPE_TO_CLASS and MEASURABLE_ISSUE_TYPE_TO_MEASURE_NAME * Fixed PR issue * Added LanguageVersion.JS * Added LanguageVersion.JS and LanguageVersion.JAVA_15 * Fixed a bug due to which all cyclomatic complexity issues were ignored * Added bad cyclomatic complexity test * Fixed test * Added LineLenIssue * Removed unnecessary time column * Update JS tests * Added README * Small fixes in README * Small fix * Fixed --allow-info-issues description * Fixed test * Fixed test * Fixed PR issue Co-authored-by: Anastasiia.Birillo <[email protected]> * Evaluation for the paper (#69) * Add evaluation for the paper: student dynamics and statistics abut the tool and the tutor tool * Fix flake8 * Rename statistics * statistics -> issues_statistics * Fixed flake8 * Undo some renaming Co-authored-by: Ilya Vlasov <[email protected]> * Script for extracting statistics from raw issues (#76) * Fixed get_output_path test data * Moved __get_total_lines to file_system.py * Added get_total_code_lines_from_file and get_total_code_lines_from_code * Added get_raw_issues_statistics * Renamed main_stats -> freq_stats and other_stats -> ratio_stats * Small fix * Added new data folders * Added tests * Removed duplicates * Removed unnecessary line * Now the script returns only one dataframe * Fixed tests * Added new tests * Added logger and small code refactoring * Added some more logging * Fixed test * Fixed test * Fixed help message * Update README.md * statistics -> issues_statistics * Fixed flake8 * Added from_value function * Added comment * Added get_ratio * Small refactoring: added get_ratio * Fixed PR issues * Small fixes * Added isnull * typo fix * Fixed tests * Added --log-output and fixed null checks * Added filemode * typo fix * Fixed tests * Fixed tests * Survey analysis (#84) * Add scripts for surveys statistics gathering * Fix flake8 * Issues fix (#88) * Fixed issue #73 * Fixed #75 * Small fix * Small fix * Sorted the whitelist.txt * Sorted whitelist.txt * Fixed tests * Fixed flake8 * Fixed get_raw_issues script (#86) * Fixed get_output_path test data * Moved __get_total_lines to file_system.py * Added get_total_code_lines_from_file and get_total_code_lines_from_code * Added get_raw_issues_statistics * Renamed main_stats -> freq_stats and other_stats -> ratio_stats * Small fix * Added new data folders * Added tests * Removed duplicates * Removed unnecessary line * Now the script returns only one dataframe * Fixed tests * Added new tests * Added logger and small code refactoring * Added some more logging * Fixed test * Fixed test * Fixed help message * Update README.md * Added more logging * statistics -> issues_statistics * Fixed flake8 * Added from_value function * Added comment * Added get_ratio * Small refactoring: added get_ratio * Fixed PR issues * Small fixes * Added isnull * typo fix * Fixed tests * Added --log-output and fixed null checks * Added filemode * typo fix * Fixed tests * Fixed tests * Code refactoring and bug fixing: - Added checks for columns 'code' and 'lang' for null - Now if flake8, pylint or WPS terminates with an error, this is logged - When creating a log file, all parent folders are created if necessary - Now the new log file overwrites the old one * Fixed test * Added new test * Replaced None with np.nan * Fixed test * Removed unnecessary space * Corrected dict errors and added types for Dict (#98) * Develop Branch Commit * Added Types for Dict * Added Any to typing * User dynamics (#95) * Add scripts for surveys statistics gathering * Fix flake8 * Add dynamics handlers * Fix flake8 * Sort whitelist * Fix evaluation run tool script * Use to_drop_nan arg * Fix tests * Fix PR#95 comments * Fix tests * `print_review.py` refactoring (#96) * Code refactoring: 1) Renamed ReviewResult -> GeneralReviewResult. 2) Added ReviewResult from which GeneralReviewResult and FileReviewResult are now inherited. 3) Put common code when printing GeneralReviewResult in different formats in the convert_review_result_to_json_dict function. 4) Also prepared the code for further addition of difficulty levels. 5) Fixed the test: now the json schema is checked more strictly. * get_influence_json_dict -> get_influence_on_penalty_json_dict * Fixed the old test and added a new one * typo fix * Renamed file * Fixed quotes * Added a new test and refactored the old one * Added typing * Update README.md * Undo Update README.md * Fixed PR issues * User dynamics visualization (#101) * Add scripts for surveys statistics gathering * Fix flake8 * Add dynamics handlers * Fix flake8 * Sort whitelist * Fix evaluation run tool script * Use to_drop_nan arg * Fix tests * Fix PR#95 comments * Fix tests * Add visualization * Fix flake8 * Fix plots * Fix plots * Delete print * Raw issues statisitcs plotter (#99) * Moved to plotters/ * Added line plot and histogram * Updated whitelist * Added plotters * Added script * Added the ability to draw horizontal lines when plotting box plot * Added boxplot * Fixed flake8 issues * Update README.md * create_line_plot -> create_line_chart * Added examples and fixed small errors * Fixed imports * Added sys.path * Difficulty levels (#103) * Code refactoring: 1) Renamed ReviewResult -> GeneralReviewResult. 2) Added ReviewResult from which GeneralReviewResult and FileReviewResult are now inherited. 3) Put common code when printing GeneralReviewResult in different formats in the convert_review_result_to_json_dict function. 4) Also prepared the code for further addition of difficulty levels. 5) Fixed the test: now the json schema is checked more strictly. * get_influence_json_dict -> get_influence_on_penalty_json_dict * Fixed the old test and added a new one * typo fix * Renamed file * Fixed quotes * Added a new test and refactored the old one * Added typing * Added group_issues_by_difficulty * Added --group-by-difficulty flag * Added IssueDifficulty * Added support for difficulty levels * Small fix * Fixed tests * Added IssueDifficulty * Added difficulty * Fixed tests * Fixed tests * Replaced literals with constants and added group_by_difficulty * Small fix * Fixed test and added a new one * Update README.md * Undo Update README.md * Fixed PR issues * Added tests * Added history * Added new tests and updated old ones * Update README.md * Fixed flake8 * Small fixes * Small fix * Small fix * Fixed flake8 * Small fix * Small fix * Added logger * Fixed todo message * Update traces and count percentage (#105) * Update traces and count percentage * Fix spellcheck * Issues fix (#107) * Update build.yml 1) actions/checkout@v1 -> actions/checkout@v2. 2) Added git. 3) Added automatic sorting of whitelists. * Update whitelist.txt * Sort whitelists (Github Actions) Co-authored-by: GirZ0n <[email protected]> * Raw issues statistics plotter fix (#108) * Added Extension.HTML * Added COLORWAY * Added new words * Added create_box_trace and create_scatter_trace * Added --group-stats and colorway * Added the ability to show several languages on one graph at once * Sort whitelists (Github Actions) * Small code refactoring * Small code refactoring * Update README.md * Small fixes * Updated examples Co-authored-by: GirZ0n <[email protected]> * Visualization of raw issues statistics for the paper (#110) * Added Extension.HTML * Added COLORWAY * Added new words * Added create_box_trace and create_scatter_trace * Added --group-stats and colorway * Added the ability to show several languages on one graph at once * Sort whitelists (Github Actions) * Small code refactoring * Small code refactoring * Update README.md * Small fixes * Updated examples * Added script * Small code refactoring * Added README and examples * Removed width * Changed width * Changed width * Fixed PR issues Co-authored-by: GirZ0n <[email protected]> * Delete evaluation part (#113) * Delete evaluation * Update build * Fix tests * Rename: review -> hyperstyle * Add tests * Delete functions * Merged `main` into `develop` (#114) * Fix GitHub actions (#46) * Update requirements and dockerfile * Use Dockerfile in Github Action * Ignore a part of flake8 issues * Ignore flake8 inspections * Separate docker file into two: for prod and for dev * Delete ` * Fix test_range_of_lines * Fix pylint and flake8 tests * Fix pylint tests * Check styles by flake8 * Fix C408 issue * Move production Dockerfile * Fix Dockefile for production * Set up Eslint in the DockerFile for production * Delete one case from flake8 tests * Add pathlib into whitelist * Delete Path from setup.py * Ignore pathlib * Don't count INFO issues * Fix flake8 tests * Small fix * Delete pathlib from the whitelist * Update docker (#54) * Use one docker file for prod and dev * Change base docker image * Fix requirements-test: delete duplicates, setup versions * Try to fix dockerfile * Fix run in subprocess function * Fix Dockerfile * Try to fix Dockerfile * Fix docker image (#58) * Change eslint installing * Update build.yaml * Introduce base docker image and auto-detect eslint path Co-authored-by: Andrey Balandin <[email protected]> Closes #43, closes #45. * Add whitelist for flake8-spellcheck (#68) * Add whitelist for flake8-spellcheck * Add a test for spellcheck and ignore unnecessary rule * Fix flake8 config * Bump version 1.1.0 -> 1.1.1 * Add Python inspections and a script for tool evaluation (#26) * wps-light support (#17) Add wps-light support * New flake8 plugins support (#18) Added support for flake8-broken-line, flake8-string-format, flake8-commas (they are WPS dependencies). Added support for cohesion. * Radon support (#19) Added new inspector: Radon. Added maintainability index. * Requirements upgrade (#21) Updated versions of dependencies Added django dictionary support Fixed tests * xlsx-run-tool (#24) Added xlsx_tool_run.py – script to run the tool on multiple code samples stored in xlsx file * Update version to 1.2.0 * Rename output_format to format * Fix double quotes * Fix small issues * Fix small issues * Fix double quotes * Fix double quotes * Inspectors fix (#39) * Add origin class for maintainability index * Add WPS518 to ignore due to collision with C0200 by Pylint * New category (#38) * Added a new category INFO, in order not to take into account some issues in the evaluation. * Fixed trailing commas * Delete create_directory function * Fix GitHub actions (#44) * Use the same environment as in the production dockerfile * Fix Dockerfiles * Install some dependencies for TeamCity * Update build * Add checks from teamcity * Update build * Fix indentation * Fix PR comments * Delete init files from resources * Add init in multi file project * Fix flake8 tests * Main upd bugs fix (#74) * Delete evaluation part * Fix bugs with detekt, pmd and flake8 * Issues fix (#80) * Fixed issue #70 * Fixed issue #72: now only spellcheck checks brackets * Fixed issue #73 * Fixed issue #77 * Fixed issue #78 * Fixed issue #71 * Fixed issue #79 * Fixed issue #81 * Added case 36: unpacking * Added case 37: wildcard import * Ignoring WPS347 and F405 * Added cases 36 (unpacking) and 37 (wildcard_import). Also removed W0622 (redefining_builtin) * Removed W0622 (redefining_builtin) * Issue #87 fix (#90) * Fixed issue #87 * Added test * Fixed issues #91 and #92 (#97) * Delete xlsx * Remove openpyxl * Issues fix (#104) * Fixed #100 * Fixed #102 * Added some more new words * Added toplevel * Update flake8 whitelist * Recovered accidentally deleted words Co-authored-by: Vlasov Ilya <[email protected]> Co-authored-by: Daria Diatlova <[email protected]> Co-authored-by: Nikolay Vyahhi <[email protected]> Co-authored-by: Ilya Vlasov <[email protected]> * Fixed typo * Sort whitelists (Github Actions) * Fixed tests * Fixed requirements * Added numpy * Removed numpy * Fixed test Co-authored-by: Nastya Birillo <[email protected]> Co-authored-by: Nikolay Vyahhi <[email protected]> Co-authored-by: Daria Diatlova <[email protected]> Co-authored-by: GirZ0n <[email protected]> * Issues fix (#117) * Fixed #109 * Fixed #112 * Fixed #116 * Updated `README` (#119) * Added output examples with --group-by-difficulty * typo fix * Delete analysis part (#120) * Pylint categorization update (#121) * Added some new words * Added W0511 * Added WPS428 * Added many exceptions * Sort whitelists (Github Actions) * Comments fix Co-authored-by: GirZ0n <[email protected]> * Detekt update (#123) * Updated detekt * Added new words * Sort whitelists (Github Actions) Co-authored-by: GirZ0n <[email protected]> * Issue fix (#127) * Fixed #122 * Sort whitelists (Github Actions) Co-authored-by: GirZ0n <[email protected]> * Delete evaluation resources * Delete Intellij and spotbugs inspectors (#131) * Delete intellij inspector * Delete spotbugs inspector * Fix teamcity build * Delete detekt sources (#132) * Delete intellij inspector * Delete spoibugs inspector * Download Detekt in Dockerfile * Install curl and unzip * Fix Dockerfile * Try to delete detekt sources * Check env variables * Try to get env variable * Use my docker image * Small code refactoring and also download detekt-jar file * Use my docker image * Use my docker image * Delete springlint inspector (#133) * Delete intellij inspector * Delete spoibugs inspector * Download Detekt in Dockerfile * Install curl and unzip * Fix Dockerfile * Try to delete detekt sources * Check env variables * Try to get env variable * Use my docker image * Small code refactoring and also download detekt-jar file * Use my docker image * Use my docker image * Delete springlint inspector * Delete checkstyle sources (#134) * Delete checkstyle sources * Fix curl command * Delete pmd sources (#135) * Delete PMD sources * Fix flake8 and echo env variables * Fix pmd tests * Delete ses evaluation resources * Add a guideline about tool installation (#137) Co-authored-by: Vlasov Ilya <[email protected]> Co-authored-by: Daria Diatlova <[email protected]> Co-authored-by: Nikolay Vyahhi <[email protected]> Co-authored-by: Ilya Vlasov <[email protected]> Co-authored-by: Nihal Shetty <[email protected]> Co-authored-by: GirZ0n <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add scripts for gathering statistics for the SIGCSE paper. In README you can find more information.