Tracking Issue: Automated Test/Support Coverage Reporting

[PhilippvK]

As discussed on Friday, we need an (automated) flow to track and report which kinds of instructions are currently being supported by Seal5. As the project is quite ambitious, I came up with a number of sub-tasks which can be implemented on individually.

Tasks

1. Properties of Instructions

• Define relevant instruction properties #118
• Extract instruction properties from metamodel & Write instruction properties to file #117

2. Status/Support of Instructions in Seal5 Flow

• Extend pass metrics to emit names of failing/skipped instructions #119
• Write status/supported patches to file #120

3. Tracking LLVM Test Results

• Map test files to instructions #128
• Parse test results #121
• Store test results in metamodel #122
• Write test results to file #123
• Extract per-Instruction Test Coverage Data #143
• [ ] Define test files in Seal5 settings

4. Analysis and Grouping

• Merge artifacts (properties, status, tests) in single table #136
• Define useful filters/hierachy/… #139
• Automatically combine similar instructions (same properties) into groups #140
• Aggregate status/tests per group #141
• Integration of Test Coverage Data in Tables

5. Generation of Reports

• Propose visualization methods (static/dynamic, table/tree/grid?) #137
• [ ] Pick suitable visualization tool
• Generate Reports
• Publish reports

6. Automation (CI/CD)

• Define triggers (weekly/push/tag/…)
• Run Properties/Status/TestResults backend in CI #130
• Export generated CSV as CI Artifacts #131
• CI Summary in Markdown #132
• Add workflows to refresh/release reports
• [ ] GitHub Pages integration?

7. Miscellaneous

• Allow combining reports from multiple Seal5 runs/flows #133
• Increase number of available tests #129
• Tracking Issue: Automatic Generation of LLVM-Tests #18
• Increase number of example instructions/extensions

[PhilippvK]

@thomasgoodfellow @dlr-jas @kebi-be @AndHager Feel free to add any comments or assign any tasks you would like to work on.

Tomorrow I can share some more notes!

[PhilippvK]

I am making good progress.

@thomasgoodfellow @kebi-be @dlr-jas
Using the example CSV snippes I have posted in #117, #120 & #123 you should be able to get started with Task 4. & 5.

I will start working on some CI/CD changes to generate the CSVs automatically and look into adding more test cases!

[PhilippvK]

@kebi-be @dlr-jas @thomasgoodfellow @AndHager

I made quite good progress this week:

• Fixed broken Core-V Demo
• Fixed broken unit tests
• Implemented more unit tests
• Generate raw data artifacts in reports
• Write data as markdown to Workflow Summary

See here for the recent run: https://github.com/tum-ei-eda/seal5/actions/runs/11630150544

Based on this, we you can always get the latest csv files by downloading the demo-*-reports artifact. Since the data is already available, it should be easy to do the combination/analysis and generation of the final reports in a CI job.

Further, I added more properties which should help grouping/filtering the instructions:

• enc_size: 16/32/48bit
• is_compressed: enc_size == 16
• op_code: (OP-IMM, LOAD, custom-0,…)
• enc_format: (r-type, i-type, …)
• {uses,defs}_custom_reg: in case non-GPR registers are used, we can not generate patterns
• has_imm_leaf (not pushed yet): If the behavior contains operations on immediate inputs which can be constant folded, the generated pattern will never match. This detection should let us skip the pattern-gen stage for incompatible instructions.

[kebi-be]

Currently using the examples of the OpenASIP_base for Task4 and Task5. This concept will be used for analysing the data in the CI as mentioned by @PhilippvK

[kebi-be]

@PhilippvK: There is an error in the "test_results.cvs"data from the https://github.com/tum-ei-eda/seal5/actions/runs/11630150544 which results to error when merging the files. Used the corev demo reports.
Columns: model_name,set,xlen,instr,test_kind,test_file,test_fmt,result,model
Fix: rename model_name to model and delete redundant column "model" at the end

[PhilippvK]

I will take care of this today, thanks!

[PhilippvK]

@PhilippvK: There is an error in the "test_results.cvs"data from the https://github.com/tum-ei-eda/seal5/actions/runs/11630150544 which results to error when merging the files. Used the corev demo reports.
Columns: model_name,set,xlen,instr,test_kind,test_file,test_fmt,result,model
Fix: rename model_name to model and delete redundant column "model" at the end

Fixed!

[PhilippvK]

I will close this as all remaining steps can be discusses in the related separate issues.

Thank you for your help @kebi-be!