Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added results for multimodal benchmark into FEDOT.docs #1115

Merged
merged 3 commits into from
Jun 29, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/benchmarks/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ We make a comparison with existing state-of-the-art AutoML libraries on various

forecasting
tabular
multimodal
12 changes: 12 additions & 0 deletions docs/source/benchmarks/multimodal.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Multimodal data
---------------

Here are overall classification and regression problems results on multimodal datasets across state-of-the-art AutoML frameworks
using `AutoML Multimodal Benchmark <https://github.com/sxjscience/automl_multimodal_benchmark>`__ test suite:

.. raw:: html
:file: multimodal_res.html


The results are obtained using sever based on Xeon Cascadelake (2900MHz)
with 12 cores and 24GB memory for experiments with the local infrastructure.
19 changes: 19 additions & 0 deletions docs/source/benchmarks/multimodal_res.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
Dataset name,Metric name,AutoGluon,FEDOT
prod,accuracy,0.895,0.897
salary,accuracy,0.528,0.402
airbnb,accuracy,0.466,0.434
channel,accuracy,0.550,0.542
wine,accuracy,0.842,0.784
imdb,auc,0.874,0.872
fake,auc,0.968,0.958
kick,auc,0.775,0.752
jigsaw,auc,0.915,0.998
qaa,r2,0.383,0.504
qaq,r2,0.426,0.554
book,r2,0.6,0.994
jc,r2,0.612,0.619
cloth,r2,0.654,0.733
ae,r2,0.979,0.974
pop,r2,0.02,0.02
house,r2,0.943,0.928
mercari,r2,0.569,0.52
318 changes: 318 additions & 0 deletions docs/source/benchmarks/multimodal_res.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,318 @@
<style type="text/css">
#T_benchmark table {
border: 1px solid #e1e4e5;
text-align: center;
font-size: .9rem;
}
#T_benchmark th {
border: 1px solid #e1e4e5;
text-align: center;
font-size: .9rem;
}
#T_benchmark td {
border: 1px solid #e1e4e5;
text-align: center;
font-size: .9rem;
}
#T_benchmark th {
padding: 8px 16px;
}
#T_benchmark td {
padding: 8px 16px;
}
#T_benchmark tr {
background-color: #fff;
}
#T_benchmark tbody tr:nth-child(odd) {
background-color: #f3f6f6;
}
#T_benchmarkrow0_col1, #T_benchmarkrow1_col0, #T_benchmarkrow2_col0, #T_benchmarkrow3_col0, #T_benchmarkrow4_col0, #T_benchmarkrow5_col0, #T_benchmarkrow6_col0, #T_benchmarkrow7_col0, #T_benchmarkrow8_col1, #T_benchmarkrow9_col1, #T_benchmarkrow10_col1, #T_benchmarkrow11_col1, #T_benchmarkrow12_col1, #T_benchmarkrow13_col1, #T_benchmarkrow14_col0, #T_benchmarkrow15_col0, #T_benchmarkrow15_col1, #T_benchmarkrow16_col0, #T_benchmarkrow17_col0 {
color: blue;
font-weight: bold;
}
</style>
<div style="overflow: auto;">
<table id="T_benchmark" style="width: 100%; border-collapse: collapse; font-family: Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;">
<thead>
<tr>
<th class="blank">
</th>
<th class="index_name level0">
framework
</th>
<th class="col_heading level0 col0">
AutoGluon
</th>
<th class="col_heading level0 col1">
FEDOT
</th>
</tr>
<tr>
<th class="index_name level0">
Dataset name
</th>
<th class="index_name level1">
Metric name
</th>
<th class="blank col0">
</th>
<th class="blank col1">
</th>
</tr>
</thead>
<tbody>
<tr>
<th class="row_heading level0 row0" id="T_benchmarklevel0_row0">
prod
</th>
<th class="row_heading level1 row0" id="T_benchmarklevel1_row0">
accuracy
</th>
<td class="data row0 col0" id="T_benchmarkrow0_col0">
0.895
</td>
<td class="data row0 col1" id="T_benchmarkrow0_col1">
0.897
</td>
</tr>
<tr>
<th class="row_heading level0 row1" id="T_benchmarklevel0_row1">
salary
</th>
<th class="row_heading level1 row1" id="T_benchmarklevel1_row1">
accuracy
</th>
<td class="data row1 col0" id="T_benchmarkrow1_col0">
0.528
</td>
<td class="data row1 col1" id="T_benchmarkrow1_col1">
0.402
</td>
</tr>
<tr>
<th class="row_heading level0 row2" id="T_benchmarklevel0_row2">
airbnb
</th>
<th class="row_heading level1 row2" id="T_benchmarklevel1_row2">
accuracy
</th>
<td class="data row2 col0" id="T_benchmarkrow2_col0">
0.466
</td>
<td class="data row2 col1" id="T_benchmarkrow2_col1">
0.434
</td>
</tr>
<tr>
<th class="row_heading level0 row3" id="T_benchmarklevel0_row3">
channel
</th>
<th class="row_heading level1 row3" id="T_benchmarklevel1_row3">
accuracy
</th>
<td class="data row3 col0" id="T_benchmarkrow3_col0">
0.550
</td>
<td class="data row3 col1" id="T_benchmarkrow3_col1">
0.542
</td>
</tr>
<tr>
<th class="row_heading level0 row4" id="T_benchmarklevel0_row4">
wine
</th>
<th class="row_heading level1 row4" id="T_benchmarklevel1_row4">
accuracy
</th>
<td class="data row4 col0" id="T_benchmarkrow4_col0">
0.842
</td>
<td class="data row4 col1" id="T_benchmarkrow4_col1">
0.784
</td>
</tr>
<tr>
<th class="row_heading level0 row5" id="T_benchmarklevel0_row5">
imdb
</th>
<th class="row_heading level1 row5" id="T_benchmarklevel1_row5">
auc
</th>
<td class="data row5 col0" id="T_benchmarkrow5_col0">
0.874
</td>
<td class="data row5 col1" id="T_benchmarkrow5_col1">
0.872
</td>
</tr>
<tr>
<th class="row_heading level0 row6" id="T_benchmarklevel0_row6">
fake
</th>
<th class="row_heading level1 row6" id="T_benchmarklevel1_row6">
auc
</th>
<td class="data row6 col0" id="T_benchmarkrow6_col0">
0.968
</td>
<td class="data row6 col1" id="T_benchmarkrow6_col1">
0.958
</td>
</tr>
<tr>
<th class="row_heading level0 row7" id="T_benchmarklevel0_row7">
kick
</th>
<th class="row_heading level1 row7" id="T_benchmarklevel1_row7">
auc
</th>
<td class="data row7 col0" id="T_benchmarkrow7_col0">
0.775
</td>
<td class="data row7 col1" id="T_benchmarkrow7_col1">
0.752
</td>
</tr>
<tr>
<th class="row_heading level0 row8" id="T_benchmarklevel0_row8">
jigsaw
</th>
<th class="row_heading level1 row8" id="T_benchmarklevel1_row8">
auc
</th>
<td class="data row8 col0" id="T_benchmarkrow8_col0">
0.915
</td>
<td class="data row8 col1" id="T_benchmarkrow8_col1">
0.998
</td>
</tr>
<tr>
<th class="row_heading level0 row9" id="T_benchmarklevel0_row9">
qaa
</th>
<th class="row_heading level1 row9" id="T_benchmarklevel1_row9">
r2
</th>
<td class="data row9 col0" id="T_benchmarkrow9_col0">
0.383
</td>
<td class="data row9 col1" id="T_benchmarkrow9_col1">
0.504
</td>
</tr>
<tr>
<th class="row_heading level0 row10" id="T_benchmarklevel0_row10">
qaq
</th>
<th class="row_heading level1 row10" id="T_benchmarklevel1_row10">
r2
</th>
<td class="data row10 col0" id="T_benchmarkrow10_col0">
0.426
</td>
<td class="data row10 col1" id="T_benchmarkrow10_col1">
0.554
</td>
</tr>
<tr>
<th class="row_heading level0 row11" id="T_benchmarklevel0_row11">
book
</th>
<th class="row_heading level1 row11" id="T_benchmarklevel1_row11">
r2
</th>
<td class="data row11 col0" id="T_benchmarkrow11_col0">
0.600
</td>
<td class="data row11 col1" id="T_benchmarkrow11_col1">
0.994
</td>
</tr>
<tr>
<th class="row_heading level0 row12" id="T_benchmarklevel0_row12">
jc
</th>
<th class="row_heading level1 row12" id="T_benchmarklevel1_row12">
r2
</th>
<td class="data row12 col0" id="T_benchmarkrow12_col0">
0.612
</td>
<td class="data row12 col1" id="T_benchmarkrow12_col1">
0.619
</td>
</tr>
<tr>
<th class="row_heading level0 row13" id="T_benchmarklevel0_row13">
cloth
</th>
<th class="row_heading level1 row13" id="T_benchmarklevel1_row13">
r2
</th>
<td class="data row13 col0" id="T_benchmarkrow13_col0">
0.654
</td>
<td class="data row13 col1" id="T_benchmarkrow13_col1">
0.733
</td>
</tr>
<tr>
<th class="row_heading level0 row14" id="T_benchmarklevel0_row14">
ae
</th>
<th class="row_heading level1 row14" id="T_benchmarklevel1_row14">
r2
</th>
<td class="data row14 col0" id="T_benchmarkrow14_col0">
0.979
</td>
<td class="data row14 col1" id="T_benchmarkrow14_col1">
0.974
</td>
</tr>
<tr>
<th class="row_heading level0 row15" id="T_benchmarklevel0_row15">
pop
</th>
<th class="row_heading level1 row15" id="T_benchmarklevel1_row15">
r2
</th>
<td class="data row15 col0" id="T_benchmarkrow15_col0">
0.020
</td>
<td class="data row15 col1" id="T_benchmarkrow15_col1">
0.020
</td>
</tr>
<tr>
<th class="row_heading level0 row16" id="T_benchmarklevel0_row16">
house
</th>
<th class="row_heading level1 row16" id="T_benchmarklevel1_row16">
r2
</th>
<td class="data row16 col0" id="T_benchmarkrow16_col0">
0.943
</td>
<td class="data row16 col1" id="T_benchmarkrow16_col1">
0.928
</td>
</tr>
<tr>
<th class="row_heading level0 row17" id="T_benchmarklevel0_row17">
mercari
</th>
<th class="row_heading level1 row17" id="T_benchmarklevel1_row17">
r2
</th>
<td class="data row17 col0" id="T_benchmarkrow17_col0">
0.569
</td>
<td class="data row17 col1" id="T_benchmarkrow17_col1">
0.520
</td>
</tr>
</tbody>
</table>
</div>