Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NLU comparison does not plot results with more than one run #5426

Closed
TEGELB opened this issue Mar 16, 2020 · 3 comments · Fixed by #5568
Closed

NLU comparison does not plot results with more than one run #5426

TEGELB opened this issue Mar 16, 2020 · 3 comments · Fixed by #5568
Assignees
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@TEGELB
Copy link

TEGELB commented Mar 16, 2020

Rasa version: 1.8.1

Python version: 3.7.5

Operating system (windows, osx, ...): Windows

Issue:
When running the comparison of different pipeline configurations with default settings out of a folder, as laid out here, the diagramm is not plotted and an error occurs.
Error occurs when comparing two as well as six configurations.
All files are saved correctly up to that point.

No error occurs when the amount of runs is set to one manually.

Error (including full traceback):

![grafik](https://user-images.githubusercontent.com/56721736/76779152-2ec5b980-67ab-11ea-92fb-73765fbdaefc.png)

Command or request that led to error:

rasa test nlu
@TEGELB TEGELB added the type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. label Mar 16, 2020
@chkoss
Copy link
Contributor

chkoss commented Mar 16, 2020

I already traced down that the following happens when running e.g. rasa test nlu --runs 2.

Here, number_of_examples is plotted vs. mean, where mean is an average over runs of what's in the file output:
https://github.com/RasaHQ/rasa/blob/master/rasa/test.py#L219

The problem is, the shapes don't match. Let’s call the value of --runs r and the number of percentages in --percentages p. Then number_of_examples has shape (r*p,), the content of output has shape (r, p) and mean therefore has shape (p,).

This is caused by how number_of_examples is determined in compare_nlu:
https://github.com/RasaHQ/rasa/blob/master/rasa/nlu/test.py#L1413
This function simply appends all the values for each run and percentage into one 1D array of shape (r*p,).

@tabergma tabergma added the area:rasa-oss 🎡 Anything related to the open source Rasa framework label Mar 18, 2020
@indam23
Copy link
Contributor

indam23 commented Mar 20, 2020

Just ran into this too - it works in 1.7, in case that helps

@m-vdb m-vdb assigned m-vdb and unassigned chkoss Apr 3, 2020
@m-vdb
Copy link
Collaborator

m-vdb commented Apr 3, 2020

I was able to reproduce the issue using the doc here: https://rasa.com/docs/rasa/user-guide/testing-your-assistant/#comparing-nlu-pipelines and the right configuration files.

@m-vdb m-vdb linked a pull request Apr 3, 2020 that will close this issue
2 tasks
@chkoss chkoss closed this as completed Apr 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants