-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Language Models from Transformers Lib #5187
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! Have a couple of comments. Also, does it makes sense to create 3 files for hf? Why don't we put all these helpers into 1 file?
Co-Authored-By: Vladimir Vlasov <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to add some tests for the components.
Can you also create a changelog entry and add some documentation? E.g. add the new components to https://rasa.com/docs/rasa/nlu/components/. |
@tabergma Yes, tests and documentation are to be added. That wasn't ready. :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we going to add when to use what in a separate PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apart from the comments I already made, it looks good 🚀 Great work!
Proposed changes:
NLP
component -HFTransformersNLP
which tokenizes and featurizes incoming messages using the Transformers Library.LanguageModelTokenizers
andLanguageModelFeaturizers
which use the information fromHFTransformersNLP
and sets them correctly for message objectPart of https://github.com/RasaHQ/research/issues/62
Status (please check what you already did):
black
(please check Readme for instructions)