Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Language Models from Transformers Lib #5187

Merged
merged 24 commits into from
Feb 12, 2020
Merged

Language Models from Transformers Lib #5187

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
5a71c57
first implementation ready.
dakshvar22 Feb 4, 2020
d5a1b85
tested all available models. implementation works
dakshvar22 Feb 4, 2020
205c7bd
refactored class name
dakshvar22 Feb 4, 2020
7eb475c
remove print statement
dakshvar22 Feb 4, 2020
cc55dfc
Apply suggestions from code review
dakshvar22 Feb 4, 2020
576d8f4
quick review comments. Tests WIP
dakshvar22 Feb 4, 2020
fab9122
Merge branch 'transformers_lm' of github.com:RasaHQ/rasa into transfo…
dakshvar22 Feb 4, 2020
4c3f218
fix imports
dakshvar22 Feb 4, 2020
49e9a15
bug fix to swap seq and sentence embeddings
dakshvar22 Feb 5, 2020
e990e8f
tests for tokenizers are in
dakshvar22 Feb 5, 2020
99bad36
added featurizer tests
dakshvar22 Feb 5, 2020
01c7de5
added documentation
dakshvar22 Feb 5, 2020
3259788
add changelog, move common method out of class
dakshvar22 Feb 5, 2020
d10d73f
refactor spacy doc name
dakshvar22 Feb 5, 2020
452368d
Apply suggestions from code review
dakshvar22 Feb 6, 2020
38b6a01
added new components to test pipelines
dakshvar22 Feb 6, 2020
7c654fd
Merge branch 'transformers_lm' of github.com:RasaHQ/rasa into transfo…
dakshvar22 Feb 6, 2020
7ccafc3
created new pipeline for failing tests
dakshvar22 Feb 9, 2020
6d9c886
separate pipeline for convert as well
dakshvar22 Feb 10, 2020
93fe116
merge tf2, resolve conflicts and resolve comments
dakshvar22 Feb 12, 2020
b88ed3e
refactored variable names
dakshvar22 Feb 12, 2020
4bfb4ed
removed unnecessary component from a test pipeline
dakshvar22 Feb 12, 2020
684836a
Merge branch 'tf2' into transformers_lm
dakshvar22 Feb 12, 2020
86ee337
added constants
dakshvar22 Feb 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions tests/nlu/training/test_train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,19 +27,16 @@ def pipelines_for_tests():
as_pipeline(
"SpacyNLP",
"MitieNLP",
"HFTransformersNLP",
"WhitespaceTokenizer",
"ConveRTTokenizer",
"MitieTokenizer",
"SpacyTokenizer",
"LanguageModelTokenizer",
"MitieFeaturizer",
"SpacyFeaturizer",
"RegexFeaturizer",
"LexicalSyntacticFeaturizer",
"CountVectorsFeaturizer",
"ConveRTFeaturizer",
"LanguageModelFeaturizer",
"MitieEntityExtractor",
"CRFEntityExtractor",
"SpacyEntityExtractor",
Expand All @@ -63,6 +60,19 @@ def pipelines_for_tests():
"KeywordIntentClassifier",
),
),
(
# Created another test pipeline because dense featurizers can't co-exist in the same pipeline together,
# because of the mismatch in sequence length their tokenizers break the incoming message into.
"en",
as_pipeline(
"HFTransformersNLP",
"LanguageModelTokenizer",
"LanguageModelFeaturizer",
"CRFEntityExtractor",
"DIETClassifier",
"ResponseSelector",
),
),
]


Expand Down