Switch to TF 2.0 and new NLU components

alt_requirements/requirements_pretrained_embeddings_convert.txt

@@ -1,5 +1,5 @@
 # Minimum Install Requirements
 -r ../requirements.txt
 
-tensorflow_text==1.15.1
-tensorflow_hub==0.6.0
+tensorflow_text==2.1.0rc0
+tensorflow_hub==0.7.0

changelog/4817.improvement.rst

@@ -1,2 +1,4 @@
 Part of Slack sanitization: 
-Multiple garbled URL's in a string coming from slack will be converted into actual strings. ``Example: health check of <http://eemdb.net|eemdb.net> and <http://eemdb1.net|eemdb1.net> to health check of eemdb.net and eemdb1.net``
+Multiple garbled URL's in a string coming from slack will be converted into actual strings.
+``Example: health check of <http://eemdb.net|eemdb.net> and <http://eemdb1.net|eemdb1.net> to health check of
+eemdb.net and eemdb1.net``

changelog/5065.feature.rst

@@ -0,0 +1,5 @@
+Add :ref:`LexicalSyntacticFeaturizer` to sparse featurizers.
+
+``LexicalSyntacticFeaturizer`` does the same featurization as the ``CRFEntityExtractor``. We extracted the
+featurization into a separate component so that the features can be reused and featurization is independent from the
+entity extraction.

changelog/5187.feature.rst

@@ -0,0 +1,7 @@
+Integrate language models from HuggingFace's `Transformers <https://github.com/huggingface/transformers>`_ Library.
+
+Add a new NLP component :ref:`HFTransformersNLP <HFTransformersNLP>` which
+tokenizes and featurizes incoming messages using a specified pre-trained model with the Transformers library as the backend.
+Add ``LanguageModelTokenizers`` and ``LanguageModelFeaturizers`` which use the information from ``HFTransformersNLP``
+and sets them correctly for message object.
+Language models currently supported: BERT, OpenAIGPT, GPT-2, XLNet, DistilBert, RoBERTa

changelog/5230.feature.rst

@@ -0,0 +1,15 @@
+Refactor how GPU and CPU environments are configured for TensorFlow 2.0.
+
+Please refer to the :ref:`documentation <tensorflow_usage>` to understand
+which environment variables to set in what scenarios. A couple of examples are shown below as well:
+
+.. code-block:: python
+
+    # This specifies to use 1024 MB of memory from GPU with logical ID 0 and 2048 MB of memory from GPU with logical ID 1
+    TF_GPU_MEMORY_ALLOC="0:1024, 1:2048"
+
+    # Specifies that at most 3 CPU threads can be used to parallelize multiple non-blocking operations
+    TF_INTER_OP_PARALLELISM_THREADS="3"
+
+    # Specifies that at most 2 CPU threads can be used to parallelize a particular operation.
+    TF_INTRA_OP_PARALLELISM_THREADS="2"

changelog/5266.feature.rst

@@ -0,0 +1,12 @@
+Added a new NLU component ``DIETClassifier`` and a new policy ``TEDPolicy``.
+
+DIET (Dual Intent and Entity Transformer) is a multi-task architecture for intent classification and entity
+recognition. You can read more about this component in our :ref:`documentation <diet-classifier>`.
+The new component will replace the ``EmbeddingIntentClassifier`` and the ``CRFEntityExtractor`` in the future.
+Those two components are deprecated from now on.
+See :ref:`migration guide <migration-to-rasa-1.8>` for details on how to
+switch to the new component.
+
+``TEDPolicy`` is the new name for ``EmbeddingPolicy``. ``EmbeddingPolicy`` is deprecated from now on.
+The functionality of ``TEDPolicy`` and ``EmbeddingPolicy`` is the same. Please update your configuration file
+to use the new name for the policy.

changelog/5266.improvement.rst

@@ -0,0 +1 @@
+We updated our code to TensorFlow 2.

changelog/5266.misc.rst

@@ -0,0 +1,9 @@
+We deprecated all existing pipeline templates, ``SklearnIntentClassifier`` and ``KerasPolicy``.
+
+Please list the components you want to use directly in your configuration file.
+Check out :ref:`Choosing a Pipeline <choosing-a-pipeline>` to decide what components to
+include in your pipeline.
+
+Use ``DIETClassifier`` instead of ``SklearnIntentClassifier``.
+
+Use ``TEDPolicy`` instead of ``KerasPolicy``.

changelog/663.feature.rst

@@ -0,0 +1,6 @@
+The sentence vector of the ``SpacyFeaturizer`` and ``MitieFeaturizer`` can be calculated using max or mean pooling.
+
+To specify the pooling operation, set the option ``pooling`` for the ``SpacyFeaturizer`` or the ``MitieFeaturizer``
+in your configuration file. The default pooling operation is ``mean``. The mean pooling operation also does not take
+into account words, that do not have a word vector.
+See our :ref:`documentation <components>` for more details.

changelog/699.misc.rst

@@ -1,6 +1,3 @@
-The `EmbeddingPolicy <https://rasa.com/docs/rasa/core/policies/#embedding-policy>`_
-replaces the ``KerasPolicy`` in new Rasa projects generated with ``rasa init``.
-The `EmbeddingPolicy <https://rasa.com/docs/rasa/core/policies/#embedding-policy>`_
-is now the recommended machine learning policy. Please see the
-`migration guide <https://rasa.com/docs/rasa/migration-guide/#rasa-1-7-to-rasa-1-8>`_
-if you want to switch to this new policy in an existing project.
+The :ref:`TEDPolicy <ted_policy>` replaces the ``KerasPolicy`` in new Rasa projects generated with ``rasa init``.
+The :ref:`TEDPolicy <ted_policy>` is now the recommended machine learning policy. Please see the
+:ref:`migration guide <migration-to-rasa-1.8>` if you want to switch to this new policy in an existing project.

data/configs_for_docs/default_config.yml

@@ -0,0 +1,14 @@
+language: "en"
+
+pipeline:
+  - name: WhitespaceTokenizer
+  - name: RegexFeaturizer
+  - name: LexicalSyntacticFeaturizer
+  - name: CountVectorsFeaturizer
+  - name: CountVectorsFeaturizer
+    analyzer: "char_wb"
+    min_ngram: 1
+    max_ngram: 4
+  - name: DIETClassifier
+  - name: EntitySynonymMapper
+  - name: DIETSelector

data/configs_for_docs/default_english_config.yml

@@ -0,0 +1,15 @@
+language: "en"
+
+pipeline:
+  - name: ConveRTTokenizer
+  - name: ConveRTFeaturizer
+  - name: RegexFeaturizer
+  - name: LexicalSyntacticFeaturizer
+  - name: CountVectorsFeaturizer
+  - name: CountVectorsFeaturizer
+    analyzer: "char_wb"
+    min_ngram: 1
+    max_ngram: 4
+  - name: DIETClassifier
+  - name: EntitySynonymMapper
+  - name: DIETSelector

data/configs_for_docs/pretrained_embeddings_convert_config_1.yml

@@ -0,0 +1,3 @@
+language: "en"
+
+pipeline: "pretrained_embeddings_convert"

data/configs_for_docs/pretrained_embeddings_convert_config_2.yml

@@ -0,0 +1,6 @@
+language: "en"
+
+pipeline:
+- name: "ConveRTTokenizer"
+- name: "ConveRTFeaturizer"
+- name: "EmbeddingIntentClassifier"

data/configs_for_docs/pretrained_embeddings_spacy_config_1.yml

@@ -0,0 +1,3 @@
+language: "en"
+
+pipeline: "pretrained_embeddings_spacy"

data/configs_for_docs/pretrained_embeddings_spacy_config_2.yml

@@ -0,0 +1,10 @@
+language: "en"
+
+pipeline:
+- name: "SpacyNLP"
+- name: "SpacyTokenizer"
+- name: "SpacyFeaturizer"
+- name: "RegexFeaturizer"
+- name: "CRFEntityExtractor"
+- name: "EntitySynonymMapper"
+- name: "SklearnIntentClassifier"

data/configs_for_docs/supervised_embeddings_config_1.yml

@@ -0,0 +1,3 @@
+language: "en"
+
+pipeline: "supervised_embeddings"

data/configs_for_docs/supervised_embeddings_config_2.yml

@@ -0,0 +1,13 @@
+language: "en"
+
+pipeline:
+- name: "WhitespaceTokenizer"
+- name: "RegexFeaturizer"
+- name: "CRFEntityExtractor"
+- name: "EntitySynonymMapper"
+- name: "CountVectorsFeaturizer"
+- name: "CountVectorsFeaturizer"
+  analyzer: "char_wb"
+  min_ngram: 1
+  max_ngram: 4
+- name: "EmbeddingIntentClassifier"

data/test/config_embedding_test.yml

@@ -2,5 +2,5 @@ language: en
 pipeline:
 - name: "CountVectorsFeaturizer"
   max_ngram: 3
-- name: "EmbeddingIntentClassifier"
-  epochs: 10
+- name: "DIETClassifier"
+  epochs: 10

sample_configs/config_crf_custom_features.yml → data/test_config/config_crf_custom_features.yml

@@ -15,7 +15,7 @@ pipeline:
     # features for word before token
     - ["low", "title", "upper", "pos", "pos2"]
     # features of token itself
-    - ["bias", "low", "word3", "word2", "upper", "title", "digit", "pos", "pos2", "pattern"]
+    - ["low", "word3", "word2", "upper", "title", "digit", "pos", "pos2"]
     # features for word after the token we want to tag
     - ["low", "title", "upper", "pos", "pos2"]
   max_iterations: 50

sample_configs/config_embedding_intent_response_selector.yml → data/test_config/config_embedding_intent_response_selector.yml

@@ -3,7 +3,7 @@ language: "en"
 pipeline:
   - name: "WhitespaceTokenizer"
   - name: "CountVectorsFeaturizer"
-  - name: "EmbeddingIntentClassifier"
+  - name: "DIETClassifier"
     epochs: 2
   - name: "ResponseSelector"
     epochs: 2

data/test_config/config_pretrained_embeddings_mitie.yml

@@ -0,0 +1,11 @@
+language: "en"
+
+pipeline:
+- name: "MitieNLP"
+  model: "data/total_word_feature_extractor.dat"
+- name: "MitieTokenizer"
+- name: "MitieEntityExtractor"
+- name: "EntitySynonymMapper"
+- name: "RegexFeaturizer"
+- name: "MitieFeaturizer"
+- name: "SklearnIntentClassifier"

data/test_config/config_pretrained_embeddings_mitie_2.yml

@@ -0,0 +1,10 @@
+language: "en"
+
+pipeline:
+- name: "MitieNLP"
+  model: "data/total_word_feature_extractor.dat"
+- name: "MitieTokenizer"
+- name: "MitieEntityExtractor"
+- name: "EntitySynonymMapper"
+- name: "RegexFeaturizer"
+- name: "MitieIntentClassifier"

sample_configs/config_supervised_embeddings_duckling.yml → data/test_config/config_supervised_embeddings_duckling.yml

@@ -3,5 +3,6 @@ language: "en"
 pipeline:
 - name: "CountVectorsFeaturizer"
 - name: "EmbeddingIntentClassifier"
+  epochs: 2
 - name: "DucklingHTTPExtractor"
   url: "http://duckling:8000"

data/test_config/embedding_random_seed.yaml

@@ -1,3 +1,4 @@
 policies:
-- name: EmbeddingPolicy
+- name: TEDPolicy
   random_seed: 42
+  epochs: 2

docker/Dockerfile_full

@@ -72,7 +72,7 @@ FROM base AS runner
 WORKDIR /app
 
 # Copy over default pipeline config
-COPY sample_configs/config_pretrained_embeddings_spacy_duckling.yml config.yml
+COPY docker/configs/config_supervised_embeddings_duckling.yml config.yml
 
 # Copy over mitie model
 COPY --from=builder /app/data/total_word_feature_extractor.dat data/total_word_feature_extractor.dat

docker/Dockerfile_pretrained_embeddings_mitie_en → docker/Dockerfile_pretrained_embeddings_convert_en

@@ -41,9 +41,6 @@ RUN apt-get update -qq \
 # Make sure we have the latest pip version
 RUN pip install -U pip
 
-# Download mitie model
-RUN wget -P /app/data/ https://s3-eu-west-1.amazonaws.com/mitie/total_word_feature_extractor.dat
-
 # Copy only what we really need
 COPY README.md .
 COPY setup.py .
@@ -54,22 +51,19 @@ COPY requirements.txt .
 COPY LICENSE.txt .
 
 # Install dependencies
-RUN pip install --no-cache-dir -r alt_requirements/requirements_pretrained_embeddings_mitie.txt
+RUN pip install --no-cache-dir -r alt_requirements/requirements_pretrained_embeddings_convert.txt
 
 # Install Rasa as package
 COPY rasa ./rasa
-RUN pip install .[sql,mitie]
+RUN pip install .[sql,convert]
 
 # Runtime stage which uses the virtualenv which we built in the previous stage
 FROM base AS runner
 
 WORKDIR /app
 
 # Copy over default pipeline config
-COPY sample_configs/config_pretrained_embeddings_mitie.yml config.yml
-
-# Copy over mitie model
-COPY --from=builder /app/data/total_word_feature_extractor.dat data/total_word_feature_extractor.dat
+COPY docker/configs/config_pretrained_embeddings_convert.yml config.yml
 
 # Copy virtualenv from previous stage
 COPY --from=builder /build /build

docker/Dockerfile_pretrained_embeddings_spacy_de

@@ -67,7 +67,7 @@ FROM base AS runner
 WORKDIR /app
 
 # Copy over default pipeline config
-COPY sample_configs/config_pretrained_embeddings_spacy_de.yml config.yml
+COPY docker/configs/config_pretrained_embeddings_spacy_de.yml config.yml
 
 # Copy virtualenv from previous stage
 COPY --from=builder /build /build

docker/Dockerfile_pretrained_embeddings_spacy_en

@@ -67,7 +67,7 @@ FROM base AS runner
 WORKDIR /app
 
 # Copy over default pipeline config
-COPY sample_configs/config_pretrained_embeddings_spacy.yml config.yml
+COPY docker/configs/config_pretrained_embeddings_spacy_en.yml config.yml
 
 # Copy virtualenv from previous stage
 COPY --from=builder /build /build

docker/configs/config_pretrained_embeddings_convert.yml

@@ -0,0 +1,15 @@
+language: "en"
+
+pipeline:
+  - name: ConveRTTokenizer
+  - name: ConveRTFeaturizer
+  - name: RegexFeaturizer
+  - name: LexicalSyntacticFeaturizer
+  - name: CountVectorsFeaturizer
+  - name: CountVectorsFeaturizer
+    analyzer: "char_wb"
+    min_ngram: 1
+    max_ngram: 4
+  - name: DIETClassifier
+  - name: EntitySynonymMapper
+  - name: DIETSelector

docker/configs/config_pretrained_embeddings_spacy_de.yml

@@ -0,0 +1,16 @@
+language: "de"
+
+pipeline:
+  - name: SpacyNLP
+  - name: SpacyTokenizer
+  - name: SpacyFeaturizer
+  - name: RegexFeaturizer
+  - name: LexicalSyntacticFeaturizer
+  - name: CountVectorsFeaturizer
+  - name: CountVectorsFeaturizer
+    analyzer: "char_wb"
+    min_ngram: 1
+    max_ngram: 4
+  - name: DIETClassifier
+  - name: EntitySynonymMapper
+  - name: DIETSelector

docker/configs/config_pretrained_embeddings_spacy_en.yml

@@ -0,0 +1,16 @@
+language: "en"
+
+pipeline:
+  - name: SpacyNLP
+  - name: SpacyTokenizer
+  - name: SpacyFeaturizer
+  - name: RegexFeaturizer
+  - name: LexicalSyntacticFeaturizer
+  - name: CountVectorsFeaturizer
+  - name: CountVectorsFeaturizer
+    analyzer: "char_wb"
+    min_ngram: 1
+    max_ngram: 4
+  - name: DIETClassifier
+  - name: EntitySynonymMapper
+  - name: DIETSelector

docker/configs/config_supervised_embeddings_duckling.yml

@@ -0,0 +1,16 @@
+language: "en"
+
+pipeline:
+  - name: WhitespaceTokenizer
+  - name: RegexFeaturizer
+  - name: LexicalSyntacticFeaturizer
+  - name: CountVectorsFeaturizer
+  - name: CountVectorsFeaturizer
+    analyzer: "char_wb"
+    min_ngram: 1
+    max_ngram: 4
+  - name: DIETClassifier
+  - name: EntitySynonymMapper
+  - name: DIETSelector
+  - name: DucklingHTTPExtractor
+    url: "http://duckling:8000"