Added support for noisy dense layers.

[LeonShams]

Description

This PR adds support for noisy dense layers. Noisy dense layers are like dense layers but random noise is injected to help agents explore in reinforcement learning environments. This is very commonly used in Deep Q Learning and was first introduced by DeepMind in their Noisy Networks for Exploration paper in 2017.
Fixes: #2127

Type of change

• Bug fix
• New Tutorial
• Updated or additional documentation
• Additional Testing
• New Activation and the changes conform to the activation contribution guidelines
• New Callback and the changes conform to the callback contribution guidelines
• New Image addition and the changes conform to the image op contribution guidelines
• New Layer and the changes conform to the layer contribution guidelines
• New Loss and the changes conform to the loss contribution guidelines
• New Metric and the changes conform to the metric contribution guidelines
• New Optimizer and the changes conform to the optimizer contribution guidelines
• New RNN Cell and the changes conform to the rnn contribution guidelines
• New Seq2seq addition and the changes conform to the seq2seq contribution guidelines
• New Text addition and the changes conform to the text op contribution guidelines

Checklist:

• I've properly formatted my code according to the guidelines
  • By running Black + Flake8
  • By running pre-commit hooks
• This PR addresses an already submitted issue for TensorFlow Addons
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• This PR contains modifications to C++ custom-ops

How Has This Been Tested?

I used PyTest to test the code and trained it on some reinforcement learning environments from the gym toolkit and compared my results to the results on the Noisy Networks for Exploration paper.

[LeonShams]

@facaiy @seanpmorgan please let me know if there are any problems or if you have any questions. Thanks!

[WindQAQ]

I just figure out that lots of codes are copied from tf.keras.layers.Dense. Therefore, a better design pattern is that

class NoisyDense(tf.keras.layers.Dense):
    @typechecked
    def __init__(
        self,
        units: int,
        sigma: float = 0.5,
        activation: types.Activation = None,
        use_bias: bool = True,
        kernel_regularizer: types.Regularizer = None,
        bias_regularizer: types.Regularizer = None,
        activity_regularizer: types.Regularizer = None,
        kernel_constraint: types.Constraint = None,
        bias_constraint: types.Constraint = None,
        **kwargs
    ):
        super().__init__(xxxx)
        # DO OTHER THINGS

    def build(self, input_shape):
        # OVERRIDE

    def reset_noise(self):
        # DO SOMETHING

    def remove_noise(self):
        # DO SOMETHING

    def call(self, inputs, reset_noise=True):
        if reset_noise:
            self.reset_noise()
        # TODO(WindQAQ): Replace this with `dense()` once public.
        # prepare your self.kernel, self.bias here.
        return super().call(inputs)

    def get_config(self):
        # OVERRIDE

[LeonShams]

Regarding the super().__init__(xxxx), what would you like me to do about self.kernel_initializer and self.bias_initializer since they are not used in NoisyDense?

[WindQAQ]

Regarding the super().__init__(xxxx), what would you like me to do about self.kernel_initializer and self.bias_initializer since they are not used in NoisyDense?

Just pass them as None. and remove them with delattr after it.

[LeonShams]

Also, what would you like me to do about super(NoisyDense, self).get_config() since it will now get the config of Dense?

[WindQAQ]

Also, what would you like me to do about super(NoisyDense, self).get_config() since it will now get the config of Dense?

config = super().get_config()
config.pop("foo") # pop things you do not have
config["bar"] = bar # update things you have additionally
return config

[LeonShams]

That will raise an error since kernel_initializer and bias_initializer were deleted.

Dense get_config():

config.update({
    ...
    'kernel_initializer':
        initializers.serialize(self.kernel_initializer),
    'bias_initializer':
        initializers.serialize(self.bias_initializer),
    ...
}

[WindQAQ]

That will raise an error since kernel_initializer and bias_initializer were deleted.

Dense get_config():

config.update({
    ...
    'kernel_initializer':
        initializers.serialize(self.kernel_initializer),
    'bias_initializer':
        initializers.serialize(self.bias_initializer),
    ...
}

Try this

config = super(tf.keras.layers.Dense, self).get_config()
config["foo"] = foo # update your stuff

or even hacking

self.bar = None
config = super().get_config()
config.pop("bar")
config["foo"] = foo

Both approaches are not so good to me, so please put TODO(WindQAQ): Get rid of this hacking way. to me 😃

https://colab.research.google.com/drive/14VzUnIZIDadej8s_fHlSKx-ieTs9rttV?usp=sharing

[LeonShams]

The first method works well!

config = super(tf.keras.layers.Dense, self).get_config()
config["foo"] = foo # update your stuff

But is there a reason why you used config["foo"] instead of config.update({})?

[WindQAQ]

The first method works well!

config = super(tf.keras.layers.Dense, self).get_config()
config["foo"] = foo # update your stuff

But is there a reason why you used config["foo"] instead of config.update({})?

Basically no. Just type in rush. config.update(xxx) LGTM.

[WindQAQ]

Some documentation improvement needed. Thanks.

[WindQAQ]

@LeonShams Sorry that this should be the last comment. Others LGTM! Thank you.

[LeonShams]

No worries, I made the changes, let me know if there is anything else.