Snake layer and activation to learn periodic functions #1967

failure-to-thrive · 2020-07-03T16:03:02Z

Closes #1939

WindQAQ

Thanks for the contribution! Generally LGTM.

WindQAQ · 2020-07-03T17:37:34Z

tensorflow_addons/activations/snake.py

+
+
+@tf.keras.utils.register_keras_serializable(package="Addons")
+def snake(inputs: types.TensorLike, freq: types.Number = 1) -> tf.Tensor:


can we use the full name frequency here?

WindQAQ · 2020-07-03T17:39:55Z

tensorflow_addons/layers/snake.py

+    """
+
+    @typechecked
+    def __init__(self, freq_initializer: types.Initializer = "ones", **kwargs):


WindQAQ · 2020-07-03T17:44:12Z

tensorflow_addons/layers/snake.py

+        self.freq = self.add_weight(initializer=freq_initializer, trainable=True)
+
+    def call(self, inputs):
+        return inputs + (1 - tf.cos(2 * self.freq * inputs)) / (2 * self.freq)


can we directly call tensorflow_addons.activations.snake here?

It imposes additional overhead. That approach used in test degraded performance by 30%.

that's weird. I don't think one convert_to_tensor and cast have such impact on performance. But anyway, because that's how core TF deals with activation and layer subclass, it'd be better to use tensorflow_addons.activations.snake IMO.

OK. They know the best.

WindQAQ · 2020-07-03T17:44:47Z

tensorflow_addons/activations/tests/snake_test.py

+import numpy as np
+
+from tensorflow_addons.activations import snake
+


nit: remove this empty line

Sure. The reason I was doing so is I was trying to somehow group similar imports since there are no strict guidelines on how to order imports, nor other layers and activations adhere some consistent pattern.

WindQAQ · 2020-07-03T17:44:55Z

tensorflow_addons/layers/tests/snake_test.py

+
+from tensorflow_addons.layers.snake import Snake
+from tensorflow_addons.activations.snake import snake
+


WindQAQ · 2020-07-03T17:46:33Z

tensorflow_addons/activations/__init__.py

@@ -22,3 +22,4 @@
 from tensorflow_addons.activations.rrelu import rrelu
 from tensorflow_addons.activations.sparsemax import sparsemax
 from tensorflow_addons.activations.tanhshrink import tanhshrink
+from tensorflow_addons.activations.snake import snake


place in alphabetical order

@WindQAQ @seanpmorgan Just as a reminder can we automate with https://pypi.org/project/flake8-isort/?

Looks good to me. Could you open and issue and PR for it? I'm fine to automate it.

WindQAQ · 2020-07-03T17:46:45Z

tensorflow_addons/layers/__init__.py

@@ -36,3 +36,4 @@
 from tensorflow_addons.layers.tlu import TLU
 from tensorflow_addons.layers.wrappers import WeightNormalization
 from tensorflow_addons.layers.esn import ESN
+from tensorflow_addons.layers.snake import Snake


WindQAQ · 2020-07-04T01:05:37Z

tensorflow_addons/activations/snake.py

+    """Snake activation to learn periodic functions.
+
+    https://arxiv.org/abs/2006.08195
+    """


could we add the arguments doc here?

I was trying, but the original academic paper is quite obscure about explanations. So it could has end-up with words like 'frequency is a frequency' and other obvious things such as 'input is an input'. If it is OK anyway, no problems, I'll add.

Got it. Maybe we can have something like

inputs: Input tensor. frequency: A scalar, frequency of the periodic part.

WindQAQ · 2020-07-04T01:05:49Z

tensorflow_addons/layers/snake.py

+    """Snake layer to learn periodic functions with the 'frequency' param trainable.
+
+    https://arxiv.org/abs/2006.08195
+    """


could we add the arguments doc here?

I put them near the init for an IDE autocomplete.

WindQAQ

Hi @failure-to-thrive, I review a paper again and see this sentence in page 4.

We also suggest the extention to make the $a$ parameter becomes a learnable parameter
for each preactivation value.

IMO, it suggests frequency to be with the same shape as the input (like PReLU). What do you think?
Also, following the layers with learnable parameter in core TF, we should also support some arguments like frequency_regularizer and frequency_constraint.

WindQAQ · 2020-07-05T19:48:39Z

tensorflow_addons/layers/snake.py

+    def __init__(self, frequency_initializer: types.Initializer = "ones", **kwargs):
+        """
+        frequency_initializer: Initializer for the 'frequency' param.
+        """


Sorry for the confusion, after a rough look at our docs, it would be better to move this near class Snake.

class Snake(tf.keras.layers.Layer): """Snake layer to learn periodic functions with the trainable `frequency` scalar. It follows: ``` f(x) = x + (1 - cos(2 * frequency * x)) / (2 * frequency) ``` where `frequency` is a learned scalar. See [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://arxiv.org/abs/2006.08195) for more details. Arguments: frequency_initializer: Initializer for the `frequency` scalar. frequency_regularizer: Regularizer for the `frequency` scalar. frequency_constraint: Constraint for the `frequency` scalar. """

I'm following doc of the parametric activation like PReLU here. Feel free to elaborate more.

IMO, it suggests frequency to be with the same shape as the input (like PReLU). What do you think?

As I said earlier, the paper is quite obscure in many aspects. The same statement could have been made about the activation. Guesswork is a Sisyphus work. So I propose to leave it as it is. We can always get back to it later.

Also, following the layers with learnable parameter in core TF, we should also support some arguments like frequency_regularizer and frequency_constraint.

Same here. I thought of it but the paper does not give a glue on how these 'regularizers and co' should be applied. If they are relevant here at all.

Got it :-) Then just move the doc under class Snake.

I had to eliminate the math formula because it looks very awkward in the plain text. In the end of the all, there is the academic paper an one must be in good knowledge before utilizing the layer.

* Snake layer and activation to learn periodic functions. * Removed @tf.function * Considering all the comments. * Black matters :) * Moved the doc under class Snake.

Snake layer and activation to learn periodic functions.

be8cf1a

failure-to-thrive requested review from facaiy and seanpmorgan as code owners July 3, 2020 16:03

boring-cyborg bot added activations github layers labels Jul 3, 2020

googlebot added the cla: yes label Jul 3, 2020

Removed @tf.function

6d5e260

WindQAQ requested changes Jul 3, 2020

View reviewed changes

WindQAQ reviewed Jul 4, 2020

View reviewed changes

failure-to-thrive added 2 commits July 5, 2020 10:37

Considering all the comments.

25e3d84

Black matters :)

add059f

WindQAQ reviewed Jul 5, 2020

View reviewed changes

Moved the doc under class Snake.

179c8ef

WindQAQ approved these changes Jul 6, 2020

View reviewed changes

WindQAQ merged commit a178071 into tensorflow:master Jul 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Snake layer and activation to learn periodic functions #1967

Snake layer and activation to learn periodic functions #1967

failure-to-thrive commented Jul 3, 2020

WindQAQ left a comment

WindQAQ Jul 3, 2020

WindQAQ Jul 3, 2020

WindQAQ Jul 3, 2020

failure-to-thrive Jul 3, 2020

WindQAQ Jul 4, 2020

failure-to-thrive Jul 5, 2020

WindQAQ Jul 3, 2020

failure-to-thrive Jul 4, 2020

WindQAQ Jul 3, 2020

WindQAQ Jul 3, 2020

bhack Jul 3, 2020

WindQAQ Jul 4, 2020

WindQAQ Jul 3, 2020

WindQAQ Jul 4, 2020

failure-to-thrive Jul 4, 2020

WindQAQ Jul 4, 2020

WindQAQ Jul 4, 2020

failure-to-thrive Jul 4, 2020

WindQAQ left a comment

WindQAQ Jul 5, 2020

failure-to-thrive Jul 6, 2020

WindQAQ Jul 6, 2020 •

edited

Loading

failure-to-thrive Jul 6, 2020

WindQAQ Jul 6, 2020



		@tf.keras.utils.register_keras_serializable(package="Addons")
		def snake(inputs: types.TensorLike, freq: types.Number = 1) -> tf.Tensor:

		import numpy as np

		from tensorflow_addons.activations import snake


		from tensorflow_addons.layers.snake import Snake
		from tensorflow_addons.activations.snake import snake

Snake layer and activation to learn periodic functions #1967

Snake layer and activation to learn periodic functions #1967

Conversation

failure-to-thrive commented Jul 3, 2020

WindQAQ left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WindQAQ left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WindQAQ Jul 6, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WindQAQ Jul 6, 2020 •

edited

Loading