tfa.image.rotate Different Fill Modes

[srilman]

Describe the feature and the current behavior/state.
Currently, when an image is rotated using the tfa.image.rotate() function, the outer regions of the image are filled with 0's, resulting in an image with black corners usually. This is usually not preferred, especially when the user is trying to create new images via data augmentation. Images with black corners are usually not a good representation of the intended creation.

tf.keras.preprocessing.image.random_rotation has a really good option where you can control how exactly the blank space is filled. The following pictures shows some of the common methods and how the resulting image looks closer to the original.

It would be nice to see the same option added to tfa.image.rotate.

Relevant information

• Are you willing to contribute it (yes/no): Yes!
• Are you willing to maintain it going forward? (yes/no): Yes!
• Is there a relevant academic paper? (if so, where): No
• Is there already an implementation in another framework? (if so, where): Keras (obviously)
• Was it part of tf.contrib? (if so, where): Not that I am aware of

Which API type would this fall under (layer, metric, optimizer, etc.)
tfa.image

Who will benefit with this feature?
Anyone who is currently using tfa.image.rotate for data / image augmentation purposes and want different methods to fill the image boundaries to create better augmented images for training.

Any other info.
While tf.keras.preprocessing.image.random_rotation (assuming you are performing rotations randomly) is capable of doing the same thing, it is generally not convenient to use it in a data pipeline. For one reason, it's not really designed for tensors in the form of (num_pictures, num_rows, num_cols, num_channels), the channels last format which is whats normally expected. Secondly, it expects inputs as Numpy arrays. Finally it is hard to use directly with @tf.function. Thus, I think a native implementation would be preferred.

[gabrieldemarmiesse]

It's widely used, already present in keras with numpy, I just hope it won't be too hard to implement. Contributions welcome!

[vinayvr11]

shall we use the same method used in keras.

[gabrieldemarmiesse]

I don't have a preference on the method as long as it's fast and the gradients and the results are correct.

[vinayvr11]

Ok so we need to make it from scratch.

[srilman]

I currently have a solution built in Python using high level Tensorflow ops for the following extensions:

• Extending the closest pixel outwards (the second picture)
• Copying the entire image along all directions (the last picture)

I'm not sure if it would be fast enough as hoped for, especially if its used in an input-bound data pipeline, but it was quick to implement and test. I can make a PR if it's good enough for now.

[gabrieldemarmiesse]

It's better to have an implementation that's correct with the right API than to have a fast one. We can always do speed optimizations afterwards. So feel free to open a pull request. If you can, first do a pull request for the simplest mode, then once the first pull request is merged, do another pull request for the other mode you have.

Having small pull requests allow us to make fast review/changes cycles.