Add correlation cost layer

[PatWie]

I just copied the files from my PR in tensorflow and slightly (+ blindly) adjusted the BUILD. Hence, this is really WIP.

References:

• Add correlation cost layer from FlowNet tensorflow#21392 (comment)

I will rebase to master when I can run those ci test locally + adding a comprehensive commit message.
The docker command seems to not contain any way to run CUDA.

[seanpmorgan]

For reference:
tensorflow/tensorflow#21392

[facaiy]

Welcome, the GPU support is still in process: #118

[PatWie]

I did a rebase to the master branch and fixed some path+lines in my code to adapt it to this addons-repo.

I will not write the tf-eager-execution-crab.
I hope it is ok to use the tf.compat.v1.disable_eager_execution() to have a proper TensorFlow code preserving my own sanity.

[seanpmorgan]

Lol thanks you're entitled to your opinion.

As part of addons it has to be TF2 eager compatible. I would like to work on this, but will probably wait until dynamic kernels are finalized. Once it's done maybe we can benchmark autograph and graph mode performance of the same layer.

I'll reach back out to you if/when I have questions if thats alright.

[seanpmorgan]

Just glanced at the code it's all the op call it seems so probably will not need to benchmark. I'll see if we can adapt it to eager in the short term. GPU support might wait for dynamic kernels so we don't package 2 addons

Thanks for the contribution!

[PatWie]

The GPU support is already in the BUILD for BAZEL (in the comments). I can do the adjustments for the GPU stuff when dynamic kernels with Cuda is described somewhere.
The unit-tests are quite slow for the CPU version.

Honestly, I tried the eager part for the tests, but I had no time to find a good way to compare CPU and GPU output.

Btw: The workflow from the contribution.md in this repo with the docker image is nice and helpful!

[olesalscheider]

Any news on this?

[seanpmorgan]

This got pushed for some other work, but GPU testing is now setup so I'll try to have this done by the end of the week

[seanpmorgan]

@tensorflow/sig-addons-maintainers @olesalscheider @PatWie Ready for review.

Tests are run in eager and graph, exposed as a Keras Layer and op if needed. GPU compiling / testing works.

There is a fail on py2-gpu that seems unrelated to this as it fails on distort_image_ops_test. It's the same fail from todays CI run and looks like its a bug in tf-core nightly or something.

[PatWie]

I had a look at the changes in the following diff. Most of them are changes I would make. Otherwise, it looks good.

[PatWie]

I would rename theoretical to gradient_from_implementation and numerical to numerical_gradient.

[seanpmorgan]

I do like that re-name, but going to leave it so its consistent with TF Core:
https://github.com/tensorflow/tensorflow/search?utf8=%E2%9C%93&q=theoretical%2C+numerical&type=

[facaiy]

@PatWie Hi, Patrick. Could you add some docstring for this class when time allows? I find its API document lacks necessary information:
https://github.com/tensorflow/addons/blob/r0.5/docs/api_docs/python/tfa/layers/CorrelationCost.md

It seems that we should hide correlation_cost function, and only expose CorrelationCost in public.
https://github.com/tensorflow/addons/blob/r0.5/docs/api_docs/python/tfa/layers/optical_flow/correlation_cost.md

Please correct me if I'm wrong, thank you :-)

[PatWie]

@facaiy . The code is documented (just the c++ part)! Does

addons/tensorflow_addons/custom_ops/layers/cc/ops/correlation_cost_op.cc

Lines 92 to 109 in 7624954

	This layer implements the correlation operation from
	FlowNet Learning Optical Flow with Convolutional Networks (Fischer et al.)
	input_a: A `Tensor` of the format specified by `data_format`.
	input_b: A `Tensor` of the format specified by `data_format`.
	kernel_size: An integer specifying the height and width of the
	patch used to compute the per-patch costs.
	max_displacement: An integer specifying the maximum search radius
	for each position.
	stride_1: An integer specifying the stride length in the input.
	stride_2: An integer specifying the stride length in the patch.
	pad: An integer specifying the paddings in height and width.
	data_format: Specifies the data format.
	Possible values are:
	"NHWC" float [batch, height, width, channels]
	"NCHW" float [batch, channels, height, width]
	Defaults to `"NHWC"`.
	)Doc");

help? Like I already stated, I'm not a big friend of Keras, TF2 and co. I create my graphs as I did since the first TF has been release.

[facaiy]

Unfortunately, I'm afraid that we only provide API document in python side for users. No worries, I filed an issue to seek help from community #509

[PyExtreme]

Hi @PatWie , Since the documentation has already been done in C++ part. I am just copying and pasting it.
As per another task in this issue, I will make the correlation_cost function private.

@seanpmorgan , Please let me know if any other changes are there. I am new to open source and will appreciate your inputs.

Thanks

[WindQAQ]

Hi @PyExtreme, many thanks for the help! What you mentioned above are the points for this issue. Just go ahead so that we could directly give you some feedback during review! Thanks again 😃