Feature: adaptive histogram equalization

[brianpopow]

Prerequisites

• I have written a descriptive pull-request title
• I have verified that there are no overlapping pull-requests open
• I have verified that I am following matches the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
• I have provided test coverage for my change (where applicable)

Description

This PR adds a sliding window implementation of the adaptive histogram equalization (AHE). It also add clipping the histogram, which is supposed to reduce the effect of over-amplifying noise in relatively homogeneous regions (which happens with AHE).

Here is an example input:

and the output:

(click on the image, at least for me the preview was not displaying the image correctly)

This algorithm is computational very demanding, especially with 65536 grey levels.
To be honest, im a bit disappointed about the performance of this implementation with 16 bit greyscale images. It can take up to 30 seconds with an image of 1000x1000 pixels on my machine.
For 256 greylevels it would take ~500 ms.

I don't know if this is good enough for ImageSharp, so i was unsure if i should open the PR in the first place. AHE can produce some pretty nice result, so i though i give it a shot anyways and see what you guys think about it.

[codecov]

Codecov Report

Merging #673 into master will increase coverage by 0.11%.
The diff coverage is 98.81%.

@@            Coverage Diff             @@
##           master     #673      +/-   ##
==========================================
+ Coverage   89.07%   89.19%   +0.11%     
==========================================
  Files        1024     1028       +4     
  Lines       44877    45417     +540     
  Branches     3211     3250      +39     
==========================================
+ Hits        39974    40509     +535     
- Misses       4203     4205       +2     
- Partials      700      703       +3

Impacted Files	Coverage Δ
src/ImageSharp/Common/Helpers/ImageMaths.cs	87.01% <ø> (ø)	⬆️
tests/ImageSharp.Tests/TestImages.cs	100% <ø> (ø)	⬆️
...Normalization/AdaptiveHistEqualizationProcessor.cs	100% <100%> (ø)
...essing/Normalization/HistogramEqualizationTests.cs	100% <100%> (ø)	⬆️
...rmalization/AdaptiveHistEqualizationSWProcessor.cs	100% <100%> (ø)
...Sharp/Processing/HistogramEqualizationExtension.cs	72.72% <72.72%> (+22.72%)	⬆️
...sors/Normalization/HistogramEqualizationOptions.cs	83.33% <83.33%> (ø)
...malization/GlobalHistogramEqualizationProcessor.cs	96.29% <96.29%> (ø)
...rs/Normalization/HistogramEqualizationProcessor.cs	97.29% <96.66%> (-2.71%)	⬇️
... and 4 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2fcba54...f194f4f. Read the comment docs.

[CLAassistant]

All committers have signed the CLA.

[JimBobSquarePants]

It can take up to 30 seconds with an image of 1000x1000 pixels on my machine.
For 256 greylevels it would take ~500 ms.

I'd like to see if we can do anything to improve this as the output is incredible!

[brianpopow]

I'd like to see if we can do anything to improve this as the output is incredible!

@JimBobSquarePants: I am glad you like it. I will try to improve it, but i think the main reason why this approach is slow, is that the distribution function needs to be calculated for each pixel and with 65k grey levels, thats a lot to chew on.

There is a different approach to this: The image would be split into n tiles dependent on how big the gridsize is chosen. For each tile the cdf is pre-calculated. Now the final grey level would be calculated by interpolating between 4 adjacent tiles. This approach should be in theory much faster.
I will give this a try over the next week and see how it goes.

[JimBobSquarePants]

Great! Looking forward to seeing what you come up with 👍

[brianpopow]

I have implemented the tile interpolation approach and it is indeed much faster. Even with 65536 grey levels it now takes around 500 ms to compute.

Here is an example output:

Its still pretty much a work in progress, but i think its going in the right direction. I have still some issues to fix. I need to figure out howto deal with the borders and i think im still doing something wrong with the interpolation. If a small gridsize is choosen like 32, the tile edges seem to be too bright and it looks kind of blocky. Here is an example:

[JimBobSquarePants]

That sounds much more promising!

It looks to me like the sum of your samples is off. When interpolating the sum of your samples should equal 1. This maintains the correct brightness. I’d choose clamping to deal with edge pixels also.

[brianpopow]

@JimBobSquarePants just a quick update: i did not had much time last week to work on this. I hope i find some time this weekend to continue with it.

[JimBobSquarePants]

@brianpopow no worries, looking forward to seeing this complete.

[JimBobSquarePants]

@brianpopow Finally getting round to having another look at this to get it finished.

Two questions:

1. Is it a requirement of the algorithm to mirror edges? We can optimize things quite a lot if we can remove the reflection and simply reuse the edge pixels (we do that when resampling in the resizer).
2. Would it be possible to gather the histogram values for each tile by row instead of per column? We could simply slice then copy.

[brianpopow]

@JimBobSquarePants:

1. Mirroring the edges is the best way i know to deal with the borders of the image where there is no data. I do not understand exactly what you mean by reusing the edge pixels. Can you point me to an example source code line where this is done in th resizer?

2. We now move the window from left to right. This means when we move one pixel, we need to remove one column from the left and add another on the right. When i was moving the window from top to bottom i could read a row (with the size of the window) when moving one pixel down, but you said this is not a good idea.
   Maybe i do not understand exactly what you mean or you see something i do not see here, but when moving the window from left to right we need to read a column when moving the window.

Im sorry that i could not be more helpful here.

[JimBobSquarePants]

Hi @brianpopow

1. Mirroring the edges is the best way i know to deal with the borders of the image where there is no data. I do not understand exactly what you mean by reusing the edge pixels. Can you point me to an example source code line where this is done in th resizer?

It's actually pretty simple, instead of mirroring you reuse the 0 and source.Width pixels when sampling. So in your current code you have:

if (x < 0)
{
    x = Math.Abs(x);
}
else if (x >= source.Width)
{
    int diff = x - source.Width;
    x = source.Width - diff - 1;
}

In the most simplistic terms you would have

if (x < 0)
{
    x = 0;
}
else if (x >= source.Width)
{
    x = x - source.Width -1;
}

This yields a different result on the image edges but the algorithm still works. I'm fairly certain we can optimize that example further though if we were going per-row instead of per-column within the tile.

2. We now move the window from left to right. This means when we move one pixel, we need to remove one column from the left and add another on the right. When i was moving the window from top to bottom i could read a row (with the size of the window) when moving one pixel down, but you said this is not a good idea.
Maybe i do not understand exactly what you mean or you see something i do not see here, but when moving the window from left to right we need to read a column when moving the window.

There must be something I am not understanding about the algorithm.

When I first suggested going top down it was because you were using a parallel for 0 => source.Width which meant you couldn't slice a row to operate on it. You're now going 0 => source.Height on the outermost loop now which is good but you're now operating per column within the tile and I don't understand why?

8f19e5e#diff-6ca0f7e3ceae4a5676d95d4de12f7b8bR62

Im sorry that i could not be more helpful here.

Never say that chum, your work is amazing.

[brianpopow]

@JimBobSquarePants:

1. ok i understand now what you mean, but we can not do that. That would over amplify the edge pixels in the border. The edge pixels would be added multiple times to the histogram, worst case is the corners of the image where it would lead adding the same value as often as the window width.

One other option is to just ignore the pixels outside of the image and do not add them. Im not sure, if that a good idea either. I have to try that and see how it looks.

Another option i have found is here: https://digitalcommons.unf.edu/cgi/viewcontent.cgi?referer=https://www.google.de/&httpsredir=1&article=1264&context=etd
See Section Image Border, Page 26.
They suggest to keep the window in place in the corners (what i call window, they call contextual region)
I think that sounds like a good alternative, but im not sure at the moment, if this will be better in performance than the mirroring approach.

2. Moving the sliding window around the image is comparable to other filtering methods, where you apply one filter mask to a region around a pixel. Lets say like a Sobel filter, but here the mask / window / contextual region is a bit bigger (typically something like 64 by 64 pixels). This window needs to be moved over each pixel. It does not matter if its from left to right or from top to bottom, but it needs to be moved over all pixels.
   All pixels which are covered by the window needs to be added to the histogram. To not add all pixels under the window to the histogram again when you move one pixel, you only have to add one column to the right and remove one column from the left (if you move from left to right).

Here is an example image, maybe that makes it clearer:

If you we move the window from top to bottom, which was the case in the commit you pointed out, we would add one row on the bottom and remove one from the top.

I hope this makes it a bit clearer, let me know if you have still any questions.

Unfortunately i think i will not find time to continue with this in this week. Maybe i can find some time next week.

[JimBobSquarePants]

ok i understand now what you mean, but we can not do that. That would over amplify the edge pixels in the border. The edge pixels would be added multiple times to the histogram, worst case is the corners of the image where it would lead adding the same value as often as the window width

Reading the linked article I understand now that sampling is more accurate away from the edge border so mirroring seems to be the valid approach.

If you we move the window from top to bottom, which was the case in the commit you pointed out, we would add one row on the bottom and remove one from the top.

Definitely preferred over columns as we can do span.Slice(x, range).Copy() for anything other than edge pixels.

Gonna have a good read through the article. It explains the process very clearly. 👍

[brianpopow]

@JimBobSquarePants: ok, so we should keep mirroring and switch back to moving the window from top to bottom? I will do that, if you agree to this.

[JimBobSquarePants]

Sorry... on holiday.

Yeah, grab per row but do a fast and slow path since we only need mirroring on edges.

[brianpopow]

@JimBobSquarePants im sorry for no update on this one for so long, i really wanted to finish it, but i cannot find time recently to continue on this, because of too much work currently at my job. I will try to get back on this next weekend.

[JimBobSquarePants]

@brianpopow No worries, thanks again for your help!

[brianpopow]

@JimBobSquarePants: I finally found some time to continue on this. As discussed, the sliding window is moved from top to bottom again, so we can add a row to the histogram when it moves down one step.

I have changed the mirroring, so it will only be done on the edges now.

[JimBobSquarePants]

@brianpopow Excellent!

Looks like we need to update the reference image again as the output is now different. I'll review this asap. 👍

[brianpopow]

@JimBobSquarePants: yes, but lets update the reference image after the review is done.

[JimBobSquarePants]

Congrats @brianpopow we've finally made it!

I ran a pass of the code and did the final performance tweaks myself removing some bounds checks so that you weren't left doing more work.

I'm very happy with this now, thanks so much for your patience and persistance. It's an amazing bit of functionality!

[brianpopow]

@JimBobSquarePants: Great, thank you! This took quite some time, but i still think it was worth it.