hicPlotMatrix cannot be --log transformed

[adklocko]

Hi,
I am running hicexplorer=2.2.2 (latest version??), with all the dependencies updated, on a MacOS 10.12.6. I can plot a heatmap from a matrix just fine (without log tranformation):

hicPlotMatrix --matrix my-matrix.h5 --colorMap coolwarm --chromosomeOrder chromosome-2 --vMin -2 --vMax 2 --dpi 300 --outFileName my-matrix-pic.png

and it gives a picture.

but if I go to --log tranform it

hicPlotMatrix --matrix my-matrix.h5 --colorMap coolwarm --chromosomeOrder chromosome-2 --vMin -2 --vMax 2 --dpi 300 --log --outFileName my-matrix-pic.png

it runs into this error:

Traceback (most recent call last): File "/Users/my-computer/anaconda2/bin/hicPlotMatrix", line 7, in <module> main() File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/hicexplorer/hicPlotMatrix.py", line 614, in main start_pos=start_pos1, start_pos2=start_pos2, pNorm=norm, pAxis=ax1, pBigwig=bigwig_info) File "/Users/my-computeranaconda2/lib/python2.7/site-packages/hicexplorer/hicPlotMatrix.py", line 241, in plotHeatmap cbar = fig.colorbar(img3, cax=cax) File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/figure.py", line 2098, in colorbar cb = cbar.colorbar_factory(cax, mappable, **cb_kw) File "/Users/my-computeranaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 1399, in colorbar_factory cb = Colorbar(cax, mappable, **kwargs) File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 945, in __init__ ColorbarBase.__init__(self, ax, **kw) File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 327, in __init__ self.draw_all() File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 353, in draw_all self._config_axes(X, Y) File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 461, in _config_axes self.update_ticks() File "/Users/my-computeranaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 378, in update_ticks ticks, ticklabels, offset_string = self._ticker() File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 622, in _ticker b = np.array(locator()) File "/Users/my-computeranaconda2/lib/python2.7/site-packages/matplotlib/ticker.py", line 2098, in __call__ return self.tick_values(vmin, vmax) File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/ticker.py", line 2124, in tick_values "Data has no positive values, and therefore can not be " ValueError: Data has no positive values, and therefore can not be log-scaled.

Even though it looks like my matplotlib is the error, I am running version 2.2.3

Any ideas would be most helpful.

[joachimwolff]

Hi,

can you try --log1p

Best,

Joachim

[LeilyR]

Dont give it vmin and vmax and run the default to see the range of your values.

[joachimwolff]

@LeilyR Right, I haven't saw that. First, Hi-C uses count data which can never be smaller 0. Second, the log function is not defined for negative values.
Moreover, a value range of in total 4 doesn't seem too meaningful for me.

[adklocko]

The -2 to 2 is an observed vs. expected matrix, and so log2 transforming it is appropirate for it, as that matrix has fully positive values ranging from 0.02 to 100 (or so). So log2 transforming it makes perfect sense.

Anyway, having a --vMin and --vMax, and a --log flag kicks back an issue:

$ hicPlotMatrix --matrix my-matrix.h5 --colorMap afmhot_r --outFileName test-plotmatrix_log.png --chromosomeOrder chromosome-2 --vMin 0 --vMax 10 --log
INFO:hicexplorer.hicPlotMatrix:min: 4, max: 28540

Traceback (most recent call last):
  File "/Users/my-computer/anaconda2/bin/hicPlotMatrix", line 7, in <module>
    main()
  File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/hicexplorer/hicPlotMatrix.py", line 614, in main
    start_pos=start_pos1, start_pos2=start_pos2, pNorm=norm, pAxis=ax1, pBigwig=bigwig_info)
  File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/hicexplorer/hicPlotMatrix.py", line 241, in plotHeatmap
    cbar = fig.colorbar(img3, cax=cax)
  File "/Users/my-computeranaconda2/lib/python2.7/site-packages/matplotlib/figure.py", line 2098, in colorbar
    cb = cbar.colorbar_factory(cax, mappable, **cb_kw)
  File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 1399, in colorbar_factory
    cb = Colorbar(cax, mappable, **kwargs)
  File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 945, in __init__
    ColorbarBase.__init__(self, ax, **kw)
  File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 327, in __init__
    self.draw_all()
  File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 349, in draw_all
    self._process_values()
  File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colorbar.py", line 705, in _process_values
    b = self.norm.inverse(self._uniform_y(self.cmap.N + 1))
  File "/Users/my-computer/anaconda2/lib/python2.7/site-packages/matplotlib/colors.py", line 1048, in inverse
    return vmin * np.ma.power((vmax / vmin), val)
ZeroDivisionError: float division by zero

Then, changing some of the other commands gave varying results:

$ hicPlotMatrix --matrix my-matrix.h5 --colorMap afmhot_r --chromosomeOrder chromosome-2 --outFileName test-plotmatrix_log-2.png --vMin 0 --vMax 10
INFO:hicexplorer.hicPlotMatrix:min: 4, max: 28540

gave a matrix that was fully black (max scale)

$ hicPlotMatrix --matrix my-matrix.h5 --colorMap afmhot_r --chromosomeOrder chromosome-2 --outFileName test-plotmatrix_log-3.png --vMin 0 --vMax 10000 INFO:hicexplorer.hicPlotMatrix:min: 4, max: 28540
gave a matrix that was more reasonably scaled, but very dark in the middle but light off diagonal.

$ hicPlotMatrix --matrix my-matrix.h5 --colorMap afmhot_r --chromosomeOrder chromosome-2 --outFileName test-plotmatrix_log-4.png INFO:hicexplorer.hicPlotMatrix:min: 4, max: 28540
so with no "--log", "--vMin", or "--vMax" gave essentially no signal off diagonal, and a moderate signal on the diagonal

$ hicPlotMatrix --matrix my-matrix.h5 --colorMap afmhot_r --chromosomeOrder chromosome-2 --outFileName test-plotmatrix_log-5.png --log INFO:hicexplorer.hicPlotMatrix:min: 4, max: 28540
Gave a heatmap that looked like it reported the data consistent with my previous results, but had a scalebar that ranged from 0 to 28000 (suggesting the scale was not log transformed). This was the condition that @LeilyR suggested.

$ hicPlotMatrix --matrix my-matrix.h5 --colorMap afmhot_r --chromosomeOrder chromosome-2 --outFileName test-plotmatrix_log-6.png --log --vMin 1 --vMax 100000 INFO:hicexplorer.hicPlotMatrix:min: 4, max: 28540
gave a heatmap that looks like previous data, had a scalebar that was consistent without being log transformed (although the "--vMin 0" is consistent with a log2 transformation, since log2 0 = inf).

It certainly seems like the log transform function is not working or not being applied correctly...

I hope these conditions help diagnose the problem.

[adklocko]

PS. the "--log1p" flag kicks back a similar error: "float division by zero"

[joachimwolff]

An observed / expect matrix can never have values < 0. The log transformed of it can have values <0, you are right here. However, using vMin and vMax the interaction counts are first transformed to vMin/vMax value range and in a second step the Log function is applied.

The Log is a log10 and not a log2.

$ hicPlotMatrix --matrix my-matrix.h5 --colorMap afmhot_r --chromosomeOrder chromosome-2 --outFileName test-plotmatrix_log-4.png INFO:hicexplorer.hicPlotMatrix:min: 4, max: 28540

Expected behaviour, the difference in the values diagonal vs off-diagonal are so large that all other, smaller differences in the count data is not displayed. Therefore log or log1p should be used.

The rest I have to test it on my own and check src, can take a while.

[adklocko]

Thanks for clarifying the order of operations within the program; that helps to make sense for why the vMin/vMax are applied first and gave heatmaps with count ranges for the vMin/vMax, and then the log10 (not log2; sorry, my mistake) function seems to have an error. I have always used the log10 function to display my data.

Thanks for looking into the error.