h5 to cool to cooltools compartments calling - KeyError: 'ignore_diags'

[kimj50]

Thank you for the amazing tool. I'm a biology phd student. So I am having a bit trouble working with tools.

#346
In light of the above issue, I wanted to try cooltools to call compartments.
I tried the following:
bowtie2 -> buildmatrix -> mergebins -> correction by ICE -> convertformat to cool -> cooltools
After cool format conversion:
File: 30kb_corrected.cool
Date: 2019-09-26T09:07:39.662113
Genome assembly: unknown
Size: 3,341
Bin_length: 30000
Number of chromosomes: 6
Non-zero elements: 5,384,106
The following columns are available: ['chrom' 'start' 'end' 'weight']

Generated by: HiCMatrix-11
Cooler library version: cooler-0.8.6.post0
HiCMatrix url: https://github.com/deeptools/HiCMatrix

Error using cooltools:
Traceback (most recent call last):
File "/share/apps/hicexplorer/3.2/bin/cooltools", line 10, in
sys.exit(cli())
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooltools/cli/call_compartments.py", line 146, in call_compartments
sort_metric=None)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooltools/eigdecomp.py", line 314, in cooler_cis_eig
if ignore_diags is None
KeyError: 'ignore_diags'

I was wondering if I skipped some steps... or is the converted cool format just not compatible with cooltools? Thank you - Jun

[gtrichard]

Hi kimj50,

You did everything well, this is a problem on the cooltools part.

I faced the same issue recently (as of yesterday actually). Since I'm a nice guy, here's the solution (although, it is not my call to fix that bug, as it is a cooltools problem):

Go to /share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooltools/eigdecomp.py line 314 and replace None by 2, it will fix your problem and call the compartments. I'm posting this issue on their github as of writing.

After some tests, HiCExplorer compartment calling is performing more or less like cooltools. In a nutshell, if my bet is not wrong, you are working in Drosophila and you will still face the issue of having entire chromosomes with the correct information stored in EV2 instead of EV1 for some chromosomes (wrong EVs are looking like all positive values on one side of a chromosome arm, then all negative values on the other side). It will happen unless you remove the centromers from your matrices. This you can do with hicAdjustMatrix, but it will break cooltools compartments calling.

[gtrichard]

Here is the link of the cooltools issue I posted if you want to check the cool people answer:

open2c/cooltools#98

[kimj50]

Thank you for the quick reply and being nice :).
I am working working with c.elegans genome, but your explanation is right and helpful. I did see that PCA --norm EV2 better correlates with active/inactive marks, although for half of the chromosomes I just get crappier version of EV1. And like you mentioned removing the arms with AdjustMatrix does improve EV1.
I still ran into the problem with cooltools:
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
Any quick fix I can try? Thanks! - Jun

[gtrichard]

Can you post the full error message?

[kimj50]

File "/share/apps/hicexplorer/3.2/bin/cooltools", line 10, in
sys.exit(cli())
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooltools/cli/call_compartments.py", line 146, in call_compartments
sort_metric=None)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooltools/eigdecomp.py", line 341, in cooler_cis_eig
eigvals_per_reg, eigvecs_per_reg = zip(*map(_each, regions))
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooltools/eigdecomp.py", line 337, in _each
sort_metric=sort_metric)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooltools/eigdecomp.py", line 114, in cis_eig
if A.shape[0] <= ignore_diags + 3 or mask.sum() <= ignore_diags+3:
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

in addition to changing "none" to "2", I also tried changing "is none" to "== 2". They give me the same error. Thank you

[nvictus]

Hi @kimj50,
I made a tiny PR (https://github.com/mirnylab/cooltools/pull/101/files). Since you're already editing the source, you could give it a try immediately.

[kimj50]

It works great!
I have another request/question...
Is there a way to perform call-compartments on a region of matrix? ie chrI:4,000,000-10,000,000?
This goes back to what Richard mentioned above: the hicexplorer hicAdjustMatrix function, which allows me to subset the matrix, breaks call-compartments function.
Essentially, I want to try: subset matrix -> call-compartment.

error msg if hicAdjustMatrix -> call-compartments:
Traceback (most recent call last):
File "/share/apps/hicexplorer/3.2/bin/cooltools", line 10, in
sys.exit(cli())
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooltools/cli/call_compartments.py", line 146, in call_compartments
sort_metric=None)
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooltools/eigdecomp.py", line 315, in cooler_cis_eig
if ignore_diags is None
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/cooler/api.py", line 109, in _load_attrs
return dict(grp[path].attrs)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/share/apps/hicexplorer/3.2/lib/python3.6/site-packages/h5py/_hl/group.py", line 264, in getitem
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'weight' doesn't exist)"

Biologically, we want to try this because we work with chromosomes that have two big flanking arms and a center. And we think there are compartments within compartments that is worth visualizing.

Thank you,
Jun

[gtrichard]

Well I think it is because hicAdjustMatrix is removing the weights somehow...

[joachimwolff]

As far as I know 'weight' is optional, so that's an error of the cooler API.