Sparse data support with example

[alemagnani]

This PR is a replacement of #937 rebased to master and with an added example

This adds some basic support for sparse data in CSR format. The main thing is a SparseBlob to store the sparse data and an extension to InnerProduct that handles both dense and sparse depending on what is presented at the input. A new data layer is added to read data from DBs.
Some more details:

• subclass of Blob that handles matrices in CSR format (SparseBlob)
• subclass of InnerProductLayer (SparseInnerProductLayer) to deal with a sparse blob as input (it handles dense at the same time)
• a new data layer (SparseDataLayer) that reads from a DB sparse vectors and creates SparseBlob
• support for CuSparse
• new sparse math operations on both GPU and CPU
• small change to Syncedmem to make it possible to own/not own GPU data.
• new ProtoBuf object SparseDatum to store sparse vectors.
• changed the way Net creates Blobs to support sparse blobs. Basically the bottom layer gets to decide what Blob to create on top. This way the SparseDataLayer produces SparseBlobs (see the new method in layer_factory.cpp). There could be other ways to achieve the same results.
• exstensive test for the new math function and the new layers
• 2 different implementations of GPU sparse kernels (one is commented out). This is because I found out that CUSparse has very poor performance at least in the way I used it. This is because it support sparse multiplication not for all transpose cases.
• simple example with a bag of word model classification with logistic and network with one hidden layer. Similar in spirit to the "hdf5_classification" example.

[sirotenko]

Thanks for a great PR. Sparsity is very promising topic.
In my experiments some time ago I found that not all sparse libs support sparse-dense operations. Also on a mnist model I found that amount of sparsity needed for similar performance as for a dense case should be about 5% when using cuSparse, dense weights and sparse feature maps.
I'm wondering if you have you tried to compare dense vs sparse nets in terms of performance?

[alemagnani]

Unfortunately I don't have a good comparison available. I could also commit a data layer that makes the sparse data into a dense blob before loading it to the network. This could be used to compare directly the 2 approaches:

1. sparse data times dense weights
2. sparse data -> dense batch times dense weights

[buaaliyi]

Great PR.

I've tried to use this patch to train my sparse data. However, I found there were so many memory re-allocations, due to the different non-zero numbers (nnz) between mini-batch data which requires to reshape the prefetch_data. All those memory alloc/free may cost extra times besides fp/bp computing. Is there any future plan for this issue? Thanks

[bhack]

What do you think of this project https://github.com/btgraham/SparseConvNet/?

[yangjunpro]

Guys, may I know the progress or plan of merging this Sparse support into Caffe master?
For our on-hand tasks, we do need sparse data support, if there isn't any short-term plan for merging this into master, we will build our own or just based on alemagnani 's nice commit.

[beniz]

@yangjunpro I'm interested and can help on / test a merge.

[beniz]

This PR is very useful. We need it to scale to sparse NLP entries, therefore I have reworked it so that it is up to date with our (customized) Caffe master, here: https://github.com/beniz/caffe/tree/master_dd_integ_sparse

It's been a bit of a struggle to refactor the code (1), but everything is working fine on both CPU and GPU (e.g. @alemagnani initial example based on 20 newsgroups), as well as unit tests.

I understand that maintainers either could not find the time to integrate this great piece by @alemagnani or judged that it should be implemented otherwise, but for us, this is great addition, and therefore we will maintain it against Caffe master branch over time.

I could provide a PR or a patch against current master if there's interest in it. Our Caffe version slightly differs from the official one in that we turn the LOG(FATAL) into exceptions, and some other non-invasive modifications, and for this reason the few commits from our custom tree would not apply as is to current Caffe master branch.

FTR, at this point there's nothing we can quantify about the experience reported by @buaaliyi but we will try to provide informative measures.

Thanks @alemagnani for the great initial work!

(1) this includes the templating of DataReader to work with SparseBatch and SparseDatum, with original code now broke down and appearing into BasePrefetchingSparseDataLayer and SparseDataLayer.