Data Store Implementation

[SidHarder]

No description provided.

[SidHarder]

@diasdavid Does it make sense that I will need to create a js-level-pull-blob-store which implements interface-pull-blob-store? I see that js-idb-pull-blob-store already exists.

[daviddias]

will need to create a js-level-pull-blob-store

Yes, it does. Here is the non pull version https://github.com/diasdavid/level-blob-store :)

[daviddias]

@SidHarder I'm not sure if I dreamed, but I was pretty confident that I got an email from you asking a bunch of stuff and I just can't find it right now, could you confirm? Apologies to keep you in the way (if it was indeed not a dream)

[SidHarder]

@diasdavid I haven't been able to spend much time in the last couple of weeks but I am back to work on this project now. I would be happy to help with other items if you need some help getting the current sprint completed.

[SidHarder]

@diasdavid David I am stuck in the mud on this one. Would you have a few minutes to talk this week?

[dignifiedquire]

@SidHarder I should be able to find some time to chat about this tomorrow. Not sure how available @diasdavid is this week

[kumavis]

@diasdavid whats datastore as opposed to blockstore ?

[SidHarder]

@kumavis The result of the command ipfs init creates some folders one of which is blockstore and another is datastore. In the Go implementation of ipfs the datastore folder is a leveldb key value store which contains a few keys. With js-ipfs the browser will use an indexdb data store and the node version will use leveldb and will contain the same keys that go-ipfs does. In js-ipfs the datastore module takes an interface-pull-blob-store and provides a way to get key/values in and out of the store using pull streams. Interface-pull-blob-store has three implementations which allow you to read and write keys to LevelDb, IndexDB, and to the file system. I think the datastore may be in the process of being deprecated, but I'm not sure what exactly the plan is.

…

On Mon, Feb 13, 2017 at 7:57 PM, kumavis ***@***.***> wrote: @diasdavid <https://github.com/diasdavid> whats datastore as opposed to blockstore ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#111 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABhliQBZPYE5yestiv8ogRDTt9jp7IOuks5rcRgqgaJpZM4LUHZv> .

[daviddias]

@SidHarder so sorry for this delay, I've been flying all over the world and super heads down with the tasks at hand, didn't get to see this thread until now.

what's the difference between datastore vs blockstore

This is an excellent question that caused a lot of confusion to myself initially too.

datastore is a really old project that @jbenet started to create a unified DB interface for multiple storage engines: https://github.com/datastore/datastore. circa 2013. It is essentially a combination of LevelDB + blob-store, although blob-store was started way after, so in reality is a reduced datastore

In IPFS land, we have two folders where we store most of the data:

• blockstore
• datastore

Initially, every block was stored in datastore, which in go-ipfs uses the datastore interface in conjunction with a LevelDB driver. However, LevelDB proved to be inefficient and so a decision was made to create blockstore to store just the blocks, keeping datastore to store DHT records, pinset root hashes, mfs root hashes and so on. blockstore is implemented using a datastore interface, but with a fs driver.

Time passes and the js-ipfs development kicks off, since in the beginning we just needed blockstore to work and since we could reuse all of the work done by blob-store and its ecosystems of adapters, the decision was to 'go ahead and use blob-stores to implement datastore'. Today we departed from that ecosystem and created our own that uses pull-streams interface-pull-blob-store.

However, this is not enough and to move forward, we really need to implement the remaining API calls that datastore interface offers, and so, the #112 endeavor is born.

Lesson of the story: Naming things is hard, but it is even harder if you give the same name to everything

[daviddias]

The comment above should be captured for the README as documentation for the future.

As for the PR, @dignifiedquire pushed an entire refactor of the module that includes datastore here: #123

@SidHarder I'm sorry that you spent so much time on this PR. It was valuable to us to understand how pressing it was and to have the discussion to actually make up our minds in our we were going to fully achieve this. Thank you.

[daviddias]

Closing in favour of #123