Document requirements for implementing a DataStore to serve ledger metadata

[tamirms]

galexie is a service which generates LedgerCloseMeta XDR objects and stores them in a repository to allow queries for ledgers which are faster and easier than running captive core.

DataStore is the Go interface which defines all storage operations required by galexie. Currently, there is only 1 concrete implementation of the DataStore interface which is backed by GCS. However, we anticipate that there will be other DataStore implementations backed by other storage engines. For example, a DataStore implementation could be backed by S3, a local file system, or even a relational database like postgres.

However, the requirements for a DataStore implementation are not documented in any specification. We should create a SEP which provides a full specification for the DataStore interface so that third party developers can create their own DataStore implementations that can be used interchangeably by galexie and galexie clients.

[urvisavla]

There are a couple of open items that could influence the datastore specification:

• Co-locate schema settings on datastore (Galexie: co-locate schema settings on datastore #5507)
• Simplifying the append functionality (resumability) (services/galexie: Simplify resumability logic #5547)

Co-locating the schema on the datastore might not be strictly necessary for the formal specification but I think it could be useful. However, the proposed changes in #5547 to simplify the append logic will definitely require the datastore to support listing objects in a specific order.

Imo, tackling these issues will help create a more robust interface that enables seamless datastore interchangeability in Galexie.

[sreuland]

The pluggable datastore design, #5526, would be a related effort that is pending, if that comes to reality, then it would redefine what constitutes the interface of a Datastore.