Skip to content
This repository has been archived by the owner on Jul 9, 2021. It is now read-only.

GSP-111: Add System Metadata In Storage Metadata #111

Merged
merged 7 commits into from
Jun 21, 2021
Merged
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions rfcs/111-add-service-metadata-in-storage-metadata.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
author: JinnyYi <github.com/JinnyYi>
status: draft
updated_at: 2021-06-17
---

# GSP-111: Add Service Metadata In Storage Metadata

## Background

In [GSP-6], we split storage meta and object meta. In [GSP-40], we split all object metadata into four groups: `global` metadata, `standard` metadata, `service` metadata and `user` metadata. For storage, there's no `standard` metadata and user defined metadata is not allowed.

For now, storage related information carried in `StorageMeta`:

```go
type StorageMeta struct {
location string
Name string
WorkDir string

// bit used as a bitmap for object value, 0 means not set, 1 means set
bit uint64
m map[string]interface{}
}
```

It only defines the `global` metadata for storage which could be used in all services, but no field to carry the metadata used in special services.

## Proposal

So I propose to add service metadata in storage metadata.

`service` metadata is provided by service, could only be used in current service.

The `global` metadata has been included in object type. `service` metadata will be stored in a struct that defined by service:

- service metadata will be `serviceMetadata interface{}` in `StorageMeta`.

For services side:

Example of adding service pair for storage metadata in `service.toml`:

```go
[infos.storage.meta.<service-meta>]
type = "<type>"
description = "<description>"
```

For service metadata, we will introduce `Strong Typed Service Metadata` like `ObjectMetadata`. We will generate following struct for specific service according to service pairs:

```go
type ServiceMetadata struct {
<service meta> <type>
...
}
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand correctly?

  • In go-storage/types, there are type Object (has field serviceMetadata interface{}), and StorageMeta
  • In go-service-* type ObjectMetadata is used to set serviceMetadata, and has getter GetObjectMetadata

Now, this proposal intends to also add the field serviceMetadata interface{} to StorageMeta. Should the go-service-* type be called StorageMetadata?

And I'm wondering why type Object and StorageMeta are at the same level 🤔

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to rename go-service-*.ObjectMetadata into something like go-service-*.ServiceObjectMetadata to avoid confusion? (and correspondingly, ServiceStorageMeta in this proposal.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The root cause is Object is a struct but Storager is an interface.

If Storage is a struct that exposed to the user, we can set serviceMetadata in Storage struct just like we do on Object.

Well, StorageMeta and StorageMetadata do look wired to me too. Do you have a better idea/name for that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the most wired one is that both go-storage.Object and go-storage.StorageMeta will have field serviceMetadata,

but go-service-*.ServiceMetadata will be the one that belongs to StorageMeta. This seems to be rather confusing

Copy link
Contributor

@Xuanwo Xuanwo Jun 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got it.

The problem is introduced by simplify

  • Service Level/Specified Object Metadata to serviceMetadata in Object
  • Service Level/Specified Storage Metadata to serviceMetadata in StorageMeta

But I don't have better ideas about the name.

Copy link
Contributor

@xxchan xxchan Jun 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about ServiceObjectMetadata and ServiceStorageMetadata?

Naming is so difficult!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can find the opposite word of global to represent all those situations? So we can:

  • Distinguish Service Pair and Service's Pair
  • Distinguish Service Metadata and Service's Metadata
  • ...

And our API will turn into:

func GetObjectXxxxMetadata(o* Object) ObjectXxxxMetadata
func GetStorageXxxxMetadata(s *StorageMeta) StorageXxxxMetadata

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Distinguish Service Pair and Service's Pair
  • Distinguish Service Metadata and Service's Metadata

Do I understand correctly?

  • Service Pair could be all the pairs in the current service. Service's Pair just means pairs related to the service implementations.
  • Service Metadata includes object service metadata and storage service metadata. Service's Metadata could corresponds to StorageMeta here.


And add following generated functions in service packages:

```go
// Only be used in service to set ServiceMetadata into StorageMeta.
func setServiceMetadata(s *StorageMeta, sm ServiceMetadata) {}
// GetServiceMetadata will get ServiceMetadata from StorageMeta.
func GetServiceMetadata(s *StorageMeta) ServiceMetadata {}
```

## Rationale

This design is highly influenced by [GSP-40].

## Compatibility

No break changes.

## Implementation

- `specs`
- Add `service-metadata` with type `any` in [info_storage_meta.toml].
- `go-storage`
- Generate `serviceMetadata` and corresponding `get/set` functions into `StorageMeta`.
- Support generate `ServiceMetadata` distinguish with `ObjectMetadata` by `Infos.Scope`.


[GSP-6]: ./6-normalize-metadata.md
[GSP-40]: ./40-unify-object-metadata.md
[info_storage_meta.toml]: ../definitions/info_storage_meta.toml