Skip to content
This repository has been archived by the owner on Oct 19, 2021. It is now read-only.

support customer-supplied encryption keys #19

Merged
merged 3 commits into from
Apr 22, 2021
Merged

support customer-supplied encryption keys #19

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2c4170e
support customer-supplied encryption keys
xxchan Apr 21, 2021
27c04c3
use []byte for customer key
xxchan Apr 21, 2021
3b124f2
rename encryption pairs
xxchan Apr 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 37 additions & 14 deletions generated.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ go 1.14
require (
cloud.google.com/go/storage v1.14.0
github.com/aos-dev/go-integration-test/v3 v3.0.0-20210205075620-0b779f4b3afc
github.com/aos-dev/go-storage/v3 v3.4.3-0.20210417162535-67db0dd18784
github.com/aos-dev/go-storage/v3 v3.5.1-0.20210421092104-4de248aed1f1
github.com/google/uuid v1.2.0
golang.org/x/oauth2 v0.0.0-20210402161424-2e8d93401602
google.golang.org/api v0.44.0
Expand Down
10 changes: 6 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ github.com/Xuanwo/templateutils v0.0.0-20201216100309-46f73cd4e4b1/go.mod h1:x0q
github.com/aos-dev/go-integration-test/v3 v3.0.0-20210205075620-0b779f4b3afc h1:s8BQqJwWe2yoTGJFOsIxnkwwMsnDqMgkO39yzEP4smw=
github.com/aos-dev/go-integration-test/v3 v3.0.0-20210205075620-0b779f4b3afc/go.mod h1:2VqxZhJVvI1jKTTERWHfCBMZQaJt10xoEJ+CxbfYZlo=
github.com/aos-dev/go-storage/v3 v3.0.1-0.20210205074802-e8a5b22166c2/go.mod h1:JznhvyhPDnETfZ3RyWm3mT/S5ic+uuYFWagSIkUZujc=
github.com/aos-dev/go-storage/v3 v3.4.3-0.20210417162535-67db0dd18784 h1:eeKDDn+8RPtYuk7lrQC48OtTNeSRF/l9P5YHa/Q3omk=
github.com/aos-dev/go-storage/v3 v3.4.3-0.20210417162535-67db0dd18784/go.mod h1:PZJT0Ta7YxVM5QoYoh8Q/X4I6e/z/7gOJqm85Aib4nY=
github.com/aos-dev/go-storage/v3 v3.5.1-0.20210421092104-4de248aed1f1 h1:Kkb/cVJRwc77lXGcnV2ThUC4QUqybiEVrFwq8qnM4V4=
github.com/aos-dev/go-storage/v3 v3.5.1-0.20210421092104-4de248aed1f1/go.mod h1:JFshvl851ZDDXtFGWDFKqkg34QEPH0xuhLJ2LjikZYc=
github.com/aos-dev/specs/go v0.0.0-20210205073047-af8ef94af73d/go.mod h1:XTNlLZtPA1inITyDH5hNnQXVjvvKUvo+lurs5GYB8NA=
github.com/aos-dev/specs/go v0.0.0-20210312090615-23109627848b h1:qIehSnBbr31ATAckM9u9h6gSz+9PkGqu79vTngx6wPw=
github.com/aos-dev/specs/go v0.0.0-20210312090615-23109627848b/go.mod h1:XTNlLZtPA1inITyDH5hNnQXVjvvKUvo+lurs5GYB8NA=
github.com/aos-dev/specs/go v0.0.0-20210420062803-1a60efa2eae3 h1:LiW0Ki0Gw6opu11JwMhxWw5M+V6I9JypJ5eAIp+Rqt4=
github.com/aos-dev/specs/go v0.0.0-20210420062803-1a60efa2eae3/go.mod h1:gNah3KaPJEfysh7uCCX+sYjQC3g2yx2VgBkFlT945Ws=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
Expand Down Expand Up @@ -170,6 +170,8 @@ github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/pelletier/go-toml v1.8.1 h1:1Nf83orprkJyknT6h7zbuEGUEjcyVlCxSUGTENmNCRM=
github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc=
github.com/pelletier/go-toml v1.9.0 h1:NOd0BRdOKpPf0SxkL3HxSQOG7rNh+4kl6PHcBPFs7Q0=
github.com/pelletier/go-toml v1.9.0/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
Expand Down
13 changes: 10 additions & 3 deletions service.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,14 @@ optional = ["default_storage_pairs", "pair_policy", "work_dir"]
optional = ["list_mode"]

[namespace.storage.op.read]
optional = ["offset", "io_callback", "size"]
optional = ["offset", "io_callback", "size", "sse_customer_key"]

[namespace.storage.op.write]
optional = ["content_md5", "content_type", "io_callback", "storage_class"]
optional = ["content_md5", "content_type", "io_callback", "storage_class", "sse_customer_key"]

[pairs.sse_customer_key]
type = "byte_array"
description = "is the customer's 32-byte AES-256 key"

[pairs.default_service_pairs]
type = "DefaultServicePairs"
Expand All @@ -32,4 +36,7 @@ type = "string"
type = "string"

[infos.object.meta.storage-class]
type = "string"
type = "string"

[infos.object.meta.sse_customer_key_sha256]
type = "string"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the content of sse_customer_key_sha256, a base64 string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

base64-encoded SHA-256 hash of the customer-supplied encryption key for the object

6 changes: 6 additions & 0 deletions storage.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,9 @@ func (s *Storage) read(ctx context.Context, path string, w io.Writer, opt pairSt
var rc io.ReadCloser

object := s.bucket.Object(rp)
if opt.HasSseCustomerKey {
object = object.Key(opt.SseCustomerKey)
}
rc, err = object.NewReader(ctx)
if err != nil {
return 0, err
Expand Down Expand Up @@ -171,6 +174,9 @@ func (s *Storage) write(ctx context.Context, path string, r io.Reader, size int6
rp := s.getAbsPath(path)

object := s.bucket.Object(rp)
if opt.HasSseCustomerKey {
object = object.Key(opt.SseCustomerKey)
}
w := object.NewWriter(ctx)
defer func() {
cerr := w.Close()
Expand Down
3 changes: 3 additions & 0 deletions utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -277,6 +277,9 @@ func (s *Storage) formatFileObject(v *gs.ObjectAttrs) (o *typ.Object, err error)
if value := v.StorageClass; value != "" {
sm[MetadataStorageClass] = value
}
if value := v.CustomerKeySHA256; value != "" {
sm[MetadataSseCustomerKeySha256] = value
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about keeping the naming style with SDK?

  • sse_customer_key => customer_key
  • sse_customer_key_sha256 => customer_key_sha256

Or the naming in GCS's docs:

  • encryption_key
  • encryption_key_sha256

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote a RFC about this: beyondstorage/specs#38

}
o.SetServiceMetadata(sm)

return
Expand Down