Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add appender support #40

Merged
merged 7 commits into from
Apr 28, 2021
Merged

Add appender support #40

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a537bd1
Add appender support
Apr 23, 2021
1b20f44
modify newObject stat parameter and add optional parameters for append
JinnyYi Apr 25, 2021
39c09bd
add the XQSNextAppendPosition pointer nil check
JinnyYi Apr 25, 2021
bad705e
change the way to get storageClass
JinnyYi Apr 25, 2021
2e4e311
Implement commitAppend
Apr 27, 2021
a666278
modify append object metadata setting and add isAppend check
Apr 28, 2021
b2fa995
set append position
Apr 28, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions generated.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ go 1.14

require (
bou.ke/monkey v1.0.2
github.com/aos-dev/go-integration-test/v3 v3.0.0-20210420062414-e972ad8801cc
github.com/aos-dev/go-storage/v3 v3.5.1-0.20210421092104-4de248aed1f1
github.com/aos-dev/go-integration-test/v3 v3.0.0
github.com/aos-dev/go-storage/v3 v3.5.1-0.20210423013030-4d27effb22ca
github.com/golang/mock v1.5.0
github.com/google/uuid v1.2.0
github.com/pengsrc/go-shared v0.2.1-0.20190131101655-1999055a4a14
Expand Down
12 changes: 5 additions & 7 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,11 @@ github.com/Xuanwo/go-bufferpool v0.0.0-20200622083641-bc954721ce54 h1:wA7f87ODtF
github.com/Xuanwo/go-bufferpool v0.0.0-20200622083641-bc954721ce54/go.mod h1:Mle++9GGouhOwGj52i9PJLNAPmW2nb8PWBP7JJzNCzk=
github.com/Xuanwo/templateutils v0.0.0-20201216100309-46f73cd4e4b1 h1:SQ8kQujQ0yl39YBMU0xZ+yyfxx/qSWfGWlCPNzHp5kw=
github.com/Xuanwo/templateutils v0.0.0-20201216100309-46f73cd4e4b1/go.mod h1:x0qS7gfgEm24b8V9U+0zBEBAu/VpxZhO/+EWtpgEgDw=
github.com/aos-dev/go-integration-test/v3 v3.0.0-20210420062414-e972ad8801cc h1:ha0GpuSyD/R19if+4D6QCKF6j7UvajiL2LF17xpqKTs=
github.com/aos-dev/go-integration-test/v3 v3.0.0-20210420062414-e972ad8801cc/go.mod h1:2i6Itxp4ScMeZbaJetPPo+b283GYExF9i9Dlcr8E9Y0=
github.com/aos-dev/go-storage/v3 v3.4.2/go.mod h1:PZJT0Ta7YxVM5QoYoh8Q/X4I6e/z/7gOJqm85Aib4nY=
github.com/aos-dev/go-storage/v3 v3.5.1-0.20210421092104-4de248aed1f1 h1:Kkb/cVJRwc77lXGcnV2ThUC4QUqybiEVrFwq8qnM4V4=
github.com/aos-dev/go-storage/v3 v3.5.1-0.20210421092104-4de248aed1f1/go.mod h1:JFshvl851ZDDXtFGWDFKqkg34QEPH0xuhLJ2LjikZYc=
github.com/aos-dev/specs/go v0.0.0-20210312090615-23109627848b h1:qIehSnBbr31ATAckM9u9h6gSz+9PkGqu79vTngx6wPw=
github.com/aos-dev/go-integration-test/v3 v3.0.0 h1:rxIc7YBfiw9JS5JiVU4BZAqQqoKJemIftFaeJpYCb5M=
github.com/aos-dev/go-integration-test/v3 v3.0.0/go.mod h1:woC3E9Ld1G/Cpo2tSEQ+iwIr4MMdgf+L/8UuyJOE11Q=
github.com/aos-dev/go-storage/v3 v3.5.0/go.mod h1:PZJT0Ta7YxVM5QoYoh8Q/X4I6e/z/7gOJqm85Aib4nY=
github.com/aos-dev/go-storage/v3 v3.5.1-0.20210423013030-4d27effb22ca h1:H7wkWwIe/7Kz1+uAFBhxDYh8GX1kcyxsUIBxPE12ksk=
github.com/aos-dev/go-storage/v3 v3.5.1-0.20210423013030-4d27effb22ca/go.mod h1:JFshvl851ZDDXtFGWDFKqkg34QEPH0xuhLJ2LjikZYc=
github.com/aos-dev/specs/go v0.0.0-20210312090615-23109627848b/go.mod h1:XTNlLZtPA1inITyDH5hNnQXVjvvKUvo+lurs5GYB8NA=
github.com/aos-dev/specs/go v0.0.0-20210420062803-1a60efa2eae3 h1:LiW0Ki0Gw6opu11JwMhxWw5M+V6I9JypJ5eAIp+Rqt4=
github.com/aos-dev/specs/go v0.0.0-20210420062803-1a60efa2eae3/go.mod h1:gNah3KaPJEfysh7uCCX+sYjQC3g2yx2VgBkFlT945Ws=
Expand Down Expand Up @@ -39,7 +38,6 @@ github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORN
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/pelletier/go-toml v1.8.1 h1:1Nf83orprkJyknT6h7zbuEGUEjcyVlCxSUGTENmNCRM=
github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc=
github.com/pelletier/go-toml v1.9.0 h1:NOd0BRdOKpPf0SxkL3HxSQOG7rNh+4kl6PHcBPFs7Q0=
github.com/pelletier/go-toml v1.9.0/go.mod h1:u1nR/EPcESfeI/szUZKdtJ0xRNbUoANCkoOuaOx1Y+c=
Expand Down
2 changes: 1 addition & 1 deletion service.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ optional = ["location"]
optional = ["location"]

[namespace.storage]
implement = ["copier", "fetcher", "mover", "multiparter", "reacher"]
implement = ["copier", "fetcher", "mover", "multiparter", "reacher", "appender"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make them in alphabet order.


[namespace.storage.new]
required = ["name"]
Expand Down
34 changes: 34 additions & 0 deletions storage.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,15 @@ func (s *Storage) create(path string, opt pairStorageCreate) (o *Object) {
return o
}

func (s *Storage) createAppend(ctx context.Context, path string, opt pairStorageCreateAppend) (o *Object, err error) {
o = s.newObject(false)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be s.newObject(true)

o.Mode = ModeRead | ModeAppend
o.ID = s.getAbsPath(path)
o.Path = path
o.SetAppendOffset(0)
return o, nil
}

func (s *Storage) createMultipart(ctx context.Context, path string, opt pairStorageCreateMultipart) (o *Object, err error) {
input := &service.InitiateMultipartUploadInput{}
if opt.HasEncryptionCustomerAlgorithm {
Expand Down Expand Up @@ -484,6 +493,31 @@ func (s *Storage) write(ctx context.Context, path string, r io.Reader, size int6
return size, nil
}

func (s *Storage) writeAppend(ctx context.Context, o *Object, r io.Reader, size int64, opt pairStorageWriteAppend) (n int64, err error) {
rp := o.GetID()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check IsAppend here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is isRead also a necessary check?

Copy link
Contributor

@Xuanwo Xuanwo Apr 28, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't need to check IsRead during WriteAppend operation.

offset, ok := o.GetAppendOffset()
if !ok {
err = fmt.Errorf("append offset is not set")
return
}

output, err := s.bucket.AppendObjectWithContext(ctx, rp, &service.AppendObjectInput{
Position: &offset,
ContentLength: &size,
Body: io.LimitReader(r, size),
})
if err != nil {
return
}

offset = *output.XQSNextAppendPosition
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we need to check it before use. The value could be a nil.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder whether the current data uploaded successfully when the err is nil but the XQSNextAppendPosition is also nil. And what kind of return values are more appropriate in this situation?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's won't happen in the normal situation because we know the behavior of qingstor service and it's SDK. But we can't trust them because the behavior could be changed in the future (like API changed or a bug in SDK).

So check the value is a safeguard: No matter how the behavior changes, it's won't lead panic in our logic and our users applications.

I plan to formalize the errors returned in service, for now, let's just returns a errors.New("next append position is empty").

o.SetAppendOffset(offset)

return offset, nil

}

func (s *Storage) writeMultipart(ctx context.Context, o *Object, r io.Reader, size int64, index int, opt pairStorageWriteMultipart) (n int64, err error) {
if o.Mode&ModePart == 0 {
return 0, fmt.Errorf("object is not a part object")
Expand Down
7 changes: 7 additions & 0 deletions tests/storage_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,10 @@ func TestMultiparter(t *testing.T) {
}
tests.TestMultiparter(t, setupTest(t))
}

func TestAppend(t *testing.T) {
if os.Getenv("STORAGE_QINGSTOR_INTEGRATION_TEST") != "on" {
t.Skipf("STORAGE_QINGSTOR_INTEGRATION_TEST is not 'on', skipped")
}
tests.TestAppender(t, setupTest(t))
}
1 change: 1 addition & 0 deletions utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ type Storage struct {
typ.UnimplementedMover
typ.UnimplementedMultiparter
typ.UnimplementedReacher
typ.UnimplementedAppender
}

// String implements Storager.String
Expand Down