Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(project): add simple function #3

Merged
merged 2 commits into from
Aug 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 44 additions & 34 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,11 @@ $ go install github.com/oomol-lab/oget
```go
import "github.com/oomol-lab/oget"

task, err := oget.CreateGettingTask(&oget.RemoteFile{
URL: "https://github.com/oomol-lab/oget/raw/main/tests/target.bin",
})
if err != nil {
panic(err)
}
_, err = task.Get(&oget.GettingConfig{
_, err := OGet{
URL: "https://github.com/oomol-lab/oget/raw/main/tests/target.bin",
FilePath: "/path/to/save/file.bin",
})
}.Get()

if err != nil {
panic(err)
}
Expand All @@ -36,20 +32,16 @@ It splits large files into multiple smaller parts for parallel downloading, then
```go
import "github.com/oomol-lab/oget"

task, err := oget.CreateGettingTask(&oget.RemoteFile{
URL: "https://github.com/oomol-lab/oget/raw/main/tests/target.bin",
})
if err != nil {
panic(err)
}
_, err = task.Get(&oget.GettingConfig{
_, err := OGet{
URL: "https://github.com/oomol-lab/oget/raw/main/tests/target.bin",
// Path to store the final file
FilePath: "/path/to/save/file.bin",
FilePath: "/path/to/save/file.bin",
// Number of file parts for splitting and parallel downloading
Parts: 4,
// If not specified, defaults to the same directory as `FilePath`
PartsPath: "/path/to/save/temp/files",
})
}.Get()

if err != nil {
panic(err)
}
Expand All @@ -62,15 +54,10 @@ if err != nil {
```go
import "github.com/oomol-lab/oget"

task, err := oget.CreateGettingTask(&oget.RemoteFile{
URL: "https://github.com/oomol-lab/oget/raw/main/tests/target.bin",
})
if err != nil {
panic(err)
}
var mux sync.Mutex
_, err = task.Get(&oget.GettingConfig{
FilePath: "/path/to/save/file.bin",
_, err := OGet{
URL: "https://github.com/oomol-lab/oget/raw/main/tests/target.bin",
FilePath: "/path/to/save/file.bin",
ListenProgress: func(event oget.ProgressEvent) {
// Callback may be invoked in multiple threads
// Use a lock to ensure thread safety
Expand All @@ -89,7 +76,8 @@ _, err = task.Get(&oget.GettingConfig{
// Total number of bytes in this step
total := event.Total
},
})
}.Get()

if err != nil {
panic(err)
}
Expand All @@ -102,16 +90,12 @@ After downloading, the library performs a SHA512 checksum on the entire file. If
```go
import "github.com/oomol-lab/oget"

task, err := oget.CreateGettingTask(&oget.RemoteFile{
URL: "https://github.com/oomol-lab/oget/raw/main/tests/target.bin",
})
if err != nil {
panic(err)
}
_, err = task.Get(&oget.GettingConfig{
_, err := OGet{
URL: "https://github.com/oomol-lab/oget/raw/main/tests/target.bin",
FilePath: "/path/to/save/file.bin",
SHA512: "d286fbb1fab9014fdbc543d09f54cb93da6e0f2c809e62ee0c81d69e4bf58eec44571fae192a8da9bc772ce1340a0d51ad638cdba6118909b555a12b005f2930",
})
}.Get()

if err != nil {
if sha512Error, ok := err.(oget.SHA512Error); ok {
// Failed due to SHA512 verification failure
Expand All @@ -124,6 +108,32 @@ if err != nil {

During a download, oget creates a temporary file with the extension `*.downloading` (regardless of whether it's split into parts). If a download fails due to network issues and the temporary file is not deleted, resuming the download will retain the progress from the previous attempt. To implement resuming downloads, ignore download failures caused by network issues and retry the download.

```go
import "github.com/oomol-lab/oget"
success := false

for i := 0; i < 10; i++ {
clean, err := OGet{
URL: "https://github.com/oomol-lab/oget/raw/main/tests/target.bin",
FilePath: "/path/to/save/file.bin",
Parts: 4,
}.Get()
if err != nil {
if sha512Error, ok := err.(oget.SHA512Error); ok {
clean()
panic(sha512Error)
}
fmt.Printf("download failed with error and retry %s", err)
} else {
success = true
}
}
if !success {
panic("download fail")
}
```
The above method will send a request to the server for the file meta each time you retry. If you don't want this meaningless request, you can use the following method.

First, create a `task` object using `oget.CreateGettingTask`. This step only fetches file metadata from the server without starting the download. If this step fails, it should be considered a complete failure of the download task.

```go
Expand Down
66 changes: 66 additions & 0 deletions get.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package oget

import (
"context"
"time"
)

type OGet struct {
// the URL of the file to download.
URL string
// the path to save the downloaded file.
FilePath string
// the context for the http request.
// if the value is nil, the context will be context.Background().
Context context.Context
// the maximum amount of time a dial will wait for a connect or copy to complete.
// the default is 10 seconds.
Timeout time.Duration
// the User-Agent header field value.
// if the value is empty, the User-Agent header will not be set.
Useragent string
// the Referer header field value.
// if the value is empty, the Referer header will not be set.
Referer string
// the maximum number of idle (keep-alive) connections to keep per-host.
// the default is 16.
MaxIdleConnsPerHost int
// the SHA512 code of the file.
// if the code is empty, the file will not be checked.
SHA512 string
// PartsPath is the path to save the temp files of downloaded parts.
// if the value is empty, the temp files will be saved in the same directory as the FilePath.
PartsPath string
// the name of the part file.
// if the value is empty, the name will be the same as the file name.
PartName string
// the number of parts to download the file.
// if the value is less than or equal to 0, the file will be downloaded in one part.
Parts int
// the progress listener.
// if the value is nil, the progress will not be listened.
ListenProgress ProgressListener
}

func (o OGet) Get() (func() error, error) {
clean := func() error { return nil }
task, err := CreateGettingTask(&RemoteFile{
Context: o.Context,
Timeout: o.Timeout,
URL: o.URL,
Useragent: o.Useragent,
Referer: o.Referer,
MaxIdleConnsPerHost: o.MaxIdleConnsPerHost,
})
if err != nil {
return clean, err
}
return task.Get(&GettingConfig{
FilePath: o.FilePath,
SHA512: o.SHA512,
PartsPath: o.PartsPath,
PartName: o.PartName,
Parts: o.Parts,
ListenProgress: o.ListenProgress,
})
}
File renamed without changes.
Loading