v2.1.8

Changelog

• c29edd2 Fix issues in the objectstorage based on JuiceFS use cases. (#2648)
• d9fb58f chore(deps): bump actions/checkout from 3 to 4 (#2695)
• 4e1e32b chore(deps): bump github.com/aliyun/aliyun-oss-go-sdk from 2.2.8+incompatible to 2.2.9+incompatible (#2676)
• f4d5cb8 chore(deps): bump github.com/aws/aws-sdk-go from 1.44.327 to 1.45.2 (#2697)
• 667abf8 chore(deps): bump github.com/go-playground/validator/v10 from 10.15.0 to 10.15.1 (#2678)
• bb3e27c chore(deps): bump github.com/google/uuid from 1.3.0 to 1.3.1 (#2696)
• f151d1f chore(deps): bump github.com/onsi/ginkgo/v2 from 2.11.0 to 2.12.0 (#2698)
• 127933f chore(deps): bump go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin from 0.40.0 to 0.43.0 (#2700)
• 854af38 chore(deps): bump go.opentelemetry.io/otel/exporters/jaeger from 1.15.1 to 1.17.0 (#2699)
• 06ce83f chore(deps): bump go.opentelemetry.io/otel/trace from 1.16.0 to 1.17.0 (#2677)
• 3282eec chore(deps): bump google.golang.org/api from 0.136.0 to 0.138.0 (#2680)
• 79095f6 chore: maintainer in alphabetical order of the company (#2687)
• 02de884 chore: update api verison to v2.0.25 (#2685)
• 6be1a4f feat: add idc and location to ListSchedulers in manager (#2674)
• 170a7f2 feat: change file mode when export task from local in dfcache (#2701)
• 85b4467 feat: initialize object storage client in the New func (#2682)
• 9c9959f feat: optimize GetObjectMetadatasInput validate in dfstore (#2686)
• c0522c1 feat: remove peer index in databae table (#2675)
• db4dcaf feat: use go-version-file: go.mod for go version detecting (#2694)
• 51596b6 fix: Vertex.DeleteInEdges and Vertex.DeleteOutEdges functions are not thread safe (#2688)

v2.1.7

Changelog

• 0c8bc5e chore(deps): bump github.com/aws/aws-sdk-go from 1.44.317 to 1.44.327 (#2659)
• 036e32c chore(deps): bump github.com/jarcoal/httpmock from 1.3.0 to 1.3.1 (#2660)
• 05aa9da chore(deps): bump gorm.io/gorm from 1.25.3 to 1.25.4 (#2658)
• bb4af5d chore(deps): bump k8s.io/component-base from 0.27.4 to 0.28.0 (#2657)
• 51c1f9b feat: add createSyncPeers to async job in manager (#2664)
• dfde8bd feat: add sync peer job for scheduler (#2663)
• fe28ba4 feat: merge sync peer with peer table in manager (#2668)
• f3b9290 feat: peer announces scheduler cluster id to scheduler (#2652)
• c04462e feat: peer information is changed from being stored in metrics to being stored in mysql (#2654)
• 0c4d779 feat: remove IsPrivate from safeSocketControl (#2672)

v2.1.6

Changelog

• 80f88d3 chore: update console version to v1.0.9 (#2656)

v2.1.5

Changelog

• 5b7ea5d Revert "feat: replace fmt.Sprintf with net.JoinHostPort" (#2649)
• bcc1286 chore: update console version to 1.0.8 (#2655)
• 4eac5e9 chore: update console version to v1.0.6 (#2650)
• bf6e08b chore: update console version to v1.0.7 (#2651)
• 35da7a7 feat: change max PerPage to 10000000 (#2653)
• a6863f9 feat: replace fmt.Sprintf with net.JoinHostPort (#2642)

v2.1.4

Changelog

• d67eb8b chore: update console to v1.0.5 (#2640)

v2.1.3

Changelog

• 0e04a56 chore: check task id (#2636)
• 0eaf429 chore: skip export to exist file (#2637)
• 860d945 chore: update console to v1.0.4 (#2639)
• 554b27c fix: add traffic (#2634)

v2.1.2

Changelog

• 5fb1ef9 chore: update console to v1.0.3 (#2638)

v2.1.1

Changelog

• 761f5ef chore(deps): bump github.com/aws/aws-sdk-go from 1.44.312 to 1.44.317 (#2606)
• 9446250 chore(deps): bump github.com/go-playground/validator/v10 from 10.14.1 to 10.15.0 (#2608)
• 47523d7 chore(deps): bump github.com/shirou/gopsutil/v3 from 3.23.6 to 3.23.7 (#2628)
• 49130df chore(deps): bump go.uber.org/zap from 1.24.0 to 1.25.0 (#2629)
• ae0b2a9 chore(deps): bump golang.org/x/oauth2 from 0.10.0 to 0.11.0 (#2605)
• c5e8f8d chore(deps): bump google.golang.org/api from 0.134.0 to 0.136.0 (#2626)
• 47807bd chore(deps): bump gorm.io/gorm from 1.25.2 to 1.25.3 (#2630)
• 342438e chore: optimize unhandled file close error (#2599)
• 93009c4 chore: update console version to v1.0.2 (#2635)
• e621fce chore: update d7y.io/api version to v2.0.18 (#2616)
• dcea13a chore: use subtle compare to verify proxy auth (#2601)
• da7e305 feat: add NewSafeDialer and fix ssrf in manager preheat api (#2611)
• 77c91cb feat: add lock to dag vertex and replace rand.Seed with rand.New (#2614)
• 7f67bc5 feat: add tls client config for preheat in manager (#2612)
• 732819b feat: case insensitive string comparison in evaluater package of the manager (#2632)
• 25d91ec feat: change cache size in manager (#2623)
• 25d12b2 feat: change cache size in manager (#2633)
• c55148b feat: change default cluster name to cluster-1 (#2604)
• a5b0f74 feat: download tiny file with https scheme (#2617)
• 9e1bb52 feat: net.JoinHostPort replace fmt.Sprintf (#2622)
• 05a6680 fix: directories created via os.MkdirAll are not checked for permissions (#2613)
• 1b9d2b0 fix: improper use strings.TrimLeft (#2603)
• afa665c fix: incorrect log message in the scheduler (#2618)
• 9cc9fa2 fix: invalid error handling (#2610)
• 931cd3f fix: list personal access token with query string (#2624)
• d9e8b28 fix: manager generates mTLS certificates for arbitrary IP addresses (#2615)
• a64140f fix: potential nil panic (#2602)
• 094b79d fix: usage of architecture-dependent int type in the scheduler (#2619)
• bd140f1 refactor: file close error

v2.1.0

Dragonfly v2.1.0 is released! 🎉🎉🎉 Thanks to the Xinxin Zhao for helping to refactor the console and the manager provides a new console for users to operate Dragonfly. Welcome to visit d7y.io website.

Features

• Console v1.0.0 is released and it provides a new console for users to operate Dragonfly.
• Add network topology feature and it can probe the network latency between peers, providing better scheduling capabilities.
• Provides the ability to control the features of the scheduler in the manager. If the scheduler preheat feature is not in feature flags, then it will stop providing the preheating in the scheduler.
• dfstore adds GetObjectMetadatas and CopyObject to supports using Dragonfly as the JuiceFS backend.
• Add personal access tokens feature in the manager and personal access token contains your security credentials for the restful open api.
• Add TLS config to manager rest server.
• Fix dfdaemon fails to start when there is no available scheduler address.
• Add cluster in the manager and the cluster contains a scheduler cluster and a seed peer cluster.
• Fix object downloads failed by dfstore when dfdaemon enabled concurrent.
• Scheduler adds database field in config and moves the redis config to database field.
• Replace net.Dial with grpc health check in dfdaemon.
• Fix filtering and evaluation in scheduling. Since the final length of the filter is the candidateParentLimit used, the parents after the filter is wrong.
• Fix storage can not write records to file when bufferSize is zero.
• Hiding sensitive information in logs, such as the token in the header.
• Use unscoped delete when destroying the manager's resources.
• Add uk_scheduler index and uk_seed_peer index in the table of the database.
• Remove security domain feature and security feature in the manager.
• Add advertise port config to manager and scheduler.
• Fix fsm changes state failed when register task.

Break Change

• The M:N relationship model between the scheduler cluster and the seed peer cluster is no longer supported. In the future, a P2P cluster will be a cluster in the manager, and a cluster will only include a scheduler cluster and a seed peer cluster.

Console

You can see Manager Console for more details.

AI Infrastructure

• Triton Inference Server uses Dragonfly to distribute model files, refer to #2185. If there are developers who are interested in the drgaonfly repository agent project, please contact [email protected].
• TorchServer uses Dragonfly to distribute model files. Developers have already participated in the dragonfly endpoint project, and the feature will be released in v2.1.1.
• Fluid downloads data through Dragonfly when running based on JuiceFS, the feature will be released in v2.1.1.
• Dragonfly helps Volcano Engine AIGC inference to accelerate image through p2p technology.
• There have been many cases in the community, using Dragonfly to distribute data in AI scenarios based on P2P technology. In the inference stage, the concurrent download model of the inference service can effectively relieve the bandwidth pressure of the model registry through Dragonfly, and improving the download speed. Community will share topic 《Dragonfly: Intro, Updates and AI Model Distribution in the Practice of Kuaishou - Wenbo Qi, Ant Group & Zekun Liu, Kuaishou Technology》 with Kuaishou in KubeCon + CloudNativeCon + Open Source Summit China 2023, please follow if interested.

Maintainers

The community has added four new Maintainers, hoping to help more contributors participate in community.

• Yiyang Huang: He works for Volcano Engine and will focus on the engineering work for Dragonfly.
• Manxiang Wen: He works for Baidu and will focus on the engineering work for Dragonfly.
• Mohammed Farooq: He works for Intel and will focus on the engineering work for Dragonfly.
• Zhou Xu: He is a PhD student at Dalian University of Technology and will focus on the intelligent scheduling algorithms.

Others

You can see CHANGELOG for more details.

Links

• Dragonfly Website: https://d7y.io/
• Dragonfly Github: https://github.com/dragonflyoss/Dragonfly2
• Dragonfly Console Github: https://github.com/dragonflyoss/console
• Dragonfly Charts Github: https://github.com/dragonflyoss/helm-charts
• Dragonfly Monitor Github: https://github.com/dragonflyoss/monitoring

v2.1.0-rc.0

Changelog

• c82d2f7 chore: update submodule version (#2596)
• db82dd9 feat: change seed peer state to active in UpdateSeedPeer api (#2592)
• a151be1 feat: create seed peer with active state in manager (#2593)
• 733b938 feat: implement DeleteSeedPeer api in manager (#2591)
• a966c04 feat: use unscoped delete for resource in manager (#2595)