OpenDAL has refined its vision to One Layer, All Storage, driven by the following core principles: Open Community, Solid Foundation, Fast Access, Object Storage First, and Extensible Architecture.
Explore the detailed vision at OpenDAL Vision.
OpenDAL v0.51 implements RFC-5313, which removes the concept of metakey.
The following structs have been removed:
Metakey
The following APIs have been removed:
list_with(path).metakey()
Users no longer need to pass the metakey into the list. Instead, services will make their best effort to return as much metadata as possible. Users can check items like Capability::list_has_etag before making a call.
The capability write_multi_align_size is not utilized by any services, and we have no plans to support it in the future; therefore, we have removed it.
OpenDAL used to perform capability checks for all services, but since v0.51, it only conducts checks that impact data correctness like write_with_if_not_exists or delete_with_version by default in the CorrectnessCheckLayer. If users wish to verify other non-critical capabilities like write_with_content_type or write_with_cache_control, they should manually enable the CapabilityCheckLayer.
OpenDAL v0.51 implements RFC-3911, which adds Deleter in OpenDAL to replace batch operation.
The following new APIs have been added:
- [
Operator::delete_iter] - [
Operator::delete_try_iter] - [
Operator::delete_stream] - [
Operator::delete_try_stream] - [
Operator::deleter] - [
Deleter::delete] - [
Deleter::delete_iter] - [
Deleter::delete_try_iter] - [
Deleter::delete_stream] - [
Deleter::delete_try_stream] - [
Deleter::flush] - [
Deleter::close] - [
Deleter::into_sink] - [
DeleteInput] - [
IntoDeleteInput] - [
FuturesDeleteSink]
The following APIs have been deprecated and will be removed in the future releases:
Operator::remove(replace with [Operator::delete_iter])Operator::remove_via(replace with [Operator::delete_stream])
As a result of this change, the limit and with_limit APIs on Operator have also been deprecated; they are currently no-ops.
To support returning key-value entries in a streaming manner instead of loading them all into memory, OpenDAL updated its adapter API to return a Scanner instead of a Vec<String>.
- async fn scan(&self, path: &str) -> Result<Vec<String>>
+ async fn scan(&self, path: &str) -> Result<Self::Scanner>All services intending to implement kv::Adapter should adhere to this API change.
OpenDAL changes it's old metadata API to info to align with the new AccessorInfo struct.
- fn metadata(&self) -> Arc<AccessorInfo>
+ fn info(&self) -> Arc<AccessorInfo>The struct RangeWriter is not utilized by any services, and we have no plans to support it in the future; therefore, we have removed it.
Previously, it supports both URL format and key-value format. After switching the implementation from tokio-postgres to sqlx, the service now supports only the URL format.
Previously, list("a/b") would not return a/b even if it does exist. Since v0.50.0, this behavior has been changed. OpenDAL will now return the path itself if it exists. This change applies to all cases, whether the path is a directory or a file.
In OpenDAL v0.50.0, we did a refactor on all metrics-related layers. They are now sharing the same underlying implementations. PrometheusLayer, PrometheusClientLayer and MetricsLayer are now have similar public APIs and exactly the same metrics value.
Configurator used to return impl Builder, but now it returns associated builder type directly. This will allow users to use the builder in a more flexible way.
impl Configurator for MemoryConfig {
- fn into_builder(self) -> impl Builder {
+ type Builder = MemoryBuilder;
+ fn into_builder(self) -> Self::Builder {
MemoryBuilder { config: self }
}
}LoggingLayer now accepts LoggingInterceptor trait instead of configuration. This change will allow users to customize the logging behavior more flexibly.
pub trait LoggingInterceptor: Debug + Clone + Send + Sync + Unpin + 'static {
fn log(
&self,
info: &AccessorInfo,
operation: Operation,
context: &[(&str, &str)],
message: &str,
err: Option<&Error>,
);
}Users can now implement the log in the way they want.
Since v0.48, the customed_credential_load function has been renamed to customized_credential_load to fix the typo of customized.
- builder.customed_credential_load(v);
+ builder.customized_credential_load(v);In 2014 Amazon switched from AWS_SECURITY_TOKEN to AWS_SESSION_TOKEN. To be consistent with the naming of AWS STS, we have renamed the security_token field to session_token in the S3 service.
- builder.security_token(v);
+ builder.session_token(v);Since v0.48, Operator's new APIs from_iter and via_iter methods have deprecated the from_map and via_map methods.
- Operator::from_map::<Fs>(map)?.finish();
+ Operator::from_iter::<Fs>(map)?.finish();New API from_iter and via_iter should cover all use cases of from_map and via_map.
Since v0.48, all service builder now takes ownership self instead of &mut self. This change will allow users to configure the service in a more flexible way.
- let mut builder = S3::default();
- builder.bucket("test");
- builder.root("/path/to/root");
+ let builder = S3::default().bucket("test").root("/path/to/root");
let op = Operator::new(builder)?.finish();Starting from version 0.48, oio::Write::write now writes the entire buffer. This update aligns the API more closely with oio::Read::read and simplifies the implementation of concurrent writing.
trait Write {
- fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<usize>>;
+ fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<()>>;
}write will now return Result<()> instead of Result<usize>. The number of bytes written can be obtained from the buffer's length.
Starting from version 0.48, Access::metadata() will return Arc<AccessInfo> instead of AccessInfo. This change is intended to improve performance and reduce memory usage.
trait Access {
- fn metadata(&self) -> AccessInfo;
+ fn metadata(&self) -> Arc<AccessInfo>;
}The MinitraceLayer has been renamed to FastraceLayer to respond to the transition from minitrace to fastrace.
- use opendal::layers::MinitraceLayer;
+ use opendal::layers::FastraceLayer;Since v0.48, the Builder::from_config and Builder::from_map method has been replaced by the Configurator trait. The Configurator trait provides a more flexible and extensible way to configure OpenDAL.
Service implementers should update their code to use the Configurator trait instead:
impl Configurator for MemoryConfig {
type Builder = MemoryBuilder;
fn into_builder(self) -> Self::Builder {
MemoryBuilder { config: self }
}
}
impl Builder for MemoryBuilder {
const SCHEME: Scheme = Scheme::Memory;
type Config = MemoryConfig;
fn build(self) -> Result<impl Access> {
...
}
}Since v0.47, Reader's into_xxx APIs requires async and returns Result instead.
- let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048);
+ let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048).await?;Affected API includes:
Reader::into_futures_async_readReader::into_bytes_streamBlockingReader::into_std_readBlockingReader::into_bytes_iterator
As explained in core: Bring Streaming Read Back, we do need read streaming back for better performance and low memory usage.
So our oio::Read changed back to streaming read instead:
trait Read {
- async fn read(&self, offset: u64, size: usize) -> Result<Buffer>;
+ async fn read(&mut self) -> Result<Buffer>;
}All services and layers should be updated to meet this change.
Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like RPITIT and AFIT.
Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags.
Additionally, we have removed all reqwest-related feature flags:
- Users must now directly use
reqwest's feature flags for options likerustls,native-tls, etc. - The
rustlsfeature is no longer enabled by default; it must be activated manually. - OpenDAL no longer offers the
trust-dnsoption; users should configure the client builder directly.
Since v0.46, OpenDAL transformed it's Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO.
opendal::Reader now have APIs like:
let r = op.reader("test.txt").await?;
let buf = r.read(1024..2048).await?;Since version 0.46, OpenDAL features a native Buffer struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO.
OpenDAL's Reader will return Buffer and Writer will accept Buffer as input. Users who have implemented their own IO traits should update their code to use the new Buffer struct.
let r = op.reader("test.txt").await?;
// read returns `Buffer`
let buf: Buffer = r.read(1024..2048).await?;
let w = op.writer("test.txt").await?;
// Buffer can be created from continues bytes.
w.write("hello, world").await?;
// Buffer can also be created from non-continues bytes.
w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?;
// Make sure file has been written completely.
w.close().await?;To enhance usability, we've integrated bridges into bytes::Buf and bytes::BufMut, allowing users to directly interact with the bytes API.
let r = op.reader("test.txt").await?;
let mut bs = vec![];
// read_into accepts bytes::BufMut
let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?;
let w = op.writer("test.txt").await?;
// write_from accepts bytes::Buf
w.write_from("hello, world".as_bytes()).await?;
// Make sure file has been written completely.
w.close().await?;OpenDAL's Reader and Writer previously implemented APIs such as AsyncRead and AsyncWrite directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance.
Since v0.46, OpenDAL provides bridge APIs for Reader and Writer instead.
let r = op.reader("test.txt").await?;
// Convert into futures AsyncRead + AsyncSeek.
let reader = r.into_futures_async_read(1024..2048);
// Convert into futures bytes stream.
let stream = r.into_bytes_stream(1024..2048);
let w = op.writer("test.txt").await?;
// Convert into futures AsyncWrite
let writer = w.into_futures_async_write();
// Convert into futures bytes sink;
let sink = w.into_bytes_sink();Since version 0.46, OpenDAL has adopted Rust's native async_in_trait for our core IO traits, including oio::Read, oio::Write, and oio::List.
This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later.
Users who have implemented their own IO traits should update their code to use the new async trait syntax.
To further enhance the optionality of tokio, we have introduced a new feature called layers-blocking. The default usage of the blocking layer has been disabled. To utilize the BlockingLayer, please enable the layers-blocking feature.
The with_speed API has been deprecated. Please use with_io_timeout instead.
No raw API changes.
- The
thread_pool_enabledoption has been removed.
After RFC: List Prefix landed, we have changed the behavior of list a path without /. OpenDAL used to return NotADirectory error, but now we will return the list of entries that start with given prefix instead.
After RFC-3526: List Recursive landed, we have changed the list API to accept recursive instead of delimiter:
Users will need to change the following usage:
op.list_with(path).delimiter("")->op.list_with(path).recursive(true)op.list_with(path).delimiter("/")->op.list_with(path).recursive(false)
delimiter other than "" and "/" is not supported anymore.
After RFC: List Prefix landed, we have changed the behavior of stat a dir path:
Here are the behavior list:
| Case | Path | Result |
|---|---|---|
| stat existing dir | abc/ |
Metadata with dir mode |
| stat existing file | abc/def_file |
Metadata with file mode |
stat dir without / |
abc/def_dir |
Error NotFound or metadata with dir mode |
stat file with / |
abc/def_file/ |
Error NotFound |
| stat not existing path | xyz |
Error NotFound |
Services like s3, azblob can handle stat("abc/") correctly by check if there are objects with prefix abc/.
We changed our internal lister implementation to align with the list public API for better performance and readability.
- trait
Page=>List - struct
Pager=>Lister - trait
BlockingPage=>BlockingList - struct
BlockingPager=>BlockingLister
Every call to next will return an entry instead a page of entries. Also, we changed our async list api into poll based instead of async_trait.
OpenDAL bumps it's MSRV to 1.67.0.
- The
enable_exact_buf_writeoption has been deprecated and is superseded byBufferedWriter, introduced in version 0.40.
- The
write_min_sizeoption has been deprecated and replaced byBufferedWriter, also introduced in version 0.40. - A new setting,
allow_anonymous, has been added. Since v0.41, OSS will now return an error if credential loading fails. Enablingallow_anonymousto fallback to request without credentials.
- The
enable_create_simulationoption has been removed. We add this option to allow ghac simulate create empty file, but it could result in unexpected behavior when users create a file with content length1. So we remove it.
wasabi service native support has been removed. Users who want to access wasabi can use our s3 service instead.
There is no public API and raw API changes.
RFC-2578 merges append into write and removes append API.
- For writing a file at once, please use
op.write()for convenience. - For appending a file, please use
op.write_with().append(true)instead ofop.append().
The same rule applies to writer() and writer_with().
RFC-2774 proposes a new lister API to replace current list and scan. And we add a new API list to return entries directly.
- For listing a directory at once, please use
list()for convenience. - For listing a directory recursively, please use
list_with().delimiter("")orlister_with().delimiter("")instead ofscan(). - For listing in streaming, please use
lister()orlister_with()instead.
RFC-2779 proposes a new op.list_with().metakey() API to allow list with metakey and removes op.metadata(&entry) API.
Please use op.list_with().metakey() instead of op.metadata(&entry), for example:
// Before
let entries: Vec<Entry> = op.list("dir/").await?;
for entry in entries {
let meta = op.metadata(&entry, Metakey::ContentLength | Metakey::ContentType).await?;
println!("{} {}", entry.name(), entry.metadata().content_length());
}
// After
let entries: Vec<Entry> = op
.list_with("dir/")
.metakey(Metakey::ContentLength | Metakey::ContentType).await?;
for entry in entries {
println!("{} {}", entry.name(), entry.metadata().content_length());
}RFC-2852 proposes new native_capability and full_capability API to allow users to check if the underlying service supports a capability natively.
native_capabilityreturnstrueif the capability is supported natively.full_capabilityreturnstrueif the capability is supported, maybe via a layer.
Most of time, you can use full_capability to replace capability call. But to check if the capability is supported natively for better performance design, please use native_capability instead.
OpenDAL v0.40 added buffered writer support!
Users don't need to specify the content_length() for writer anymore!
- let mut w = op.writer_with("path/to/file").content_length(1024).await?;
+ let mut w = op.writer_with("path/to/file").await?;Users can specify the buffer() to control the size we call underlying storage:
let mut w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?;If buffer is not specified, we will call underlying storage everytime we call write. Otherwise, we will make sure to call underlying storage when buffer is full or close is called.
OpenDAL v0.40 removed the origin range_read and range_reader interfaces, please use read_with().range() or reader_with().range().
- op.range_read(path, range_start..range_end).await?;
+ op.read_with(path).range(range_start..range_end).await?;- let reader = op.range_reader(path, range_start..range_end).await?;
+ let reader = op.reader_with(path).range(range_start..range_end).await?;RFC-3017 removes copy_from API from the oio::Write trait. Users who implements services and layers by hand should remove this API.
In PR #2687, OpenDAL changed the behavior when role_arn has been specified.
OpenDAL used to override role_arn simply. But since this version, OpenDAL will make sure to use assume_role with specified role_arn and external_id (if supplied).
In PR #2666, RetryLayer supports RetryInterceptor. To implement this change, RetryLayer changed it's in-memory layout by adding a new generic parameter I to RetryLayer<I>.
Users who stores RetryLayer in struct or enum will need to change the type if they don't want to use default behavior.
In PR #2698, OpenDAL re-org the internal structure of opendal::raw::oio and changed some APIs name.
There are no public API changes.
OpenDAL add the Write::sink API to enable streaming writing. This is a breaking change for users who depend on the raw API.
For a quick fix, users who have implemented opendal::raw::oio::Write can return an Unsupported error for Write::sink().
More details could be found at [RFC: Writer sink API][crate::docs::rfcs::rfc_2083_writer_sink_api].
In v0.37.0, OpenDAL bump the version of reqsign to v0.13.0.
There are no public API and raw API changes.
In v0.36, OpenDAL improving the xxx_with API by allow it to be called in chain:
After this change, all xxx_with alike call will be changed from
let bs = op.read_with(
"path/to/file",
OpRead::new()
.with_range(0..=1024)
.with_if_match("<etag>")
.with_if_none_match("<etag>")
.with_override_cache_control("<cache_control>")
.with_override_content_disposition("<content_disposition>")
).await?;to
let bs = op.read_with("path/to/file")
.range(0..=1024)
.if_match("<etag>")
.if_none_match("<etag>")
.override_cache_control("<cache_control>")
.override_content_disposition("<content_disposition>")
.await?;For blocking API calls, we will need a call() at the end:
let bs = bop.read_with("path/to/file")
.range(0..=1024)
.if_match("<etag>")
.if_none_match("<etag>")
.override_cache_control("<cache_control>")
.override_content_disposition("<content_disposition>")
.call()?;Along with this change, users don't need to call OpXxx anymore so we moved it to raw API.
More details could be found at [RFC: Chain Based Operator API][crate::docs::rfcs::rfc_2299_chain_based_operator_api].
Migrated opendal::ops to opendal::raw::ops.
- OpenDAL removes rarely used
Operator::from_envandOperator::from_iterAPIs- Users can use
Operator::via_mapinstead.
- Users can use
- OpenDAL adds
appendsupport with could break existing layers. Please make sureappendrequests have been forward correctly. - After the merging of
scanandlist, OpenDAL removes thescanfrom raw API. Please uselist_without_delimiterinstead.
- OpenDAL raises it's MSRV to 1.65 for dependencies changes
OperatorInfo::can_scanhas been removed, to check if underlying services support scan a dir natively, please useCapability::list_without_delimiterinstead.
After Capability introduced, we have added delimiter in OpList. Users can specify the delimiter to "" or "/" to control the list behavior.
Along with this change, Operator::scan() becomes a short alias of Operator::list_with(OpList::new().with_delimiter("")).
In v0.34, OpenDAL adds a typed kv adapter for zero-copy read and write. If you are implemented kv adapter for a rust in-memory data struct, please consider migrate.
OpenDAL 0.33 has redesigned the Writer API, replacing all instances of writer.append() with writer.write(). For more information, please refer to Writer.
In addition to the redesign of the Writer API, we have removed append from oio::Write. Therefore, users who implement services and layers should also remove it.
After v0.33 landing, services should handle OpWrite::content_length correctly by following these guidelines:
- If the writer does not support uploading unsized data, return a response of
NotSupportedif content length isNone. - Otherwise, continue writing data until either
closeoraborthas been called.
Furthermore, OpenDAL 0.33 introduces a new concept called Capability which replaces AccessorCapability. Services must adapt to this change.
OpenDAL 0.32 doesn't have much breaking changes.
We changed Accessor::create into Accessor::create_dir. Only users who implement Layer need to change.
In version v0.31 of OpenDAL, we made some internal refactoring to improve its compatibility with the ecosystem.
We increased the MSRV to 1.64 from v0.31 onwards. Although it is still possible to build OpenDAL under older rustc versions, we cannot guarantee that any issues related to them will be fixed.
Previously, OpenDAL accepted time::Duration as input for presign_xxx. However, since v0.31, we have changed this to accept std::time::Duration so that users do not need to depend on time. Internally, we migrated from time to chrono for better integration with other parts of the ecosystem.
We have added a new configuration option called disable_ec2_metadata for the S3 service in response to a mistake where it was mixed up with another option called disable_config_load. Users who want to disable loading credentials from EC2 metadata should set this option instead.
Starting from v0.31, all services in OpenDAL are split into different feature flags. To enable only S3 support, use the following TOML configuration:
opendal = {
version = "0.31",
default-features = false,
features = ["services-s3"]
}In version 0.30, we made significant breaking changes by removing objects. Our goal in doing so was to provide our users with APIs that are easier to understand and maintain.
More details could be found at [RFC: Remove Object Concept][crate::docs::rfcs::rfc_1477_remove_object_concept].
To upgrade to OpenDAL v0.30, users need to make the following changes:
- regex replace
object\((.*)\).reader\(\)toreader($1)- replace the function on your case, it's recommended to do it one by one
- rename
ObjectMetakey=>Metakey - rename
ObjectMode=>EntryMode - replace
ErrorKind::ObjectXxxtoErrorKind::Xxx - rename
AccessorMetadata=>AccessorInfo - rename
ObjectMetadata=>Metadata - replace
operator.metadata()=>operator.info()
In v0.29, we introduced [Object Writer][crate::docs::rfcs::rfc_1420_object_writer] to replace existing Multipart related APIs.
Users can now append multiparts bytes into object via:
let mut w = o.writer().await?;
w.write(bs1).await?;
w.write(bs2).await?;
w.close()Along with this change, we cleaned up a lot of internal structs and traits. Users who used to depend on opendal::raw::io::{input,output} should use opendal::raw::oio instead.
Also, decompress related feature also removed. Users can use async-compression with ObjectReader directly.
In v0.28, we introduced [Query Based Metadata][crate::docs::rfcs::rfc_1398_query_based_metadata]. Users can query cached metadata with ObjectMetakey to make sure that OpenDAL always makes the best decision.
- pub async fn metadata(&self) -> Result<ObjectMetadata>;
+ pub async fn metadata(
+ &self,
+ flags: impl Into<FlagSet<ObjectMetakey>>,
+ ) -> Result<Arc<ObjectMetadata>>;Please visit Object::metadata()'s example for more details.
In v0.27, we refactored our list related logic and added scan support. So make Pager and BlockingPager associated types in Accessor too!
pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
type Reader: output::Read;
type BlockingReader: output::BlockingRead;
+ type Pager: output::Page;
+ type BlockingPager: output::BlockingPage;
}Due to this change, all layers implementation should be changed. If there is not changed over pager, they can be changed like the following:
impl<A: Accessor> LayeredAccessor for MyAccessor<A> {
type Inner = A;
type Reader = MyReader<A::Reader>;
type BlockingReader = MyReader<A::BlockingReader>;
+ type Pager = A::Pager;
+ type BlockingPager = A::BlockingPager;
+ async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Pager)> {
+ self.inner.list(path, args).await
+ }
+ async fn scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::Pager)> {
+ self.inner.scan(path, args).await
+ }
+ fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingPager)> {
+ self.inner.blocking_list(path, args)
+ }
+ fn blocking_scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::BlockingPager)> {
+ self.inner.blocking_scan(path, args)
+ }
}To reduce the understanding overhead, we move all OpXxx into opendal::ops now. User may need to change:
- use opendal::OpWrite;
+ use opendal::ops::OpWrite;backon is the implementation detail of our RetryLayer, so we hide it from our public API. Users of RetryLayer need to change the code like:
- RetryLayer::new(backon::ExponentialBackoff::default())
+ RetryLayer::new()In v0.26 we have replaced all internal dynamic dispatch usage with static dispatch. With this change, we can ensure that all operations performed inside OpenDAL are zero cost.
Due to this change, we have to refactor the logic of Operator's init logic. In v0.26, we added opendal::Builder trait and opendal::OperatorBuilder. For the first glance, the only change to existing code will be like:
- let op = Operator::new(builder.build()?);
+ let op = Operator::new(builder.build()?).finish();By adding a finish() call, we will erase all generic types so that Operator can still be easily used everywhere as before.
In v0.26, Accessor has been changed into trait with associated types.
All services need to declare the types returned as Reader or BlockingReader:
pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
type Reader: output::Read;
type BlockingReader: output::BlockingRead;
}If your service doesn't support read or blocking_read, we can use () to represent a dummy reader:
impl Accessor for MyDummyAccessor {
type Reader = ();
type BlockingReader = ();
}As described before, OpenDAL prefer to use static dispatch. Layers are required to implement the new Layer and LayeredAccessor trait:
pub trait Layer<A: Accessor> {
type LayeredAccessor: Accessor;
fn layer(&self, inner: A) -> Self::LayeredAccessor;
}
#[async_trait]
pub trait LayeredAccessor: Send + Sync + Debug + Unpin + 'static {
type Inner: Accessor;
type Reader: output::Read;
type BlockingReader: output::BlockingRead;
}LayeredAccessor is a wrapper of Accessor with the typed Innder. All methods that not implemented will be forward to inner instead.
Since v0.26, we implement opendal::Builder for all services, and services' mod will not be exported.
- use opendal::services::s3::Builder;
+ use opendal::services::S3;Sorry again for the big changes in this release. It's a big step for OpenDAL to work in more critical systems.
In v0.25, we bring the same feature sets from ObjectReader to BlockingObjectReader.
Due to this change, all code that depends on BlockingBytesReader should be refactored.
BlockingBytesReader=>input::BlockingReaderBlockingOutputBytesReader=>output::BlockingReader
Most changes only happen inside. Users not using opendal::raw::* will not be affected.
Apart from this change, we refactored s3 credential loading logic. After this change, we can disable the config load instead of the credential methods.
builder.disable_credential_loader=>builder.disable_config_load
In v0.24, we made a big refactor on our internal IO-related traits. In this version, we split our IO traits into input and output versions:
Take Reader as an example:
input::Reader is the user input reader, which only requires futures::AsyncRead + Send.
output::Reader is the reader returned by OpenDAL, which implements futures::AsyncRead, futures::AsyncSeek, and futures::Stream<Item=io::Result<Bytes>>. Besides, output::Reader also implements Send + Sync, which makes it useful for users.
Due to this change, all code that depends on BytesReader should be refactored.
BytesReader=>input::ReaderOutputBytesReader=>output::Reader
Thanks to the change of IO trait split, we make ObjectReader implements all needed traits:
futures::AsyncReadfutures::AsyncSeekfutures::Stream<Item=io::Result<Bytes>>
Thus, we removed the seekable_reader API. They can be replaced by range_reader:
o.seekable_reader=>o.range_reader
Most changes only happen inside. Users not using opendal::raw::* will not be affected.
Sorry for the inconvenience. I think those changes are required and make OpenDAL better! Welcome any comments at Discussion.
v0.21 is an internal refactor version of OpenDAL. In this version, we refactored our error handling and our Accessor APIs. Thanks to those internal changes, we added an object-level metadata cache, making it nearly zero cost to reuse existing metadata continuously.
Let's start with our errors.
As described in RFC-0977: Refactor Error, we refactor opendal error by a new error
called opendal::Error.
This change will affect all APIs that are used to return io::Error.
To migrate this, please replace std::io::Error with opendal::Error:
- use std::io::Result;
+ use opendal::Result;And the following error kinds should be updated:
std::io::ErrorKind::NotFound=>opendal::ErrorKind::ObjectNotFoundstd::io::ErrorKind::PermissionDenied=>opendal::ErrorKind::ObjectPermissionDenied
And since v0.21, we will return errors ObjectIsADirectory and ObjectNotADirectory instead of anyhow::Error.
In v0.21, we refactor the whole Accessor's API:
- async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<u64>
+ async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<RpWrite>Since v0.21, we will return a reply struct for different operations called RpWrite instead of an exact type. We can split OpenDAL's public API and raw API with this change.
Since v0.21, Accessor will return Pager for List:
- async fn list(&self, path: &str, args: OpList) -> Result<ObjectStreamer>
+ async fn list(&self, path: &str, args: OpList) -> Result<(RpList, output::Pager)>And Object will return an ObjectLister which is built upon Page:
pub async fn list(&self) -> Result<ObjectLister> { ... }ObjectLister can be used as an object stream as before. It also provides the function next_page to get the underlying pages directly:
impl ObjectLister {
pub async fn next_page(&mut self) -> Result<Option<Vec<Object>>>;
}Since v0.21, we have categorized all APIs into public and raw.
Public APIs are exposed under opendal::Xxx; they are user-face APIs that are easy to use and understand.
Raw APIs are exposed under opendal::raw::Xxx; they are implementation details for underlying services and layers.
Please replace all usage of opendal::io_util::* and opendal::http_util::* to opendal::raw::* instead.
With this change, new users of OpenDAL maybe be it easier to get started.
Sorry for introducing too much breaking change in a single version. This version can be a solid version for preparing OpenDAL v1.0.
v0.20 is a big release that we introduce a lot of performance related changes.
To make the best of information from read operation, we propose and implemented RFC-0926: Object Reader. By this RFC, we can fetch content length from ObjectReader now!
pub struct ObjectReader {
inner: BytesReader
meta: ObjectMetadata,
}
impl ObjectReader {
pub fn content_length(&self) -> u64 {}
pub fn last_modified(&self) -> Option<OffsetDateTime> {}
pub fn etag(&self) -> Option<String> {}
}To make this happen, we changed our Accessor API:
- async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {}
+ async fn read(&self, path: &str, args: OpRead) -> Result<ObjectReader> {}All layers should be updated to meet this change. Also, it's required to return content_length while building ObjectReader. Please make sure the returning ObjectMetadata is used correctly.
OpenDAL deprecate some features:
serde: We will enable it by default.layers-retry: We will enable retry support by default.layers-metadata-cache: We will enable it by default.
Deprecated types like DirEntry has been removed.
OpenDAL v0.18 introduces the following breaking changes:
- Deprecated feature flag
services-httphas been removed. - All
DirXxxitems have been renamed toObjectXxxto make them more consistent.DirEntry->EntryDirStream->ObjectStreamDirStreamer->ObjectStreamDirIterate->ObjectIterateDirIterator->ObjectIterator
Besides, we also make a big change to our Entry API. Since v0.18, we can fully reuse the metadata that fetched during list. Take entry.content_length() for example:
- If
content_lengthis already known, we will return directly. - If not, we will check if the object entry is
complete:- If
complete, the entry already fetched all metadata that it could have, return directly. - If not, we will send a
statcall to get themetadataand refresh our cache.
- If
This change means:
- All API like
content_lengthwill be changed into async functions. metadataandblocking_metadatawill not return errors anymore.- To retrieve the latest meta, please use
entry.into_object().metadata()instead.
OpenDAL v0.17 refactor the Accessor to make space for future features.
We move path String out of the OpXxx to function args so that we don't need to clone twice.
- async fn read(&self, args: OpRead) -> Result<BytesReader>
+ async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader>For more information about this change, please refer to RFC-0661: Path In Accessor.
And since OpenDAL v0.17, we will use rustls as default tls engine for our underlying http client. Since this release, we will not depend on openssl anymore.
OpenDAL v0.16 refactor the internal implementation of http service. Since v0.16, http service can be used directly without enabling services-http feature. Accompany by these changes, http service has the following breaking changes:
services-httpfeature has been deprecated. Enablingservices-httpis a no-op now.- http service is read only services and can't be used to
listorwrite.
OpenDAL introduces a new layer ImmutableIndexLayer that can add list capability for services:
use opendal::layers::ImmutableIndexLayer;
use opendal::Operator;
use opendal::Scheme;
async fn main() {
let mut iil = ImmutableIndexLayer::default();
for i in ["file", "dir/", "dir/file", "dir_without_prefix/file"] {
iil.insert(i.to_string())
}
let op = Operator::from_env(Scheme::Http)?.layer(iil);
}For more information about this change, please refer to RFC-0627: Split Capabilities.
OpenDAL v0.14 removed all deprecated APIs in previous versions, including:
Operator::with_backoffin v0.13- All services
Builder::finish()in v0.12 - All services
Backend::build()in v0.12
Please visit related version's upgrade guide for migration.
And in OpenDAL v0.14, we introduce a break change for write operations.
pub trait Accessor {
- async fn write(&self, args: &OpWrite) -> Result<BytesWriter> {}
+ async fn write(&self, args: &OpWrite, r: BytesReader) -> Result<u64> {}
}The following APIs have affected by this change:
Object::writenow acceptimpl Into<Vec<u8>>instead ofAsRef<&[u8]>Object::writerhas been removed.Object::write_fromhas been added to support write from a reader.- All layers should be refactored to adapt new
Accessortrait.
For more information about this change, please refer to RFC-0554: Write Refactor.
OpenDAL deprecate Operator::with_backoff since v0.13.
Please use RetryLayer instead:
use anyhow::Result;
use backon::ExponentialBackoff;
use opendal::layers::RetryLayer;
use opendal::Operator;
use opendal::Scheme;
let _ = Operator::from_env(Scheme::Fs)
.expect("must init")
.layer(RetryLayer::new(ExponentialBackoff::default()));OpenDAL introduces breaking changes for services initiation.
Since v0.12, Operator::new will accept impl Accessor + 'static instead of Arc<dyn Accessor>:
impl Operator {
pub fn new(accessor: impl Accessor + 'static) -> Self { .. }
}Every service's Builder now have a build() API which can be run without async:
let mut builder = fs::Builder::default();
let op: Operator = Operator::new(builder.build()?);Along with these changes, Operator::from_iter and Operator::from_env now is a blocking API too.
For more information about this change, please refer to RFC-0501: New Builder.
The following APIs have been deprecated:
- All services
Builder::finish()(replaced byBuilder::build()) - All services
Backend::build()(replace byBuilder::default())
The following APIs have been removed:
- public struct
Metadata(deprecated in v0.8, replaced byObjectMetadata)
OpenDAL introduces a breaking change of list related operations in v0.8.
Since v0.8, list will return DirStreamer instead:
pub trait Accessor: Send + Sync + Debug {
async fn list(&self, args: &OpList) -> Result<DirStreamer> {}
}DirStreamer streams DirEntry which carries ObjectMode, so that we don't need an extra call to get object mode:
impl DirEntry {
pub fn mode(&self) -> ObjectMode {
self.mode
}
}And DirEntry can be converted into Object without overhead:
let o: Object = de.into()Since v0.8, opendal::Metadata has been deprecated by opendal::ObjectMetadata.
OpenDAL introduces a breaking change of decompress_read related in v0.7.
Since v0.7, decompress_read and decompress_reader will return Ok(None) while OpenDAL can't detect the correct compress algorithm.
impl Object {
pub async fn decompress_read(&self) -> Result<Option<Vec<u8>>> {}
pub async fn decompress_reader(&self) -> Result<Option<impl BytesRead>> {}
}So users should match and check the None case:
let bs = o.decompress_read().await?.expect("must have valid compress algorithm");OpenDAL introduces many breaking changes in v0.4.
Since v0.4, Object::reader() will return impl BytesRead instead of Reader that implements AsyncRead and AsyncSeek. Users who want AsyncSeek please wrapped with opendal::io_util::seekable_read:
use opendal::io_util::seekable_read;
let o = op.object("test");
let mut r = seekable_read(&o, 10..);
r.seek(SeekFrom::Current(10)).await?;
let mut bs = vec![0;10];
r.read(&mut bs).await?;Since v0.4, the following APIs will be removed.
Object::limited_reader(size: u64)Object::offset_reader(offset: u64)Object::range_reader(offset: u64, size: u64)
Instead, OpenDAL is providing a more general range_reader powered by RangeBounds:
pub async fn range_reader(&self, range: impl RangeBounds<u64>) -> Result<impl BytesRead>Users can use their familiar rust range syntax:
let r = o.range_reader(1024..2048).await?;Since v0.4, all functions in OpenDAL will return std::io::Result instead.
Please check via std::io::ErrorKind directly:
use std::io::ErrorKind;
if let Err(e) = op.object("test_file").metadata().await {
if e.kind() == ErrorKind::NotFound {
println!("object not exist")
}
}Since v0.4, Credential has been removed, please use the API provided by Builder directly.
builder.access_key_id("access_key_id");
builder.secret_access_key("secret_access_key");Since v0.4, Accessor::write will return a BytesWriter instead accepting a BoxedAsyncReader.
Along with this change, the old Writer has been replaced by a new set of write functions:
pub async fn write(&self, bs: impl AsRef<[u8]>) -> Result<()> {}
pub async fn writer(&self, size: u64) -> Result<impl BytesWrite> {}Users can write into an object more easily:
let _ = op.object("path/to/file").write("Hello, World!").await?;Since v0.4, mod io_util will replace readers. In io_utils, OpenDAL provides helpful functions like:
into_reader: ConvertBytesStreamintoBytesReadinto_sink: ConvertBytesWriteintoBytesSinkinto_stream: ConvertBytesReadintoBytesStreaminto_writer: ConvertBytesSinkintoBytesWriteobserve_read: Add callback forBytesReaderobserve_write: Add callback forBytesWrite
For better naming, types that OpenDAL returns have been renamed:
AsyncRead + Unpin + Send=>BytesReadBoxedAsyncReader=>BytesReaderAsyncWrite + Unpin + Send=>BytesWriteBoxedAsyncWriter=>BytesWriterObjectStream=>ObjectStreamer