Implement Object Store for Viceroy #167
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit implements support for Object Store in Viceroy allowing for
users to simulate locally being able to query for blobs or add blobs to
the store. This commit does so with the following important changes:
- Bump the version of the Rust SDK to 0.8.7 which contains support for
the Object Store with higher level APIs rather than having users use
raw host calls
- Add config support to `fastly.toml` in the following format:
```
[local_server]
object_store.empty_store = []
object_store.store_one = [{key = "first", data = "This is some data"},{key = "second", path = "../test-fixtures/data/object-store.txt"}]
```
or alternatively with the following TOML syntax
```
[local_server]
[object_store]
empty_store = []
[[object_store.store_one]]
key = "first"
data = "This is some data"
[[object_store.store_one]]
key = "second"
path = "../test-fixtures/data/object-store.txt"
```
Semantically these are the same and it comes down to style preference
as the TOML parsing should work the same.
`object_store` is a new field in the `local_server` config option. It
takes any string value which is the name of the object store. This
then contains an array of objects that contain a `key` field which is
the name of the object stored in the store and either a `path` *or* a
`data` key value not both. `path` points to a file relative to the
project root and data is a string of bytes. This lets the user either
point to some file, useful for cases where the file might be really
large, or inline if the data is small.
The object store must be defined in the `fastly.toml` file beforehand
as new Object Stores cannot be created on the fly, however data can be
inserted after opening the file. The store will maintain the state
across requests and will lose it when Viceroy is stopped.
- Add Object Store hostcalls to Viceroy
- Implement the Object Store hostcalls for Viceroy
With this users can now locally test their code as if they were running
in production.
TODO: Add validation checks for the object key names
TODO: Test more aspects of the SDK/Object Store
acw
requested changes
Aug 11, 2022
| use fastly::ObjectStore; | ||
|
|
||
| fn main() { | ||
| let store_one = ObjectStore::open("store_one").unwrap().unwrap(); |
There was a problem hiding this comment.
Can we add a negative version of this test, too?
|
|
||
| #[tokio::test(flavor = "multi_thread")] | ||
| async fn object_store() -> TestResult { | ||
| const FASTLY_TOML: &str = r#" |
There was a problem hiding this comment.
Can we add some negative checks for mis-formatted toml files, to make sure your nice error messages pop out?
|
@acw I've updated the PR with fixes in the implementation due to adding more tests (yaaaaay) and more tests to cover all the failure cases for configs |
acw
requested changes
Aug 16, 2022
lib/src/object_store.rs
Outdated
|
|
||
| #[derive(Debug, PartialOrd, Ord, PartialEq, Eq, thiserror::Error)] | ||
| pub enum ObjectStoreError { | ||
| #[error("lol it's missing ya dingus")] |
There was a problem hiding this comment.
yeah, we probably shouldn't publish this error message. 😆
There was a problem hiding this comment.
And this is why we do code review
|
@acw I think with this error message fix and having run cargo fmt everywhere we just need a release of the new sdk. I can confirm that locally these tests pass at least with the new one. |
|
Ah wait I need to add validation for object names like we do in the SDK locally |
acw
previously approved these changes
Aug 17, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit implements support for Object Store in Viceroy allowing for
users to simulate locally being able to query for blobs or add blobs to
the store. This commit does so with the following important changes:
Bump the version of the Rust SDK to 0.8.7 which contains support for
the Object Store with higher level APIs rather than having users use
raw host calls
Add config support to
fastly.tomlin the following format:or alternatively with the following TOML syntax
Semantically these are the same and it comes down to style preference
as the TOML parsing should work the same.
object_storeis a new field in thelocal_serverconfig option. Ittakes any string value which is the name of the object store. This
then contains an array of objects that contain a
keyfield which isthe name of the object stored in the store and either a
pathor adatakey value not both.pathpoints to a file relative to theproject root and data is a string of bytes. This lets the user either
point to some file, useful for cases where the file might be really
large, or inline if the data is small.
The object store must be defined in the
fastly.tomlfile beforehandas new Object Stores cannot be created on the fly, however data can be
inserted after opening the file. The store will maintain the state
across requests and will lose it when Viceroy is stopped.
Add Object Store hostcalls to Viceroy
Implement the Object Store hostcalls for Viceroy
With this users can now locally test their code as if they were running
in production.
I wanted to get this up to have a second pair of eyes, but the following are tasks I still need to finish up before getting approval. Note CI will fail until we release the new version of the SDK.
TODO List before merging:
-[x] Add validation checks for the object key names
-[x] Test more aspects of the SDK/Object Store
cc: @JakeChampion since you were working on the JS SDK support I figured you'd also want to kick the tires with this!