diff --git a/open-grants/ipfs-rust/README.md b/open-grants/ipfs-rust/phase-1/README.md similarity index 100% rename from open-grants/ipfs-rust/README.md rename to open-grants/ipfs-rust/phase-1/README.md diff --git a/open-grants/ipfs-rust/media/fig1-ipfs-rust-risk-assessment.png b/open-grants/ipfs-rust/phase-1/media/fig1-ipfs-rust-risk-assessment.png similarity index 100% rename from open-grants/ipfs-rust/media/fig1-ipfs-rust-risk-assessment.png rename to open-grants/ipfs-rust/phase-1/media/fig1-ipfs-rust-risk-assessment.png diff --git a/open-grants/ipfs-rust/media/fig2-ipfs-rust-implementation-schedule.png b/open-grants/ipfs-rust/phase-1/media/fig2-ipfs-rust-implementation-schedule.png similarity index 100% rename from open-grants/ipfs-rust/media/fig2-ipfs-rust-implementation-schedule.png rename to open-grants/ipfs-rust/phase-1/media/fig2-ipfs-rust-implementation-schedule.png diff --git a/open-grants/ipfs-rust/media/phase-1-0-gantt.png b/open-grants/ipfs-rust/phase-1/media/phase-1-0-gantt.png similarity index 100% rename from open-grants/ipfs-rust/media/phase-1-0-gantt.png rename to open-grants/ipfs-rust/phase-1/media/phase-1-0-gantt.png diff --git a/open-grants/ipfs-rust/media/phase-1-1-gantt.png b/open-grants/ipfs-rust/phase-1/media/phase-1-1-gantt.png similarity index 100% rename from open-grants/ipfs-rust/media/phase-1-1-gantt.png rename to open-grants/ipfs-rust/phase-1/media/phase-1-1-gantt.png diff --git a/open-grants/ipfs-rust/media/phase-1-2-gantt.png b/open-grants/ipfs-rust/phase-1/media/phase-1-2-gantt.png similarity index 100% rename from open-grants/ipfs-rust/media/phase-1-2-gantt.png rename to open-grants/ipfs-rust/phase-1/media/phase-1-2-gantt.png diff --git a/open-grants/ipfs-rust/reports/phase-1.0.md b/open-grants/ipfs-rust/phase-1/reports/phase-1.0.md similarity index 100% rename from open-grants/ipfs-rust/reports/phase-1.0.md rename to open-grants/ipfs-rust/phase-1/reports/phase-1.0.md diff --git a/open-grants/ipfs-rust/reports/phase-1.1.md b/open-grants/ipfs-rust/phase-1/reports/phase-1.1.md similarity index 100% rename from open-grants/ipfs-rust/reports/phase-1.1.md rename to open-grants/ipfs-rust/phase-1/reports/phase-1.1.md diff --git a/open-grants/ipfs-rust/phase-2/README.md b/open-grants/ipfs-rust/phase-2/README.md new file mode 100644 index 00000000..9cc4448e --- /dev/null +++ b/open-grants/ipfs-rust/phase-2/README.md @@ -0,0 +1,267 @@ +# Open Grant Proposal: Rust IPFS Phase 2 +> Category: core-dev | Proposed by: Mark Henderson and Joonas Koivunen of [Equilibrium](https://equilibrium.co) + +The goal of the [Rust IPFS Phase 1 devgrant](https://github.com/ipfs/devgrants/tree/master/open-grants/ipfs-rust) +was to allow developers to write IPLD applications. Since the approval and inception of the +grant work in Q1 2020, the grant team has delivered a Rust implementation with over a dozen HTTP +endpoints and over 100 passing conformance tests. + +As such, we believe that IPLD application developers have what they need to start creating apps +that leverage the flexibility and connectivity of IPLD, along with the performance and reliability +of Rust. + +Two of the pillars for a ready-to-use implementation of IPFS are there now: peer-to-peer networking +and the blockstore. What remains is UnixFS. This proposal is a small fast-follow to Phase 1, which +completes the foundation and provides something that can be tested in the wild. + +Implementing UnixFS will enable future work on persistence. Following this grant, our plans focus on +the creation of benchmarks using bitswap and UnixFS exporting. This will establish a baseline to +guide exploration of multihash keyed blockstore following the [ipfs/specs #242 resolution]. + +[ipfs/specs #242 resolution]: https://github.com/ipfs/specs/issues/242#issuecomment-627533056 + +# Project Description + +### Summary + +The Rust IPFS grant team successfully completed Phase 1 of the +[Rust IPFS devgrant](https://github.com/ipfs/devgrants/tree/master/open-grants/ipfs-rust), and +plans to continue working towards a fully conformant Rust implementation of IPFS. + +Next, we would implement UnixFS and the minimum-viable set of endpoints, determined to be +`/get`, and `/cat` or the UnixFS exporter side. + +Adding content as UnixFS or the importer side was left out of the scope to be +realistic on the timeframe but we expect to lay enough groundwork during this +grant that development of the importer could be provided by new contributors. + +For more information on how the grant process has been going so far, please read the grant +reports for more information: +* [Phase 1.0](https://github.com/ipfs/devgrants/blob/master/open-grants/ipfs-rust/reports/phase-1.0.md) +* [Phase 1.1](https://github.com/ipfs/devgrants/blob/master/open-grants/ipfs-rust/reports/phase-1.1.md) +* [Phase 1.2](https://github.com/ipfs/devgrants/blob/master/open-grants/ipfs-rust/reports/phase-1.2.md) + +### Value + +The grant team has already brought both technical and administrative value to Rust IPFS. Since the +beginning of Phase 1, the team has: + +* Implemented all planned endpoints +* Set up GitHub repositories according to both Github and PL community standards +* Applied continuous integration via GitHub Actions for tests, linting, and language idioms +* Published two blog posts, one on the +[IPFS blog](https://blog.ipfs.io/2020-03-18-announcing-rust-ipfs/) and the other on the +[Equilibrium blog](https://medium.com/equilibriumco/announcing-rust-ipfs-af8358f90beb) +leading to: + * The number of stars on the repo growing by over 400, as well as: + * Two separate community contributors lending their talents, unsolicited, to pull requests + +Overall, the grant team has shown that they can meet milestones and deliverables, +even throughout internal, community-wide, external, and of course global challenges. + +As for Phase 2, the value of UnixFS would be the final pillar in the foundation allowing for use +cases such as low-resourced devices in IoT settings. + +### Risk Assessment + +Some of the risks from Phase 1 came to light, others did not. Still others that we +did not foresee occurred, making the progress challenging at times. + +The grant team was able to mitigate many risks, and are in the process of mitigating the rest. +The following headings are what we see as the current landscape of risks as we move forward, +along with some basic rankings of likelihood and impact. + +#### Reliance on rust-libp2p and its dependency chain + +**Likelihood**: Certain
**Impact**: Medium/Low + +Overall this risk was fairly mitigated, not causing too any practical issues over +the course of Phase 1. We made one required PR that was quickly merged. As we move through the list +of potential upstream features from `go-ipfs` 0.5.0 and beyond, this may require closer +collaboration with the `rust-libp2p` team. + +#### Community efforts can move and change very fast + +**Likelihood**: Medium/Low
**Impact**: Low + +In the end, this was not as much of a risk as we had anticipated. In fact, the community seemed to +coalesce around `rust-ipfs`. For example, `ipfs-rust/rust-ipld` became the ad-hoc "official" IPLD +repository in Rust. We were not notified about any other critical work in the ecosystem during +Phase 1. We will continue to keep our eyes and ears out for any other meaningful community projects +during Phase 2. + +#### The code could still “bit rot” even with community support + +**Likelihood**: Low
**Impact**: Medium/Low + +So far, so good. The code base and community are more active than ever. We plan on focusing more +heavily on community engagement and best practices around fostering a healthy open source project. + +#### New implementation == greater attack surface for exploits + +**Likelihood**: High
**Impact**: Medium + +Focus so far has been on implementation, so undivided attention to security concerns was +unfortunately not available. For the time being, more documentation, testing, and benchmarking +will invariably lead to insights about security. Additionally, the project is still marked at +`Pre-Alpha`, so expectations are set rather low in terms of security. + +#### Another implementation also means more known bugs + +**Likelihood**: High
**Impact**: Medium/High + +This risk can easily be turned into a benefit. While working towards the Phase 1 goals we've also +worked towards getting the known implementation differences fixed in ecosystem crates like +`aes-ctr`. Putting in the work to identify more of such issues through continued interoperability +testing (even if manual) will help the larger Rust ecosystem in future to interoperate with +Go and JS counterparts. + +#### Kademlia Unknown Unknowns + +**Likelihood**: High
**Impact**: High + +In phase 1 we didn't get to implementing `/api/v0/id?arg=other_peer_id` or +`Ipfs::identify_other(&self, peer: PeerId)`. As a result we know very little of how the +`libp2p_kad` implementation works together with `{go,js}-ipfs`. + +_This, coupled with the reliance on libp2p, poses the single largest risk to the project._ + +#### Open Source Governance + +**Likelihood**: Medium/High
**Impact**: High + +Open source governance is not easy, as it requires many “soft” skills outside of simple development +and engineering management. The grant team plans on working closely with Protocol Labs to ensure +that governance happens transparently, and that all interests can be represented fairly. + +### Project Team + +| Mark Henderson | Joonas Koivunen | +| -------------- | --------------- | +| | | +| Rust Developer, Project Manager | Rust Developer | +| [+github](https://github.com/aphelionz) [+website](https://mrh.io) [+resume](https://ipfs.io/ipfs/QmcHxD94cvJgq5ZZxQkEi7SRMwD5dBnkhQ3zzaVFqNWFJb) | [+github](https://github.com/koivunej) [+linkedin](https://www.linkedin.com/in/joonas-koivunen-70273412/) | +| Core contributor, OrbitDB. Current rust-ipfs contributor. Implemented an [Ambients Protocol parser/compiler](https://github.com/aphelionz/ambients) in Rust | Current rust-ipfs contributor + Rust-lang tooling contributor + experience with database implementations in Rust | + +### Maintenance and Upgrade Plan + +We want to make a code base that will last into the future. Equilibrium and MRH.io, along with the +support of the community, pledge to continue to maintain the Rust IPFS to the best of their ability +and within any financial constraints that exist. + +Much like we will build upon community efforts, we will also enable and encourage others to build +upon our work. This will be a twofold effort that includes both permissive licensing and community +outreach: on-boarding as many new contributors as possible, mapping the work out into issues of +different levels of difficulty, and by providing mentoring. + +## Project Plan + +### Summary + +The grant team will continue our phased approach, with each deliverable continuing to build upon +the last. + +* **Phase 2.0** will be time-boxed to 2 weeks and will add the necessary UnixFS support for +`/cat`, and `/get` endpoints. + +#### Updates to Project Management Processes + +We had adopted and had success with the use of organization-wide GitHub projects, which allows us +a coordinated, high-level view of each grant phase. You can see the rs-ipfs projects here: +https://github.com/orgs/rs-ipfs/projects. + +Additionally, instead of a gantt chart, we are opting for more roughly estimated timeline with +careful research done for each set of endpoints to properly scope out the requirements ahead of +implementation. An example of this is the +[block endpoint issue](https://github.com/rs-ipfs/rust-ipfs/issues/90), which combined a number +of documentation sources, tests, and independent investigation to determine conformance +requirements. + +Finally, as mentioned above, we are trying out time-boxing for Phase 2.0. + +### Implementation Details + +#### Phase 2.0 + +UnixFSv1 support is necessary for many use cases. We will start with basic files (non-hamt +directories), supporting single-block directories and multi-block files with up to +[174 links](https://github.com/ipfs/specs/blob/master/UNIXFS.md#layout) per folder. +That will allow us to land `/get` as well as `/cat`. + +This will be a time-boxed phase of two weeks. + +### Success Metric +Previously we used [HTTP endpoints](https://github.com/ipfs/devgrants/tree/master/open-grants/ipfs-rust#metric-number-of-http-endpoints-implemented) implemented as the key performance indicator. Given the lessons learned from Phase 1, we’d like to stick closely to this, but change the metric slightly to **passing conformance tests** for endpoints. + +As of the writing of this proposal, the grant team's work passes 102 conformance tests. Analysis of the conformance tests that cover Phase 2's endpoints suggest there could be well over 135 passingtests by the end of this grant cycle. + +Additionally, we will track the test coverage (currently 56% according to `cargo-tarpaulin`) to ensure that it, at the very least, does not decrease. Increasing the test coverage will be a target for the grant team but will not be guaranteed during this particular grant cycle. + +### Definition of Done +We had success with the definition of done used in Phase 1, and as such we will continue to use +it. Each command will be considered done when the following requirements are met: + +1. There is a working Rust implementation of the command’s functionality + 1. Code is “linted” i.e. code formatting via rustfmt and language idioms via clippy +2. There is an HTTP binding for said command exposed via the IPFS daemon +3. There are functional and/or unit tests written, and they are passing +4. There is suitable documentation. In our case, this means: + 1. Each command has a usage example and API specification + 2. Top-level commands with subcommands display usage instructions + 3. Rustdoc tests are passing on all code-level comments + 4. Differences between Rust’s implementation and Go or JS are explained +5. There are passing conformance tests, approved by Protocol Labs + +Note that we removed one item: +> *(Optional) There is a CLI command that utilizes either the Rust APIs or the HTTP APIs* + +### Out of Scope + +The following endpoints have been deemed out of scope for this proposal and will not be included in +Phase 2. The endpoints have been grouped by their respective reasonings for omission below. + +The following commands require on-the-fly config changes, though we could potentially implement +the read-only versions such as `/config/show`: + +- `/bootstrap` +- `/config` + +Finally, the following endpoints are simply not required by our use case: +- `/stats` +- `/cid` +- `/diag` +- `/mount` +- `/p2p` +- `/swarm/filters` +- `/files` (regular and mfs) + +The following endpoints are experimental: +- `/filestore` +- `/urlstore` + +The following endpoints are or will be deprecated: +- `/file` +- `/object` + +## Project Schedule + +### Phase 2.0 + +The focus of this phase is UnixFSv1 support and the `/get` and `/cat` endpoints. See the +[implementation details](#implementation-details) section for more information. + +#### Deliverables +1. UnixFSv1 support +2. `/get` +3. `/cat` +4. Project milestone report + 1. Interop and performance test results + +#### Development Schedule +**This will be a time-boxed phase.** This is to ensure we don't go too far down the "rabbit hole" +of implementing UnixFS and the endpoint details. For example - `/get` requires `tar` support. + +#### Estimated Budget +| | Number of Hours | Hourly Rate | Total (Euros) | Total (USD) | +| --- | ---- | ---- | ---- | --- | +| Software Development and Project Management | 70 hours | 120€ | 8.400€ | $9096.11 |