Audit and Revise Storage Providers Section in Lotus docs

[LexLuthr]

This issue is part of overall lotus docs audit project.

Goal - User should be able to run and maintain a miner successfully with just this section of lotus-docs.

Check for content and correctness.
Validate all links
Add more relevant content if required.
Run all commands locally to verify input and output.
Add missing steps or variables and OS based information if needed.
Identify the opportunity and create new articles/solutions/blogs/tutorials.

• Overview
• Hardware requirements
• Mining architectures
• Setup
• Configuration
• Custom storage layout
• Split-markets miners
• Dagstore
• Lifecycle
• Benchmarks
• Addresses
• Connectivity
• Backup and restore
• Upgrades
• Sector pledging
• Manage storage deals
• Dynamic retrieval pricing
• Miner journal
• Message pool
• Lotus Miner CLI
• Troubleshooting
• Seal workers - @LexLuthr Audit SP seal-worker #137

[Annamarie2019]

Great summary above: Docs start at Overview

(I'm putting this at medium difficulty, because, on the one hand, it's a great learning experience for a new person, but on the other hand it's a large, complex set of docs. But looks like Lex Luthr knows what he's doing here. To balance out: Effort is weeks.)

[LexLuthr]

https://filecoinproject.slack.com/archives/CPFTWMY7N/p1648645482412959

Seems like there's a few locations that the flag --call-on-market is in that help doc. Needs to be --call-on-markets

[LexLuthr]

https://filecoinproject.slack.com/archives/CPFTWMY7N/p1649080143953649

[rjan90]

I have dived into the Storage Provider section of the docs the last couple of days, and tried to look at it from the perspective of someone who is a new storage provider. To me it seems like we add quite a lot of complexity very early in the docs, which can create a very steep learning curve for potential new storage providers.

Most new storage providers will initially just want to initilize their Storage Provider on the calibration network and try to pledge a couple of sectors, and do not necessarily care about splitting their markets-node, DAG-store, SnapDeals, etc.

I think we therefore should have a Setup/section in the docs that are very easy, and rather add complexity later on in sections called: Operate/, Workers/ and Advanced configurations/. Some initial ideas for restructuring the docs a bit:

Storage Providers/
├─ Get Started/
│  ├─ Overview
│  ├─ Tasks
│  ├─ Architecture / Hardware requirements
│  ├─ Economics
├─ Setup/
│  ├─ Pre-requisites
│  ├─ Initialize
│  ├─ Lifecycles
├─ Operate/
│  ├─ Overview
│  ├─ Batching
│  ├─ Get storage power
│  ├─ Addresses
│  ├─ Backup & Restore
│  ├─ Connectivity
│  ├─ CLI
│  ├─ Upgrades
│  ├─ Index Provider
│  ├─ Snap-Deals
│  ├─ Troubleshooting
├─ Workers/
│  ├─ Seal Workers
│  ├─ PoSt Workers
├─ Advanced configurations/
│  ├─ Configurations
│  ├─ Config-file

Where Get Started/ is a very high-level of what a storage provider does and how it functions. Talk a bit around architectures/hardware specs, and emphasize that lotus can be used by Storage Providers ranging from a couple of TiB to probably 100PiB+ so it´s highly configurable.

Setup/ - very easy walkthrough of initlizing a storage provider on the Filecoin network. Only mention the most important configs, and let most stay at default.

Operate/ - bring in more advanced concepts of being a Storage Provider and its functions -> Snap-Deals, Index-provider, etc.

Workers/ - Section for talking about the workers, and how one can create complex architectures with these. An idea might also be to put the split-marketssection into this category as well, since running a seperate markets-node is essentially a "worker". Though in that case, we should maybe rename it to something else.

Advanced configurations/ - like the title, only configurations Storage Providers need to take into account after having tested running a Storage Provider on the calibration network.

[jennijuju]

great write ups! a couple of suggestions for you to consider:

• lifecycle may go well with the tasks?
• connection, address can goi to the setup after initialize may be�?
• in the intialize we should mention something like " we will get you started with the default config, for advacned configrution or config for each operation, please see xyz"
• i would furthur breakdown the operate by operation type
  • overview page: getting power, which can be done by pledge CC sector / making deals
  • PC batching and C aggregation might go under pledge CC sector -> the next suggested page on this should be workers?
  • in making deals for regular deals, after MRA(which will go under worker), ask setting, dt & PCD such we might circle back to the last point as they go through the regular sealing flow
  • snap deal might go under making deals

[jennijuju]

in th eoperation, we can also mention things like sector extention, 1:N replica and such

[rjan90]

lifecycle may go well with the tasks?

Yeah, I think some of the content can be consolidated into it (Safely restart, reducing time offline). The idea was that the tasks-part would go over the different jobs a lotus-miner process is tasked with, and show some of the data-flow through the different tasks (for example the PC1->PC2-WaitSeed->C2 cycle, windowPoSt, etc).

in the intialize we should mention something like " we will get you started with the default config, for advacned configrution or config for each operation, please see xyz"

Agree 👍 .

i would furthur breakdown the operate by operation type

Also 👍 on this.

[LexLuthr]

Proposal:

├─ Get Started/
│ ├─ Overview
│ ├─ Sealing Pipeline <---- Sounds more accurate than tasks
│ ├─ Architecture / Hardware requirements
│ ├─ Economics

├─ Setup/
│ ├─ Pre-requisites
│ ├─ Initialize
│ ├─ Lifecycles
│ ├─ Connectivity

├─ Operate/
│ ├─ Overview
│ ├─ Batching
│ ├─ Get storage power
│ ├─ Backup & Restore
│ ├─ Addresses
│ ├─ Daily Operations - This needs to cover extending sector and other operations that are done on daily basis but are not a part of docs yet. We have many user asking how to run certain commands correctly. This section will cover that.
│ ├─ CLI
│ ├─ Upgrades
│ ├─ Index Provider
│ ├─ Snap-Deals
│ ├─ Troubleshooting <---- Time to get rid of this and move content to KB section

├─ Advanced configurations/
│ ├─ Market config. <---- We should let users know boost is also an option under this section
│ ├─ Worker config
│ ├─ Sealing config
│ ├─ libp2p config
etc etc. I would prefer to break the current monolith config into subsections based on the subsystem and big features like indexer, snap-deals, market section etc.

[rjan90]

│ ├─ Troubleshooting <---- Time to get rid of this and move content to KB section

I was thinking of moving the current Journal docs into Troubleshooting-section, get rid of everything else and point to the kb for common issues.

etc etc. I would prefer to break the current monolith config into subsections based on the subsystem and big features like indexer, snap-deals, market section etc.

Yeah, I´m very in favor for this 👍

[LexLuthr]

First we need to create new KB to move the current troubleshooting section there. I don't want to lose the content. Troubleshooting is an umbrella word and users might expect it to contain all types of troubleshooting methods. I think it might create more confusion because of the word.

BTW, I also want to add logging section under advanced config to cover how to record logs to a file, change to debug etc. We need full details of what log list actually contains. It is too complicated in the CLI with zero explanation.

[rjan90]

I missed this comment on my last look:

│ ├─ Sealing Pipeline <---- Sounds more accurate than tasks

In my opinion sealing pipeline is just a narrow view of what tasks the lotus-miner is doing (it´s also doing windowPoSt, winningPoSt, scheduler). We can maybe iterate on the wording (jobs, chore, etc?)

[LexLuthr]

Yeah, makes sense. Sealing pipeline is just one task. Jobs might be a better fit. I leave this to you. I think we set on most of the things. Any other changes you would propose? We also need to talk to Johnny about aliases as we would be moving a lot of content around and getting rid of few pages. I will loop you in the conversation.

[LexLuthr]

https://filecoin.io/blog/posts/the-economics-of-storage-providers/

[rjan90]

So taking in the feedback, here is an updated tree-stucture of what the new storage provider section could look like. I would emphazise could, since we will definetely find some content underway that might not fit under its current sub-section, pages that can be consollidated, wording might be a bit off, etc.

Storage Providers/
├─ Get Started/
│  ├─ Overview
│  ├─ Tasks    <-- Including safely restart, reducing time offline
│  ├─ Architecture / Hardware requirements
│  ├─ Economics
├─ Setup/
│  ├─ Pre-requisites
│  ├─ Initialize
│  ├─ Configuration
├─ Operate/
│  ├─ Benchmarks
│  ├─ Addresses <-- Owner / worker, Control addresses
│  ├─ Connectivity <-- lotus-miner reachability, peer count,
│  ├─ Get storage power
│  ├─ Batching
│  ├─ Backup & Restore
│  ├─ Upgrades
│  ├─ Chores
│  ├─ Index Provider
│  ├─ Snap-Deals
│  ├─ Troubleshooting
│  ├─ CLI
├─ Workers/
│  ├─ Seal Workers
│  ├─ PoSt Workers
├─ Advanced configurations/
│  ├─ Markets
│  ├─ Workers
│  ├─ Sealing
│  ├─ Logging

Maybe we should start with the workers sections first. It´s an easy target, and gets us a little bit into the flow. We would also need to add the Advanced configurations/ for this since we´re splitting some of Seal Worker content out.

So next steps would be:

• Create the Advanced Configuration section Advanced Configuration section #146
• Create the Workers page under the Advanced Configuration section Advanced Configuration section #146
• Publish the PoSt Workers page PoSt workers page #114
• Audit and revise the Seal Workers page Audit SP seal-worker #137

After that I would suggest we aim to finish the Get Started-section & Setup-section.

Updated 23.05.2022 - Current state of the Audit/Revision of the SP-sections of the docs

Get Started:

• Audit and revise the Overview page Audit and revise get-started/overview #171
• Create the tasks page Get-Started/Task page #166
• Audit and revise the Architecture & Hardware requirements page
• Audit and revise the Economics page Audit of economics-page #172

Setup:

• Create the setup section Audit config section, populate advanced-config section #167
• Create/Audit/Revise the pre-requisites page Audit/Revision of Setup/Prerequisities #225
• Create/Audit/Revise the Initialize page Audit/Revision of Setup/Initialize #228
• Create/Audit/Revise the Configuration page Audit/revision of Setup/Configruation #243

Operate:

• Audit and revise Addresses Audit/Revision for Operate/Addresses #245
• Audit and revise Connectivity
• Audit and revise Backup & Restore
• Audit and revise Upgrades Audit/Revision of Operate/Upgrades #244
• Audit and revise Benchmarks Audit/Revision of Operate/Benchmarks #265
• Audit and revise Disputer Audit/Revision of Operate/Disputer #267
• Audit and revise Troubleshooting / Journal Audit/sp-operate-lexluthr #259

Workers:

• Audit and revise the Seal Worker page Audit SP seal-worker #137
• Create/Audit/Revise the PoSt Worker page PoSt workers #158

Advanced Configuration:

• Create the Advanced Configuration section Audit config section, populate advanced-config section #167

[LexLuthr]

https://filecoinproject.slack.com/archives/CPFTWMY7N/p1649422025668839

[rjan90]

Closing this issue now that almost all of the parts outlined here have been done and completed. There are still a small number of pages that are currently being audited/revised in draft PRs, but those should be finished soon.

Any further issues about the content in the storage-provider section of the docs, and auditing/revising those, should be kept in separate issues to make it easier to track them now that we have a much better structure in place.

Next up we should take a look at the Lotus section and its structure. We got some good notes from the Launchpad group that we can use as a starting point.