Move online documentation to Github Pages

[ccordoba12]

The old PyPI doesn't allow to upload new documentation to pythonhosted.org, so we need to remove the docs hosted there and move them to readthedocs.

CAM-Gerlach Edit:

• Move docs from both repo branches
• Create/add other key files
  • README
  • CONTRIBUTING
  • LICENSE
  • AUTHORS
  • .gitignore
  • .gitattributes
• Minor fixes to existing docs
  • Harmonize line endings to LF
  • Remove extra spaces
  • Fix build warnings
• Add sphinx building and enable Travis
  • Add/Setup sphinx makefiles
  • Create/setup .travis.yml
  • Enable Travis and test builds
• Setup repo settings
  • Permissions
  • Tags
  • Description
  • Protected branches
  • Other settings as needed
• Setup and enable doctr deployment
  • Create gh-pages and .nojekyll
  • Configure doctr settings and keys
  • Add doctr deployment to Travis
  • Test functionality and deploy
• Other work items
  • Update conf.py with modernization and fixes
  • Minor doc updates
• Before initial public release
  • Update out of date screenshots
  • Baseline editing pass to update and copyedit
    • Initial cross-file changes
    • Per-file updates, copyediting, line breaks and minimal expansion
    • Apply remaining cross-file style guide formatting and fix links
  • Decide on modern theme
  • Add Profiler doc: PR REVIEW
• Final steps to release
  • Decide on what to do about multiple branches and implement
  • Redirect to docs.spyder-ide.org subdomain
  • Add banner and fonts, etc. from our main website
  • Final layout and link fixes
  • Font size harmonization
  • Add common footer (?)
  • Update other links throughout Spyder to point to new docs instead of PyHosted: PR REVIEW
  • Make layout adaptive/non-broken on narrower screens/windows
• Following release
  • Inform/submit PRs to other sources to add/update Spyder website and doc links
  • Redirect or eventually remove PyHosted docs (in a few months)

[jnsebgosselin]

Looks like it is already on rdt: http://spyderide.readthedocs.io/en/latest/

Last build was generated 3 months ago.

[ccordoba12]

We don't have control on that. I really don't who has it.

[jnsebgosselin]

The RTD user that has control of it is maxgam : https://readthedocs.org/profiles/maxgam/

[jnsebgosselin]

Unfortunately, i think there is no free organization option for open source project on RTD. There is only a paid option: https://readthedocs.com/pricing/.

So I think we will have to contact a RTD admin and ask them to delete the current project that is own by maxgam. Then we will be able to claim back the ownership of the spyder repo on RTD. Maybe we can use Spyder bot for this purpose, or you can use your personal account.

[ccordoba12]

@CAM-Gerlach, please open an issue in Readthedocs so they can put us in contact with maxgam, so he/she give us admin permissions to our docs.

After that, we could set up a webhook to update our docs every time new content is pushed to Github, and then we could remove our old docs (which can't be updated anymore).

[CAM-Gerlach]

please open an issue in Readthedocs so they can put us in contact with maxgam

Done, as requested. Do we have a contingency plan in case we aren't able to immediately resolve it in this fashion? Aside from picking a different name, I can think of a few potential legal strategies, though IANAL.

After that, we could set up a webhook to update our docs every time new content is pushed to Github, and then we could remove our old docs (which can't be updated anymore).

Seems quite simple enough, once we get control of the project...

[CAM-Gerlach]

@ccordoba12 Hey, any progress on your end with this? What's the current status?

[ccordoba12]

I just created the new repository for our docs:

https://github.com/spyder-ide/spyder-docs

@CAM-Gerlach, please move our current docs from here to that repo. There you can update the Sphinx theme to use a more modern one and add new content.

Once that's ready, please use doctr:

https://github.com/drdoctr/doctr

to deploy the docs to the GitHub pages of the repo.

The last step of the plan is to add a link in our new homepage to the generated documentation.

[ccordoba12]

One thing I'm not sure we should do or not is to stop distributing our docs with the package. I mean, we can certainly do that for Spyder 4, but I think we should not do it for Spyder 3 because it's in maintenance mode and I'm sure some people are using that feature.

[jnsebgosselin]

Once that's ready, please use doctr to deploy the docs to the GitHub pages of the repo.

Do I understand correctly or we won't be using RTD to deploy the docs? It's going to be directly hosted on github?

[CAM-Gerlach]

@ccordoba12 Will do, thanks.

There you can update the Sphinx theme to use a more modern one

Want me to offer up a few good options in a PR or Issue on that repo and have the team vote/give feedback?

add new content.

Anything particular in mind that's high priority before we deploy them?

One thing I'm not sure we should do or not is to stop distributing our docs with the package.

I guess the biggest major downside is if we make any changes before we drop support for 3.x, we'll have to submit two PRs against two different repos, right?

[jnsebgosselin]

@CAM-Gerlach even if we don't use RTD (which I'm not sure) maybe that would be a good idea to claim the name anyway just in case?

[ccordoba12]

maybe that would be a good idea to claim the name anyway just in case?

We can't claim it. We can just become co-admins of the Spyder project (which we already are). But once the docs are removed from the Spyder repo, the project will cease to be valid and we'll be admins of the spyder-docs one.

[ccordoba12]

Want me to offer up a few good options in a PR or Issue on that repo and have the team vote/give feedback?

Sure, thanks!

Anything particular in mind that's high priority before we deploy them?

We should probably go with our current content to start. Then you could add new content in different PRs. There are a lot of new things we could talk about, but let's leave that for later.

I guess the biggest major downside is if we make any changes before we drop support for 3.x, we'll have to submit two PRs against two different repos, right?

We could have two branches in spyder-docs, one for master and another one for 3.x. Then before every 3.x release we could create a new PR in spyder to update the docs contents. I don't see a need to do a PR in spyder after every change in spyder-docs.

[jnsebgosselin]

We can't claim it. We can just become co-admins of the Spyder project (which we already are). But once the docs are removed from the Spyder repo, the project will cease to be valid and we'll be admins of the spyder-docs one.

That's what I meant. Claiming the spyder-docs one. I can already claim it if I want I think. So better you do it now before someone else does?

[ccordoba12]

Yes, please claim it.

[jnsebgosselin]

Yes, please claim it.

Done. I've added you and @CAM-Gerlach as admins.

[CAM-Gerlach]

@ccordoba12 So are we still deploying directly to github pages with doctr, to RtD as discussed previously or both? If both, which should be the preferred/canonical outlet to link users to, e.g. from our site?

[ccordoba12]

Let's try with Github pages first.

[CAM-Gerlach]

@ccordoba12 I've done everything except for the final deployment with doctr, as that's currently blocked by needing to build with Travis. Can you (or someone else who has admin rights) enable Travis on spyder-docs ? It should be ready to test building the docs with my latest PR spyder-ide/spyder-docs#2 there, then I just need to add doctr and #5986 will be complete , but I need to make sure Travis is working before I do that.

Also, since I apparently can't see the settings page on the repo likely due to lack of permissions, can you setup the proper tags, description line, branch protection, and other settings for that repo; or, if possible, grant me the appropriate permissions on that repo so I can do so? Thanks.

[CAM-Gerlach]

@ccordoba12 Okay; all that's done and the current docs are live on Github pages. However, there remains the question of what to do about supporting docs for multiple versions of Spyder simultaneously, given we have separate docs for 3.x (Spyder 3) and master (Spyder 4). They currently only have a few small differences, and we only have a need to publish those for 3.x fight now as Spyder 4 isn't even in beta yet, but both of those will rapidly change as the latter gets closer to release.

As of now, I have it set up to build and test-deploy docs with PRs to both branches, so we know they work, but to only actually deploy them to the live site from the 3.x. That could work for now, but at least for the long term if not sooner we'll need to do something about that. The options I can think of:

• Keep the 3.x docs up until Spyder 4 is released, and then do a hard swich-over to the latter, simply setting the docs to deploy automatically from that branch instead.
  Advantages: Requires zero additional effort now, and a nearly trivial amount to switch over. Keeps things simple and avoids near-term user confusion.
  Disadvantages: No ability for users to view docs for any release other than whatever we consider the "current" one.

• Have doctr output each branch to a separate directory on the gh-pages branch, and manually create a main index page in the root of the repo, or on the main Spyder website linking to the two versions; link to that page from the documentation link on our homepage/main site.).
  Advantages: Shouldn't be too difficult to get set up, and even less effort than the first option to "switch" which one is "live" if necessary at all—just changing what is linked to, or the text on the main index page, if that. Allows users to view either set of docs as they choose, so long as they make it to the main index page.
  Disadvantages: Somewhat inelegant; requires creating a main index page either from scratch on the spyder-docs gh-pages and maintaining it by hand (possibly cleaner, but much worse from a design perspective and we'll have to use yet a third theme from our website and docs themselves), or probably better just creating a dedicated page on the main Spyder website to link to both doc version indices. No ability for the user to easily switch from one version to another while within one version's docs, only from the main doc page, unless we hack something in.

• Abandon doctr and gh-pages and setup and use Read the Docs with the spyder-docs repo instead, and link to that.
  Advantages: Has built in support for not only versioned branches, but also can automatically use tags to organize docs for each released version, and automatically categorize them as stable or development based on PEP 440. Can customize handling much of this, including what version is the default, etc. without much work or changes to the code. Users should be able to easily switch between different versions from any page, right from the familiar popup menu. Also, would make the choice of theme straightforward—just the standard, fairly nice RTD one.
  Disadvantages: Would need a (hopefully modest) amount of additional setup. Reliant on RTD as a platform. Less direct control than github pages.

Or, there might be another, better option entirely...thoughts @ccordoba12 @jnsebgosselin ?

[ccordoba12]

Keep the 3.x docs up until Spyder 4 is released

I'm in favor of this option, that's what we've done so far. Besides, from our issue tracker here on Github we can conclude very few people is using master.

Let's start with this and see how things evolve (now that everything is setup in spyder-docs). Another thing I like about using our own deployment, instead of RTD, is that can render our docs inside our website.

Finally, where our docs live is not the most important thing but its content. So @CAM-Gerlach, we rely on you to expand our docs and better explain people how to use the many features Spyder has to offer. We also need a more modern Sphinx theme, with a color scheme similar to the one we're using in our website.

[CAM-Gerlach]

@ccordoba12

I'm in favor of this option, that's what we've done so far.

Cool; that's the current status quo as of now if and until we want to change it.

that can render our docs inside our website.

Especially since its on a subpath of our nominal domain, is there an easy way to set this up? Or have them in an iframe beneath the top bar or something? I have very little in the way of actual web glueing; I'm only done some stuff on the very high level (design, consulting, CMS, etc), and a very little bit on the scripting side (a taste of PHP and JS). At least for starters, absent something better, I'll just link it to a "documentation" link on the top bar and then solicit feedback on people's preferences for a more modern sphinx theme that matches the site.

we rely on you to expand our docs and better explain people how to use the many features Spyder has to offer.

Yep, will do right after I get all the legal stuff taken care of, help get us funded, fix the outstanding bugs I'm assigned, and hopefully pitch in getting the website into better shape, alongside the usual tide of user issue reports, PR reviews and repo management...oh and don't forget my courses, internship job, personal research, funded research, and volunteer outreach :)

Hopefully this summer I can do a big push on it, especially if I'm not already too busy with my next big funded project designing new sensors and ML/ANN processing algorithms for NOAA's satellite fleet. In the meantime, I'll try to get in incremental improvements, but it seems my primary short term focus is funding (both Spyder's and my own) and getting as much as practicable low hanging fruit done for 3.2.9.

[jitseniesen]

I can help with writing the docs once we get to that part.

[ccordoba12]

Especially since its on a subpath of our nominal domain, is there an easy way to set this up?

Once @dalthviz finish our blog (which is almost done), he'll add an entry in our nav bar to easily access them. After that, we could remove the deployment in pythonhosted.org, which can't be updated anymore.

then solicit feedback on people's preferences for a more modern sphinx theme that matches the site.

Yes, please open an issue in spyder-docs to discuss about it.

Hopefully this summer I can do a big push on it

Great! I think we can set up some tasks and ask people what they want to work on.

[ccordoba12]

I can help with writing the docs once we get to that part.

Thanks a lot @jitseniesen! I'd say you can start right now, if you have time, by opening a document in Dropbox paper first, then ask for feedback and finally, when the discussion has finished, open a PR in spyder-docs.

I think we should start with a Dropbox paper because we have attested that reviewing content in a PR is very cumbersome.

[CAM-Gerlach]

@ccordoba12

Once @dalthviz finish our blog (which is almost done), he'll add an entry in our nav bar to easily access them.

Okay, so this is in work right now? Any idea on an ETA? I'm guessing quick enough that you're implicitly recommending that I just wait for that to proceed rather than going ahead and submitting a PR to do it myself, since this is the last blocker toward resolving this work item?

Yes, please open an issue in spyder-docs to discuss about it.

Will do shortly. Should I notify our core and juniors on the issue directly with @ to weigh in, or announce it with @ all on the Gitter?

Great! I think we can set up some tasks and ask people what they want to work on.

I'd say you can start right now, if you have time

I'll try to get a basic framework setup with issues on that repo. Right now, my proposed plan for 3.2.9

1. Do a baseline editing pass through all the docs to make sure they're at least up to date and free of major grammar/style issues—I can do this pretty quickly
2. Implement the new theme once people decide on one
3. Update the out of date screenshots and perhaps add a few more (in overview, and possibly elsewhere) showing different layouts, themes and features/panels as you suggested on the other issue; and as I mentioned over there, I can take them all in one go with those.

I can commit to at least doing all of that for the time being; beyond that I would think the next shorter-term priority for me and/or @jitseniesen is hitting some of the low-hanging fruit content improvements like filling out features that aren't described, and describing more of the "how" and not just the "what" when explaining a feature or capability. But really, almost anything is helpful at this point.

[CAM-Gerlach]

I think we should start with a Dropbox paper

@ccordoba12 Are we sure Dropbox paper won't mangle any of the formatting, spacing, characters in our rsts? I know it has good markdown support and can input and output md but none that I'm aware of for rst. Would setting the font to monospaced and disabling any automatic formatting get us there?

[ccordoba12]

Okay, so this is in work right now? Any idea on an ETA?

It should be ready by the end of next week, but I'd like to have a new theme in place for our docs before doing the transition.

Should I notify our core and juniors on the issue directly with @ to weigh in, or announce it with @ all on the Gitter?

As you wish ;-)

Right now, my proposed plan for 3.2.9

Good plan. Please go ahead with it.

hitting some of the low-hanging fruit content improvements like filling out features that aren't described, and describing more of the "how" and not just the "what" when explaining a feature or capability

That would be a great start!

[CAM-Gerlach]

It should be ready by the end of next week, but I'd like to have a new theme in place for our docs before doing the transition.

That's quite a while, but I suppose it makes sense if it takes us that long to decide on a theme. Hopefully I'll have the baseline editing pass and the screenshots done sooner, as well.

That would be a great start!

I can't promise how much in-depth content work I can prioritize for 3.2.9 considering my many other tasks for that and other near-term milestones (website, funding, etc), but hopefully @jitseniesen can pitch in there at some point.