Upgrade Website (#3294)

Summary:
Pull Request resolved: #3294

## Context

This is the Ax counterpart to the same upgrade in Botorch: pytorch/botorch#2653

## Motivation

We are already using most of the correct tooling but it has become very out of date and is tied together with scripts that are difficult to maintain. Upgrade our tooling by updating to latest versions and dramatically simplifying our setup/deploy scripts.

Includes:
- Upgrade Docusaurus from V1 to V3
- Utilize new MDX support in Docusaurus by converting our tutorials to MDX (previously converted to html)
- Tutorials link to Colab for easy execution
- Website versioning is now handled natively through Docusaurus
- New website versions are only created for major and minor releases (not patches)
- Build and Deploy workflows have been dramatically simplified
- Change Sphinx theme to ReadTheDocs
- API Reference is still built using Sphinx but is now hosted by ReadTheDocs. ReadTheDocs also handles versioning.
- Legacy website will be frozen and available at [archive.ax.dev](archive.ax.dev)
    - Served by Vercel from the `archive` branch
- Website now supports dark mode

| Before    | After |
| -------- | ------- |
|        <img width="1325" alt="image" src="https://github.com/user-attachments/assets/4af73120-f125-47e2-8285-bd01f4251ee9" />         |      <img width="1330" alt="image" src="https://github.com/user-attachments/assets/ea17c47c-5012-47ab-a9d2-4b7e9c103cb0" />      |
| <img width="1328" alt="image" src="https://github.com/user-attachments/assets/3dbde70a-8b84-43a4-8374-362f594de7fc" /> | <img width="1332" alt="image" src="https://github.com/user-attachments/assets/c779b7dd-76b9-4eb3-9b08-ca663732d7ec" />  |
| <img width="1328" alt="image" src="https://github.com/user-attachments/assets/25d74ed7-4536-43a3-8654-ed291ae50cf3" /> | <img width="1331" alt="image" src="https://github.com/user-attachments/assets/e2438746-d975-4a09-96b5-d73e5d1a9980" /> |
| <img width="1327" alt="image" src="https://github.com/user-attachments/assets/bcf71617-a537-45e4-a5e8-452de946a238" /> | <img width="1277" alt="image" src="https://github.com/user-attachments/assets/71b34ff9-1aaf-4c39-ade1-7bb346aad317" /> |

## Considerations
- Our tutorials are natively embedded and versioned in the website using Docusaurus, but our sphinx-generated API reference is not. ReadTheDocs will create new version of the sphinx docs for every new release tag in our github repo.
- Continue using same integrations for Algolia Search and Google Analytics
- The number and type of github workflows we have remain the same, but the complexity of the workflows and the supporting scripts has been simplified.

Pull Request resolved: #3289

Test Plan:
- I've enabled all the workflows in my PR and everything is passing [https://github.com/CristianLara/botorch/actions](https://github.com/CristianLara/botorch/actions)
    - Pypy deploy and codecov are failing as expected due to missing credentials
- Website for my fork is being actively built and served at [cristianlara.me/Ax](cristianlara.me/Ax)
- API Reference is being built and served by ReadTheDocs at [https://ax.readthedocs.io/](https://ax.readthedocs.io/)

Reviewed By: saitcakmak

Differential Revision: D68916590

Pulled By: CristianLara

fbshipit-source-id: 7b1d4a3bffff72a89b16acea7231b0c4bf7ddc1e

.github/workflows/build-and-test.yml

@@ -23,7 +23,7 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v5
       with:
-        python-version: "3.10"
+        python-version: "3.12"
     - name: Install dependencies
       env:
         ALLOW_BOTORCH_LATEST: true

.github/workflows/cron.yml

@@ -37,29 +37,9 @@ jobs:
       pinned_botorch: false
 
   publish-latest-website:
-
-    runs-on: ubuntu-latest
-
-    steps:
-    - uses: actions/checkout@v4
-    - name: Set up Python
-      uses: actions/setup-python@v5
-      with:
-        python-version: "3.10"
-    - name: Install dependencies
-      env:
-        ALLOW_BOTORCH_LATEST: true
-        ALLOW_LATEST_GPYTORCH_LINOP: true
-      run: |
-        # use latest BoTorch
-        pip install git+https://github.com/cornellius-gp/gpytorch.git
-        pip install git+https://github.com/pytorch/botorch.git
-        pip install -e ".[tutorial]"
-    - name: Publish latest website
-      env:
-        DOCUSAURUS_PUBLISH_TOKEN: ${{ secrets.DOCUSAURUS_PUBLISH_TOKEN }}
-      run: |
-        bash scripts/publish_site.sh -d
+    name: Publish latest website
+    uses: ./.github/workflows/publish_website.yml
+    secrets: inherit
 
   deploy-test-pypi:
 
@@ -72,7 +52,7 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v5
       with:
-        python-version: "3.10"
+        python-version: "3.12"
     - name: Install dependencies
       env:
         ALLOW_BOTORCH_LATEST: true

.github/workflows/deploy.yml

@@ -22,38 +22,53 @@ jobs:
       pinned_botorch: true
     secrets: inherit
 
-  publish-stable-website:
-
-    needs: tests-and-coverage-pinned # only run if test step succeeds
+  check-versions:
+    needs: tests-and-coverage-pinned
+    name: Check if major or minor version changed
     runs-on: ubuntu-latest
-
+    outputs:
+      major_minor_changed: ${{ steps.compare.outputs.major_minor_changed }}
     steps:
     - uses: actions/checkout@v4
-    - name: Set up Python
-      uses: actions/setup-python@v5
       with:
-        python-version: "3.10"
-    - name: Install dependencies
-      run: |
-        # use stable Botorch
-        pip install -e ".[tutorial]"
-    - name: Publish latest website
-      env:
-        DOCUSAURUS_PUBLISH_TOKEN: ${{ secrets.DOCUSAURUS_PUBLISH_TOKEN }}
+        fetch-depth: 0
+        fetch-tags: true
+        ref: ${{ github.sha }}
+    - name: Check if major or minor version changed
+      id: compare
       run: |
-        bash scripts/publish_site.sh -d -v ${{ github.event.release.tag_name }}
+        git fetch --tags --force
+        previous_version=$(git describe --tags --abbrev=0 ${{ github.event.release.tag_name }}^)
+        prev=$(cut -d '.' -f 1-2 <<< $previous_version) # remove patch number
+        prev=${prev#v}                                  # remove optional "v" prefix
+        next=$(cut -d '.' -f 1-2 <<< ${{ github.event.release.tag_name }})
+        next=${next#v}
 
-  deploy:
+        echo "Updating from version $previous_version to ${{ github.event.release.tag_name }}"
+        if [[ "$prev" == "$next" ]]; then
+          echo "::warning::Major/Minor version was not changed. Skipping website & docs generation step."
+        else
+          echo major_minor_changed=true >> $GITHUB_OUTPUT
+        fi
 
+  version-and-publish-website:
+    needs: check-versions
+    name: Version and Publish website
+    if: ${{ needs.check-versions.outputs.major_minor_changed == 'true' }}
+    uses: ./.github/workflows/publish_website.yml
+    with:
+      new_version: ${{ github.event.release.tag_name }}
+    secrets: inherit
+
+  deploy:
     needs: tests-and-coverage-pinned # only run if test step succeeds
     runs-on: ubuntu-latest
-
     steps:
     - uses: actions/checkout@v4
     - name: Set up Python
       uses: actions/setup-python@v5
       with:
-        python-version: "3.10"
+        python-version: "3.12"
     - name: Install dependencies
       run: |
         # use stable Botorch

.github/workflows/publish_website.yml

@@ -0,0 +1,74 @@
+name: Publish Website
+
+on:
+  workflow_call:
+    inputs:
+      new_version:
+        required: false
+        type: string
+      run_tutorials:
+        required: false
+        type: boolean
+        default: false
+  workflow_dispatch:
+
+
+jobs:
+
+  build-website:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+      with:
+        ref: 'docusaurus-versions' # release branch
+        fetch-depth: 0
+    - name: Sync release branch with main
+      run: |
+        git config --global user.name "github-actions[bot]"
+        git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
+        git merge origin/main
+        # To avoid a large number of commits we don't push this sync commit to github until a new release.
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: "3.12"
+    - name: Install dependencies
+      run: |
+        pip install -e ".[tutorial]"
+    - if: ${{ inputs.run_tutorials }}
+      name: Run Tutorials
+      run: |
+        python scripts/run_tutorials.py -w $(pwd)
+    - if: ${{ inputs.new_version }}
+      name: Create new docusaurus version
+      run: |
+        python3 scripts/convert_ipynb_to_mdx.py --clean
+        cd website
+        yarn
+        yarn docusaurus docs:version ${{ inputs.new_version }}
+
+        git add --all
+        git commit -m "Create version ${{ inputs.new_version }} of site in Docusaurus"
+        git push --force origin HEAD:docusaurus-versions
+    - name: Build website
+      run: |
+        bash scripts/make_docs.sh -b
+    - name: Upload website build as artifact
+      id: deployment
+      uses: actions/upload-pages-artifact@v3
+      with:
+        path: website/build/
+
+  deploy-website:
+    needs: build-website
+    permissions:
+      pages: write
+      id-token: write
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

.github/workflows/reusable_tutorials.yml

@@ -28,7 +28,7 @@ jobs:
     - name: Set up Python
       uses: actions/setup-python@v5
       with:
-        python-version: "3.10"
+        python-version: "3.12"
 
     - if: ${{ inputs.pinned_botorch }}
       name: Install dependencies with pinned BoTorch
@@ -46,10 +46,13 @@ jobs:
         pip install -e ".[tutorial]"
 
     - if: ${{ inputs.smoke_test }}
-      name: Build tutorials with smoke test
+      name: Run tutorials with smoke test
       run: |
-        python scripts/make_tutorials.py -w $(pwd) -e -s
+        python scripts/run_tutorials.py -w $(pwd) -s
     - if: ${{ !inputs.smoke_test }}
-      name: Build tutorials without smoke test
+      name: Run tutorials without smoke test
       run: |
-        python scripts/make_tutorials.py -w $(pwd) -e
+        python scripts/run_tutorials.py -w $(pwd)
+    - name: Build tutorials
+      run : |
+        python scripts/convert_ipynb_to_mdx.py --clean

.gitignore

@@ -131,18 +131,16 @@ sphinx/build/
 tutorials/experiment.json
 tutorials/ax_client_snapshot.json
 tutorials/*.db
+docs/tutorials/*
+!docs/tutorials/index.mdx
 
 # Docusaurus site
-website/yarn.lock
+**/yarn.lock
 website/build/
 website/i18n/
-website/node_modules/
-
-## Generated for tutorials
-website/_tutorials/
-website/static/files/
-website/pages/tutorials/*
-!website/pages/tutorials/index.js
+**/node_modules/
+.docusaurus
+.cache-loader
 
 ## Generated for Sphinx
 website/pages/api/

.readthedocs.yaml

@@ -0,0 +1,25 @@
+version: "2"
+
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.12"
+  jobs:
+    post_install:
+      # Install latest botorch if not on a released version
+      - |
+        tag=$(eval "git name-rev --name-only --tags HEAD")
+        if [ $tag = "undefined" ]; then
+          pip install git+https://github.com/cornellius-gp/gpytorch.git
+          pip install git+https://github.com/pytorch/botorch.git
+        fi
+
+python:
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - dev
+
+sphinx:
+  configuration: sphinx/source/conf.py

docs/algo-overview.md

@@ -2,8 +2,8 @@
 id: algo-overview
 title: Overview
 ---
-
 Ax supports:
-* Bandit optimization
-  * Empirical Bayes with Thompson sampling
-* Bayesian optimization
+
+-   Bandit optimization
+    -   Empirical Bayes with Thompson sampling
+-   Bayesian optimization