Skip to content

Commit

Permalink
Add sphinx documentation (#249)
Browse files Browse the repository at this point in the history 
* Initial sphinx starter pack setup

* Add .gitignore for doc builds

* Resolve most TODO's in conf.py

* Address TODO's in contributing-myst.md

* Removed last mention to acme in file

* Addressed TODO in contributing.rst

* Update acme

* Set working directory for markdownlint

* Remove line length rule for markdown as it doesn't make sense

* Update doc workflows to trigger on push

* Add words to spelling wordlist

* Add more words to spelling wordlist

* Remove starter pack documentation

* Add introduction

* Add submit-a-test how-to

* Remove unnecessary section

* Add glossary

* Mention TO api docs

* Remove duplicate mention of API docs and VPN

* Update link checks

* Fix spelling issues

* Move main readme into terraform directory

* Fix spelling issues

* Remove doc cheat sheet files

* Remove unused links

* Bring back links.txt file

* Add comment about ignored link checks

* Resolve duplicate frontend warning

* Update docs/content/explanation/glossary.rst

Co-authored-by: Jonathan Cave <[email protected]>

* Add mention of Test Results

* Remove `content` directory

* Add check to only build read the docs if there are changes to docs directory

---------

Co-authored-by: Jonathan Cave <[email protected]>
  • Loading branch information
@omar-selo @jocave
omar-selo and jocave authored Feb 10, 2025
1 parent b111cab commit 869219e
Show file tree
Hide file tree
Showing 26 changed files with 1,213 additions and 177 deletions.
21 changes: 21 additions & 0 deletions .github/workflows/automatic-doc-checks.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
name: Automatic doc checks

on:
push:
branches-ignore:
- 'main'
paths:
- 'docs/**'
- '.github/workflows/automatic-doc-checks.yml'
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
documentation-checks:
uses: canonical/documentation-workflows/.github/workflows/documentation-checks.yaml@main
with:
working-directory: "docs"
fetch-depth: 0
22 changes: 22 additions & 0 deletions .github/workflows/markdown-style-checks.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: Markdown style checks

on:
push:
branches-ignore:
- 'main'
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
markdown-lint:
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: DavidAnson/markdownlint-cli2-action@v16
with:
config: "docs/.sphinx/.markdownlint.json"
52 changes: 52 additions & 0 deletions .github/workflows/sphinx-python-dependency-build-checks.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# The purpose of this workflow file is to confirm that the Sphinx
# virtual environment can be built from source, consequently documenting
# the packages required in the build environment to do that.
#
# This is needed because some projects embeds the documentation into built
# artifacts which involves rendering the documentation on the target
# architecture.
#
# Depending on the architecture, pip may or may not have already built wheels
# available, and as such we need to make sure building wheels from source can
# succeed.
name: Sphinx python dependency build checks
on:
push:
branches-ignore:
- 'main'
paths:
- 'docs/**'
- '.github/workflows/sphinx-python-dependency-build-checks.yml'
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
name: build
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Install dependencies
run: |
set -ex
sudo apt -y install \
cargo \
libpython3-dev \
libxml2-dev \
libxslt1-dev \
make \
python3-venv \
rustc \
libtiff5-dev libjpeg8-dev libopenjp2-7-dev zlib1g-dev libfreetype6-dev liblcms2-dev libwebp-dev tcl8.6-dev tk8.6-dev python3-tk libharfbuzz-dev libfribidi-dev libxcb1-dev
- name: Build Sphinx venv
working-directory: "docs"
run: |
set -ex
make install \
PIPOPTS="--no-binary :all:" \
|| ( cat .sphinx/venv/pip_install.log && exit 1 )
40 changes: 40 additions & 0 deletions .readthedocs.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details

# Required
version: 2

# Set the version of Python and other tools you might need
build:
os: ubuntu-22.04
tools:
python: "3.11"
jobs:
pre_install:
- git fetch --unshallow || true
post_checkout:
# Cancel building pull requests when there aren't changed in the docs directory.
# If there are no changes (git diff exits with 0) we force the command to return with 183.
# This is a special exit code on Read the Docs that will cancel the build immediately.
# https://docs.readthedocs.io/en/stable/build-customization.html#cancel-build-based-on-a-condition
- |
if [ "$READTHEDOCS_VERSION_TYPE" = "external" ] && git diff --quiet origin/main -- docs/;
then
exit 183;
fi
# Build documentation in the docs/ directory with Sphinx
sphinx:
builder: dirhtml
configuration: docs/conf.py
fail_on_warning: true

# If using Sphinx, optionally build your docs in additional formats such as PDF
formats:
- pdf

# Optionally declare the Python requirements required to build your docs
python:
install:
- requirements: docs/.sphinx/requirements.txt
4 changes: 4 additions & 0 deletions .wokeignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# the cheat sheets contain a link to a repository with a block word which we
# cannot avoid for now, ie
# https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html
doc-cheat-sheet*
182 changes: 5 additions & 177 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,183 +1,11 @@
# Test Observer

Observe the status and state of certification tests for various artefacts
Test Observer (TO) is a dashboard for viewing the results of tests run on different environments for a particular artefact. A user interested in testing an artefact (a deb, snap, charm or image) under different environments (particular machines or cloud setups) can use TO as means for storing, viewing and comparing results with previous runs or versions of an artefact. The last use case is particularly useful for catching regressions. Additionally, TO provides a mechanism to assign reviewers that can look at results and mark artefacts as approved or failed to gate updates. It is important to note that TO does not run the tests itself, but provides an API with which users can report the results to.

## Prerequisites for deploying locally
Certification currently deploys an instance of TO that they used for reviewing Stable Release Updates (SRUs). Other teams also use this instance for their tests. You can visit the `frontend <https://test-observer.canonical.com/>`_and view the `API docs <https://test-observer-api.canonical.com/docs>`_, although this currently requires Canonical VPN access. There's also a staging deployment of `frontend <https://test-observer-staging.canonical.com/>`_and `API <https://test-observer-api-staging.canonical.com/docs>`_ that teams can use to test their integration.

- `juju` 3.1 or later (`sudo snap install juju --channel=3.1/stable`)
- `microk8s` 1.27 or later (`sudo snap install microk8s --channel=1.27-strict/stable`) + [permission setup steps after install](https://juju.is/docs/sdk/set-up-your-development-environment#heading--install-microk8s)
- `terraform` 1.4.6 or later (`sudo snap install terraform --classic`)
- `lxd` 5.19 or later (`sudo snap install lxd --channel=5.19/stable` or `sudo snap refresh lxd --channel=5.19/stable` if already installed) + `lxd init --auto` after install.
- `charmcraft` 2.3.0 or later (`sudo snap install charmcraft --channel=2.x/stable --classic`)
- optional: `jhack` for all kinds of handy Juju and charm SDK development and debugging operations (`sudo snap install jhack`)
## Run Locally

## Deploying a copy of the system with terraform / juju in microk8s
For development look at the [backend](/backend/README.md) and [frontend](/frontend/README.md).

Workaround for juju bug https://bugs.launchpad.net/juju/+bug/1988355

```
mkdir -p ~/.local/share
```

Fist configure microk8s with the needed extensions:

```
sudo microk8s enable dns hostpath-storage metallb ingress# metallb setup involves choosing a free IP range for the load balancer.
```

Setup juju:

```bash
juju bootstrap microk8s
juju model-config logging-config="<root>=DEBUG"
```

### Deploy the system locally with Terraform

In the `terraform` directory of your working copy, complete the one-time initialisation:

```bash
cd terraform
terraform init
```

After initialization (or after making changes to the terraform configuration) you can deploy the whole system with:

```bash
TF_VAR_environment=development TF_VAR_external_ingress_hostname="mah-domain.com" terraform apply -auto-approve
```

At the time of writing, this will accomplish deploying the following:

- the backend API server
- the frontend served using nginx
- a postgresql database
- nginx as ingress
- backend connected to frontend (the backend's public facing base URI passed to the frontend app)
- backend connected to database
- backend connected to load balancer
- frontend connected to load balancer

Terraform works by applying changes between the current state of the system and what is in the plan (the test-observer.tf configuration file). When `terraform apply` is run the 1st time, there is no state -> it will create the Juju model and all resources inside it. When it is run with a pre-existing model already in place, it will instead set / unset config values that have changed, add / remove relations, add / remove applications, etc. Basically, it makes working with Juju declarative - yay!

The terraform juju provider is documented over here: https://registry.terraform.io/providers/juju/juju/latest/docs

Terraform tracks its state with a .tfstate file which is created as a result of running `terraform apply` -- for production purposes this will be stored in an S3-like bucket remotely, and for local development purposes it sits in the `terraform` directory aftery you have done a `terraform apply`).

After all is up, you can run `juju switch test-observer-development` to use the development juju model. Then `juju status --relations` should give you output to the direction of the following:

```bash
$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp
test-observer-development juju-controller microk8s/localhost 3.1.2 unsupported 15:38:51+03:00

App Version Status Scale Charm Channel Rev Address Exposed Message
api active 1 test-observer-api latest/edge 15 10.152.183.182 no
db 14.7 active 1 postgresql-k8s 14/stable 73 10.152.183.172 no Primary
frontend active 1 test-observer-frontend latest/edge 8 10.152.183.79 no
ingress 25.3.0 active 1 nginx-ingress-integrator stable 59 10.152.183.103 no Ingress IP(s): 127.0.0.1, 127.0.0.1, Service IP(s): 10.152.183.72, 10.152.183.34

Unit Workload Agent Address Ports Message
api/0* active idle 10.1.131.142
db/0* active idle 10.1.131.132 Primary
frontend/0* active idle 10.1.131.169
ingress/0* active idle 10.1.131.167 Ingress IP(s): 127.0.0.1, 127.0.0.1, Service IP(s): 10.152.183.72, 10.152.183.34

Relation provider Requirer Interface Type Message
api:test-observer-rest-api frontend:test-observer-rest-api http regular
db:database api:database postgresql_client regular
db:database-peers db:database-peers postgresql_peers peer
db:restart db:restart rolling_op peer
ingress:nginx-route api:nginx-route nginx-route regular
ingress:nginx-route frontend:nginx-route nginx-route regular
```

## Add /etc/hosts entries

To test the application, you need to create some aliases in `/etc/hosts` to the IP address that the ingress got from `metallb` (`juju status` above will find you the ingress IP). Let's assume you have a domain `mah-domain.com` that you want to expose service under, the backend and frontend will be present as subdomains `test-observer.mah-domain.com` and `test-observer-api.mah-domain.com`, respectively:

```bash
$ cat /etc/hosts
192.168.0.202 test-observer.mah-domain.com test-observer-api.mah-domain.com
...
```

Note that without this step the frontend will fail to connect to api as it's trying to use `test-observer-api.mah-domain.com`

## Developing the charm

To develop and test updates to the backend and frontend charms, you would typically want to first complete the above steps to deploy a working system. Once you have done that, proceed with the following steps.

### Build and refresh the backend charm

You can make edits to the backend charm and refresh it in the running system on the fly with:

```bash
cd backend/charm
charmcraft pack
juju refresh test-observer-api --path ./test-observer-api_ubuntu-22.04-amd64.charm

# to update the OCI image that runs the backend
juju attach-resource test-observer-api api-image=ghcr.io/canonical/test_observer/backend:[tag or sha]
```

### Build and refresh the frontend charm

Same thing with the frontend:

```bash
cd frontend/charm
charmcraft pack

juju refresh test-observer-frontend ./test-observer-frontend_ubuntu-22.04-amd64.charm

# to update the OCI image that runs the backend
juju attach-resource test-observer-frontend frontend-image=ghcr.io/canonical/test_observer/frontend:[tag or sha]
```

Note that the frontend app is made aware of the backend URL to connect to using the global `window.testObserverAPIBaseURI`, which is set at runtime with some nginx config level trickery based on...

- the `test-observer-api` charm's `hostname` config value.
- the frontend charm's `test-observer-api-scheme` config value.

These in turn can be set using the terraform plan (`terraform/test-observer.tf` and associated variables).

## Running tests

To run the unit and integration tests for the frontend charms, do the following:

```bash
cd frontend/charm
tox -e unit
tox -e integration
```

## Releasing the charms

Charms are released through GitHub actions on push to main. If however you need to release charms on your branch before merging with main you could always just add your branch as a trigger to those same GitHub actions.

## VS Code & charm libraries

VS Code fails to find (for autocompletions and code navigation purposes) the charm libraries under `lib` in each of `backend/charm` and `frontend/charm`. There is a .vscode-settings-default.json found under each of these directories which you can copy to the `.gitignore`d path `.vscode/settings.json` to make them fly. Taking the backend charm as an example:

```bash
mkdir -p backend/charm/.vscode
cp backend/charm/.vscode-settings-default.json backend/charm/.vscode/settings.json

mkdir -p frontend/charm/.vscode
cp frontend/charm/.vscode-settings-default.json frontend/charm/.vscode/settings.json
```

Now if you use as your project the directory `backend/charm` and `frontend/charm` respectively (which you'll want to do also for them to keep their own virtual environments), VS Code should be happy.

## Handy documentation pointers about charming

- [Integrations (how to provide and require relations)](https://juju.is/docs/sdk/integration)

### Enable the K8s Dashboard

You need an auth token in case you want to connect to the kubernetes dashboard:

```bash
microk8s kubectl describe secret -n kube-system microk8s-dashboard-token
```
To run via Terraform, juju and charms simulating production and staging environments, look at [terraform](README.md)
33 changes: 33 additions & 0 deletions docs/.custom_wordlist.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Leave a blank line at the end of this file to support concatenation
backend
backend's
Backend
cjk
cryptographically
dvipng
fonts
freefont
frontend
frontend's
Frontend
github
GPG
GPLv
gyre
https
lang
latexmk
md
otf
plantuml
schemas
SRU
SRUs
tex
texlive
TOC
utils
VPN
WCAG
xetex
xindy
5 changes: 5 additions & 0 deletions docs/.gitignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
_build
venv
.doctrees
warnings.txt
.wordlist.dic
21 changes: 21 additions & 0 deletions docs/.sphinx/.markdownlint.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"default": false,
"MD003": {
"style": "atx"
},
"MD014": true,
"MD018": true,
"MD022": true,
"MD023": true,
"MD026": {
"punctuation": ".,;。，；"
},
"MD031": {
"list_items": false
},
"MD032": true,
"MD035": true,
"MD042": true,
"MD045": true,
"MD052": true
}
Loading

0 comments on commit 869219e

Please sign in to comment.