docs: move and adapt the other chapters of Stan's docs suitable for public docs

also added `just serve-docs`

Author: alehander92

.gitignore

@@ -11,7 +11,7 @@ build-*/
 .direnv/
 result
 
-docs/experimental-documentation/build/*
+docs/book/book
 src/index.js
 ui.js
 debugger.js

ci/deploy/docs.sh

@@ -2,7 +2,7 @@
 
 set -e
 
-pushd docs/experimental-documentation/mdbook-ct
+pushd docs/book/
 
 # TODO: eventually: a more fine-grained nix shell/env?
 # nix-shell --command "mdbook build"

docs/experimental-documentation/.htaccess docs/book/.htaccess

@@ -0,0 +1,36 @@
+## Summary
+
+- [Introduction](./introduction.md)
+
+## Usage guide
+
+- [Intro to usage guide](./usage_guide/intro_to_usage.md)
+- [CLI](./usage_guide/CLI.md)
+- [Basic GUI](./usage_guide/basic_gui.md)
+- [Tracepoints](./usage_guide/tracepoints.md)
+- [CodeTracer Shell](./usage_guide/codetracer_shell.md)
+
+## Backends
+
+- [DB backend](./backends/db_backend.md)
+  - [Noir](./backends/db-backend/noir.md)
+  - [Ruby](./backends/db-backend/ruby.md)
+  - [Python](./backends/db-backend/py.md)
+  - [Lua](./backends/db-backend/lua.md)
+  - [small](./backends/db-backend/small.md)
+  
+- [RR backend](./backends/rr_backend.md)
+  - [C & C++](./backends/rr-backend/c_and_cpp.md)
+  - [Rust](./backends/rr-backend/rust.md)
+  - [Nim](./backends/rr-backend/nim.md)
+  - [Go](./backends/rr-backend/go.md)
+
+## Building & packaging
+
+- [Build systems](./building_and_packaging/build_systems.md)
+
+### Misc
+
+- [Troubleshooting](./misc/troubleshooting.md)
+- [Environment variables](./misc/environment_variables.md)
+- [Building the documentation](./misc/building_docs.md)

docs/experimental-documentation/mdbook-ct/book.toml docs/book/book.toml

@@ -0,0 +1,5 @@
+## Lua
+
+We still don't have Lua support, however we've had a very minimal proof of concept experiment with supporting Lua (the interpreter, not LuaJIT) with our RR backend years ago.
+
+We can change that patch to produce traces suitable for the DB backend, or at least publish some kind of basis for that(the RR approach remains possible, but with different tradeoffs)

docs/experimental-documentation/mdbook-ct/shell.nix docs/book/shell.nix

@@ -0,0 +1,7 @@
+## Noir
+
+The initial release of CodeTracer has MVP support for the Noir programming language. It has been developed in collaboration with the Blocksense team and currently requires the use of the [Blocksense Noir Compiler](https://github.com/blocksense-network/noir), which is included in the CodeTracer distribution.
+
+We support many of Noir's features, but not all: e.g. we don't support mutable references currently, we don't serialize struct values and some other cases.
+
+We are planning on adding support for the missing features.

docs/book/src/SUMMARY.md

@@ -0,0 +1,8 @@
+## python
+
+We currently have started adding support for recording python programs for the DB backend, but the prototype isn't finished yet. 
+
+The recorder is currently hosted in the [codetracer-python-recorder](https://github.com/metacraft-labs/codetracer-python-recorder) repo.
+
+You can read its README for more details. We are welcoming contributors!
+

docs/book/src/backends/db-backend/lua.md

@@ -0,0 +1 @@
+# Python

docs/book/src/backends/db-backend/noir.md

@@ -0,0 +1,9 @@
+## Ruby
+
+We currently have partial support for the Ruby programming language in the DB backend. 
+We support many cases, you can use it like `ct record <path to rb file> [<args>]` and `ct replay <name of rb file>` (or directly `ct run <path to rb file> [<args>]`)
+
+The recorder for Ruby is currently hosted in the [codetracer-ruby-recorder](https://github.com/metacraft-labs/codetracer-ruby-recorder) repo.
+
+You can read its README for more details. We are welcoming contributors!
+

docs/book/src/backends/db-backend/py.md

@@ -1,5 +1,8 @@
-SmallLang is, as the name says, a small, lisp-like language, that's used for general testing of the DB backend.
+## small lang
+
+small lang is, as the name says, a small, lisp-like language, that's used for general dogfooding of new tracing features and testing of the DB backend.
 It's purely experimental.
 
 You can find its source code under `src/small-lang`. There, you can also find multiple files with the `.small`
 extension, which can be used as examples of the language.
+

docs/book/src/backends/db-backend/python.md

@@ -0,0 +1,30 @@
+## DB backend
+
+The DB backend is currently used for interpreted languages, like scripting and blockchain/zero knowledge languages:
+
+1. [Noir](./db-backend/noir.md)
+1. [Ruby](./db-backend/ruby.md)
+1. [Python(prototype not finished yet)](./db-backend/py.md)
+1. [Lua(not done: just a plan)](./db-backend/lua.md)
+1. [small(a toy interpreter for dogfooding)](./db-backend/small.md)
+
+If you're interested in db-backend support for those languages or for now ones, you can discuss with us on our Github issue tracker,
+in the discussions or in our [chat server](https://discord.gg/aH5WTMnKHT). We welcome contributors!
+
+However it might be used for emulators(and this way for native languages) in the future.
+
+Also, languages can have both native(AOT) and interpreted implementations: 
+so we can more easily support e.g. interpreters/VM-s of certain languages with this backend currently.
+
+As the name suggests, this backend is implemented as a database. Due to its properties, this backend provides
+a more complete feature set - one of the reasons it was ready first.
+
+For each language, we define a tracer, which instruments or hooks into the interpreter, VM or program and 
+records all the data, that we need in a trace folder. 
+
+We postprocess and index that trace into a database-like structure. Currently this happens in the beginning of the replay.
+For now the data in the database is represented by a runtime-only Rust structure, however
+we plan on storing it as a file, which can be memory-mapped and on separating the postprocessing/indexing step as 
+an optionally independent action.
+
+

docs/book/src/backends/db-backend/ruby.md

@@ -0,0 +1 @@
+# Noir

docs/experimental-documentation/Introduction/Backends/DBBackend/SmallLang.md docs/book/src/backends/db-backend/small.md

@@ -0,0 +1,6 @@
+## C & C++
+
+We have partial support for C and C++ in the RR backend. C generally has good support, though it is less advanced than
+the Rust implementation.
+
+Support for C++ is still experimental, due to missing support for many C++ constructs.

docs/book/src/backends/db_backend.md

@@ -1,2 +1,4 @@
+# Go
+
 We have developed а basic integration that supports small parts of the Golang language in the Core backend. 
-Development on a more complete revision is scheduled to start soon!
+Development on a more complete revision is scheduled to start soon!

docs/book/src/backends/db_backend/noir.md

@@ -0,0 +1,18 @@
+# Nim
+
+We have partial support for the Nim backend, with it being somewhat behind in features compared to C, but ahead,
+compared to C++.
+
+The Nim backend only supports Nim 1 as of today, but there are plans to add support for Nim 2.
+
+
+Some additional future goals that have been announced in the Nim forum are:
+
+* First-class support for macro-generated code - macro generated code should be indistinguishable from regular code. You can expand it layer by layer, set breakpoints and tracepoints anywhere within it, and step through it in all the familiar ways. (or at least part of those)
+* Precise debug info - you should be able to set breakpoints and tracepoints at the level of individual sub-expressions and you can switch between debugging Nim, C or assembly code at any time.
+* Compile-time debugging - with few clicks in your text editor, you should be able to trace the execution of the Nim VM itself.
+
+> [!NOTE]
+> For now we're focused on the beta and initial releases of the RR backend though, and on initial releases for some of the language supports.
+> This means the initial support for Nim would be focused on the same features as the other languages: stepping, tracepoints, event log, calltrace, omniscience, state, history. We do hope we'll be able to continue with some of the more advanced goals afterwards.
+

docs/book/src/backends/rr-backend/c_and_cpp.md

@@ -0,0 +1,3 @@
+# Rust
+
+Our backend for Rust is the most developed out of all languages in the RR backend, with it being closest to MVP status(but still non trivial amount of work left).

docs/experimental-documentation/Introduction/Backends/CoreBackend/Golang.md docs/book/src/backends/rr-backend/go.md

@@ -0,0 +1,32 @@
+## RR backend
+
+The RR backend is used for systems programming languages, like:
+
+1. [C & C++](./rr-backend/c_and_cpp.md)
+1. [Rust](./rr-backend/rust.md)
+1. [Nim](./rr-backend/nim.md)
+1. [Go](./rr-backend/go.md)
+
+Under the hood, it uses a patched version of the [RR debugger](https://rr-project.org/), as well as the standard
+GDB debugger.
+
+We are still not distributing it, but we might need to rethink the GDB usage before that.
+
+The Core backend is based on a dispatcher that maintains a pool of rr replay processes. These processes allow you to jump back and forward in time, while also
+preloading information, such as the information, needed for omniscience and calltrace, as well as managing other features, such as tracepoints. This logic is
+currently implemented using python code using the gdb API. 
+(If required, this can be changed to use the lldb API or something more custom, like a dwarf lib-based solution. We are currently using the rust `gimli` DWARF lib for a tool, that helps with recording/metadata in the RR backend).
+
+These are the main differences, when compared to the DB Backend: based on the way RR works:
+
+1. Only non-deterministic events are recorded.
+1. The program's different states are navigated through re-execution.
+1. More practical for recording real-world software(RR was originally created by Mozilla to debug Firefox)
+1. Some features of CodeTracer are more challenging to implement, compared to the [DB Backend](./backends/db-backend.md)
+
+> ![NOTE]
+> This backend is in active development, and is yet to be released to the wider public. 
+>
+> It is currently proprietary and it might remain closed source. 
+> Open sourcing it is also possible, if we find a suitable business model.
+

docs/book/src/backends/rr-backend/nim.md

@@ -1,12 +1,16 @@
+## Build systems
+
 Codetracer uses nix, just, tup and direnv as parts of its build system.
 
 ## Breakdown of the different components
 ### Nix
 The Nix package manager/build system deals with managing all dependencies, required by CodeTracer, as well as packaging
-it for NixOS, and other operating systems through an AppImage generator utility.
+it for NixOS. It can package for other operating systems through an AppImage generator utility, however currently we are building our
+appimages with custom shell scripts in `appimage-scripts`.
 
 ### Just
-The Just build system is used to trigger build commands. It's semi-analogous to a Makefile. The following commands are
+
+The `just` tool is used to trigger build commands and various other actions. It's semi-analogous to a Makefile. The following commands are
 the most widely-used:
 
 1. `just build` - Builds the project with Tup and starts the automatic build process, which is used for active development
@@ -27,10 +31,13 @@ The Tup build system is used for local builds of CodeTracer and deals with calli
 The `direnv` utility sets up your local environment for using CodeTracer.
 
 ## Packaging
-### Detailed breakdown of the Nix package
+
+### More detailed breakdown of the Nix package
+
 Coming soon!
 
 ### Packaging for non-NixOS distributions
+
 > [!TIP]
 > If you're a user that wants a package for your distribution contact us. We're currently in the process of creating
 > packages for popular distributions, such as Debian/Ubuntu, Fedora/RHEL, Arch Linux, Gentoo, Void, etc. 
@@ -40,7 +47,10 @@ which you can install to `/usr/bin`. Along with it, you should also install our
 from `resources/` to the needed directories, such as `/usr/share/pixmaps` and `/usr/share/applications`.
 
 ### Packaging for Windows(DB-backend only)
+
 Coming soon!
 
 ### Packaging for macOS(DB-backend only)
+
 Coming soon!
+

docs/book/src/backends/rr-backend/rust.md

@@ -3,13 +3,9 @@
 Welcome to the codetracer-desktop wiki. Here you can find information on almost every topic
 regarding codetracer development and usage.
 
-## What is codetracer
-Codetracer is a debugging environment, based on the concept of record and replay, developed as a powerful tool to easily
-debug complex applications.
-
 ## Installation
 > [!CAUTION]
-> Codetracer can only be installed on Linux and macOS.
+> Codetracer can only be installed on Linux and macOS currently.
 
 ### Prerequisites
 On systems that are not NixOS, you need to install `direnv` and `nix`.
@@ -50,39 +46,14 @@ TODO:
 > 1. Run `direnv allow`
 
 ### Building and running the tests
-To start running tests do the following:
 
-1. Run `tester build` - Builds tester
-1. Run `tester parallel` - Runs the tests
+Coming soon!
 
 ### Enabling `cachix`
-> [!NOTE]
-> This step is optional
-
-<!-- TODO: Scrap or completely rewrite the cachix instructions for end users. Alternatively, make this an internal developer guide -->
-
-Cachix is a cache for nix that allows you to save time on compiling codetracer and related projects. To enable `cachix` do the following:
-
-1. Log into [cachix](https://www.cachix.org/) with your personal GitHub account
-1. Create an authentication token
-1. Run `cachix authtoken --stdin`
-1. Paste the token and click enter
-1. Press `CTRL + D` to save the token
-1. Run `cachix use metacraft-labs-codetracer`
-1. Run `direnv allow`, `nix develop`, or `just build-nix` to auto-download cached binaries if available
-
-### Explicit `cachix` setup
-> [!NOTE]
-> You have to be an admin for the private cache. Ask an administrator to get added.
-
-Our current `cachix` setup pushes binaries from CI, but if you want to manually push to `cachix` as well, or want to know how pushing works, you can do the following:
-
-1. Go to [this page](https://app.cachix.org/organization/metacraft-labs/cache/metacraft-labs-codetracer/settings/authtokens)
-1. Create an auth token with `Read+Write` permissions
-1. Locally register it as described in the above heading
 
-To push the dev shell to `cachix` use either one of the following commands:
+<!-- TODO(alexander): I removed the detailed cachix guide, as it's sensitive, and we don't have a public codetracer cache yet  -->
+<!-- either include it in an internal docs in the rr-backend, or re-include here when this is discussed again -->
 
-1. Automatically: `just cachix-push-devshell`
-1. Manually: `cachix push metacraft-labs-codetracer "$(nix build --print-out-paths .#devshells.x86_64-linux.default)"`
+Cachix is a cache for nix that allows you to save time on compiling codetracer and related projects. We'll discuss using a public codetracer cache
+for the open sourced parts, however this is available only internally for now.
 

docs/book/src/backends/rr_backend.md

@@ -0,0 +1,8 @@
+## Building the documentation
+
+The documenation for codetracer is written in Markdown using [mdbook](https://rust-lang.github.io/mdBook/).
+We use some extensions from [GitHub Flavoured Markdown](https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax) : specifically alert boxes with [mdbook-alerts](https://crates.io/crates/mdbook-alerts).
+
+To build the documentation run `just build-docs`. If you want to iterate on the documentation for local development, run `just serve-docs [<hostname>] [<port>]` or by going in the docs directory and running `mdbook serve [--hostname <hostname>] [--port <port>]` and a web server will be started, by default on http://localhost:3000 .
+
+The built doc files are stored under `docs/experimental-documentation/build`, while the markdown files are under `docs/experimental-documentation` and its child directories.

docs/experimental-documentation/Introduction/BuildSystem.md docs/book/src/building_and_packaging/build_systems.md

@@ -1,3 +1,5 @@
+## Environment variables
+
 CodeTracer exposes a number of environment variables that you can use to override some of its behaviours:
 
 1. `CODETRACER_ELECTRON_ARGS` - adds arguments for launching Electron. Useful for debugging production builds
@@ -16,4 +18,4 @@ These are generally not functional right now, since they affect CodeTracer Shell
 1. `CODETRACER_SHELL_RECORDS_OUTPUT` - ?
 1. `CODETRACER_SHELL_EXPORT` - ?
 1. `CODETRACER_SHELL_CLEANUP_OUTPUT_FOLDER` - ?
-1. `CODETRACER_SHELL_SOCKET` and `CODETRACER_SHELL_ADDRESS` - they override the socket location and address respectively
+1. `CODETRACER_SHELL_SOCKET` and `CODETRACER_SHELL_ADDRESS` - they override the socket location and address respectively

docs/experimental-documentation/mdbook-ct/src/chapter_1.md docs/book/src/introduction.md

@@ -1,19 +1,26 @@
+## Troubleshooting
+
 CodeTracer is currently in an experimental state, so we expect that there are many bugs that have not been found as of now.
 If you find any bug, please report it as an issue on GitHub.
 
 In the meantime, you can use this page to fix some issues that are somewhat common.
 
-## Fixing outdated configuration/layout files
-You can find more information [here](https://dev-docs.codetracer.com/Introduction/Configuration#Layout).
+### Fixing outdated configuration/layout files
+
+You can reset them by deleting your `~/.config/codetracer` folder currently.
+
+You will be able to find more information in the configuration docs eventually: work in progress.
+
+### Resetting the local trace database
 
-## Resetting the local trace database
 There are 2 commands that can be used to completely wipe all traces from your user's data:
 
 1. `just reset-db` - Resets the local user's trace database
 1. `just clear-local-traces` - Clears the local user's traces
 
-## Broken local build
+### Broken local build
+
 Sometimes your local build might break. In most cases, a simple `direnv reload` and `just build` should be able to fix it.
-Otherwise, you can ask for help on our [issues tracker](github.com/metacraft-labs/codetracer), or in our [chat server](https://discord.gg/aH5WTMnKHT)
+Otherwise, you can ask for help on our [issues tracker](https://github.com/metacraft-labs/codetracer), or in our [chat server](https://discord.gg/aH5WTMnKHT)
 
 <!-- TODO: Add more info here -->