Add crate documentation (#106)

- Separate out public, unstable and internal APIs.
- Cleanup README.md and include it as the crate documentation.

Signed-off-by: Anand Krishnamoorthi <[email protected]>

README.md

@@ -1,135 +1,184 @@
 # Regorus
 
-**Regorus** is 
+**Regorus** is
 
   - *Rego*-*Rus(t)*  - A fast, light-weight [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/) interpreter written in Rust.
   - *Rigorous* - A rigorous enforcer of well-defined Rego semantics.
 
 Regorus is available as a library that can be easily integrated into your Rust projects.
 
+```rust
+use anyhow::Result;
+use regorus::*;
+use serde_json;
+
+fn main() -> Result<()> {
+  // Create an engine for evaluating Rego policies.
+  let mut engine = Engine::new();
+
+  // Add policy to the engine.
+  engine.add_policy(
+    // Filename to be associated with the policy.
+    "hello.rego".to_string(),
+
+	// Rego policy that just sets a message.
+	r#"
+	   package test
+	   message = "Hello, World!"
+	"#.to_string()
+  )?;
+
+  // Evaluate the policy, fetch the message and print it.
+  let results = engine.eval_query("data.test.message".to_string(), false)?;
+  println!("{}", serde_json::to_string_pretty(&results)?);
+
+  Ok(())
+}
+```
 
 Regorus passes the [OPA v0.60.0 test-suite](https://www.openpolicyagent.org/docs/latest/ir/#test-suite) barring a few builtins.
-See [OPA Conformance][#opa-conformance] below.
+See [OPA Conformance](#opa-conformance) below.
 
 ## Getting Started
 
-[examples/regorus](examples/regorus.rs) is an example program that shows how to integrate Regorus into your project and evaluate Rego policies.
+[examples/regorus](https://github.com/microsoft/regorus/blob/main/examples/regorus.rs) is an example program that shows how to integrate Regorus into your project and evaluate Rego policies.
 
 To build and install it, do
 
-    cargo install --example regorus --path .
-
+```bash
+$ cargo install --example regorus --path .
+```
 
 Check that the regorus example program is working
 
-    $ regorus
-    Usage: regorus <COMMAND>
-
-    Commands:
-      eval   Evaluate a Rego Query
-      lex    Tokenize a Rego policy
-      parse  Parse a Rego policy
-      help   Print this message or the help of the given subcommand(s)
-
-    Options:
-      -h, --help     Print help
-      -V, --version  Print versionUsage: regorus <COMMAND>
+```bash
+$ regorus
+Usage: regorus <COMMAND>
 
+Commands:
+  eval   Evaluate a Rego Query
+  lex    Tokenize a Rego policy
+  parse  Parse a Rego policy
+  help   Print this message or the help of the given subcommand(s)
+
+Options:
+  -h, --help     Print help
+  -V, --version  Print version
+```
 
 
 First, let's evaluate a simple Rego expression `1*2+3`
 
-    regorus eval "1*2+3"
+```bash
+$ regorus eval "1*2+3"
+```
 
 This produces the following output
 
+```json
+{
+  "result": [
     {
-      "result": [
+      "expressions": [
         {
-          "expressions": [
-            {
-               "value": 5,
-               "text": "1*2+3",
-               "location": {
-                 "row": 1,
-                 "col": 1
-              }
+           "value": 5,
+           "text": "1*2+3",
+           "location": {
+              "row": 1,
+              "col": 1
             }
-          ]
         }
       ]
     }
+  ]
+}
+```
 
 Next, evaluate a sample [policy](examples/example.rego) and [input](examples/input.json) (borrowed from [Rego tutorial](https://www.openpolicyagent.org/docs/latest/#2-try-opa-eval)):
 
-    regorus eval -d examples/example.rego -i examples/input.json data.example
+```bash
+$ regorus eval -d examples/example.rego -i examples/input.json data.example
+```
 
 Finally, evaluate real-world [policies](tests/aci/) used in Azure Container Instances (ACI)
 
-    regorus eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.policy.mount_overlay=x
+```bash
+$ regorus eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.policy.mount_overlay=x
+```
 
 
 ## ACI Policies
 
 Regorus successfully passes the ACI policy test-suite. It is fast and can run each of the tests in a few milliseconds.
 
-    $ cargo test -r --test aci
-        Finished release [optimized + debuginfo] target(s) in 0.05s
-         Running tests/aci/main.rs (target/release/deps/aci-2cd8d21a893a2450)
-    aci/mount_device                                  passed    3.863292ms
-    aci/mount_overlay                                 passed    3.6905ms
-    aci/scratch_mount                                 passed    3.643041ms
-    aci/create_container                              passed    5.046333ms
-    aci/shutdown_container                            passed    3.632ms
-    aci/scratch_unmount                               passed    3.631333ms
-    aci/unmount_overlay                               passed    3.609916ms
-    aci/unmount_device                                passed    3.626875ms
-    aci/load_fragment                                 passed    4.045167ms
+```bash
+$ cargo test -r --test aci
+    Finished release [optimized + debuginfo] target(s) in 0.05s
+    Running tests/aci/main.rs (target/release/deps/aci-2cd8d21a893a2450)
+aci/mount_device                                  passed    3.863292ms
+aci/mount_overlay                                 passed    3.6905ms
+aci/scratch_mount                                 passed    3.643041ms
+aci/create_container                              passed    5.046333ms
+aci/shutdown_container                            passed    3.632ms
+aci/scratch_unmount                               passed    3.631333ms
+aci/unmount_overlay                               passed    3.609916ms
+aci/unmount_device                                passed    3.626875ms
+aci/load_fragment                                 passed    4.045167ms
+```
 
 Run the ACI policies in the `tests/aci` directory, using data `tests/aci/data.json` and input `tests/aci/input.json`:
 
-    regorus eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.policy.mount_overlay=x
-
+```bash
+$ regorus eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.policy.mount_overlay=x
+```
 
 Verify that [OPA](https://github.com/open-policy-agent/opa/releases) produces the same output
 
-    diff <(regorus eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x) \
-         <(opa eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x)
+```bash
+$ diff <(regorus eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x) \
+       <(opa eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x)
+```
+
 
 ## Performance
 
 To check how fast Regorus runs on your system, first install a tool like [hyperfine](https://github.com/sharkdp/hyperfine).
 
-    cargo install hyperfine
+```bash
+$ cargo install hyperfine
+```
 
 Then benchmark evaluation of the ACI policies,
 
-    $ hyperfine "regorus eval -b tests/aci -d tests/aci/data.json -i   tests/aci/input.json data.framework.mount_overlay=x"
-    Benchmark 1: regorus eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x
-      Time (mean ± σ):       4.6 ms ±   0.2 ms    [User: 4.1 ms, System: 0.4 ms]
-      Range (min … max):     4.4 ms …   6.0 ms    422 runs
-
-Compare it with OPA
+```bash
+$ hyperfine "regorus eval -b tests/aci -d tests/aci/data.json -i   tests/aci/input.json data.framework.mount_overlay=x"
+Benchmark 1: regorus eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x
+  Time (mean ± σ):       4.6 ms ±   0.2 ms    [User: 4.1 ms, System: 0.4 ms]
+  Range (min … max):     4.4 ms …   6.0 ms    422 runs
+```
 
-    $  hyperfine "opa eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x"
-    Benchmark 1: opa eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x
-      Time (mean ± σ):      45.2 ms ±   0.6 ms    [User: 68.8 ms, System: 5.1 ms]
-      Range (min … max):    43.8 ms …  46.7 ms    62 runs
+Compare it with OPA
 
+```bash
+$ hyperfine "opa eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x"
+Benchmark 1: opa eval -b tests/aci -d tests/aci/data.json -i tests/aci/input.json data.framework.mount_overlay=x
+  Time (mean ± σ):      45.2 ms ±   0.6 ms    [User: 68.8 ms, System: 5.1 ms]
+  Range (min … max):    43.8 ms …  46.7 ms    62 runs
 
+```
 ## OPA Conformance
 
-Regorus has been verified to be compliant with [OPA v0.60.0](https://github.com/open-policy-agent/opa/releases/tag/v0.60.0) 
-using a [test driver](tests/opa.rs) that loads and runs the OPA testsuite using Regorus, and verifies that expected outputs
+Regorus has been verified to be compliant with [OPA v0.60.0](https://github.com/open-policy-agent/opa/releases/tag/v0.60.0)
+using a [test driver](https://github.com/microsoft/regorus/blob/main/tests/opa.rs) that loads and runs the OPA testsuite using Regorus, and verifies that expected outputs
 are produced.
 
 The test driver can be invoked by running:
 
-```
-cargo test -r --test opa
+```bash
+$ cargo test -r --test opa
 ```
 
-Currently, Regorus passes all the non-builtin specific tests. See [passing tests suites](tests/opa.passing).
+Currently, Regorus passes all the non-builtin specific tests. See [passing tests suites](https://github.com/microsoft/regorus/blob/main/tests/opa.passing).
 
 The following test suites don't pass fully due to mising builtins:
 - `cryptoparsersaprivatekeys`
@@ -167,15 +216,16 @@ The following test suites don't pass fully due to mising builtins:
 
 They are captured in the following [github issues](https://github.com/microsoft/regorus/issues?q=is%3Aopen+is%3Aissue+label%3Alib).
 
+
 ### Grammar
 
-The grammar used by Regorus to parse Rego policies is described in [grammar.md](docs/grammar.md) in both EBNF and RailRoad Diagram formats.
+The grammar used by Regorus to parse Rego policies is described in [grammar.md](https://github.com/microsoft/regorus/blob/main/docs/grammar.md) in both [W3C EBNF](https://www.w3.org/Notation.html) and [RailRoad Diagram](https://en.wikipedia.org/wiki/Syntax_diagram) formats.
 
 ## Contributing
 
 This project welcomes contributions and suggestions.  Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
-the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
+the rights to use your contribution. For details, visit <https://cla.opensource.microsoft.com>.
 
 When you submit a pull request, a CLA bot will automatically determine whether you need to provide
 a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
@@ -187,8 +237,8 @@ contact [[email protected]](mailto:[email protected]) with any additio
 
 ## Trademarks
 
-This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 
-trademarks or logos is subject to and must follow 
+This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
+trademarks or logos is subject to and must follow
 [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
 Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
 Any use of third-party trademarks or logos are subject to those third-party's policies.

examples/regorus.rs

@@ -2,7 +2,6 @@
 // Licensed under the MIT License.
 
 use anyhow::{bail, Result};
-use clap::{Parser, Subcommand};
 
 fn rego_eval(
     bundles: &[String],
@@ -75,16 +74,18 @@ fn rego_eval(
 }
 
 fn rego_lex(file: String, verbose: bool) -> Result<()> {
+    use regorus::unstable::*;
+
     // Create source.
-    let source = regorus::Source::from_file(file)?;
+    let source = Source::from_file(file)?;
 
     // Create lexer.
-    let mut lexer = regorus::Lexer::new(&source);
+    let mut lexer = Lexer::new(&source);
 
     // Read tokens until EOF.
     loop {
         let token = lexer.next_token()?;
-        if token.0 == regorus::TokenKind::Eof {
+        if token.0 == TokenKind::Eof {
             break;
         }
 
@@ -100,18 +101,20 @@ fn rego_lex(file: String, verbose: bool) -> Result<()> {
 }
 
 fn rego_parse(file: String) -> Result<()> {
+    use regorus::unstable::*;
+
     // Create source.
-    let source = regorus::Source::from_file(file)?;
+    let source = Source::from_file(file)?;
 
     // Create a parser and parse the source.
-    let mut parser = regorus::Parser::new(&source)?;
+    let mut parser = Parser::new(&source)?;
     let ast = parser.parse()?;
     println!("{ast:#?}");
 
     Ok(())
 }
 
-#[derive(Subcommand)]
+#[derive(clap::Subcommand)]
 enum RegorusCommand {
     /// Evaluate a Rego Query.
     Eval {
@@ -164,6 +167,7 @@ struct Cli {
 }
 
 fn main() -> Result<()> {
+    use clap::Parser;
     env_logger::builder()
         .format_level(false)
         .format_timestamp(None)

src/interpreter.rs

@@ -226,10 +226,6 @@ impl Interpreter {
         self.modules = modules.to_vec();
     }
 
-    pub fn get_modules_mut(&mut self) -> &mut Vec<Ref<Module>> {
-        &mut self.modules
-    }
-
     pub fn set_init_data(&mut self, init_data: Value) {
         self.init_data = init_data;
     }

src/lib.rs

@@ -1,22 +1,30 @@
 // Copyright (c) Microsoft Corporation.
 // Licensed under the MIT License.
 
-pub mod ast;
+// Use README.md as crate documentation.
+#![doc = include_str!(concat!(env!("CARGO_MANIFEST_DIR"), "/README.md"))]
+
+mod ast;
 mod builtins;
-pub mod engine;
-pub mod interpreter;
-pub mod lexer;
-pub mod number;
-pub mod parser;
-pub mod scheduler;
+mod engine;
+mod interpreter;
+mod lexer;
+mod number;
+mod parser;
+mod scheduler;
 mod utils;
-pub mod value;
+mod value;
+
+pub use engine::Engine;
+pub use interpreter::{QueryResult, QueryResults};
+pub use value::Value;
+
+/// Items in `unstable` are likely to change.
+pub mod unstable {
+    pub use crate::ast::*;
+    pub use crate::lexer::*;
+    pub use crate::parser::*;
+}
 
-pub use ast::*;
-pub use engine::*;
-pub use interpreter::*;
-pub use lexer::*;
-pub use number::*;
-pub use parser::*;
-pub use scheduler::*;
-pub use value::*;
+#[cfg(test)]
+mod tests;