Skip to content

Commit

Permalink
update README
Browse files Browse the repository at this point in the history
  • Loading branch information
@shikokuchuo
shikokuchuo committed Sep 25, 2023
1 parent a9a3767 commit 7640107
Show file tree
Hide file tree
Showing 2 changed files with 59 additions and 146 deletions.
72 changes: 18 additions & 54 deletions README.Rmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,10 +47,9 @@ Features efficient task scheduling, fast inter-process communications, and Trans
7. [Distributed Computing: TLS Secure Connections](#distributed-computing-tls-secure-connections)
8. [Compute Profiles](#compute-profiles)
9. [Errors, Interrupts and Timeouts](#errors-interrupts-and-timeouts)
10. [Deferred Evaluation Pipe](#deferred-evaluation-pipe)
11. [Integrations with Crew, Targets, Shiny](#integrations-with-crew-targets-shiny)
12. [Thanks](#thanks)
13. [Links](#links)
10. [Integrations with Crew, Targets, Shiny](#integrations-with-crew-targets-shiny)
11. [Thanks](#thanks)
12. [Links](#links)

### Installation

Expand Down Expand Up @@ -187,7 +186,7 @@ library(mirai)
run_iteration <- function(i) {
if (runif(1) < 0.1) stop("random error", call. = FALSE) # simulates a stochastic error rate
if (runif(1) < 0.1) stop("random error\n", call. = FALSE) # simulates a stochastic error rate
sprintf("iteration %d successful\n", i)
}
Expand Down Expand Up @@ -280,7 +279,7 @@ The daemons interface may also be used to send tasks for computation to remote d

Call `daemons()` specifying 'url' as a character string the host network address and a port that is able to accept incoming connections.

The examples below use an illustrative local network IP address of '10.75.48.93'.
The examples below use an illustrative local network IP address of '10.75.37.40'.

A port on the host machine also needs to be open and available for inbound connections from the local network, illustratively '5555' in the examples below.

Expand All @@ -293,15 +292,15 @@ The default `dispatcher = TRUE` creates a background `dispatcher()` process on t
It is recommended to use a websocket URL starting `ws://` instead of TCP in this scenario (used interchangeably with `tcp://`). A websocket URL supports a path after the port number, which can be made unique for each daemon. In this way a dispatcher can connect to an arbitrary number of daemons over a single port.

```{r localqueue}
daemons(n = 4, url = "ws://10.75.48.93:5555")
daemons(n = 4, url = "ws://10.75.37.40:5555")
```

Above, a single URL was supplied, along with `n = 4` to specify that the dispatcher should listen at 4 URLs. In such a case, an integer sequence is automatically appended to the path `/1` through `/4` to produce these URLs.

Alternatively, supplying a vector of URLs allows the use of arbitrary port numbers / paths, e.g.:

```{r vectorqueue, eval=FALSE}
daemons(url = c("ws://10.75.48.93:5566/cpu", "ws://10.75.48.93:5566/gpu", "ws://10.75.48.93:7788/1"))
daemons(url = c("ws://10.75.37.40:5566/cpu", "ws://10.75.37.40:5566/gpu", "ws://10.75.37.40:7788/1"))
```

Above, 'n' is not specified, in which case its value is inferred from the length of the 'url' vector supplied.
Expand All @@ -311,19 +310,19 @@ Above, 'n' is not specified, in which case its value is inferred from the length
On the remote resource, `daemon()` may be called from an R session, or directly from a shell using Rscript. Each daemon instance should dial into one of the unique URLs that the dispatcher is listening at:

```
Rscript -e 'mirai::daemon("ws://10.75.48.93:5555/1")'
Rscript -e 'mirai::daemon("ws://10.75.48.93:5555/2")'
Rscript -e 'mirai::daemon("ws://10.75.48.93:5555/3")'
Rscript -e 'mirai::daemon("ws://10.75.48.93:5555/4")'
Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/1")'
Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/2")'
Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/3")'
Rscript -e 'mirai::daemon("ws://10.75.37.40:5555/4")'
```

Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available.

`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.48.100 accepts SSH connections over port 22:
`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.37.100 accepts SSH connections over port 22:

```{r launchremote, eval=FALSE}
launch_remote(1:4, command = "ssh", args = c("-p 22 10.75.48.100", .))
launch_remote(1:4, command = "ssh", args = c("-p 22 10.75.37.100", .))
```
```{r launchremotereal, echo=FALSE}
launch_remote(1:4)
Expand Down Expand Up @@ -362,7 +361,7 @@ Closing the connection causes the dispatcher to exit automatically, and in turn
By specifying `dispatcher = FALSE`, remote daemons connect directly to the host process. The host listens at a single URL, and distributes tasks to all connected daemons.

```{r remote, eval=FALSE}
daemons(url = "tcp://10.75.48.93:0", dispatcher = FALSE)
daemons(url = "tcp://10.75.37.40:0", dispatcher = FALSE)
```

Alternatively, simply supply a colon followed by the port number to listen on all interfaces on the local host, for example:
Expand All @@ -378,17 +377,17 @@ Note that above, the port number is specified as zero. This is a wildcard value
On the network resource, `daemon()` may be called from an R session, or an Rscript invocation from a shell. This sets up a remote daemon process that connects to the host URL and receives tasks:

```
Rscript -e 'mirai::daemon("tcp://10.75.48.93:0")'
Rscript -e 'mirai::daemon("tcp://10.75.37.40:0")'
```
Note that `daemons()` should be set up on the host machine before launching `daemon()` on remote resources, otherwise the daemon instances will exit if a connection is not immediately available. Alternatively, specifying `daemon(asyncdial = TRUE)` will allow daemons to wait (indefinitely) for a connection to become available.

`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.48.100 accepts SSH connections over port 22:
`launch_remote()` may also be used to launch daemons directly on a remote machine. For example, if the remote machine at 10.75.37.100 accepts SSH connections over port 22:

```{r launchremoteu, eval=FALSE}
launch_remote("tcp://10.75.48.93:0", command = "ssh", args = c("-p 22 10.75.48.100", .))
launch_remote("tcp://10.75.37.40:0", command = "ssh", args = c("-p 22 10.75.37.100", .))
```
```{r launchremoteureal, echo=FALSE}
launch_remote("tcp://10.75.48.93:0")
launch_remote("tcp://10.75.37.40:0")
```

The returned vector comprises the shell commands executed on the remote machine.
Expand Down Expand Up @@ -518,41 +517,6 @@ is_error_value(m3$data)

[&laquo; Back to ToC](#table-of-contents)

### Deferred Evaluation Pipe

`mirai` implements a deferred evaluation pipe `%>>%` for working with potentially unresolved values.

Pipe a 'mirai' or mirai `$data` value forward into a function or series of functions and initially an 'unresolvedExpr' will be returned.

The result may be queried at `$data`, which will return another 'unresolvedExpr' whilst unresolved. However when the original value resolves, the 'unresolvedExpr' will simultaneously resolve into a 'resolvedExpr', for which the evaluated result will then be available at `$data`.

A piped expression should be wrapped in `.()` to ensure that the return value is always an 'unresolvedExpr' or 'resolvedExpr' as the case may be.

It is possible to use `unresolved()` around an expression, or its `$data` element, to test for resolution, as in the example below.

The pipe operator semantics are similar to R's base pipe `|>`:

`x %>>% f` is equivalent to `f(x)` <br />
`x %>>% f()` is equivalent to `f(x)` <br />
`x %>>% f(y)` is equivalent to `f(x, y)`

```{r pipe}
m <- mirai({nanonext::msleep(500); 1})
b <- .(m %>>% c(2, 3) %>>% as.character)
unresolved(b)
b
b$data
nanonext::msleep(1000)
unresolved(b)
b
b$data
```

[&laquo; Back to ToC](#table-of-contents)

### Integrations with Crew, Targets, Shiny

The [`crew`](https://wlandau.github.io/crew/) package is a distributed worker-launcher that provides an R6-based interface extending `mirai` to different distributed computing platforms, from traditional clusters to cloud services. The [`crew.cluster`](https://wlandau.github.io/crew.cluster/) package is a plug-in that enables mirai-based workflows on traditional high-performance computing clusters using LFS, PBS/TORQUE, SGE and SLURM.
Expand Down
Loading

0 comments on commit 7640107

Please sign in to comment.