Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native Julia engine based on QuartoNotebookRunner.jl #8645

Merged
merged 107 commits into from
Apr 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
107 commits
Select commit Hold shift + click to select a range
d0fc6c5
add basic scaffolding to use QuartoNotebookRunner for running Julia q…
jkrumbiegel Jan 23, 2024
b8d8771
change default content to julia
jkrumbiegel Jan 23, 2024
1c78e04
add note
jkrumbiegel Jan 23, 2024
ca10f51
simply julia code
jkrumbiegel Jan 23, 2024
1bb0f14
first working version with transport file
jkrumbiegel Jan 24, 2024
8137ce8
fix julia server bug by using Deno.Command
jkrumbiegel Jan 24, 2024
7ca08c2
add oneShot option to exec options
jkrumbiegel Jan 24, 2024
9d9e60c
replace console.log with trace
jkrumbiegel Jan 24, 2024
93b2475
cleanups
jkrumbiegel Jan 25, 2024
7974a36
restart daemon if requested
jkrumbiegel Jan 25, 2024
ba02f1c
move sleeps
jkrumbiegel Jan 25, 2024
12b8a2b
use custom env for server
jkrumbiegel Jan 25, 2024
558fe37
add notice when server starts
jkrumbiegel Jan 25, 2024
c79e5b2
also check that julia version in manifest matches
jkrumbiegel Jan 25, 2024
9a544c1
allow override of julia binary via QUARTO_JULIA env var
jkrumbiegel Feb 1, 2024
11347f1
bump quartonotebookrunner
jkrumbiegel Feb 5, 2024
503230f
send options along with "run" command
jkrumbiegel Feb 5, 2024
3dd6af9
Merge branch 'main' into jk/quartonotebookrunner
jkrumbiegel Feb 26, 2024
2153023
register julia engine again
jkrumbiegel Feb 26, 2024
c2c2012
reject if notebook exec returns undefined
jkrumbiegel Feb 26, 2024
c8ac7f9
bump QuartoNotebookRunner
jkrumbiegel Feb 26, 2024
4de287c
remove commented out code
jkrumbiegel Feb 27, 2024
9f11e35
bump QuartoNotebookRunner to 0.3.2
jkrumbiegel Feb 27, 2024
9f8cb5b
let connection timeout if `isready` isn't returned quickly enough
jkrumbiegel Feb 27, 2024
8bb791b
formatting
jkrumbiegel Feb 27, 2024
2e18503
don't claim julia automatically
jkrumbiegel Feb 27, 2024
dcc21b3
add julia engine test
jkrumbiegel Feb 27, 2024
61c0c1b
timeout after 3000 ms
jkrumbiegel Feb 27, 2024
af00717
take out `info` as tests expect no output
jkrumbiegel Feb 27, 2024
32509fd
use jupyter file for old test
jkrumbiegel Feb 27, 2024
3adf782
don't always trace timeout message
jkrumbiegel Feb 27, 2024
762c22c
re-add native julia commonmark smoke test
jkrumbiegel Feb 28, 2024
e96a341
add native julia crossref test
jkrumbiegel Feb 28, 2024
906d6c1
rethrow error if server connection fails to establish
jkrumbiegel Feb 28, 2024
468583b
try if 30 seconds timeout removes the timeout errors
jkrumbiegel Feb 28, 2024
654633d
try if specifying `localhost` makes a connection difference
jkrumbiegel Feb 28, 2024
334c15a
remove localhost again
jkrumbiegel Feb 29, 2024
f34d2c8
change timeout back to 3 seconds
jkrumbiegel Feb 29, 2024
be684ed
clarify command creation code and add console.log for CI check
jkrumbiegel Feb 29, 2024
171e688
remove logging, not visible in CI
jkrumbiegel Feb 29, 2024
f2c23f2
make server process detached through julia indirection
jkrumbiegel Feb 29, 2024
761fc8a
look at `lsof` output in CI
jkrumbiegel Mar 1, 2024
e38b03a
add debug log level
jkrumbiegel Mar 4, 2024
f77bdbb
run only linux
jkrumbiegel Mar 4, 2024
75e5f3b
try tmate session for debugging
jkrumbiegel Mar 4, 2024
9ebba87
move session before failure
jkrumbiegel Mar 4, 2024
60c537f
increase timeout to 10 seconds
jkrumbiegel Mar 4, 2024
e280ee8
try warming up the julia native server before tests run
jkrumbiegel Mar 4, 2024
8c4c027
continue on error
jkrumbiegel Mar 4, 2024
3ef741e
remove `lsof` used for debugging
jkrumbiegel Mar 4, 2024
5051783
format timeout error better
jkrumbiegel Mar 4, 2024
0ca91d2
bump quartonotebookrunner version to 0.4
jkrumbiegel Mar 4, 2024
a7319af
add julianative subfig test
jkrumbiegel Mar 4, 2024
8c22e07
rename files consistently
jkrumbiegel Mar 4, 2024
4efeb61
take out warmup again
jkrumbiegel Mar 4, 2024
e822d46
add test connection before writing transport file
jkrumbiegel Mar 4, 2024
1419e87
put windows back
jkrumbiegel Mar 4, 2024
ebadf3c
add tmate action and only test windows
jkrumbiegel Mar 5, 2024
b4f803b
use command.spawn and log transport file accesses
jkrumbiegel Mar 5, 2024
a3e190a
remove tmate session
jkrumbiegel Mar 5, 2024
20957d4
call exit manually to see if that truly stops the spawned process
jkrumbiegel Mar 5, 2024
d0e3ea1
try starting the detached julia the way that detached python is start…
jkrumbiegel Mar 5, 2024
ac3d2f2
import re
jkrumbiegel Mar 5, 2024
0bdea7e
delete unnecessary lines
jkrumbiegel Mar 5, 2024
159b7e7
delete accidental double Popen
jkrumbiegel Mar 5, 2024
b956433
try detached julia with devnull'ed file descriptors
jkrumbiegel Mar 5, 2024
ee65d59
use `uv_unref` on the process handle
jkrumbiegel Mar 5, 2024
c458113
remove uv_unref again, didn't work
jkrumbiegel Mar 6, 2024
3e8a2eb
try spawning julia server through powershell on windows
jkrumbiegel Mar 6, 2024
8c22000
fix window style
jkrumbiegel Mar 6, 2024
729a572
add explanatory comment
jkrumbiegel Mar 6, 2024
6e3abe1
remove printouts
jkrumbiegel Mar 6, 2024
cdfbef8
use outputSync to wait until spawning the detached process has succeeded
jkrumbiegel Mar 6, 2024
1df135f
put ubuntu runners back in
jkrumbiegel Mar 6, 2024
5c2ce5a
Merge branch 'main' into jk/quartonotebookrunner
jkrumbiegel Mar 6, 2024
c43c136
add markdownForFile method
jkrumbiegel Mar 6, 2024
dc72ecd
remove unused python script
jkrumbiegel Mar 6, 2024
caa7250
actually make julianative test use julia engine
jkrumbiegel Mar 6, 2024
c81c624
bump quartonotebookrunner
jkrumbiegel Mar 7, 2024
268d389
startup file no
jkrumbiegel Mar 7, 2024
1279112
use existing project for faster CI
jkrumbiegel Mar 7, 2024
1f1b46f
comment out most of CI, reenable windows tmate
jkrumbiegel Mar 7, 2024
fbbb9f8
comment out more
jkrumbiegel Mar 7, 2024
6cbbe4e
fix bug and actually throw if spawning fails
jkrumbiegel Mar 7, 2024
5d92d70
restore CI components
jkrumbiegel Mar 7, 2024
2457ec0
one more startup file no
jkrumbiegel Mar 7, 2024
f6a9657
print useful julia internal server errors
jkrumbiegel Mar 7, 2024
9a90e7e
move to quartonotebookrunner 0.5
jkrumbiegel Mar 8, 2024
333ed00
give visible notice about starting the julia server
jkrumbiegel Mar 8, 2024
cb166df
fix patch version
jkrumbiegel Mar 8, 2024
6699a91
implement hmac
jkrumbiegel Mar 14, 2024
4ec4c70
bump to quartonotebookrunner 0.7
jkrumbiegel Mar 14, 2024
1c70fe6
simplify init script, server should already be `listen`ing
jkrumbiegel Mar 14, 2024
d5b10a9
improve messages and show server init process output
jkrumbiegel Mar 14, 2024
5776be8
time control process out after 5 minutes of inactivity
jkrumbiegel Mar 14, 2024
c51ffc6
fix error printing
jkrumbiegel Mar 14, 2024
cf0e0e4
Merge branch 'main' into jk/quartonotebookrunner
jkrumbiegel Mar 14, 2024
f4f0d8a
QuartoNotebookRunner 0.7.1
jkrumbiegel Mar 15, 2024
31c8b9e
remove windows special behavior by using `spawn` and `unref`
jkrumbiegel Mar 15, 2024
61bc4b4
bump QuartoNotebookRunner
jkrumbiegel Mar 21, 2024
f71a88b
Revert "remove windows special behavior by using `spawn` and `unref`"
jkrumbiegel Mar 25, 2024
ccae95b
Merge branch 'main' into jk/quartonotebookrunner
jkrumbiegel Mar 25, 2024
33e6ad9
QuartoNotebookRunner 0.8.1
jkrumbiegel Mar 25, 2024
67fb806
Merge branch 'main' into jk/quartonotebookrunner
jkrumbiegel Mar 26, 2024
cfd6f20
explicitly `precompile` to surface potential errors in the console
jkrumbiegel Mar 26, 2024
192f0f4
bump to QNR 0.9 for render progress updates
jkrumbiegel Apr 2, 2024
082f7fe
Merge branch 'main' into jk/quartonotebookrunner
jkrumbiegel Apr 2, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/test-smokes.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ jobs:
run: |
if (!requireNamespace('renv', quietly = TRUE)) install.packages('renv')
renv::restore()
# Install dev versions for our testing
# Install dev versions for our testing
# Use r-universe to avoid github api calls
try(install.packages('knitr', repos = 'https://yihui.r-universe.dev'))
try(install.packages('rmarkdown', repos = 'https://rstudio.r-universe.dev'))
Expand Down Expand Up @@ -164,7 +164,7 @@ jobs:
run: |
# Setup IJulia with the jupyter from the Python environment
# https://julialang.github.io/IJulia.jl/stable/manual/installation/
export JUPYTER=$(find $(dirname $(pipenv run which jupyter))/ -type f -name "jupyter.exe" -o -name "jupyter")
export JUPYTER=$(find $(dirname $(pipenv run which jupyter))/ -type f -name "jupyter.exe" -o -name "jupyter")
pipenv run julia --color=yes --project=. -e "import Pkg; Pkg.instantiate(); Pkg.build(\"IJulia\"); Pkg.precompile()"
echo "Julia Jupyter:"
julia --project=. -e "import IJulia;println(IJulia.JUPYTER);println(IJulia.find_jupyter_subcommand(\"notebook\"))"
Expand All @@ -174,6 +174,9 @@ jobs:
run: |
echo "QUARTO_TEST_TIMING=timing-for-ci.txt" >> "$GITHUB_ENV"

# - name: Setup tmate session
# uses: mxschmitt/action-tmate@v3

- name: Run all Smoke Tests Windows
if: ${{ runner.os == 'Windows' && format('{0}', inputs.buckets) == '' && matrix.time-test == false }}
env:
Expand All @@ -197,6 +200,7 @@ jobs:
env:
# Useful as TinyTeX latest release is checked in run-test.sh
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
QUARTO_LOG_LEVEL: DEBUG
run: |
haserror=0
readarray -t my_array < <(echo '${{ inputs.buckets }}' | jq -rc '.[]')
Expand Down
5 changes: 4 additions & 1 deletion src/execute/engine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import { mergeConfigs } from "../core/config.ts";
import { ProjectContext } from "../project/types.ts";
import { pandocBuiltInFormats } from "../core/pandoc/pandoc-formats.ts";
import { gitignoreEntries } from "../project/project-gitignore.ts";
import { juliaEngine } from "./julia.ts";

const kEngines: Map<string, ExecutionEngine> = new Map();

Expand All @@ -40,7 +41,9 @@ export function executionEngine(name: string) {
return kEngines.get(name);
}

for (const engine of [knitrEngine, jupyterEngine, markdownEngine]) {
for (
const engine of [knitrEngine, jupyterEngine, markdownEngine, juliaEngine]
) {
registerExecutionEngine(engine);
}

Expand Down
Loading
Loading