clojurebridge swallos exceptions

[behrica]

at least when using like this:

python -c 'from clojurebridge import cljbridge;cljbridge.load_clojure_file(clj_file="train.clj",mvn_local_repo="/home/carsten/.m2/repository")'

[behrica]

This is quire sever, I would say.
Any type of exception in a clj file run via:

python -c 'from clojurebridge import cljbridge;cljbridge.load_clojure_file(clj_file="file.clj")'

is not visible to a user. Additionaly the process hangs, it seems to me.

[behrica]

Not true for all exception. But this for example:

Execution error at libpython-clj2.python.ffi/check-error-throw (ffi.clj:708).
Traceback (most recent call last):
  File "/home/carsten/.conda/envs/libpython-clj--194/lib/python3.9/site-packages/simpletransformers/classification/classification_model.py", line 361, in __init__
    raise ValueError(
ValueError: 'use_cuda' set to True when cuda is unavailable. Make sure CUDA is available or set use_cuda=False.

[behrica]

To reproduce :

crash.clj

(ns train
  (:require [libpython-clj2.python :refer [py.- py.] :as py]
            [libpython-clj2.python.ffi :as ffi]))
            ;; [libpython-clj2.require :refer [require-python]]
            ;; [libpython-clj2.python.gc]
(println "-------- manual-gil: " ffi/manual-gil)

(def locked (ffi/lock-gil))
(println :gil-locked)

(println :before-crash)
(py/import-module "not existing")
(println "crash fininished")

Run as embedded:

python -c 'from clojurebridge import cljbridge;cljbridge.load_clojure_file(clj_file="src/crash.clj",mvn_local_repo="/home/carsten/.m2/repository")'

[cnuernber]

I think we might be speaking past each other. I did add an api - clj-python/libpython-clj#221 (comment)

(py/with-manual-gil
  (println :gil-locked)
  (println :before-crash)
  (py/import-module "not existing")
  (println "crash fininished"))

• This is the try-with-resources pattern.

[behrica]

yes, but this does not prevent the hanging, if somebody does not use teh API.

[behrica]

For me the "hanging" is a very frustrating experience for a beginner with libpython-clj, so I think we should prevent it, if we can. (or document it more prominently at least)

[behrica]

The libpython-clj API has now three different with-*** type of calls.
Maybe we could summarize in one place when they should be used, and what they could protect against.

Because on "first sight" none of it is needed, and the code seems to work anyway.

[cnuernber]

Totally agreed that documentation could be updated, changed, or added. I honestly think usage.md is a bit out of date - when I get tmd updated to latest dtype (nontrivial full version change) I will circle back and spend some time on libpython-clj.

[cnuernber]

I also would like to know what exactly is crashing honestly. Some interaction with check-gil and multiple interpreters is my best guess but I have no solid proof. If you look at the code to checkgil in libpython it does juggle some thread-specific variables.

Honestly though that type of work is hard and detailed enough that I would want to be paid for it. Auditing C code and debugging that type of thing really just eats time like crazy.

[cnuernber]

In this case the crashing is just something is trying to lock the gil when something else has it so in this case the crash is easy but the cause for for manual gil management in the first place, aside from perf, is that it reduces the crashiness of the system by quite a lot. It is that, the original crashiness, that I would still like to know more about.