Refactor SHAP inference functionality

[bdewilde]

changes

• auto-formats / lints / tidies up inference src and tests modules
• refactors single-batch shap values calculation logic into a separate function, and renames the spark udf that applies that batch func to chunks of a larger dataset
• adds a unit test for the single-batch shap values func

context

Prior art: PR #48

Btw @vishpillai123 , I had started putting together an inference nb using code in this repo, but was waiting until I could chat with Daniel (Wednesday!) about how this is meant to work in practice, given confusion about the dbutils nb widgets and such.

questions

[vishpillai123]

Refactoring provides more clarity, though my only concern is whether it has the same SHAP values output when we test on a particular school (it should). Either way, we'll have the notebook eventually to provide test cases/sanity checks.

So, I realized db.utils is used in the context of Databricks jobs! It's quite elegant actually. I'm using the one Kelsey created and it's really nice once you get the privs sorted out.

[bdewilde]

So, I realized db.utils is used in the context of Databricks jobs! It's quite elegant actually. I'm using the one Kelsey created for Lee College and it's really nice once you get the privs sorted out.

I'm chatting with Daniel about this stuff in an hour, hopefully we'll both be clear on how the pipeline is meant to run in production by EoD. :)