[Feature] Enable pure python transforms in new spark runtime.

[daw3rd]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/Other

Feature

With the recent PR to the spark runtime, to allow checkpointing and pure-python transforms to run in spark, we should begin enabling transforms listed below to run in the spark runtime. Largely this means creating the boilerplate main() and associated Spark configuration classes. See the new filter transform for an example.

The transforms suggested this work are as follows:

Universal

• profiler
• resize
• tokenization

Code

• code2parquet
• code_quality
• header_cleanser
• malware
• proglang_select
• repo_level_ordering

Language

• doc_chunk
• doc_quality
• lang_id
• pdf2parquet
• pii_redactor
• text_encoder

Are you willing to submit a PR?

• Yes I am willing to submit a PR!