Lighter is an opensource application for interacting with Apache Spark on Kubernetes or Apache Hadoop YARN. It is hevily inspired by Apache Livy and has some overlaping features.
Lighter supports:
- Interactive Python Sessions through Sparkmagic kernel
- Batch application submissions through the REST API
⚠️ If you are using interactive sessions: While we have tested batch applications quite extensively, there might be some problems with interactive sessions, consider current release of Lighter as alpha.
You can read a breaf description on how Lighter works here.
- Installation on Kubernetes
- Installation on Docker
- Configuration Properties
- Configuring Sparkmagic
- Using REST API
To build Docker image, containing application, you need to run
docker build -t lighter .
Lighter will always try to support the latest patch version for all officially recommended Apache Spark releases i.e.: 3.1.3, 3.2.2 and 3.3.0.
Lighter is MIT licensed.