A basic use case for how to leverage Kafka and Faust for streaming events from a FastAPI application.
This Project uses docker and docker compose to create a development environment.
Run the docker compose file with:
docker compose up -d
The idea for this project was to keep the use case simple and focus on how these technologies can work together
to create a real-time data streaming application.
We are creating a Users collection in Arangodb to store user information.
The REST API (a.k.a Inarious) only has a GET and POST REST endpoint for the Users resource.
Anytime a client uses an endpoint an activity event message is created, serialized with protobuf
and send to the Users Kafka topic.
Then the Faust worker service will be consuming the Users topic
and deserializing the incoming messages using protobuf.
Once the Faust worker has received the message it can proceed to perform a task or event based on that message.
For example in this case it will be receiving the below structured data:
message BackendApiActivity {
string client_host = 1;
string client_port = 2;
string endpoint = 3;
string http_method = 4;
}
This data could then be aggregated to track the users api usage and populate a dashboard graph.
These graphs could then be used to see in real time what resources the user is using the most.
This was a fun mini-project to help me learn more about Kafka, pub-sub systems and protobuf.
I hope this can be helpful to someone else as well, and I have linked to some resources I needed to use below
to build this application.
Need to install the protoc compiler first.
I am using an Ubuntu 20.04 OS, so you may need to adjust the below command accordingly,
see Protobuf Releases
to find the release compatible with your OS and replace the PROTOC_ZIP and PROTOC_VERSION variables in the below
command:
PROTOC_ZIP=protoc-23.4-linux-x86_64.zip
PROTOC_VERSION=v23.4
curl -OL https://github.com/protocolbuffers/protobuf/releases/download/$PROTOC_VERSION/$PROTOC_ZIP
sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc
sudo unzip -o $PROTOC_ZIP -d /usr/local 'include/*'
rm -f $PROTOC_ZIP
To generate usable integration code, we use the proto compiler which compiles a given
.proto file into language-specific integration classes.
For Python we can use the below command:
protoc -I=. --python_out=. ./example.proto
I will continue to update this repo and use it to expand the current simple use case with:
- CI
- dashboard interface for the real-time data stream.
- kafka schema registry.
- more friendly development setup environment (i.e. mounting the fast api project folder).
- expand more into Arangodb's multi-model system and build a basic graph database.