examples: add chunker job to support configurable chunking

[yonitoo]

What: Extend the pgvector-embedder by adding configurable chunking mechanism.

Why: Until now, the whole documents were embedded and ingested into the database but their size sometimes exceed the token limit imposed by the LLM used for inference. This change introduces a configurable document chunking mechanism to overcome this problem.

Testing Done: Ran the pipeline jobs locally.

Closes #3084

Signed-off by: Yoan Salambashev [email protected]

[duyguHsnHsn]

Looks amazing! Good work

[antoniivanov]

How did you test it?

Looks ok. I just want to make sure it's tested properly before approving. All my other comments are minor things about documentation or adding comments.

[antoniivanov]

Let's merge it in whatever stable state it could be.

But a few things to note:

The current fixed splitting is splitting the text in the middle of a word/sentence which I am not sure how good it would affect the RAG app. We should test end to end.

It would be really good idea to add the metadata to each chunk (when creating embeddings ).

We should see to change the ID - as you can have multiple chunks per document so the ID should be composite

The default CHUNK_SIZE should be probably like 2048 and default overlap 64. As most LLMs have token limit like 4000.