feat: [Experimental] New VLM Pipeline leveraging vision models

[maxmnemonic]

Preliminary integration with SmolDocling model and VLM Pipeline:

• SmolDocling inference model
• New VLM Pipeline that uses SmolDocling model
• Assembly code that builds Docling document from Doc-tags format predicted by SmolDocling
• Example of how to use
• Rudimentary speed measurement logging

Checklist:

• Documentation has been updated, if necessary.
• Examples have been added, if necessary.
• Tests have been added, if necessary.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[maxmnemonic]

merged all the proposed changes

[dolfim-ibm]

Initial comments submitted

[dolfim-ibm]

I'm summarizing here the target of this PR, I will submit code proposals later.

VlmPipeline

Specs of the new pipeline

• Input: (PDF) Document
• Processing: using a vision language model
• Output: DoclingDocument

Implementations

SmolDocling

Here the model will produce accurate DocTags which are converted (in the assemble step) to a DoclingDocument.

Other DocTags models

In the future we expect more models producing DocTags, which would go through the same assembling step of SmolDocling.

Other intermediate outputs

The pipeline will also support the case of VLMs producing a different intermediate representation. For example, models producing Markdown output, then we internally reuse the Markdown backend to create the DoclingDocument.

Wrap up

We definitely don't have to implement more than what it is nicely done in the PR, but a few naming (specially in the options) could be tuned for being ready for the next steps.

My suggestion is to use the vlm_options as the discriminator which, in the future, will decide things like 1) which model to call, 2) which type of internal assemble.

I would at least introduce from the beginning the kind in the options.

[dolfim-ibm]

I'm reading this as a wish for experimenting. What I'm really asking is if experimenting would not imply also using a different model?

Meaning, wouldn't it be better to

• either both question and repo_id
• or none of those?

[maxmnemonic]

The same model can be instructed differently to perform different tasks, default option is conversion to docling doc tags

[dolfim-ibm]

This is currently allowing a free question. Only a few, very specific prompts, will produce processable output by the pipeline.

Whenever we would consider making a nice wrapper for running interesting prompts with SmolDocling, I think it would deserve its own place in the docling-ibm-models, not in the model class for the processing pipeline.

[dolfim-ibm]

We should either allow both question and repo_id or none of the two.

[cau-git]

This was merged on a derived PR.