Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: [Experimental] Introduce VLM pipeline using HF AutoModelForVision2Seq, featuring SmolDocling model #1054

Merged
merged 45 commits into from
Feb 26, 2025
Merged

feat: [Experimental] Introduce VLM pipeline using HF AutoModelForVision2Seq, featuring SmolDocling model #1054

merged 45 commits into from
Feb 26, 2025
+1,248 −316

Conversation

cau-git
Copy link
Contributor

@cau-git cau-git commented Feb 25, 2025 •
edited by maxmnemonic
Loading

Checklist:

  • Documentation has been updated, if necessary.
  • Examples have been added, if necessary.
  • Tests have been added, if necessary.

cau-git and others added 30 commits February 24, 2025 11:46
@cau-git
Skeleton for SmolDocling model and VLM Pipeline
dc3a388 
Signed-off-by: Christoph Auer <[email protected]>
Signed-off-by: Maksym Lysak <[email protected]>
wip smolDocling inference and vlm pipeline
03c8d45 
Signed-off-by: Maksym Lysak <[email protected]>
WIP, first working code for inference of SmolDocling, and vlm pipelin…
3c4c647 
…e assembly code, example included.

Signed-off-by: Maksym Lysak <[email protected]>
Fixes to preserve page image and demo export to html
1b968e4 
Signed-off-by: Maksym Lysak <[email protected]>
Enabled figure support in vlm_pipeline
ef079e4 
Signed-off-by: Maksym Lysak <[email protected]>
Fix for table span compute in vlm_pipeline
01c46e2 
Signed-off-by: Maksym Lysak <[email protected]>
Properly propagating image data per page, together with predicted tag…
61bb9db 
…s in VLM pipeline. This enables correct figure extraction and page numbers in provenances

Signed-off-by: Maksym Lysak <[email protected]>
Cleaned up logs, added pages to vlm_pipeline, basic timing per page m…
2a43c19 
…easurement in smol_docling models

Signed-off-by: Maksym Lysak <[email protected]>
Replaced hardcoded otsl tokens with the ones from docling-core tokens…
4370535 
….py enum

Signed-off-by: Maksym Lysak <[email protected]>
Added tokens/sec measurement, improved example
e092978 
Signed-off-by: Maksym Lysak <[email protected]>
Added capability for vlm_pipeline to grab text from preconfigured bac…
0dc3ac4 
…kend

Signed-off-by: Maksym Lysak <[email protected]>
Exposed "force_backend_text" as pipeline parameter
9901729 
Signed-off-by: Maksym Lysak <[email protected]>
Flipped keep_backend to True for vlm_pipeline assembly to work
f6d123a 
Signed-off-by: Maksym Lysak <[email protected]>
Updated vlm pipeline assembly and smol docling model code to support …
0fe12d8 
…updated doctags

Signed-off-by: Maksym Lysak <[email protected]>
Fixing doctags starting tag, that broke elements on first line during…
88b9ac6 
… assembly

Signed-off-by: Maksym Lysak <[email protected]>
Introduced SmolDoclingOptions to configure model parameters (such as …
f2751e1 
…query and artifacts path) via client code, see example in minimal_smol_docling. Provisioning for other potential vlm all-in-one models.

Signed-off-by: Maksym Lysak <[email protected]>
Moved artifacts_path for SmolDocling into vlm_options instead of glob…
7c4ab5c 
…al pipeline option

Signed-off-by: Maksym Lysak <[email protected]>
New assembly code for latest model revision, updated prompt and parsi…
479ee23 
…ng of doctags, updated logging

Signed-off-by: Maksym Lysak <[email protected]>
Updated example of Smol Docling usage
d7abe1b 
Signed-off-by: Maksym Lysak <[email protected]>
Added captions for the images for SmolDocling assembly code, improved…
b1df461 
… provenance definition for all elements

Signed-off-by: Maksym Lysak <[email protected]>
@cau-git
Update minimal smoldocling example
6f9f4f4 
Signed-off-by: Christoph Auer <[email protected]>
@cau-git
Fix repo id
55fa4eb 
Signed-off-by: Christoph Auer <[email protected]>
Cleaned up unnecessary logging
e486eb1 
Signed-off-by: Maksym Lysak <[email protected]>
More elegant solution in removing the input prompt
66532ea 
Signed-off-by: Maksym Lysak <[email protected]>
removed minimal_smol_docling example from CI checks
b12f5ba 
Signed-off-by: Maksym Lysak <[email protected]>
Removed special html code wrapping when exporting to docling document…
b0935da 
…, cleaned up comments

Signed-off-by: Maksym Lysak <[email protected]>
Addressing PR comments, added enabled property to SmolDocling, and re…
853544b 
…lated VLM pipeline option, few other minor things

Signed-off-by: Maksym Lysak <[email protected]>
Moved keep_backend = True to vlm pipeline
0c60ef1 
Signed-off-by: Maksym Lysak <[email protected]>
removed pipeline_options.generate_table_images from vlm_pipeline (dep…
1dbedcb 
…recated in the pipelines)

Signed-off-by: Maksym Lysak <[email protected]>
Added example on how to get original predicted doctags in minimal_smo…
a7a1f32 
…l_docling

Signed-off-by: Maksym Lysak <[email protected]>
@cau-git
Move imports
84c77c9 
Signed-off-by: Christoph Auer <[email protected]>
dolfim-ibm
dolfim-ibm previously approved these changes Feb 25, 2025
View reviewed changes
@cau-git cau-git changed the base branch from mly/smol-docling-integration to main February 26, 2025 11:48
@cau-git cau-git dismissed dolfim-ibm’s stale review February 26, 2025 11:48

The base branch was changed.

@mergify Mergify
Copy link

mergify bot commented Feb 26, 2025

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@cau-git cau-git changed the title feat: Generalize VLM pipeline and models to work with any HF model, include Granite Vision feat: [Experimental] Introduce VLM pipeline using HF AutoModelForVision2Seq, featuring SmolDocling model Feb 26, 2025
@cau-git
chore: fix leftover .to(device)
b88440a 
Signed-off-by: Christoph Auer <[email protected]>
dolfim-ibm
dolfim-ibm previously approved these changes Feb 26, 2025
View reviewed changes
@cau-git
fix: add proper table provenance
483d1bf 
Signed-off-by: Christoph Auer <[email protected]>
@cau-git cau-git dismissed stale reviews from dolfim-ibm and nikos-livathinos via 483d1bf February 26, 2025 13:02
maxmnemonic
maxmnemonic approved these changes Feb 26, 2025
View reviewed changes
Copy link
Contributor

@maxmnemonic maxmnemonic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PeterStaar-IBM
PeterStaar-IBM approved these changes Feb 26, 2025
View reviewed changes
Copy link
Contributor

@PeterStaar-IBM PeterStaar-IBM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work!

dolfim-ibm
dolfim-ibm approved these changes Feb 26, 2025
View reviewed changes
Copy link
Contributor

@dolfim-ibm dolfim-ibm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@cau-git cau-git merged commit 3c9fe76 into main Feb 26, 2025
10 checks passed
@cau-git cau-git deleted the cau/vlm-pipeline-generalizations branch February 26, 2025 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PeterStaar-IBM PeterStaar-IBM PeterStaar-IBM approved these changes

@dolfim-ibm dolfim-ibm dolfim-ibm approved these changes

@nikos-livathinos nikos-livathinos nikos-livathinos approved these changes

@maxmnemonic maxmnemonic maxmnemonic approved these changes

@nassarofficial nassarofficial Awaiting requested review from nassarofficial

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@cau-git @PeterStaar-IBM @dolfim-ibm @nikos-livathinos @maxmnemonic