Refactoring of ImageProcessorFast #35069

yonigozlan · 2024-12-04T01:10:40Z

What does this PR do?

This PR introduces a significant refactoring of how fast image processors can be implemented in Transformers.

Motivation

The primary goal of this refactoring is to simplify the process of creating fast image processors when a slow image processor already exists.

Unlike the current BaseImageProcessor, which provides only a minimal skeleton for image processor classes, the newly introduced BaseImageProcessorFast includes all the basic functionalities needed by a typical image processor. This design allows contributors to focus primarily on modifying default parameters, such as the image mean, std, or default resizing dimensions, rather than rewriting foundational code.

Key Advantages:

Ease of Contribution: Contributors no longer need to rely on copy-paste from an arbitrary slow image processor (which I feel is what is happening currently for some slow image processors). Instead, the BaseImageProcessorFast provides a natural starting point with predefined functionalities.
Consistency: Contributors are encouraged to use a common structure. Whether they only modify default parameters, leverage mixins, or add custom code, they are likely to follow a consistent syntax and logic.
Automatic Optimizations: Improvements made to BaseImageProcessorFast are automatically propagated to all derived fast image processors.
Reduced Diffs: The new approach minimizes added diffs compared to the existing "# Copied from" philosophy in slow image processors. While the "repeat yourself" philosophy is an important part of modeling in Transformers, I feel that it might not be as necessary for image processing, as the model's uniqueness is rarely found in the image processing logic.

Implementation

Functional or class transforms

Following the torchvision approach to defining image transforms, there are two main ways to write the processing logic for image processors: using functional transforms or class-based transforms.

This PR showcases both approaches:

BaseImageProcessorFast: implemented using functional transforms.
ViTImageProcessorFast: almost equivalent but uses piped class-based transforms.
LlavaNextImageProcessorFast: implemented using functionals.
LlavaOnevisionImageProcessorFast: implemented using a mix of class-based transforms and functionals. equivalent to LlavaNextImageProcessorFast

The choice is entirely open for debate at this point. I see advantages to both approaches, but I’m sure I haven’t considered everything, so please share your thoughts if you have a preference one way or the other.

To me, the advantages/drawbacks of functionals are the following:

🟢 Less abstraction so potentially easier to read
🟢 Easier for contributors to write or adapt from existing slow image processors (which currently use functionals).
🟢 Allows more flexibility in processing logic, as transforms do not need to be sequential. For more complex processors, using piped class-based transforms would likely require mixing functionals and class-based transforms, or adding logic outside the transforms pipeline instead of a simple one-liner.
🔴 The logic can be more verbose than for class transforms

For class transforms:

🟢 Aligns with practices in other libraries like Albumentations.
🟢 Generally cleaner and more structured, easier to add/remove simple transforms.
🔴 As mentioned before, the logic is restricted by the sequential nature of the pipeline. For complex processors (e.g., involving patching), mixing functionals and class-based transforms or adding logic around the pipeline seems unavoidable (as seen in LlavaOnevisionImageProcessorFast).
🔴 There appear to be compilation issues: LlavaOnevisionImageProcessorFast fails to compile, as it seemingly gets stuck in an infinite compilation loop, while its functional equivalent, LlavaNextImageProcessorFast, compiles without problems. Hover this needs more thorough investigation.

New `add-fast-image-processor` Command in `transformers-cli`

This PR introduces a new CLI command that automates the creation of fast image processors. Given a model folder name, it generates all necessary imports, documentation, dummy objects, and a fast image processor file where default parameters are parsed from the slow image processor file.
The new fast image processor class is also added to the image processor test file by the CLI. However, some tests may still need manual adjustments to properly include and validate the new fast image processor class.

Example Usage:

transformers-cli add-fast-image-processor --model-name blip

Example Output:

In transformers/src/models/blip/image_processing_blip_fast.py:

# coding=utf-8
# Copyright 2024 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Fast Image processor class for BLIP."""

from ...image_processing_utils_fast import BaseImageProcessorFast
from ...image_utils import OPENAI_CLIP_MEAN, OPENAI_CLIP_STD, PILImageResampling


class BlipImageProcessorFast(BaseImageProcessorFast):
    r"""
    Constructs a fast BLIP image processor.

    Args:
        do_resize (`bool`, *optional*):
            Whether to resize the image's (height, width) dimensions to the specified `size`. Can be overridden by the
            `do_resize` parameter in the `preprocess` method.
        size (`dict`, *optional*):
            Size of the output image after resizing. Can be overridden by the `size` parameter in the `preprocess`
            method.
        resample (`PILImageResampling`, *optional*):
            Resampling filter to use if resizing the image. Only has an effect if `do_resize` is set to `True`. Can be
            overridden by the `resample` parameter in the `preprocess` method.
        do_center_crop (`bool`, *optional*, defaults to `True`):
            Whether to center crop the image to the specified `crop_size`. Can be overridden by `do_center_crop` in the
            `preprocess` method.
        crop_size (`Dict[str, int]` *optional*, defaults to 224):
            Size of the output image after applying `center_crop`. Can be overridden by `crop_size` in the `preprocess`
            method.
        do_rescale (`bool`, *optional*):
            Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by the
            `do_rescale` parameter in the `preprocess` method.
        rescale_factor (`int` or `float`, *optional*, defaults to `1/255`):
            Scale factor to use if rescaling the image. Only has an effect if `do_rescale` is set to `True`. Can be
            overridden by the `rescale_factor` parameter in the `preprocess` method.
        do_normalize (`bool`, *optional*):
            Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess`
            method. Can be overridden by the `do_normalize` parameter in the `preprocess` method.
        image_mean (`float` or `List[float]`, *optional*):
            Mean to use if normalizing the image. This is a float or list of floats the length of the number of
            channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be
            overridden by the `image_mean` parameter in the `preprocess` method.
        image_std (`float` or `List[float]`, *optional*):
            Standard deviation to use if normalizing the image. This is a float or list of floats the length of the
            number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method.
            Can be overridden by the `image_std` parameter in the `preprocess` method.
        do_convert_rgb (`bool`, *optional*):
            Whether to convert the image to RGB.
    """


    # This generated class can be used as a starting point for the fast image processor.
    # if the image processor is only used for simple augmentations, such as resizing, center cropping, rescaling, or normalizing,
    # only the default values should be set in the class.
    # If the image processor requires more complex augmentations, methods from BaseImageProcessorFast can be overridden.
    # For an example of a fast image processor requiring more complex augmentations, see `LlavaOnevisionImageProcessorFast`.

    # Default values should be checked against the slow image processor
    # None values left after checking can be removed
    resample = PILImageResampling.BICUBIC
    image_mean = OPENAI_CLIP_MEAN
    image_std = OPENAI_CLIP_STD
    size = {"height": 384, "width": 384}
    default_to_square = None
    crop_size = None
    do_resize = True
    do_center_crop = None
    do_rescale = True
    do_normalize = True
    do_convert_rgb = True

In this case, this is enough to get a fully working fast Blip image processor!

New Mixins for Common Logic

To handle shared preprocessing and post-processing logic, this PR introduces reusable mixins ~~(only LlavaPatchingMixin is present as an example in this PR)~~. Additional mixins are planned for other common patterns, such as:

Video processing
DETR-like processing
Segmentation post-processing
Depth estimation post-processing

Edit: Removing the Mixins for patching in favor of # Copied from or modular in the futur, as such preprocessing techniques don't usually stick for a long time, and adding Mixins every time a technique is used twice or more wouldn't scale well.

Summary: Three Types of Fast Image Processors

Basic Processors:
- These support standard operations like resizing, rescaling, normalizing, and cropping.
- They require minimal customization, mostly overriding default parameters.
- Examples in this PR: blip, clip, deit, siglip, and vit.

~~2. Mixin-Based Processors:~~
~~- These rely heavily on predefined mixins to implement shared logic.~~
~~- Examples in this PR: llava_next and llava_onevision.~~

Edit: Removing Mixins in favor of # Copied From or modular for now, see earlier comment.

Exotic Processors:
- These have unique processing logic that differs significantly from the base class or existing mixins.
- Contributors need to override functions like __init__ and preprocess while reusing the syntax and structure of BaseImageProcessorFast.
- Example in this PR: convnext

Miscellaneous Issues and Questions

The CLI currently needs an existing slow image processor to work. If the library aims to eventually fully deprecate slow image processors or at least for new models, this will need to change, and the CI should maybe be integrated into the add-new-model-like.
There is a significant design difference between slow and fast image processors. For slow processors, contributors need to rewrite most of the logic, while for fast processors, the goal is to rewrite as little code as possible. This might get confusing for contributors, especially since as mentioned in the previous point, users will most likely start with writing a slow image processor.
Adding lots of mixins to image_processing_utils_fast might make the file large and difficult to read. This also raises the question of when a mixin should be created for repeated patterns across image processors. Should it be created when a pattern is shared by two or more models? Or when it presents an idea likely to be reused in the future?
Padding functionality is currently not included in BaseImageProcessorFast because the image processors that use padding implement it in slightly different ways. This makes it challenging to standardize the padding logic. A more consistent approach should maybe be added in the future.
The center_crop functions in Transformers' image_transforms and in torchvision have different implementations, namely the cropping boundaries in transformers are defined as: top = (orig_height - crop_height) // 2, left = (orig_width - crop_width) // 2 and top = int(round((orig_height - crop_height) / 2.0)), left = int(round((orig_width - crop_width) / 2.0)), which can result in shifted cropping when the size before cropping is odd. I don't think this should be much of a problem for users, but when comparing outputs of slow and fast image processors in tests, this results in huge differences.

Who can review?

@qubvel @molbap @zucchini-nlp

HuggingFaceDocBuilderDev · 2024-12-04T01:53:35Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

tests/test_image_processing_common.py

yonigozlan · 2024-12-11T16:48:47Z

@ArthurZucker This PR is still in a rough state but pinging you for visibility as there are lots of refactoring choices that are up for discussion!

qubvel

I made an initial quick pass. My main concern here is how common these methods are. I mean, for preprocess, does it scale to most models, or will we need to override it somewhere?

I think (but might be wrong) that the better design choice would be to follow standards in image processing, where we define each transformation as a separate class and then can pipe them. This approach is followed by torchvision, albumentations, kornia, and imgaug. So I suppose this is a pretty robust approach that should work well in our case too.

We can reuse the original torch transforms if they are compatible with torch compile, otherwise, we can rewrite them ourselves. The most common transforms will live in the base file, while custom will live in image_processing_* model's file and can be moved to common for reuse.

Let me know what you think regarding this? Do you see any difficulties using this approach?

src/transformers/image_processing_utils_fast.py

yonigozlan · 2024-12-11T17:45:41Z

I think (but might be wrong) that the better design choice would be to follow standards in image processing, where we define each transformation as a separate class and then can pipe them. This approach is followed by torchvision, albumentations, kornia, and imgaug. So I suppose this is a pretty robust approach that should work well in our case too.

We can reuse the original torch transforms if they are compatible with torch compile, otherwise, we can rewrite them ourselves. The most common transforms will live in the base file, while custom will live in image_processing_* model's file and can be moved to common for reuse.

@qubvel The problem I see with this is that some image processors needs additional logic in-between transforms, that depends on the result of the previous transforms (like the padding operation in detr-like image processors), so unless I'm mistaken I don't think piping class transforms would be possible here.
It feels to me that having some processors using piping of class transforms and other functional transforms would be confusing and break consistency. I also think that using functional transforms is closer to what is done in slow processors, so maybe easier to adapt for contributors when they need to implement "exotic" fast image processing operations.

That said, I might be misunderstanding or missing something if this is a standard approach in other image preprocessing library, so happy to discuss this further.

zucchini-nlp

Very cool PR, happy to see fast image processors getting propagated!

I left a few questions here and there, mostly nits. Overall I like the "less code for users" and moving redundant parts for the base class idea. But seems like we are hiding the core processing logic in base class. So I agree here with @qubvel that having a pipe of transforms looks nicer and easier to consume.

And for weird model, prob we don't need separate patching mixin, as it doesn't scale much because not many of those patch methods will stick for long time and new methods will continue to be invented. I am also thinking if all this repeated code can be squeezed when we add modular to those models? For example llava-next style patching is used in three models, so we need to define it once and then let modular copy the main code body

src/transformers/image_processing_utils_fast.py

src/transformers/models/convnext/image_processing_convnext_fast.py

src/transformers/image_processing_utils_fast.py

yonigozlan · 2024-12-18T03:16:37Z

Hello @qubvel @zucchini-nlp ! Thanks again for your reviews. I added examples of how I would write processors using class-based transforms, and a section in the PR with the advantages and drawbacks I see for functional vs class transforms. If you have some time I would be glad to have your opinions on this :)

qubvel

Great work! Thanks for providing both options for review 👍 I agree, the functional approach looks more flexible right now.

I would differentiate image processors for the following cases:

Basic image transforms: operate only on raw images, and usually, in the original code, we might see the torchvision defined preprocessing pipeline.
Patching transforms: operate on patches but still work with images only. The preprocessing usually looks as follows: split into patches -> apply basic image transforms to patches (normalize, rescale, etc.).
Annotation-aware image transforms: operate on images and annotations (object detection, image segmentation). These are the most complicated, I suppose, because for each transform, we have to handle annotation transformation like bounding boxes/masks/keypoints.

It migth be oversimplified, am I missing something?

I think we can have different styles of image processors for these cases. And the thing we have to take into consideration is that it's common to have transforms written in torchvision, so it would be nice to have a clear way to port the original transforms to be transformers-like.

src/transformers/models/convnext/__init__.py

src/transformers/models/convnext/image_processing_convnext_fast.py

src/transformers/models/llava_onevision/image_processing_llava_onevision_fast.py

src/transformers/models/llava_onevision/processing_llava_onevision.py

src/transformers/image_transforms.py

yonigozlan · 2024-12-19T22:31:15Z

Great work! Thanks for providing both options for review 👍 I agree, the functional approach looks more flexible right now.

I believe so too for the functional approach. Using class transforms might look nicer in some cases, but they are limiting and add a layer of complexity for contributors that I don't think is necessary.
I can leave the two class transforms based fast image processors (ViTImageProcessorFast and LlavaOnevisionImageProcessorFast) as they are for now, in case future reviewers have stronger opinion on this, but I'll focus on improving the functional based ones.

I would differentiate image processors for the following cases:

Basic image transforms: operate only on raw images, and usually, in the original code, we might see the torchvision defined preprocessing pipeline.
Patching transforms: operate on patches but still work with images only. The preprocessing usually looks as follows: split into patches -> apply basic image transforms to patches (normalize, rescale, etc.).
Annotation-aware image transforms: operate on images and annotations (object detection, image segmentation). These are the most complicated, I suppose, because for each transform, we have to handle annotation transformation like bounding boxes/masks/keypoints.
It migth be oversimplified, am I missing something?

I think we can have different styles of image processors for these cases. And the thing we have to take into consideration is that it's common to have transforms written in torchvision, so it would be nice to have a clear way to port the original transforms to be transformers-like.

Agreed on the three different categories, but I'm not sure how to handle this differentiation in the code.
I think having the BaseImageProcessorFast as the template for the first category (as it is currently) is a nice way to reduce diffs and force consistency. For the two others, as we discussed before, having a mixin for each doesn't seem to be the way to go, as they are so many slight variations between all models belonging to each categories that it would be impossible to have a Mixin that generalizes to all model in the category and will stand the test of time.

What I was thinking was to add some guidance in the fast image processor file generated by the transformers-cli, explaining what existing processor to use as a baseline depending on what sort of image processing the model needs

yonigozlan · 2025-01-22T18:52:52Z

@ArthurZucker, @qubvel , @zucchini-nlp Here's a new iteration where I implemented @ArthurZucker suggestions.
Now the only function fast image processors need to override in most cases is _preprocess. The only exceptions (for now) are the DETR processors as they work a little differently for backward compatibility, and Qwen2VL as it takes images and videos as inputs.
I'm also waiting on this PR #35122 to uniformize the Pixtral image processor.

The "pre"-preprocessing functions now can handle arbitrary kwargs, as long as they are specified in the valid_extra_kwargs attributes of the child class.
Remaining issues that I see are the class docstrings that are less informative now, as the extra kwargs can't be in the docstring anymore. To mitigate this I wrote detailed doctrings for image processors that override _preprocess, which include descriptions of the extra kwargs.

…processor-base

ArthurZucker

Much better thanks!
Would be nice to use Unpack[DefaultFastImageProcessorKwargs] rather than having to move around the 20 + arguments for both init and forward

src/transformers/image_processing_utils_fast.py

src/transformers/models/llava_next/image_processing_llava_next_fast.py

src/transformers/image_processing_utils_fast.py

src/transformers/models/convnext/image_processing_convnext_fast.py

src/transformers/models/deformable_detr/image_processing_deformable_detr_fast.py

…processor-base

yonigozlan · 2025-01-31T16:22:07Z

Would be nice to use Unpack[DefaultFastImageProcessorKwargs] rather than having to move around the 20 + arguments for both init and forward

Agreed but if I do that the CI won't let me use a start docstrings describing all the parameters, and the type hints when instantiating an image processor will be much less useful. What do you think we should prioritize @ArthurZucker? Or do you see a way to use both Unpack[DefaultFastImageProcessorKwargs] and have a useful docstring?

ArthurZucker

Super cool! Last nits IMO and we should be alright

ArthurZucker · 2025-02-04T09:59:39Z

src/transformers/image_processing_utils_fast.py

-        }
+        data_format = kwargs.pop("data_format", None)
+        data_format = data_format if data_format is not None else ChannelDimension.FIRST
+
        images = self._prepare_input_images(


here you will have a n issue: you are poping way too many time from the kwargs, so you pop then you use self, values will be different.
This should be simplified. Just update self with kwargs, then do the rest with always self. something like that

ArthurZucker · 2025-02-04T10:00:00Z

src/transformers/models/convnext/image_processing_convnext_fast.py

-            crop_pct=crop_pct,
-            **kwargs,
-        )
+    def __init__(self, **kwargs: Unpack[valid_init_kwargs]):


Suggested change

def __init__(self, **kwargs: Unpack[valid_init_kwargs]):

def __init__(self, **kwargs: Unpack[ConvNextFastImageProcessorInitKwargs]):

this is better and explicit

ArthurZucker · 2025-02-04T10:00:10Z

src/transformers/models/convnext/image_processing_convnext_fast.py

@@ -114,8 +95,8 @@ def __init__(
            overridden by `crop_pct` in the`preprocess` method.
        """,
    )
-    def preprocess(self, *args, **kwargs) -> BatchFeature:
-        return super().preprocess(*args, **kwargs)
+    def preprocess(self, images: ImageInput, **kwargs: Unpack[valid_preprocess_kwargs]) -> BatchFeature:


same comment let's be explicit if possible

ArthurZucker · 2025-02-04T10:00:18Z

src/transformers/models/deformable_detr/image_processing_deformable_detr_fast.py

+    valid_init_kwargs = DeformableDetrFastImageProcessorInitKwargs
+    valid_preprocess_kwargs = DeformableDetrFastImageProcessorPreprocessKwargs
+
+    def __init__(self, **kwargs: Unpack[valid_init_kwargs]) -> None:


same comment

… kwargs

…m/yonigozlan/transformers into improve-fast-image-processor-base

ArthurZucker · 2025-02-10T08:50:57Z

src/transformers/models/llava_next/image_processing_llava_next_fast.py

+    valid_init_kwargs = LlavaNextFastImageProcessorInitKwargs
+    valid_preprocess_kwargs = LlavaNextFastImageProcessorPreprocessKwargs


this is probably a ditto

ArthurZucker · 2025-02-10T15:02:14Z

src/transformers/image_processing_utils_fast.py

+    valid_init_kwargs = DefaultFastImageProcessorInitKwargs
+    valid_preprocess_kwargs = DefaultFastImageProcessorPreprocessKwargs
+


should be removed!

* add init and base image processing functions * add add_fast_image_processor to transformers-cli * add working fast image processor clip * add fast image processor to doc, working tests * remove "to be implemented" SigLip * fix unprotected import * fix unprotected vision import * update ViTImageProcessorFast * increase threshold slow fast ewuivalence * add fast img blip * add fast class in tests with cli * improve cli * add fast image processor convnext * add LlavaPatchingMixin and fast image processor for llava_next and llava_onevision * add device kwarg to ImagesKwargs for fast processing on cuda * cleanup * fix unprotected import * group images by sizes and add batch processing * Add batch equivalence tests, skip when center_crop is used * cleanup * update init and cli * fix-copies * refactor convnext, cleanup base * fix * remove patching mixins, add piped torchvision transforms for ViT * fix unbatched processing * fix f strings * protect imports * change llava onevision to class transforms (test) * fix convnext * improve formatting (following Pavel review) * fix handling device arg * improve cli * fix * fix inits * Add distinction between preprocess and _preprocess, and support for arbitrary kwargs through valid_extra_kwargs * uniformize qwen2_vl fast * fix docstrings * add add fast image processor llava * remove min_pixels max_pixels from accepted size * nit * nit * refactor fast image processors docstrings * cleanup and remove fast class transforms * update add fast image processor transformers cli * cleanup docstring * uniformize pixtral fast and make _process_image explicit * fix prepare image structure llava next/onevision * Use typed kwargs instead of explicit args * nit fix import Unpack * clearly separate pops and gets in base preprocess. Use explicit typed kwargs * make qwen2_vl preprocess arguments hashable

yonigozlan force-pushed the improve-fast-image-processor-base branch from 9e31e43 to c393647 Compare December 7, 2024 00:18

yonigozlan commented Dec 9, 2024

View reviewed changes

tests/test_image_processing_common.py Outdated Show resolved Hide resolved

yonigozlan commented Dec 9, 2024

View reviewed changes

tests/test_image_processing_common.py Outdated Show resolved Hide resolved

qubvel reviewed Dec 11, 2024

View reviewed changes

src/transformers/image_processing_utils_fast.py Outdated Show resolved Hide resolved

src/transformers/image_processing_utils_fast.py Outdated Show resolved Hide resolved

yonigozlan force-pushed the improve-fast-image-processor-base branch from 867b1f5 to 3f8674f Compare December 11, 2024 22:15

zucchini-nlp reviewed Dec 13, 2024

View reviewed changes

yonigozlan force-pushed the improve-fast-image-processor-base branch from f12c5d7 to da6de2e Compare December 16, 2024 19:35

qubvel reviewed Dec 18, 2024

View reviewed changes

yonigozlan changed the title ~~[WIP] Refactoring of ImageProcessorFast~~ Refactoring of ImageProcessorFast Jan 6, 2025

yonigozlan requested a review from ArthurZucker January 6, 2025 23:13

molbap mentioned this pull request Jan 8, 2025

Add Molmo (7B-D, 7B-O, 70B) #33962

Open

yonigozlan mentioned this pull request Jan 16, 2025

image_transforms preprocess quite slow when run large image with qwen2vl #34272

Closed

4 tasks

yonigozlan added 13 commits January 16, 2025 15:21

add init and base image processing functions

2f00f0c

add add_fast_image_processor to transformers-cli

cfadb72

add working fast image processor clip

2cd73cb

add fast image processor to doc, working tests

932bd68

remove "to be implemented" SigLip

23d79ce

fix unprotected import

3f2d8a6

fix unprotected vision import

6a9d332

update ViTImageProcessorFast

a1e2663

increase threshold slow fast ewuivalence

fa74e7e

add fast img blip

9dbd765

add fast class in tests with cli

d39ff52

improve cli

f609730

add fast image processor convnext

8f7774d

yonigozlan added 4 commits January 22, 2025 17:14

add add fast image processor llava

e5c1e01

remove min_pixels max_pixels from accepted size

aef2fb4

nit

7078a14

nit

aa94873

yonigozlan removed the request for review from stevhliu January 22, 2025 18:53

yonigozlan added 5 commits January 28, 2025 17:47

refactor fast image processors docstrings

13a125b

Merge remote-tracking branch 'upstream/main' into improve-fast-image-…

8adb893

…processor-base

cleanup and remove fast class transforms

67d65f2

update add fast image processor transformers cli

d225448

cleanup docstring

80c6824

ArthurZucker approved these changes Jan 30, 2025

View reviewed changes

yonigozlan added 4 commits January 30, 2025 18:04

Merge remote-tracking branch 'upstream/main' into improve-fast-image-…

b96adfa

…processor-base

uniformize pixtral fast and make _process_image explicit

dbaacd1

Merge remote-tracking branch 'upstream/main' into improve-fast-image-…

b660e9d

…processor-base

fix prepare image structure llava next/onevision

b43ede1

yonigozlan and others added 3 commits February 4, 2025 00:18

Use typed kwargs instead of explicit args

3b05cbd

nit fix import Unpack

95db4a9

Merge branch 'main' into improve-fast-image-processor-base

d9e1fcd

ArthurZucker reviewed Feb 4, 2025

View reviewed changes

yonigozlan and others added 4 commits February 4, 2025 17:34

clearly separate pops and gets in base preprocess. Use explicit typed…

6bd7a1b

… kwargs

Merge branch 'main' into improve-fast-image-processor-base

565e482

make qwen2_vl preprocess arguments hashable

f85c06f

Merge branch 'improve-fast-image-processor-base' of https://github.co…

1a7b0c4

…m/yonigozlan/transformers into improve-fast-image-processor-base

yonigozlan merged commit fa56dcc into huggingface:main Feb 4, 2025
23 of 25 checks passed

ArthurZucker reviewed Feb 10, 2025

View reviewed changes

ArthurZucker mentioned this pull request Feb 13, 2025

Mask2former & Maskformer Fast Image Processor #35685

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactoring of ImageProcessorFast #35069

Refactoring of ImageProcessorFast #35069

yonigozlan commented Dec 4, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 4, 2024

yonigozlan commented Dec 11, 2024

qubvel left a comment

yonigozlan commented Dec 11, 2024 •

edited

Loading

zucchini-nlp left a comment

yonigozlan commented Dec 18, 2024

qubvel left a comment

yonigozlan commented Dec 19, 2024

yonigozlan commented Jan 22, 2025

ArthurZucker left a comment

yonigozlan commented Jan 31, 2025

ArthurZucker left a comment

ArthurZucker Feb 4, 2025

ArthurZucker Feb 4, 2025

ArthurZucker Feb 4, 2025

ArthurZucker Feb 4, 2025

ArthurZucker Feb 10, 2025

ArthurZucker Feb 10, 2025

	def __init__(self, **kwargs: Unpack[valid_init_kwargs]):
	def __init__(self, **kwargs: Unpack[ConvNextFastImageProcessorInitKwargs]):

		valid_init_kwargs = LlavaNextFastImageProcessorInitKwargs
		valid_preprocess_kwargs = LlavaNextFastImageProcessorPreprocessKwargs

		valid_init_kwargs = DefaultFastImageProcessorInitKwargs
		valid_preprocess_kwargs = DefaultFastImageProcessorPreprocessKwargs

Refactoring of ImageProcessorFast #35069

Refactoring of ImageProcessorFast #35069

Conversation

yonigozlan commented Dec 4, 2024 • edited Loading

What does this PR do?

Motivation

Key Advantages:

Implementation

Functional or class transforms

New add-fast-image-processor Command in transformers-cli

Example Usage:

Example Output:

New Mixins for Common Logic

Summary: Three Types of Fast Image Processors

Miscellaneous Issues and Questions

Who can review?

HuggingFaceDocBuilderDev commented Dec 4, 2024

yonigozlan commented Dec 11, 2024

qubvel left a comment

Choose a reason for hiding this comment

yonigozlan commented Dec 11, 2024 • edited Loading

zucchini-nlp left a comment

Choose a reason for hiding this comment

yonigozlan commented Dec 18, 2024

qubvel left a comment

Choose a reason for hiding this comment

yonigozlan commented Dec 19, 2024

yonigozlan commented Jan 22, 2025

ArthurZucker left a comment

Choose a reason for hiding this comment

yonigozlan commented Jan 31, 2025

ArthurZucker left a comment

Choose a reason for hiding this comment

ArthurZucker Feb 4, 2025

Choose a reason for hiding this comment

ArthurZucker Feb 4, 2025

Choose a reason for hiding this comment

ArthurZucker Feb 4, 2025

Choose a reason for hiding this comment

ArthurZucker Feb 4, 2025

Choose a reason for hiding this comment

ArthurZucker Feb 10, 2025

Choose a reason for hiding this comment

ArthurZucker Feb 10, 2025

Choose a reason for hiding this comment

yonigozlan commented Dec 4, 2024 •

edited

Loading

New `add-fast-image-processor` Command in `transformers-cli`

yonigozlan commented Dec 11, 2024 •

edited

Loading