Porting PaliGemma transformers checkpoint

[ariG23498]

Porting Pali Gemma transformers checkpoints in Keras NLP

[ariG23498]

The working colab notebook: Notebook

[ariG23498]

With this I got to run a fine-tuned captioning model

!pip install -U -q git+https://github.com/ariG23498/keras-nlp@aritra/paligemma-hf-port
!pip install -U -q keras

import os
os.environ["KERAS_BACKEND"] = "jax"

import keras_nlp

import requests
from PIL import Image
import numpy as np

pali_gemma_lm = keras_nlp.models.PaliGemmaCausalLM.from_preset(
    "hf://gokaygokay/sd3-long-captioner"
)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"

image = Image.open(requests.get(url, stream=True).raw).resize((224, 224))
image = np.array(image)

pali_gemma_lm.generate({
    "images": image,
    "prompts": "caption en "
})

Which generates

caption en captured from a high-angle perspective, a light blue vintage car is parked on a cobblestone street,
its side facing the viewer. the car's body is adorned with a silver trim, adding a touch of elegance to its
appearance. the windows of the car are tinted, providing a glimpse of the interior. the car's wheels, a striking
feature, are adorned with a silver hub cap, adding a touch of elegance to the overall design. the backdrop
features a weathered yellow wall, its texture enhanced by the peeling paint on its surface. to the left of the
frame, a brown wooden door with arched top panels is visible, adding a touch of rustic charm to the scene.

The image in question 👇

[mattdangerw]

Thanks!