upload vertexai multimodal example notebook

examples/vertexai/README.md

@@ -9,6 +9,15 @@ We provide an example experiment file in [data/input/vertexai-example.jsonl](htt
 prompto_run_experiment --file data/input/vertexai-example.jsonl --max-queries 30
 ```
 
+## Multimodal prompting
+
+Multimodal prompting is available with the VertexAI API. We provide an example notebook in the [Multimodal prompting with Vertex AI notebook](./vertexai-multimodal.ipynb) and example experiment file in [data/input/vertexai-multimodal-example.jsonl](https://github.com/alan-turing-institute/prompto/blob/main/examples/vertexai/data/input/vertexai-multimodal-example.jsonl). You can run it with the following command:
+```bash
+prompto_run_experiment --file data/input/vertexai-multimodal-example.jsonl --max-queries 30
+```
+
+## Environment variables
+
 To run the experiment, you will need to set the following environment variables first:
 ```bash
 export VERTEXAI_PROJECT_ID=<YOUR-VERTEXAI-PROJECT-ID>

examples/vertexai/data/input/vertexai-multimodal-example.jsonl

@@ -0,0 +1,3 @@
+{"id": 0, "api": "vertexai", "model_name": "gemini-1.5-flash-002", "prompt": [{"role": "user", "parts": ["describe what is happening in this image", {"type": "image", "media": "pantani_giro.jpg"}]}], "parameters": {"candidate_count": 1, "temperature": 1, "max_output_tokens": 1000}}
+{"id": 1, "api": "vertexai", "model_name": "gemini-1.5-flash-002", "prompt": [{"role": "user", "parts": [{"type": "image", "media": "mortadella.jpg"}, "what is this?"]}], "parameters": {"candidate_count": 1, "temperature": 1, "max_output_tokens": 1000}}
+{"id": 2, "api": "vertexai", "model_name": "gemini-1.5-flash-002", "prompt": [{"role": "user", "parts": ["what is in this image?", {"type": "image", "media": "pantani_giro.jpg"}]}, {"role": "model", "parts": "This is image shows a group of cyclists."}, {"role": "user", "parts": "are there any notable cyclists in this image? what are their names?"}], "parameters": {"candidate_count": 1, "temperature": 1, "max_output_tokens": 1000}}

examples/vertexai/data/output/vertexai-multimodal-example/21-10-2024-11-56-49-completed-vertexai-multimodal-example.jsonl

@@ -0,0 +1,3 @@
+{"id": 1, "api": "vertexai", "model_name": "gemini-1.5-flash-002", "prompt": [{"role": "user", "parts": [{"type": "image", "media": "mortadella.jpg"}, "what is this?"]}], "parameters": {"candidate_count": 1, "temperature": 1, "max_output_tokens": 1000}, "timestamp_sent": "21-10-2024-11-56-54", "response": "That's **Mortadella**.  Specifically, the image shows whole and sliced mortadella, a large Italian sausage known for its distinctive marbling of fat throughout the meat.  The string tied around it is a common presentation.\n", "safety_attributes": {"HARM_CATEGORY_HATE_SPEECH": "1", "HARM_CATEGORY_DANGEROUS_CONTENT": "1", "HARM_CATEGORY_HARASSMENT": "1", "HARM_CATEGORY_SEXUALLY_EXPLICIT": "1", "blocked": "[False, False, False, False]", "finish_reason": "STOP"}}
+{"id": 0, "api": "vertexai", "model_name": "gemini-1.5-flash-002", "prompt": [{"role": "user", "parts": ["describe what is happening in this image", {"type": "image", "media": "pantani_giro.jpg"}]}], "parameters": {"candidate_count": 1, "temperature": 1, "max_output_tokens": 1000}, "timestamp_sent": "21-10-2024-11-56-51", "response": "Here's a description of the image:\n\nThe photo depicts a group of professional cyclists in a road race, riding closely together in a peloton.\u00a0\n\n\nHere's a breakdown of the scene:\n\n* **The Setting:** The cyclists are riding alongside a low stone wall, with a metal fence visible behind it. There's some greenery beyond the fence, suggesting a roadside or urban setting.\n\n* **The Cyclists:** The cyclists are wearing brightly colored, highly visible cycling jerseys representing different teams.  One cyclist is easily identifiable by his pink jersey, possibly indicating a leader's position or stage win. The others are in various colors, including yellow, red, green, and blue.  Their concentration is evident in their postures.\n\n* **The Bicycles:**  The bicycles are sleek racing bikes with thin tires. The bikes all appear to be high-end racing models.\n\n* **The Action:** The cyclists are clearly in the middle of a race,  riding at a high pace. Their close proximity and intense focus suggests a competitive moment in the race.  There's a sense of urgency and speed in the image.\n\n\nThe overall impression is one of intense athletic competition and the energy of a cycling road race. The colors of the jerseys and the setting are vivid and sharp.\n", "safety_attributes": {"HARM_CATEGORY_HATE_SPEECH": "1", "HARM_CATEGORY_DANGEROUS_CONTENT": "1", "HARM_CATEGORY_HARASSMENT": "1", "HARM_CATEGORY_SEXUALLY_EXPLICIT": "1", "blocked": "[False, False, False, False]", "finish_reason": "STOP"}}
+{"id": 2, "api": "vertexai", "model_name": "gemini-1.5-flash-002", "prompt": [{"role": "user", "parts": ["what is in this image?", {"type": "image", "media": "pantani_giro.jpg"}]}, {"role": "model", "parts": "This is image shows a group of cyclists."}, {"role": "user", "parts": "are there any notable cyclists in this image? what are their names?"}], "parameters": {"candidate_count": 1, "temperature": 1, "max_output_tokens": 1000}, "timestamp_sent": "21-10-2024-11-56-56", "response": "That's a photo from the 1992 Giro d'Italia.  The most prominent cyclist in the image is **Claudio Chiappucci** in the pink jersey.  He's leading the pack.\n\nWhile it's difficult to definitively identify all the other riders with certainty from this angle and image quality,  identifying other notable cyclists in this particular snapshot would require more information or a higher-resolution image.\n", "safety_attributes": {"HARM_CATEGORY_HATE_SPEECH": "1", "HARM_CATEGORY_DANGEROUS_CONTENT": "1", "HARM_CATEGORY_HARASSMENT": "1", "HARM_CATEGORY_SEXUALLY_EXPLICIT": "1", "blocked": "[False, False, False, False]", "finish_reason": "STOP"}}

examples/vertexai/data/output/vertexai-multimodal-example/21-10-2024-11-56-49-input-vertexai-multimodal-example.jsonl

@@ -0,0 +1,3 @@
+{"id": 0, "api": "vertexai", "model_name": "gemini-1.5-flash-002", "prompt": [{"role": "user", "parts": ["describe what is happening in this image", {"type": "image", "media": "pantani_giro.jpg"}]}], "parameters": {"candidate_count": 1, "temperature": 1, "max_output_tokens": 1000}}
+{"id": 1, "api": "vertexai", "model_name": "gemini-1.5-flash-002", "prompt": [{"role": "user", "parts": [{"type": "image", "media": "mortadella.jpg"}, "what is this?"]}], "parameters": {"candidate_count": 1, "temperature": 1, "max_output_tokens": 1000}}
+{"id": 2, "api": "vertexai", "model_name": "gemini-1.5-flash-002", "prompt": [{"role": "user", "parts": ["what is in this image?", {"type": "image", "media": "pantani_giro.jpg"}]}, {"role": "model", "parts": "This is image shows a group of cyclists."}, {"role": "user", "parts": "are there any notable cyclists in this image? what are their names?"}], "parameters": {"candidate_count": 1, "temperature": 1, "max_output_tokens": 1000}}

examples/vertexai/data/output/vertexai-multimodal-example/21-10-2024-11-56-49-log-vertexai-multimodal-example.txt

@@ -0,0 +1 @@
+21-10-2024, 11:57: Completed experiment: vertexai-multimodal-example.jsonl! Experiment processing time: 11.011 seconds, Average time per query: 3.67 seconds