Tutorial - LangSmith.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "727e6646-79e6-4219-9553-20363a14fba5",
   "metadata": {},
   "source": [
    "# LangSmith Hands-on Tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b24840a-68ed-4c83-a102-43ac5eb60fd1",
   "metadata": {},
   "source": [
    "### Installing the Required Library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "dcfa099b-ea36-4812-b4cf-3e2720cf3056",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U langsmith -q\n",
    "!pip install langchain -q\n",
    "!pip install openai -q\n",
    "!pip install python-dotenv -q"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d26002a-eca7-48a5-b4dc-7385cb142965",
   "metadata": {},
   "source": [
    "### Load Environment Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f40b88b8-bcf6-4e1c-860e-3bd7ab694421",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OpenAI API Key Loaded:  True\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import openai\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "load_dotenv(find_dotenv(), override=True)\n",
    "\n",
    "os.environ.get('OPENAI_API_KEY')\n",
    "\n",
    "print('OpenAI API Key Loaded: ', os.environ.get('OPENAI_API_KEY') is not None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4cf906f-447b-4b12-8084-9e7bdc9ed916",
   "metadata": {},
   "source": [
    "### Building Simple LLM Application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "d73bfc18-a3be-41d1-bfd3-726f40b6d445",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The weather in the UAE can vary depending on the time of year. In general, the UAE has a hot desert climate with high temperatures year-round. The summers are extremely hot, with temperatures often exceeding 40 degrees Celsius (104 degrees Fahrenheit). The winters are milder, with temperatures ranging from 15-25 degrees Celsius (59-77 degrees Fahrenheit). Rainfall is minimal and mostly occurs during the winter months.\\n\\nOverall, the weather in the UAE is characterized by sunny skies, low humidity, and little precipitation, making it a popular destination for those seeking warm weather and sunshine.'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI()\n",
    "llm.predict(\"How is the weather in UAE?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7bf3f90-4dbc-4435-9cae-7b349e1d3ba9",
   "metadata": {},
   "source": [
    "### Using LangSmith LangChain Tracer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "db9ba2d5-8621-4e6d-a2c3-58ca1f6f7ffe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'As of 2021, the estimated population of the United Arab Emirates (UAE) is around 9.9 million people.'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.callbacks import LangChainTracer\n",
    "\n",
    "tracer = LangChainTracer(project_name=\"LangChainTracerProject\")\n",
    "llm.predict(\"How many people live in UAE?\",callbacks=[tracer])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57345308-445f-4775-a67d-87f76656c567",
   "metadata": {},
   "source": [
    "### Using Context Manager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "bb2f9879-2a5a-41fb-9ad9-bc038621bb34",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.callbacks import tracing_v2_enabled\n",
    "\n",
    "with tracing_v2_enabled(project_name=\"LangChainTracerProject\"):\n",
    "    llm.predict(\"How many people live in France?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47c8a682-55c5-49f9-988c-0f2b1bf2f65d",
   "metadata": {},
   "source": [
    "### Using Tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "5743dcc9-0d72-466c-a02a-09c425c89492",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input': 'Hello, World!', 'text': 'Hello! How can I assist you today?'}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.chains import LLMChain\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.prompts import PromptTemplate\n",
    "\n",
    "llm = ChatOpenAI(temperature=0, tags=[\"llm-tag\"])\n",
    "prompt = PromptTemplate.from_template(\"{input}\")\n",
    "chain = LLMChain(llm=llm, prompt=prompt, tags=[\"chain-tag\",\"another-tag\"])\n",
    "\n",
    "chain(\"Hello, World!\", tags=[\"shared-tags\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "876a6d50-acee-4f88-b5c6-19174efc5cca",
   "metadata": {},
   "source": [
    "### Using Groups "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "75ac9661-b2d1-43af-85a5-8c57f23b84fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.callbacks.manager import (trace_as_chain_group)\n",
    "\n",
    "with trace_as_chain_group(\"ataya_group_name\") as group_manager:\n",
    "    pass\n",
    "\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.prompts import PromptTemplate\n",
    "\n",
    "llm=ChatOpenAI(temperature=0.7)\n",
    "prompt= PromptTemplate(\n",
    "    input_variables=[\"question\"],\n",
    "    template=\"What is the answer to {question}\"\n",
    ")\n",
    "chain = LLMChain(llm=llm, prompt=prompt)\n",
    "with trace_as_chain_group(\"ataya_group_name\") as group_manager:\n",
    "    chain.run(question=\"What is the your name?\", callbacks=group_manager)\n",
    "    chain.run(question=\"What is your age?\", callbacks=group_manager)\n",
    "    chain.run(question=\"Where do you work?\", callbacks=group_manager)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fea3b004-0ae0-4355-85ab-f3bd955e6fd2",
   "metadata": {},
   "source": [
    "### Using Client class from LangSmith"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "c0206c0b-41b1-4547-9d5d-a0ec0dd954ef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object Client.list_runs at 0x12578d8a0>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langsmith import Client\n",
    "\n",
    "client = Client()\n",
    "project_runs=client.list_runs(project_name=\"default\")\n",
    "project_runs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "7914d2bc-f626-46c5-a79d-55a40c70eb78",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object Client.list_runs at 0x12578d300>"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime, timedelta\n",
    "\n",
    "today_runs = client.list_runs(\n",
    "    project_name=\"default\",\n",
    "    start_time=datetime.now() - timedelta(days=1),\n",
    "    run_type=\"llm\"\n",
    ")\n",
    "\n",
    "today_runs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "2be27d3d-fc65-478e-98ac-a73d3f04d7cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id=UUID('17a20712-1c15-4073-a386-22007cc5944b') name='ChatOpenAI' start_time=datetime.datetime(2024, 7, 2, 21, 42, 53, 724266) run_type='llm' end_time=datetime.datetime(2024, 7, 2, 21, 42, 54, 829655) extra={'invocation_params': {'model': 'gpt-3.5-turbo', 'model_name': 'gpt-3.5-turbo', 'stream': False, 'n': 1, 'temperature': 0.7, '_type': 'openai-chat', 'stop': None}, 'options': {'stop': None}, 'batch_size': 1, 'runtime': {'sdk': 'langsmith-py', 'sdk_version': '0.1.82', 'library': 'langsmith', 'platform': 'macOS-14.6-arm64-arm-64bit', 'runtime': 'python', 'py_implementation': 'CPython', 'runtime_version': '3.11.7', 'langchain_version': '0.1.20', 'langchain_core_version': '0.1.52', 'thread_count': 22.0, 'mem': {'rss': 280526848.0}, 'cpu': {'time': {'sys': 3.72275584, 'user': 8.425246208}, 'ctx_switches': {'voluntary': 127141.0, 'involuntary': 0.0}, 'percent': 0.0}}, 'metadata': {'revision_id': '9bb07c0-dirty', 'system_fingerprint': None}} error=None serialized=None events=[{'name': 'start', 'time': '2024-07-02T21:42:53.724266+00:00'}, {'name': 'end', 'time': '2024-07-02T21:42:54.829655+00:00'}] inputs={'messages': [{'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'HumanMessage'], 'kwargs': {'content': 'What is the answer to Where do you work?', 'type': 'human'}}]} outputs={'llm_output': {'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None}, 'run': None, 'generations': [{'text': 'The answer to \"Where do you work?\" would depend on the individual\\'s place of employment.', 'generation_info': {'finish_reason': 'stop', 'logprobs': None}, 'type': 'ChatGeneration', 'message': {'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'AIMessage'], 'kwargs': {'content': 'The answer to \"Where do you work?\" would depend on the individual\\'s place of employment.', 'response_metadata': {'token_usage': {'completion_tokens': 19, 'prompt_tokens': 17, 'total_tokens': 36}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, 'type': 'ai', 'id': 'run-17a20712-1c15-4073-a386-22007cc5944b-0', 'tool_calls': [], 'invalid_tool_calls': []}}}]} reference_example_id=None parent_run_id=UUID('8ebc2721-07e9-420b-b8aa-44964dd3f229') tags=[] session_id=UUID('dfccefec-106a-431e-9448-ce8c857f96e4') child_run_ids=None child_runs=None feedback_stats=None app_path='/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/projects/p/dfccefec-106a-431e-9448-ce8c857f96e4/r/17a20712-1c15-4073-a386-22007cc5944b?trace_id=49ad980e-9add-4dc6-a0f8-61dd6b07236d&start_time=2024-07-02T21:42:51.489370' manifest_id=None status='success' prompt_tokens=17 completion_tokens=19 total_tokens=36 first_token_time=None total_cost=Decimal('0.000037') prompt_cost=Decimal('0.0000085') completion_cost=Decimal('0.0000285') parent_run_ids=[UUID('49ad980e-9add-4dc6-a0f8-61dd6b07236d'), UUID('8ebc2721-07e9-420b-b8aa-44964dd3f229')] trace_id=UUID('49ad980e-9add-4dc6-a0f8-61dd6b07236d') dotted_order='20240702T214251489370Z49ad980e-9add-4dc6-a0f8-61dd6b07236d.20240702T214253722808Z8ebc2721-07e9-420b-b8aa-44964dd3f229.20240702T214253724266Z17a20712-1c15-4073-a386-22007cc5944b' in_dataset=False\n",
      "id=UUID('8d10e3fe-abe2-49e1-964f-17788f0bfba1') name='ChatOpenAI' start_time=datetime.datetime(2024, 7, 2, 21, 42, 53, 34474) run_type='llm' end_time=datetime.datetime(2024, 7, 2, 21, 42, 53, 720327) extra={'invocation_params': {'model': 'gpt-3.5-turbo', 'model_name': 'gpt-3.5-turbo', 'stream': False, 'n': 1, 'temperature': 0.7, '_type': 'openai-chat', 'stop': None}, 'options': {'stop': None}, 'batch_size': 1, 'runtime': {'sdk': 'langsmith-py', 'sdk_version': '0.1.82', 'library': 'langsmith', 'platform': 'macOS-14.6-arm64-arm-64bit', 'runtime': 'python', 'py_implementation': 'CPython', 'runtime_version': '3.11.7', 'langchain_version': '0.1.20', 'langchain_core_version': '0.1.52', 'thread_count': 22.0, 'mem': {'rss': 280526848.0}, 'cpu': {'time': {'sys': 3.72275584, 'user': 8.425246208}, 'ctx_switches': {'voluntary': 127141.0, 'involuntary': 0.0}, 'percent': 0.0}}, 'metadata': {'revision_id': '9bb07c0-dirty', 'system_fingerprint': None}} error=None serialized=None events=[{'name': 'start', 'time': '2024-07-02T21:42:53.034474+00:00'}, {'name': 'end', 'time': '2024-07-02T21:42:53.720327+00:00'}] inputs={'messages': [{'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'HumanMessage'], 'kwargs': {'content': 'What is the answer to What is your age?', 'type': 'human'}}]} outputs={'llm_output': {'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None}, 'run': None, 'generations': [{'text': 'I am a language model AI and do not have an age.', 'generation_info': {'finish_reason': 'stop', 'logprobs': None}, 'type': 'ChatGeneration', 'message': {'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'AIMessage'], 'kwargs': {'content': 'I am a language model AI and do not have an age.', 'response_metadata': {'token_usage': {'completion_tokens': 13, 'prompt_tokens': 17, 'total_tokens': 30}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, 'type': 'ai', 'id': 'run-8d10e3fe-abe2-49e1-964f-17788f0bfba1-0', 'tool_calls': [], 'invalid_tool_calls': []}}}]} reference_example_id=None parent_run_id=UUID('e97d4729-1eb2-47d7-983f-b627103f268f') tags=[] session_id=UUID('dfccefec-106a-431e-9448-ce8c857f96e4') child_run_ids=None child_runs=None feedback_stats=None app_path='/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/projects/p/dfccefec-106a-431e-9448-ce8c857f96e4/r/8d10e3fe-abe2-49e1-964f-17788f0bfba1?trace_id=49ad980e-9add-4dc6-a0f8-61dd6b07236d&start_time=2024-07-02T21:42:51.489370' manifest_id=None status='success' prompt_tokens=17 completion_tokens=13 total_tokens=30 first_token_time=None total_cost=Decimal('0.000028') prompt_cost=Decimal('0.0000085') completion_cost=Decimal('0.0000195') parent_run_ids=[UUID('49ad980e-9add-4dc6-a0f8-61dd6b07236d'), UUID('e97d4729-1eb2-47d7-983f-b627103f268f')] trace_id=UUID('49ad980e-9add-4dc6-a0f8-61dd6b07236d') dotted_order='20240702T214251489370Z49ad980e-9add-4dc6-a0f8-61dd6b07236d.20240702T214253032878Ze97d4729-1eb2-47d7-983f-b627103f268f.20240702T214253034474Z8d10e3fe-abe2-49e1-964f-17788f0bfba1' in_dataset=False\n",
      "id=UUID('bc59a564-1fff-40e5-8a23-7f9f9bf666bc') name='ChatOpenAI' start_time=datetime.datetime(2024, 7, 2, 21, 42, 51, 503085) run_type='llm' end_time=datetime.datetime(2024, 7, 2, 21, 42, 53, 28239) extra={'invocation_params': {'model': 'gpt-3.5-turbo', 'model_name': 'gpt-3.5-turbo', 'stream': False, 'n': 1, 'temperature': 0.7, '_type': 'openai-chat', 'stop': None}, 'options': {'stop': None}, 'batch_size': 1, 'runtime': {'sdk': 'langsmith-py', 'sdk_version': '0.1.82', 'library': 'langsmith', 'platform': 'macOS-14.6-arm64-arm-64bit', 'runtime': 'python', 'py_implementation': 'CPython', 'runtime_version': '3.11.7', 'langchain_version': '0.1.20', 'langchain_core_version': '0.1.52', 'thread_count': 22.0, 'mem': {'rss': 280526848.0}, 'cpu': {'time': {'sys': 3.72275584, 'user': 8.425246208}, 'ctx_switches': {'voluntary': 127141.0, 'involuntary': 0.0}, 'percent': 0.0}}, 'metadata': {'revision_id': '9bb07c0-dirty', 'system_fingerprint': None}} error=None serialized=None events=[{'name': 'start', 'time': '2024-07-02T21:42:51.503085+00:00'}, {'name': 'end', 'time': '2024-07-02T21:42:53.028239+00:00'}] inputs={'messages': [{'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'HumanMessage'], 'kwargs': {'content': 'What is the answer to What is the your name?', 'type': 'human'}}]} outputs={'llm_output': {'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None}, 'run': None, 'generations': [{'text': 'I am a language model assistant created by OpenAI and I do not have a specific name. You can refer to me as Assistant or ChatGPT. How can I assist you today?', 'generation_info': {'finish_reason': 'stop', 'logprobs': None}, 'type': 'ChatGeneration', 'message': {'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'AIMessage'], 'kwargs': {'content': 'I am a language model assistant created by OpenAI and I do not have a specific name. You can refer to me as Assistant or ChatGPT. How can I assist you today?', 'response_metadata': {'token_usage': {'completion_tokens': 38, 'prompt_tokens': 18, 'total_tokens': 56}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, 'type': 'ai', 'id': 'run-bc59a564-1fff-40e5-8a23-7f9f9bf666bc-0', 'tool_calls': [], 'invalid_tool_calls': []}}}]} reference_example_id=None parent_run_id=UUID('9aeb47e5-69f6-4943-9a4f-d95edb119647') tags=[] session_id=UUID('dfccefec-106a-431e-9448-ce8c857f96e4') child_run_ids=None child_runs=None feedback_stats=None app_path='/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/projects/p/dfccefec-106a-431e-9448-ce8c857f96e4/r/bc59a564-1fff-40e5-8a23-7f9f9bf666bc?trace_id=49ad980e-9add-4dc6-a0f8-61dd6b07236d&start_time=2024-07-02T21:42:51.489370' manifest_id=None status='success' prompt_tokens=18 completion_tokens=38 total_tokens=56 first_token_time=None total_cost=Decimal('0.000066') prompt_cost=Decimal('0.000009') completion_cost=Decimal('0.000057') parent_run_ids=[UUID('49ad980e-9add-4dc6-a0f8-61dd6b07236d'), UUID('9aeb47e5-69f6-4943-9a4f-d95edb119647')] trace_id=UUID('49ad980e-9add-4dc6-a0f8-61dd6b07236d') dotted_order='20240702T214251489370Z49ad980e-9add-4dc6-a0f8-61dd6b07236d.20240702T214251502137Z9aeb47e5-69f6-4943-9a4f-d95edb119647.20240702T214251503085Zbc59a564-1fff-40e5-8a23-7f9f9bf666bc' in_dataset=False\n",
      "id=UUID('52cc1d8e-cedc-4d27-9904-df5fa352934b') name='ChatOpenAI' start_time=datetime.datetime(2024, 7, 2, 21, 20, 35, 187839) run_type='llm' end_time=datetime.datetime(2024, 7, 2, 21, 20, 36, 135488) extra={'invocation_params': {'model': 'gpt-3.5-turbo', 'model_name': 'gpt-3.5-turbo', 'stream': False, 'n': 1, 'temperature': 0.0, '_type': 'openai-chat', 'stop': None}, 'options': {'stop': None}, 'batch_size': 1, 'runtime': {'sdk': 'langsmith-py', 'sdk_version': '0.1.82', 'library': 'langsmith', 'platform': 'macOS-14.6-arm64-arm-64bit', 'runtime': 'python', 'py_implementation': 'CPython', 'runtime_version': '3.11.7', 'langchain_version': '0.1.20', 'langchain_core_version': '0.1.52', 'thread_count': 22.0, 'mem': {'rss': 203849728.0}, 'cpu': {'time': {'sys': 3.40529664, 'user': 7.128896512}, 'ctx_switches': {'voluntary': 108198.0, 'involuntary': 0.0}, 'percent': 0.0}}, 'metadata': {'revision_id': '9bb07c0-dirty', 'system_fingerprint': None}} error=None serialized=None events=[{'name': 'start', 'time': '2024-07-02T21:20:35.187839+00:00'}, {'name': 'end', 'time': '2024-07-02T21:20:36.135488+00:00'}] inputs={'messages': [{'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'HumanMessage'], 'kwargs': {'content': 'Hello, World!', 'type': 'human'}}]} outputs={'llm_output': {'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None}, 'run': None, 'generations': [{'text': 'Hello! How can I assist you today?', 'generation_info': {'finish_reason': 'stop', 'logprobs': None}, 'type': 'ChatGeneration', 'message': {'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'AIMessage'], 'kwargs': {'content': 'Hello! How can I assist you today?', 'response_metadata': {'token_usage': {'completion_tokens': 9, 'prompt_tokens': 11, 'total_tokens': 20}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, 'type': 'ai', 'id': 'run-52cc1d8e-cedc-4d27-9904-df5fa352934b-0', 'tool_calls': [], 'invalid_tool_calls': []}}}]} reference_example_id=None parent_run_id=UUID('a8e86722-63d1-4e17-81c0-da377429434f') tags=['shared-tags', 'llm-tag'] session_id=UUID('dfccefec-106a-431e-9448-ce8c857f96e4') child_run_ids=None child_runs=None feedback_stats=None app_path='/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/projects/p/dfccefec-106a-431e-9448-ce8c857f96e4/r/52cc1d8e-cedc-4d27-9904-df5fa352934b?trace_id=a8e86722-63d1-4e17-81c0-da377429434f&start_time=2024-07-02T21:20:35.187267' manifest_id=None status='success' prompt_tokens=11 completion_tokens=9 total_tokens=20 first_token_time=None total_cost=Decimal('0.000019') prompt_cost=Decimal('0.0000055') completion_cost=Decimal('0.0000135') parent_run_ids=[UUID('a8e86722-63d1-4e17-81c0-da377429434f')] trace_id=UUID('a8e86722-63d1-4e17-81c0-da377429434f') dotted_order='20240702T212035187267Za8e86722-63d1-4e17-81c0-da377429434f.20240702T212035187839Z52cc1d8e-cedc-4d27-9904-df5fa352934b' in_dataset=False\n",
      "id=UUID('5e0b3b7e-70cd-4dd2-bbf2-6b026546df3f') name='ChatOpenAI' start_time=datetime.datetime(2024, 7, 2, 21, 20, 15, 792159) run_type='llm' end_time=datetime.datetime(2024, 7, 2, 21, 20, 18, 435683) extra={'invocation_params': {'model': 'gpt-3.5-turbo', 'model_name': 'gpt-3.5-turbo', 'stream': False, 'n': 1, 'temperature': 0.7, '_type': 'openai-chat', 'stop': None}, 'options': {'stop': None}, 'batch_size': 1, 'runtime': {'sdk': 'langsmith-py', 'sdk_version': '0.1.82', 'library': 'langsmith', 'platform': 'macOS-14.6-arm64-arm-64bit', 'runtime': 'python', 'py_implementation': 'CPython', 'runtime_version': '3.11.7', 'langchain_version': '0.1.20', 'langchain_core_version': '0.1.52', 'thread_count': 22.0, 'mem': {'rss': 202801152.0}, 'cpu': {'time': {'sys': 3.393734656, 'user': 7.04520192}, 'ctx_switches': {'voluntary': 107474.0, 'involuntary': 0.0}, 'percent': 0.0}}, 'metadata': {'revision_id': '9bb07c0-dirty', 'system_fingerprint': None}} error=None serialized=None events=[{'name': 'start', 'time': '2024-07-02T21:20:15.792159+00:00'}, {'name': 'end', 'time': '2024-07-02T21:20:18.435683+00:00'}] inputs={'messages': [{'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'HumanMessage'], 'kwargs': {'content': 'How is the weather in UAE?', 'type': 'human'}}]} outputs={'llm_output': {'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None}, 'run': None, 'generations': [{'text': 'The weather in the UAE can vary depending on the time of year. In general, the UAE has a hot desert climate with high temperatures year-round. The summers are extremely hot, with temperatures often exceeding 40 degrees Celsius (104 degrees Fahrenheit). The winters are milder, with temperatures ranging from 15-25 degrees Celsius (59-77 degrees Fahrenheit). Rainfall is minimal and mostly occurs during the winter months.\\n\\nOverall, the weather in the UAE is characterized by sunny skies, low humidity, and little precipitation, making it a popular destination for those seeking warm weather and sunshine.', 'generation_info': {'finish_reason': 'stop', 'logprobs': None}, 'type': 'ChatGeneration', 'message': {'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'AIMessage'], 'kwargs': {'content': 'The weather in the UAE can vary depending on the time of year. In general, the UAE has a hot desert climate with high temperatures year-round. The summers are extremely hot, with temperatures often exceeding 40 degrees Celsius (104 degrees Fahrenheit). The winters are milder, with temperatures ranging from 15-25 degrees Celsius (59-77 degrees Fahrenheit). Rainfall is minimal and mostly occurs during the winter months.\\n\\nOverall, the weather in the UAE is characterized by sunny skies, low humidity, and little precipitation, making it a popular destination for those seeking warm weather and sunshine.', 'response_metadata': {'token_usage': {'completion_tokens': 117, 'prompt_tokens': 14, 'total_tokens': 131}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, 'type': 'ai', 'id': 'run-5e0b3b7e-70cd-4dd2-bbf2-6b026546df3f-0', 'tool_calls': [], 'invalid_tool_calls': []}}}]} reference_example_id=None parent_run_id=None tags=[] session_id=UUID('dfccefec-106a-431e-9448-ce8c857f96e4') child_run_ids=None child_runs=None feedback_stats=None app_path='/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/projects/p/dfccefec-106a-431e-9448-ce8c857f96e4/r/5e0b3b7e-70cd-4dd2-bbf2-6b026546df3f?trace_id=5e0b3b7e-70cd-4dd2-bbf2-6b026546df3f&start_time=2024-07-02T21:20:15.792159' manifest_id=None status='success' prompt_tokens=14 completion_tokens=117 total_tokens=131 first_token_time=None total_cost=Decimal('0.0001825') prompt_cost=Decimal('0.000007') completion_cost=Decimal('0.0001755') parent_run_ids=[] trace_id=UUID('5e0b3b7e-70cd-4dd2-bbf2-6b026546df3f') dotted_order='20240702T212015792159Z5e0b3b7e-70cd-4dd2-bbf2-6b026546df3f' in_dataset=False\n",
      "id=UUID('8230e0cf-04dd-4220-9dd0-f61b88f0fba5') name='ChatOpenAI' start_time=datetime.datetime(2024, 7, 2, 21, 19, 46, 850284) run_type='llm' end_time=datetime.datetime(2024, 7, 2, 21, 19, 49, 71566) extra={'invocation_params': {'model': 'gpt-3.5-turbo', 'model_name': 'gpt-3.5-turbo', 'stream': False, 'n': 1, 'temperature': 0.7, '_type': 'openai-chat', 'stop': None}, 'options': {'stop': None}, 'batch_size': 1, 'runtime': {'sdk': 'langsmith-py', 'sdk_version': '0.1.82', 'library': 'langsmith', 'platform': 'macOS-14.6-arm64-arm-64bit', 'runtime': 'python', 'py_implementation': 'CPython', 'runtime_version': '3.11.7', 'langchain_version': '0.1.20', 'langchain_core_version': '0.1.52', 'thread_count': 22.0, 'mem': {'rss': 200474624.0}, 'cpu': {'time': {'sys': 3.380490496, 'user': 6.963059712}, 'ctx_switches': {'voluntary': 106831.0, 'involuntary': 0.0}, 'percent': 0.0}}, 'metadata': {'revision_id': '9bb07c0-dirty', 'system_fingerprint': None}} error=None serialized=None events=[{'name': 'start', 'time': '2024-07-02T21:19:46.850284+00:00'}, {'name': 'end', 'time': '2024-07-02T21:19:49.071566+00:00'}] inputs={'messages': [{'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'HumanMessage'], 'kwargs': {'content': 'How is the weather in UAE?', 'type': 'human'}}]} outputs={'llm_output': {'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None}, 'run': None, 'generations': [{'text': 'The weather in the UAE is typically hot and dry, with temperatures often exceeding 40°C (104°F) during the summer months. The country experiences very little rainfall, especially in the desert regions, and humidity levels can be quite high, particularly along the coast. Winter temperatures are more moderate, ranging from around 15-30°C (59-86°F). It is important to stay hydrated and protect yourself from the sun when visiting or living in the UAE.', 'generation_info': {'finish_reason': 'stop', 'logprobs': None}, 'type': 'ChatGeneration', 'message': {'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'AIMessage'], 'kwargs': {'content': 'The weather in the UAE is typically hot and dry, with temperatures often exceeding 40°C (104°F) during the summer months. The country experiences very little rainfall, especially in the desert regions, and humidity levels can be quite high, particularly along the coast. Winter temperatures are more moderate, ranging from around 15-30°C (59-86°F). It is important to stay hydrated and protect yourself from the sun when visiting or living in the UAE.', 'response_metadata': {'token_usage': {'completion_tokens': 93, 'prompt_tokens': 14, 'total_tokens': 107}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, 'type': 'ai', 'id': 'run-8230e0cf-04dd-4220-9dd0-f61b88f0fba5-0', 'tool_calls': [], 'invalid_tool_calls': []}}}]} reference_example_id=None parent_run_id=None tags=[] session_id=UUID('dfccefec-106a-431e-9448-ce8c857f96e4') child_run_ids=None child_runs=None feedback_stats=None app_path='/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/projects/p/dfccefec-106a-431e-9448-ce8c857f96e4/r/8230e0cf-04dd-4220-9dd0-f61b88f0fba5?trace_id=8230e0cf-04dd-4220-9dd0-f61b88f0fba5&start_time=2024-07-02T21:19:46.850284' manifest_id=None status='success' prompt_tokens=14 completion_tokens=93 total_tokens=107 first_token_time=None total_cost=Decimal('0.0001465') prompt_cost=Decimal('0.000007') completion_cost=Decimal('0.0001395') parent_run_ids=[] trace_id=UUID('8230e0cf-04dd-4220-9dd0-f61b88f0fba5') dotted_order='20240702T211946850284Z8230e0cf-04dd-4220-9dd0-f61b88f0fba5' in_dataset=False\n"
     ]
    }
   ],
   "source": [
    "for run in today_runs:\n",
    "    print(run)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e7d0114-ac45-4067-8209-7dda71dddade",
   "metadata": {},
   "source": [
    "### Using Metadata in LangSmith"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "8135330c-49f6-4956-93da-34d1de9f39e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input': 'What is the capital of France',\n",
       " 'text': 'The capital of France is Paris.'}"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.chains import LLMChain\n",
    "\n",
    "chat_model=ChatOpenAI()\n",
    "chain = LLMChain.from_string(llm= chat_model, template=\"What is the answer to {input}\")\n",
    "\n",
    "chain({\"input\":\"What is the capital of France\"}, metadata={\"source\":\"interent\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "fb0ce68e-d3dd-4edd-89cb-2a008b4a8d0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Run(id=UUID('e9723d54-9109-4344-840e-c584e396bbb2'), name='ChatOpenAI', start_time=datetime.datetime(2024, 7, 2, 22, 41, 46, 967724), run_type='llm', end_time=datetime.datetime(2024, 7, 2, 22, 41, 48, 117410), extra={'invocation_params': {'model': 'gpt-3.5-turbo', 'model_name': 'gpt-3.5-turbo', 'stream': False, 'n': 1, 'temperature': 0.7, '_type': 'openai-chat', 'stop': None}, 'options': {'stop': None}, 'batch_size': 1, 'metadata': {'source': 'interent', 'revision_id': '9bb07c0-dirty', 'system_fingerprint': None}, 'runtime': {'sdk': 'langsmith-py', 'sdk_version': '0.1.82', 'library': 'langsmith', 'platform': 'macOS-14.6-arm64-arm-64bit', 'runtime': 'python', 'py_implementation': 'CPython', 'runtime_version': '3.11.7', 'langchain_version': '0.1.20', 'langchain_core_version': '0.1.52', 'thread_count': 25.0, 'mem': {'rss': 304906240.0}, 'cpu': {'time': {'sys': 4.72084736, 'user': 12.724870144}, 'ctx_switches': {'voluntary': 206428.0, 'involuntary': 0.0}, 'percent': 0.0}}}, error=None, serialized=None, events=[{'name': 'start', 'time': '2024-07-02T22:41:46.967724+00:00'}, {'name': 'end', 'time': '2024-07-02T22:41:48.117410+00:00'}], inputs={'messages': [{'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'HumanMessage'], 'kwargs': {'content': 'What is the answer to What is the capital of France', 'type': 'human'}}]}, outputs={'llm_output': {'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None}, 'run': None, 'generations': [{'text': 'The capital of France is Paris.', 'generation_info': {'finish_reason': 'stop', 'logprobs': None}, 'type': 'ChatGeneration', 'message': {'lc': 1, 'type': 'constructor', 'id': ['langchain', 'schema', 'messages', 'AIMessage'], 'kwargs': {'content': 'The capital of France is Paris.', 'response_metadata': {'token_usage': {'completion_tokens': 7, 'prompt_tokens': 18, 'total_tokens': 25}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, 'type': 'ai', 'id': 'run-e9723d54-9109-4344-840e-c584e396bbb2-0', 'tool_calls': [], 'invalid_tool_calls': []}}}]}, reference_example_id=None, parent_run_id=UUID('8e9632cd-70cd-459a-9735-d886d7e9f8a8'), tags=[], session_id=UUID('dfccefec-106a-431e-9448-ce8c857f96e4'), child_run_ids=None, child_runs=None, feedback_stats=None, app_path='/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/projects/p/dfccefec-106a-431e-9448-ce8c857f96e4/r/e9723d54-9109-4344-840e-c584e396bbb2?trace_id=8e9632cd-70cd-459a-9735-d886d7e9f8a8&start_time=2024-07-02T22:41:46.966922', manifest_id=None, status='success', prompt_tokens=18, completion_tokens=7, total_tokens=25, first_token_time=None, total_cost=Decimal('0.0000195'), prompt_cost=Decimal('0.000009'), completion_cost=Decimal('0.0000105'), parent_run_ids=[UUID('8e9632cd-70cd-459a-9735-d886d7e9f8a8')], trace_id=UUID('8e9632cd-70cd-459a-9735-d886d7e9f8a8'), dotted_order='20240702T224146966922Z8e9632cd-70cd-459a-9735-d886d7e9f8a8.20240702T224146967724Ze9723d54-9109-4344-840e-c584e396bbb2', in_dataset=False), Run(id=UUID('8e9632cd-70cd-459a-9735-d886d7e9f8a8'), name='LLMChain', start_time=datetime.datetime(2024, 7, 2, 22, 41, 46, 966922), run_type='chain', end_time=datetime.datetime(2024, 7, 2, 22, 41, 48, 117822), extra={'metadata': {'source': 'interent', 'revision_id': '9bb07c0-dirty'}, 'runtime': {'sdk': 'langsmith-py', 'sdk_version': '0.1.82', 'library': 'langsmith', 'platform': 'macOS-14.6-arm64-arm-64bit', 'runtime': 'python', 'py_implementation': 'CPython', 'runtime_version': '3.11.7', 'langchain_version': '0.1.20', 'langchain_core_version': '0.1.52', 'thread_count': 25.0, 'mem': {'rss': 304906240.0}, 'cpu': {'time': {'sys': 4.72084736, 'user': 12.724870144}, 'ctx_switches': {'voluntary': 206428.0, 'involuntary': 0.0}, 'percent': 0.0}}}, error=None, serialized=None, events=[{'name': 'start', 'time': '2024-07-02T22:41:46.966922+00:00'}, {'name': 'end', 'time': '2024-07-02T22:41:48.117822+00:00'}], inputs={'input': 'What is the capital of France'}, outputs={'text': 'The capital of France is Paris.'}, reference_example_id=None, parent_run_id=None, tags=[], session_id=UUID('dfccefec-106a-431e-9448-ce8c857f96e4'), child_run_ids=[UUID('e9723d54-9109-4344-840e-c584e396bbb2')], child_runs=None, feedback_stats=None, app_path='/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/projects/p/dfccefec-106a-431e-9448-ce8c857f96e4/r/8e9632cd-70cd-459a-9735-d886d7e9f8a8?trace_id=8e9632cd-70cd-459a-9735-d886d7e9f8a8&start_time=2024-07-02T22:41:46.966922', manifest_id=None, status='success', prompt_tokens=18, completion_tokens=7, total_tokens=25, first_token_time=None, total_cost=Decimal('0.0000195'), prompt_cost=Decimal('0.000009'), completion_cost=Decimal('0.0000105'), parent_run_ids=[], trace_id=UUID('8e9632cd-70cd-459a-9735-d886d7e9f8a8'), dotted_order='20240702T224146966922Z8e9632cd-70cd-459a-9735-d886d7e9f8a8', in_dataset=False)]\n"
     ]
    }
   ],
   "source": [
    "from langsmith import Client\n",
    "\n",
    "client = Client()\n",
    "\n",
    "runs = list(client.list_runs(\n",
    "    project_name=\"default\",\n",
    "    filter='has(metadata, \\'{\"source\":\"interent\"}\\')',\n",
    "))\n",
    "\n",
    "print(list(runs))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c02bded-e036-40a9-aa06-5859db22156c",
   "metadata": {},
   "source": [
    "### Dataset and Evaluation in LangSmith"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "abc95cae-192b-425d-99fb-857ecd652c61",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import Client\n",
    "\n",
    "client = Client()\n",
    "\n",
    "dataset = client.upload_csv(\n",
    "    csv_file=\"./Files/dataset.csv\",\n",
    "    input_keys=[\"Question\"],\n",
    "    output_keys=[\"Answer\"],\n",
    "    name=\"First CSV Dataset\",\n",
    "    description=\"Dataset with 5 questions and answers in CSV format\",\n",
    "    data_type=\"kv\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "636b9d07-e984-4b67-b1ac-645e8ab0c2d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langsmith import Client\n",
    "from langchain.smith import RunEvalConfig, run_on_dataset\n",
    "\n",
    "evaluation_config = RunEvalConfig(\n",
    "    evaluators=[\n",
    "        \"qa\",\n",
    "        \"context_qa\",\n",
    "        \"cot_qa\"\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "0a9fca54-743f-49f1-afcf-891f0ce4cb4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "View the evaluation results for project 'Evaluation Project' at:\n",
      "https://smith.langchain.com/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/datasets/9c904d45-e5bd-43f2-bba1-ffa05329f517/compare?selectedSessions=f41c9566-5ab6-4ef1-9c7b-b072f690a337\n",
      "\n",
      "View all tests for Dataset First CSV Dataset at:\n",
      "https://smith.langchain.com/o/b1c74b64-69f9-5c8f-975c-ba13560c50a8/datasets/9c904d45-e5bd-43f2-bba1-ffa05329f517\n",
      "[------------------------------------------------->] 5/5"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'project_name': 'Evaluation Project',\n",
       " 'results': {'01ad7ef2-5050-48a4-9528-4a11724d3d29': {'input': {'Question': 'What is the key difference between North Indian and South Indian cuisine?'},\n",
       "   'feedback': [EvaluationResult(key='correctness', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('82d0d255-9a04-4cfc-ab9a-19a9439b9b03'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='Contextual Accuracy', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('fe973884-cb01-4526-8f17-5da93a94f562'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment=\"The student's answer is in line with the context provided. The student correctly identifies that North Indian cuisine features more wheat-based dishes and richer gravies, and that South Indian cuisine uses more rice and incorporates coconut and tamarind for flavor. The student also provides additional accurate information about the cuisines that does not conflict with the context. \\nGRADE: CORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('2e0ac65a-fc29-4c12-b648-58e37beba2ff'))}, feedback_config=None, source_run_id=None, target_run_id=None)],\n",
       "   'execution_time': 4.097788,\n",
       "   'run_id': '558314f2-0243-4c33-b6d8-918612def77e',\n",
       "   'output': AIMessage(content='The key difference between North Indian and South Indian cuisine lies in the use of ingredients, spices, and cooking techniques. \\n\\nNorth Indian cuisine is characterized by the use of dairy products such as ghee, milk, paneer, and yogurt. It also features a wide variety of breads like roti, naan, and paratha. The cuisine is known for its rich and creamy curries, tandoori dishes, and kebabs. North Indian food is typically more spicy and uses a variety of spices like cumin, coriander, and garam masala.\\n\\nOn the other hand, South Indian cuisine is known for its more vegetarian-friendly dishes, with a focus on rice, lentils, and coconut. Coconut oil and coconut milk are commonly used in cooking. South Indian food is known for its use of tangy and spicy flavors, with dishes like dosas, idlis, sambhar, and rasam being popular. South Indian cuisine also features a variety of chutneys and pickles made from local ingredients like tamarind and curry leaves.\\n\\nOverall, North Indian cuisine tends to be richer and heavier, while South Indian cuisine is lighter and more flavorful.', response_metadata={'token_usage': {'completion_tokens': 239, 'prompt_tokens': 20, 'total_tokens': 259}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-558314f2-0243-4c33-b6d8-918612def77e-0'),\n",
       "   'reference': {'Answer': 'North Indian cuisine features more wheat-based dishes (breads like naan and roti) and richer gravies, while South Indian cuisine uses more rice and incorporates coconut and tamarind for flavor.'}},\n",
       "  '328ce7d4-2051-4399-ae87-2f802be447c4': {'input': {'Question': 'What is the main ingredient in Middle Eastern food?'},\n",
       "   'feedback': [EvaluationResult(key='correctness', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('a29a386a-df25-4b8c-abf9-58b8a781a6f4'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='Contextual Accuracy', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('b5853351-e99f-48b7-8c6e-795899ca4553'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment=\"The student's answer includes rice as a main ingredient in Middle Eastern food, which aligns with the context provided. Although the student also mentions other ingredients like bulgur, couscous, meat, vegetables, and various spices, these do not conflict with the context. Therefore, the student's answer is factually accurate.\\nGRADE: CORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('c181bd9d-0d49-4bff-a5a3-0b9f71900669'))}, feedback_config=None, source_run_id=None, target_run_id=None)],\n",
       "   'execution_time': 1.494791,\n",
       "   'run_id': 'b1f5b097-f1eb-4d53-8df8-25488ac638d1',\n",
       "   'output': AIMessage(content='The main ingredient in Middle Eastern food is typically grains, such as rice, bulgur, or couscous. Meat, vegetables, and various spices are also commonly used in Middle Eastern cuisine.', response_metadata={'token_usage': {'completion_tokens': 40, 'prompt_tokens': 17, 'total_tokens': 57}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-b1f5b097-f1eb-4d53-8df8-25488ac638d1-0'),\n",
       "   'reference': {'Answer': 'Rice'}},\n",
       "  '01273b4a-3278-4761-943d-9435270f85c5': {'input': {'Question': 'What are the main ingredients in tiramisu?'},\n",
       "   'feedback': [EvaluationResult(key='correctness', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('925e5b86-69de-425a-82f1-9a34fc55b3bf'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='Contextual Accuracy', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('cc6606b4-991f-4580-8670-3f11a0bc0858'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment=\"The student's answer includes all the main ingredients mentioned in the context: ladyfingers (savoiardi), coffee, mascarpone cheese, eggs, sugar, and cocoa powder. The student also mentions that sometimes liquor such as Marsala wine or rum is used. This additional information does not conflict with the context, as it does not state that these ingredients are never used. Therefore, the student's answer is factually accurate.\\nGRADE: CORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('b194341b-8c54-4145-8ae2-f3914a510a8f'))}, feedback_config=None, source_run_id=None, target_run_id=None)],\n",
       "   'execution_time': 1.524901,\n",
       "   'run_id': '281c9dc6-8228-4fe1-97c8-bef7ea7c9284',\n",
       "   'output': AIMessage(content='The main ingredients in tiramisu are ladyfinger biscuits (savoiardi), espresso coffee, mascarpone cheese, eggs, sugar, cocoa powder, and sometimes liquor such as Marsala wine or rum.', response_metadata={'token_usage': {'completion_tokens': 42, 'prompt_tokens': 17, 'total_tokens': 59}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-281c9dc6-8228-4fe1-97c8-bef7ea7c9284-0'),\n",
       "   'reference': {'Answer': 'Ladyfingers (savoiardi), coffee, mascarpone cheese, eggs, sugar, and cocoa powder.'}},\n",
       "  '54db61ba-20d9-48fb-8bef-b6caec4e85e7': {'input': {'Question': 'What are the main ingredients in a traditional Peking duck dish?'},\n",
       "   'feedback': [EvaluationResult(key='correctness', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('ae0893ad-721c-430b-b7ff-00690137f6f7'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='Contextual Accuracy', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('a01d08e2-670b-4976-bed3-85082b3191e8'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment=\"The student's answer includes all the ingredients mentioned in the context: roasted duck, thin pancakes, hoisin sauce, scallions, and cucumbers. The student also mentions steamed buns and sugar for the crispy skin, which are not mentioned in the context, but these do not conflict with the context. Therefore, the student's answer is correct.\\nGRADE: CORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('9c0861e8-31c7-41f6-9266-a532a6d47819'))}, feedback_config=None, source_run_id=None, target_run_id=None)],\n",
       "   'execution_time': 1.499215,\n",
       "   'run_id': '1ea8a0f2-d77f-4b8d-8676-5a6d796ceb57',\n",
       "   'output': AIMessage(content='The main ingredients in a traditional Peking duck dish are duck, hoisin sauce, scallions, cucumbers, pancakes or steamed buns, and sometimes sugar for the crispy skin.', response_metadata={'token_usage': {'completion_tokens': 40, 'prompt_tokens': 20, 'total_tokens': 60}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-1ea8a0f2-d77f-4b8d-8676-5a6d796ceb57-0'),\n",
       "   'reference': {'Answer': 'Roasted duck, thin pancakes, hoisin sauce, scallions, and cucumbers.'}},\n",
       "  '7090bcd9-04fc-4932-bac9-245322889ff8': {'input': {'Question': ' Explain the difference between \"foie gras\" and \"pâté.\"'},\n",
       "   'feedback': [EvaluationResult(key='correctness', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('c7a4fd27-f990-4b45-ab8b-9dea0dc14464'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='Contextual Accuracy', score=1, value='CORRECT', comment='CORRECT', correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('d7d7e6a8-edd3-4206-b11a-faf69f55a1cd'))}, feedback_config=None, source_run_id=None, target_run_id=None),\n",
       "    EvaluationResult(key='COT Contextual Accuracy', score=1, value='CORRECT', comment=\"The student's answer correctly identifies that foie gras is made from the fattened liver of a duck or goose, and that pâté is a spreadable paste made from meat or seafood. The student also correctly explains the differences in preparation and texture between the two. The student's answer does not conflict with the context and provides additional accurate information.\\nGRADE: CORRECT\", correction=None, evaluator_info={'__run': RunInfo(run_id=UUID('ca1afb37-88cf-4407-940b-1c9e6494b9ea'))}, feedback_config=None, source_run_id=None, target_run_id=None)],\n",
       "   'execution_time': 3.579432,\n",
       "   'run_id': '020a0788-9fee-4b18-80a9-b961b8987e20',\n",
       "   'output': AIMessage(content=\"Foie gras and pâté are both French delicacies made from liver, but they are different in terms of their preparation and texture.\\n\\nFoie gras is made from the liver of a duck or goose that has been specially fattened through force-feeding. This results in a rich, buttery texture and a unique flavor that is highly prized in French cuisine. Foie gras is typically served in slices or whole pieces and is often enjoyed as a luxurious appetizer or as part of a gourmet dish.\\n\\nPâté, on the other hand, is a spreadable paste made from finely ground liver (usually pork, chicken, or duck), along with other ingredients such as herbs, spices, and sometimes alcohol. Pâté has a smoother texture than foie gras and is often served on toast or crackers as an hors d'oeuvre or as a sandwich filling. Pâté is more accessible and affordable than foie gras, making it a popular choice for everyday meals and casual entertaining.\", response_metadata={'token_usage': {'completion_tokens': 200, 'prompt_tokens': 22, 'total_tokens': 222}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': None, 'finish_reason': 'stop', 'logprobs': None}, id='run-020a0788-9fee-4b18-80a9-b961b8987e20-0'),\n",
       "   'reference': {'Answer': 'Foie gras is a luxury food product made from the fattened liver of a duck or goose, while pâté is a spreadable paste made from meat (often liver) or seafood.'}}},\n",
       " 'aggregate_metrics': None}"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = Client()\n",
    "llm = ChatOpenAI()\n",
    "run_on_dataset(\n",
    "    dataset_name=\"First CSV Dataset\",\n",
    "    llm_or_chain_factory=llm,\n",
    "    client=client,\n",
    "    evaluation=evaluation_config,\n",
    "    project_name=\"Evaluation Project\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e642853e-1ddf-488f-92d1-37c998914a2d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e80275f7-f1a7-46a5-8ea3-6670a345c6f9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40597227-2088-41a9-a9ee-6ec68867514d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "961f4069-a14f-472b-8890-1bad55140316",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "575b904b-916c-4e8a-a94e-429daaa0c6d8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f4f5b3b-2862-4a1a-bfde-7ab58f90501c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de8f0c99-44a2-4697-a082-b20468a2e675",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "051b369b-b424-4a88-953a-80e6f460ab08",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01a56dff-5bfc-4742-86cb-8e8cf38cbc63",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64a70da3-779f-410e-8378-73a7e7ff29e9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26cb1b5f-ba61-4f00-b533-d2d7cc2ee4d6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3dffc6d3-69c0-47d8-b2f4-205e70f7ac0c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e10c5af6-0d22-4d81-aef5-002f7d88a436",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e52c3bb-ba4e-4449-89a6-058646fce11b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "438bb152-2589-4237-a4f5-e9fe98cf3bea",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}