Add retriever evaluation tutorial and update question generation note…

…book (mlflow#10419)

Signed-off-by: Liang Zhang <[email protected]>
Signed-off-by: Sunish Sheth <[email protected]>
Co-authored-by: Sunish Sheth <[email protected]>

dev/mlflow-typo.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
 
-if grep -nP '(?<!import\s)\bM(lf|LF|lF)low\b(?!\()' "$@"; then
+if grep -InP '(?<!import\s)\bM(lf|LF|lF)low\b(?!\()' "$@"; then
     exit 1
 else
     exit 0

docs/source/index.rst

@@ -145,10 +145,10 @@ LLM Guides and Tutorials
             <div class="simple-card">
                 <a href="llms/rag/index.html" >
                     <div class="header">
-                        Question Generation for RAG
+                        Evaluation for RAG
                     </div>
                     <p>
-                        Learn how to leverage LLMs to generate a question dataset for use in Retrieval Augmented Generation applications.
+                        Learn how to evaluate Retrieval Augmented Generation applications by leveraging LLMs to generate a evaluation dataset and evaluate it using the built-in metrics in the MLflow Evaluate API.
                     </p>
                 </a>
             </div>

docs/source/llms/index.rst

@@ -411,10 +411,10 @@ Note that there are additional tutorials within the `Native Integration Guides a
             <div class="simple-card">
                 <a href="rag/index.html" >
                     <div class="header">
-                        Question Generation for RAG
+                        Evaluation for RAG
                     </div>
                     <p>
-                        Learn how to leverage LLMs to generate a question dataset for use in Retrieval Augmented Generation applications.
+                        Learn how to evaluate Retrieval Augmented Generation applications by leveraging LLMs to generate a evaluation dataset and evaluate it using the built-in metrics in the MLflow Evaluate API.
                     </p>
                 </a>
             </div>

docs/source/llms/rag/index.rst

@@ -41,7 +41,7 @@ Explore the Tutorial
 
 .. raw:: html
 
-    <a href="notebooks/index.html" class="download-btn">View the RAG Question Generation Tutorial</a><br/>
+    <a href="notebooks/index.html" class="download-btn">View RAG Tutorials</a><br/>
 
 .. toctree::
     :maxdepth: 1

docs/source/llms/rag/notebooks/index.rst

@@ -1,23 +1,28 @@
-============================================
-Question Generation for Retrieval Evaluation
-============================================
+=============
+RAG Tutorials
+=============
 
-This notebook is a step-by-step tutorial on how to generate a question dataset with 
-LLMs for retrieval evaluation within RAG. It will guide you through getting a document dataset,
-generating diverse and relevant questions through prompt engineering on LLMs, and analyzing the 
-question dataset. The question dataset can then be used for the subsequent task of evaluating the 
-retriever model, which is a part of RAG that collects and ranks relevant document chunks based on
-the user's question.
+You can find a list of tutorials for RAG below. These tutorials are designed to help you
+get started with RAG evaluation and walk you through a concrete example of how to evaluate
+a RAG application that answers questions about MLflow documentation.
 
 .. toctree::
     :maxdepth: 1
     :hidden:
 
     question-generation-retrieval-evaluation.ipynb
+    retriever-evaluation-tutorial.ipynb
 
-Question Generation for RAG Notebook
+Question Generation for RAG Tutorial
 ------------------------------------
 
+This notebook is a step-by-step tutorial on how to generate a question dataset with 
+LLMs for retrieval evaluation within RAG. It will guide you through getting a document dataset,
+generating relevant questions through prompt engineering on LLMs, and analyzing the 
+question dataset. The question dataset can then be used for the subsequent task of evaluating the 
+retriever model, which is a part of RAG that collects and ranks relevant document chunks based on
+the user's question.
+
 If you would like a copy of this notebook to execute in your environment, download the notebook here:
 
 .. raw:: html
@@ -29,3 +34,28 @@ To follow along and see the sections of the notebook guide, click below:
 .. raw:: html
 
     <a href="question-generation-retrieval-evaluation.html" class="download-btn">View the Notebook</a><br/>
+
+
+Retriever Evaluation Tutorial
+-----------------------------
+
+This tutorial walks you through a concrete example of how to build and evaluate
+a RAG application that answers questions about MLflow documentation.
+
+In this tutorial you will learn:
+
+- How to prepare an evaluation dataset for your RAG application.
+- How to call your retriever in the MLflow evaluate API.
+- How to evaluate a retriever's capacity for retrieving relevant documents based on a series of queries using MLflow evaluate.
+
+If you would like a copy of this notebook to execute in your environment, download the notebook here:
+
+.. raw:: html
+
+    <a href="https://raw.githubusercontent.com/mlflow/mlflow/master/docs/source/llms/rag/notebooks/retriever-evaluation-tutorial.ipynb" class="notebook-download-btn">Download the notebook</a><br/>
+
+To follow along and see the sections of the notebook guide, click below:
+
+.. raw:: html
+
+    <a href="retriever-evaluation-tutorial.html" class="download-btn">View the Notebook</a><br/>