information_retrieval.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eddea025-fb05-4d45-8fde-5060a8b6b388",
   "metadata": {},
   "source": [
    "# Information retrieval\n",
    "\n",
    "Information Retrieval (IR) is a Natural Language Processing (NLP) task, in which the objective is to retrieve relevant information within a corpus of text, based on a query. For this purpose, text encoder models are trained to represent texts using `embeddings` (i.e., vectors of numbers), aiming to correctly represent the meaning of the text. Finally, the model can also encode the query, and hopefully, the query embedding will be closer to text embeddings that are meaningful to answer the query. \n",
    "\n",
    "This notebooks show how to perform Information Retrieval to retrieve relevant spans of texts within a set of documents. This will be divided into 3 different sections : \n",
    "1. **Document processing** : the 1st step is to extract spans of texts from all the desired documents.\n",
    "2. **Text encoding** : the 2nd step is to encode all the spans of texts using an appropriate embedding model. For this demonstration, we will use the [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) model,\n",
    "3. **Search query** : finally, the last step is to encode the query by using the same model, and retrieve the spans with the lowest distance (or higher similarity) with the query embedding. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cf8ec35-3787-4ecc-9cfd-ec1eb8436af5",
   "metadata": {},
   "source": [
    "## 1. Document processing\n",
    "\n",
    "The document processing step aims to extract texts from documents. The `parse_document` method accepts filenames, directories and filename format (as below), and returns a list of paragraphs. This is more convenient compared to full text extraction for futher processing, like encoding the texts ;) The method currently accepts `.txt`, `.md`, `.pdf` and `.docs` file formats, and more will be added in the future !\n",
    "\n",
    "For this demonstration, I will use the `README` files from all my github repositories. This will also be easier to evaluate the relevance of the retrieved documents ! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c852972-e22b-43bf-ad3f-34385d35334a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "from utils.text import parse_document\n",
    "\n",
    "documents = parse_document('../**/README.md')\n",
    "documents = pd.DataFrame(documents)\n",
    "print('# texts : {}'.format(len(documents)))\n",
    "documents.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5935181-3802-4327-accc-17c0e7ef0417",
   "metadata": {},
   "source": [
    "## 2. Text encoding\n",
    "\n",
    "Now that we have all the texts extracted with additional information (like section title / filename), we can encode them using embeddings ! For this purpose, let's initialize a `TextEncoder` model with the [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) model. Then, we can use this model to encode the texts by using the `embed` method. \n",
    "\n",
    "`embed` is a batched function, meaning that you can provide the `batch_size` argument to control the number of texts to encode in parallel. An important aspect to consider is that texts are padded when passing in parallel, in order to form a rectangular matrix (i.e., the smaller texts have zero-values at the end so that all texts within a batch have the same length). The function has been optimized by sorting the texts by length, in order to minimize padding. However, it remains interesting to correctly tune the `batch_size`, as it has a large impact on performances ! My recommandation would be to use a small value, around 8. \n",
    "\n",
    "The model is compiled using `XLA` by default, which explains why some calls are slower than subsequent ones, due to retracing. \n",
    "\n",
    "At the 1st call, the official `transformers` model will be downloaded and converted to my `keras` implementation of the `XLMRoberta` architecture. For this purpose, you will need the `torch` library to be installed. Once done, the model will be saved in regular keras format under the `pretrained_models/{name}` folder for subsequent loading. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b314a0b8-7b17-4a88-99b2-ba44ae7a8e73",
   "metadata": {},
   "outputs": [],
   "source": [
    "from models.encoder.text_encoder import TextEncoder\n",
    "\n",
    "model = TextEncoder(pretrained = 'BAAI/bge-m3', name = 'bge-m3')\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0f5e2c91-26c8-43da-9f8c-4d3bc3b288b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████| 16/16 [00:01<00:00,  9.63it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========== DenseVectors ==========\n",
      "- # data    : 127\n",
      "- Dimension : 1024\n",
      "- Columns (primary ('filename', 'text')) : ('section_titles', 'filename', 'text', 'chunks', 'type', 'section')\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "vectors = model.predict(\n",
    "    documents,\n",
    "    batch_size = 8,\n",
    "\n",
    "    save         = False,\n",
    "    chunk_size   = 256,\n",
    "    group_by    = ('filename', 'section_titles'),\n",
    "    primary_key = ('filename', 'text'),\n",
    "    missmatch_mode = 'ignore',\n",
    "    \n",
    "    tqdm = tqdm,\n",
    ")\n",
    "print(vectors)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bc93ccc-1eee-4e36-a9a9-6be7a06eb2aa",
   "metadata": {},
   "source": [
    "## 3. Search query\n",
    "\n",
    "The final step is to encode the query, then compute the `cosine similarity` (or any other distance/similarity metric) between the embedded query and all embedded texts, and take the top-k with the best score ! All these steps are performed internally by the `search` method of the `DenseVectors` class ;)\n",
    "\n",
    "We can observe that the best results are correctly related to `embeddings`, and even more, the best retrieved passage correctly defines the notion of embeddings !\n",
    "\n",
    "It is worth mentioning that the model will retrieve passages no matter if they are relevant or not, as it simply provides a score for each passage. Therefore, if the query does not have any relevant span in the provided text, it will return irrelevant spans. Nonetheless, as it can be observed in the 2nd example, scores for such irreevant passages (in the 2nd example) is lower than relevant ones (in the 1st example) ;) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36dbb7c8-ef9a-40bb-9969-fd8dc1e25761",
   "metadata": {},
   "source": [
    "### Example 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c10158c9-7469-4636-a1b1-66c6be814a82",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========== DenseVectors ==========\n",
      "- # data    : 3\n",
      "- Dimension : 1024\n",
      "- Columns (primary ('filename', 'text')) : ('section_titles', 'filename', 'text', 'chunks', 'type', 'section', 'score')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "query = 'What is an embedding ?'\n",
    "\n",
    "res = vectors.search(query, k = 3)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cb2760e1-94f9-4150-947c-b47a7461d363",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text from file `../text_to_speech/README.md` - section [':yum: Text To Speech (TTS)', 'Multi-speaker Text-To-Speech', 'Automatic voice cloning with the `SV2TTS` architecture', 'The basic intuition']\n",
      "Score : 0.505\n",
      "This model basically takes as input an audio sample (5-10 sec) from a speaker, and encodes it on a *d*-dimensional vector, named the `embedding`. This embedding aims to capture relevant information about the speaker's voice (e.g., `frequencies`, `rythm`, `pitch`, ...). \n",
      "2. This pre-trained `Speaker Encoder (SE)` is then used to encode the voice of the speaker to clone.\n",
      "3. The produced embedding is then concatenated with the output of the `Tacotron-2` encoder part, such that the `Decoder` has access to both the encoded text and the speaker embedding.\n",
      "\n",
      "The objective is that the `Decoder` will learn to use the `speaker embedding` to copy its prosody / intonation / ... to read the text with the voice of this speaker.\n",
      "\n",
      "\n",
      "\n",
      "Text from file `../text_to_speech/README.md` - section [':yum: Text To Speech (TTS)', 'Multi-speaker Text-To-Speech', 'Automatic voice cloning with the `SV2TTS` architecture', 'The basic intuition']\n",
      "Score : 0.464\n",
      "Note : in the next paragraphs, `encoder` refers to the `Tacotron Encoder` part, while `SE` refers to a `speaker encoder` model (detailed below).\n",
      "\n",
      "#### The basic intuition\n",
      "\n",
      "The `Speaker Encoder-based Text-To-Speech` is inspired from the \"From Speaker Verification To Text-To-Speech (SV2TTS)\" paper. The authors have proposed an extension of the `Tacotron-2` architecture to include information about the speaker's voice.\n",
      "\n",
      "Here is a short overview of the proposed procedure :\n",
      "1. Train a model to identify speakers based on short audio samples : the `speaker verification` model. This model basically takes as input an audio sample (5-10 sec) from a speaker, and encodes it on a *d*-dimensional vector, named the `embedding`. \n",
      "\n",
      "Text from file `../data_processing/README.md` - section [':yum: Data processing utilities', 'Project structure']\n",
      "Score : 0.459\n",
      "├── example_data        : data used for the demonstrations\n",
      "├── loggers             : custom utilities for the `logging` module\n",
      "│   ├── utils       : subset of the utils module to make loggers fully independant\n",
      "│   ├── __init__.py         : defines useful utilities to control `logging`\n",
      "│   ├── telegram_handler.py : custom logger using the telegram bot api\n",
      "│   ├── time_logging.py     : custom timer features\n",
      "│   └── tts_handler.py      : custom logger using the Text-To-Speech models\n",
      "├── unitests            : custom unit-testing for the different `utils` modules\n",
      "│   ├── __init__.py\n",
      "│   ├── test_custom_objects.py  : not executed in this project (requires `custom_train_objects`)\n",
      "│   ├── test_layers_masking.py  : not executed in this project (requires `custom_layers`)\n",
      "│   ├── test_transformers.py    : test `transformers.AutoTokenizer` convertion to `TextEncoder`\n",
      "│   ├── test_utils_audio.py\n",
      "│   ├── test_utils_boxes.py\n",
      "│   ├── test_utils_compile.py\n",
      "│   ├── test_utils_clustering.py\n",
      "│   ├── test_utils_distance.py\n",
      "│   ├── test_utils_embeddings.py\n",
      "│   ├── test_utils_image.py\n",
      "│   ├── test_utils_ops.py\n",
      "│   ├── test_utils_text.py\n",
      "│   └── test_utils_threading.py\n",
      "├── utils\n",
      "│   ├── audio                   : audio utilities\n",
      "│   │   ├── __init__.py\n",
      "│   │   ├── audio_annotation.py     : annotation features for new TTS/STT dataset creation\n",
      "│   │   ├── audio_augmentation.py   : augmentation methods for audio / mel data\n",
      "│   │   ├── audio_io.py             : audio loading / writing\n",
      "│   │   ├── audio_processing.py     : audio normalization / processing\n",
      "│   │   ├── audio_search.py         : custom search in audio / video based on transcript\n",
      "│   │   ├── mkv_utils.py            : processing for .mkv video format\n",
      "│   │   ├── noisereducev1.py        : maintained version of the old `noisereduce` library\n",
      "│   │   └── stft.py                 : implementations of various mel-spectrogram methods\n",
      "│   ├── distance                : distance / similarity functions\n",
      "│   │   ├── __init__.py\n",
      "│   │   ├── clustering.py           : clustering wrappers / base features\n",
      "│   │   ├── distance_method.py      : distance / similarity metrics computation\n",
      "│   │   ├── kmeans_method.py        : K-Means implementation\n",
      "│   │   ├── knn_method.py           : K-Nearest Neighbor implementation\n",
      "│   │   ├── label_propagation_method.py : custom clustering algorithm\n",
      "│   │   ├── spectral_clustering_method.py   : spectral-clustering implementation\n",
      "│   │   └── text_distance_method.py     : text-based similarity metrics (e.g., F1)\n",
      "│   ├── image                   : image features\n",
      "│   │   ├── bounding_box            : features for bounding box manipulation (for object detection)\n",
      "│   │   │   ├── __init__.py\n",
      "│   │   │   ├── combination.py          : (experimental) combines group of boxes\n",
      "│   │   │   ├── converter.py            : box convertion format\n",
      "│   │   │   ├── iou.py                  : Intersection over Union implementation\n",
      "│   │   │   ├── locality_aware_nms.py   : (experimental) LA-NMS implementation\n",
      "│   │   │   ├── non_max_suppression.py  : (experimental) non-max suppression (NMS) implementation\n",
      "│   │   │   ├── polygons.py             : polygon manipulation for the EAST model\n",
      "│   │   │   ├── processing.py           : box processing\n",
      "│   │   │   └── visualization.py        : box extraction / drawing\n",
      "│   │   ├── __init__.py\n",
      "│   │   ├── custom_cameras.py       : custom camera for the HTTPScreenMirror app\n",
      "│   │   ├── image_augmentation.py   : image augmentation methods\n",
      "│   │   ├── image_io.py             : image loading / writing / camera streaming features\n",
      "│   │   ├── image_normalization.py  : normalization schema\n",
      "│   │   ├── image_utils.py          : custom image functions (resizing, padding, ...)\n",
      "│   │   ├── mask_utils.py           : masking utilities\n",
      "│   │   └── video_utils.py          : (experimental) basic functions for video manipulation\n",
      "│   ├── keras_utils             : custom keras operations (see `example_custom_operations.ipynb`)\n",
      "│   │   ├── ops                     : interfaces the main keras / numpy operations\n",
      "│   │   │   ├── __init__.py\n",
      "│   │   │   ├── core.py\n",
      "│   │   │   ├── image.py\n",
      "│   │   │   ├── linalg.py\n",
      "│   │   │   ├── math.py\n",
      "│   │   │   ├── nn.py\n",
      "│   │   │   ├── numpy.py\n",
      "│   │   │   ├── ops_builder.py\n",
      "│   │   │   └── random.py\n",
      "│   │   ├── __init__.py\n",
      "│   │   ├── compile.py          : custom graph compilation features\n",
      "│   │   └── gpu_utils.py        : (experimental) custom gpu features\n",
      "│   ├── search              : utilities related to information retrieval\n",
      "│   │   ├── vectors\n",
      "│   │   │   ├── __init__.py\n",
      "│   │   │   ├── base_vectors_db.py  : BaseVectorsDB abstraction class\n",
      "│   │   │   └── dense_vectors.py    : DenseVectors class\n",
      "│   ├── text                : text-related features\n",
      "│   │   ├── abreviations\n",
      "│   │   │   └── en.json\n",
      "│   │   ├── document_parser         : text extraction from documents\n",
      "│   │   │   ├── pdf_parser\n",
      "│   │   │   │   ├── __init__.py         : main parsing method\n",
      "│   │   │   │   ├── combination.py      : subset of utils/image/bounding_box/combination.py\n",
      "│   │   │   │   ├── pdfminer_parser.py  : parser based on the pdfminer extraction library\n",
      "│   │   │   │   ├── post_processing.py  : post processing for pypdfium2_parser\n",
      "│   │   │   │   ├── pypdf_parser.py     : parser based on the pypdf extraction library\n",
      "│   │   │   │   └── pypdfium2_parser.py : parser based on the pypdfium2 extraction library\n",
      "│   │   │   ├── __init__.py\n",
      "│   │   │   └── docx_parser.py\n",
      "│   │   │   ├── html_parser.py\n",
      "│   │   │   └── md_parser.py\n",
      "│   │   │   ├── parser.py\n",
      "│   │   │   ├── parser_utils.py\n",
      "│   │   │   └── txt_parser.py\n",
      "│   │   ├── __init__.py\n",
      "│   │   ├── byte_pair_encoding.py   : BPE implementation\n",
      "│   │   ├── cleaners.py             : text cleaning methods\n",
      "│   │   ├── ctc_decoder.py          : (experimental) CTC-decoding\n",
      "│   │   ├── numbers.py              : numbers cleaning methods\n",
      "│   │   ├── sentencepiece_encoder.py    : custom encoder interfacing with the sentencepiece library\n",
      "│   │   ├── text_augmentation.py    : (experimental) token masking methods\n",
      "│   │   ├── text_encoder.py         : TextEncoder class\n",
      "│   │   └── text_processing.py      : custom text / logits processing functions\n",
      "│   ├── threading               : custom producer-consumer methods\n",
      "│   │   ├── __init__.py\n",
      "│   │   ├── consumer.py         : multi-threaded consumer with observers\n",
      "│   │   ├── priority_queue.py   : custom `PriorityQueue` with order consistency\n",
      "│   │   ├── producer.py         : multi-threaded generator with observers\n",
      "│   │   └── threaded_dict.py    : thread-safe `dict` with blocking get\n",
      "│   ├── __init__.py\n",
      "│   ├── comparison_utils.py     : convenient comparison features for various data types\n",
      "│   ├── embeddings.py           : embeddings saving / loading\n",
      "│   ├── file_utils.py           : data saving / loading\n",
      "│   ├── generic_utils.py        : generic features \n",
      "│   ├── pandas_utils.py         : pandas custom features\n",
      "│   ├── plot_utils.py           : plotting functions\n",
      "│   ├── sequence_utils.py       : sequence manipulation\n",
      "│   ├── stream_utils.py         : function streaming interface\n",
      "│   └── wrapper_utils.py        : custom wrappers\n",
      "├── LICENSE\n",
      "├── Makefile\n",
      "├── README.md\n",
      "├── example_audio.ipynb\n",
      "├── example_clustering.ipynb\n",
      "├── example_custom_operations.ipynb\n",
      "├── example_generic.ipynb\n",
      "├── example_image.ipynb\n",
      "├── example_producer_consumer.ipynb\n",
      "├── example_text.ipynb\n",
      "└── requirements.txt\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for paragraph in res:\n",
    "    print('Text from file `{filename}` - section {section_titles}\\nScore : {score:.3f}\\n{text}\\n'.format(** paragraph))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fbde3d0-0832-4a46-a6cd-740f50d0e116",
   "metadata": {},
   "source": [
    "### Example 2 : irrelevant query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6bc3b922-3386-4a6b-bf79-d22da07b6faf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========== DenseVectors ==========\n",
      "- # data    : 3\n",
      "- Dimension : 1024\n",
      "- Columns (primary ('filename', 'text')) : ('section_titles', 'filename', 'text', 'chunks', 'type', 'section', 'score')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "query = 'What is the meaning of life ?'\n",
    "\n",
    "res = vectors.search(query, k = 3)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d6a7a488-a53f-42ba-adc3-9a65c02395e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text from file `../yui-mhcp/README.md` - section ['Covered topics', 'Contacts and licence', 'Terms of use']\n",
      "Score : 0.423\n",
      "- x] [Search text in audios / videos\n",
      "- x] [Live transcription / subtitle generation : some models tend to be accurate enough for transcription, like the `Whisper` family of models !\n",
      "- x] [Text-To-Speech logger : `logging`-based logger that reads your logs with `TTS` models\n",
      "- x] [Optical Character Recognition (OCR) : this projects allows to detect text in an image, and performs OCR on the detected text\n",
      "\n",
      "All topics are released in separate repositories to make it easier to learn / experiments with dedicated codes and ressources.\n",
      "\n",
      "\\* It is a demonstration code to show how to subclass `BaseModel`. I will add a dedicated repository later for general classification (text / image / ... ).\n",
      "\n",
      "## Contacts and licence\n",
      "\n",
      "Contacts :\n",
      "- **Mail** : `yui-mhcp@tutanota.com`\n",
      "- **Discord** : yui0732\n",
      "\n",
      "### Terms of use\n",
      "\n",
      "\n",
      "\n",
      "Text from file `../base_dl_project/README.md` - section [':yum: Base Deep Learning project', 'Project structure']\n",
      "Score : 0.407\n",
      "defines functions for image processing\n",
      "│   │   ├── base_model.py           : main BaseModel class\n",
      "│   │   └── base_text_model.py      : defines functions for text encoding / decoding / processing\n",
      "│   ├── saving.py           : utilities functions on models\n",
      "│   ├── model_utils.py      : utilities functions on models\n",
      "│   └── weights_converter.py    : utilities to convert weights from 2 different models\n",
      "├── pretrained_models   : main directory where all trained models are saved\n",
      "├── unitests        : *\n",
      "├── utils           : *\n",
      "├── example_classifier.ipynb\n",
      "└── example_classifier_2.ipynb\n",
      "\n",
      "\\* Check the data processing repository for more information on these modules.\n",
      "\n",
      "\n",
      "\n",
      "Text from file `../yui-mhcp/README.md` - section None\n",
      "Score : 0.401\n",
      "<h2 aligne = \"center\">\n",
      "<p> :yum: Yui-mhcp :yum: </p>\n",
      "</h2>\n",
      "\n",
      "<h2 aligne = \"center\">\n",
      "<p> A centralization of Deep Learning projects </p>\n",
      "</h2>\n",
      "\n",
      "Welcome to my GitHub profile ! The objective of my projects is to make Deep Learning more accessible, and provide real-world examples to enhance education and research in the field ! :yum:\n",
      "\n",
      "The goal is to aggregate a wide range of Deep Learning topics in one place with a common abstraction, making it easier for you to dive into this fascinating field. Each repository not only focuses on a specific area of Deep Learning, but also includes links to tutorials and reference papers. These resources are carefully selected to help you grasp both the practical and theoretical aspects of Deep Learning, which can sometimes be challenging to find.\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for paragraph in res:\n",
    "    print('Text from file `{filename}` - section {section_titles}\\nScore : {score:.3f}\\n{text}\\n'.format(** paragraph))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "026e5fa3-3a92-42ab-bb5d-6a90de5250b1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}