workflow_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preparation\n",
    "### Install prerequisites\n",
    "1. Install python (windows app store v3.9)\n",
    "2. Install git (https://git-scm.com/downloads)\n",
    "3. install visual C++ build tools 14.0 (Visual Studio installer from https://visualstudio.microsoft.com/visual-cpp-build-tools/; individual components --> VS 2015 C++ build tools (v14.00))\n",
    "\n",
    "### install aparts package and dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "vscode": {
     "languageId": "powershell"
    }
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (2079595370.py, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  Cell \u001b[1;32mIn[3], line 1\u001b[1;36m\u001b[0m\n\u001b[1;33m    pip install git+https://github.com/SamBoerlijst/aparts.git\u001b[0m\n\u001b[1;37m        ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "python3.9 -m pip install git+https://github.com/SamBoerlijst/aparts.git\n",
    "python3.9 -m pip install notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In meantime download a relevant article list or continue using the example csv\n",
    "The package uses a csv containing titles and abstracts (and optionally author given keywords) to generate a keyword list to scan pdf files with. Such a dataset may be acquired using the following methods. To increase relevancy, one could use the [[#Query expansion]] function to propose an optimized query using tag co-occurrence. \n",
    "*  **Web of science**\n",
    "\t1. Go to [Web of Science](https://www-webofscience-com.ezproxy.leidenuniv.nl/wos/) and login with your institution.\n",
    "\t2. Enter a query broad enough to capture the scope of the bilbiography you want to tag lateron and press search.\n",
    "\t3. Click on ‘Export’ and select ‘Excel’.\n",
    "\t4. select the amount of records you want to import, and ‘Full record’ in Record content. About 200 articles are recommended, but the only true limitation is computation time during keyword generation.\n",
    "\t5. Convert the xslx to csv using excel or equivalent.\n",
    "* **Publish or perish**\n",
    "\t1. Download [Publish or Perish](https://harzing.com/resources/publish-or-perish)\n",
    "\t2. Open the software and select Pubmed as search engine. Many search engines are available, but pubmed is one of the few including abstracts in their records.\n",
    "\t3. Enter a query broad enough to capture the scope of the bilbiography you want to tag later on and press search.\n",
    "\t4. Save the results as csv.\n",
    "* **semantic scholar**\n",
    "\t1. Although web of science and publish or perish can retrieve elaborate datasets as described here, semantic scholar can be used to download a simplified dataset as alternative. \n",
    "\tThe following function downloads a simplified dataset from semantic scholar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.semantic_scholar import query_to_csv\n",
    "\n",
    "query_to_csv(query=\"Culex pipiens AND population dynamics\", output = 'C:/Users/boerlijs/aparts/input/papers.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nb. The package expects a comma delimited csv file with a \"Article Title\" and \"Abstract\" column. You may either manually change the column names, or define the columns in the extract_tags function.\n",
    "\n",
    "### Install model for spacy\n",
    "Spacy uses a language model to detect common words and their respective type. To download the english language model, run the following command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (3892070002.py, line 1)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  Cell \u001b[1;32mIn[1], line 1\u001b[1;36m\u001b[0m\n\u001b[1;33m    python3.9 -m spacy download en_core_web_sm\u001b[0m\n\u001b[1;37m           ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "python3.9 -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install model for nltk\n",
    "NLTK, or 'natural language tokenizer' uses two libraries to tokenize words, which will be used to algorithmically identify tags. To download the tokenizers, run the following commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "python3.9 -m nltk nltk.download('punkt')\n",
    "python3.9 -m nltk nltk.download('punkt_tab')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "nltk.download('punkt_tab')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setting variables\n",
    "To make working with the package easier, define some information on the input that will be used in the following codeblock:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define where the input and output folders should be stored\n",
    "working_directory = \"C:/Users/boerlijs/aparts\"\n",
    "\n",
    "#Define the query used for the input csv\n",
    "query = \"ecology AND vector OR mosquito* OR population dynamics OR Culex\"\n",
    "#Define the filename of the input csv, without its extension\n",
    "input_csv = \"savedrecs\"\n",
    "#Define the column names used in the input csv\n",
    "title_column = \"Article Title\"\n",
    "author_column = \"Author\"\n",
    "abstract_column = \"Abstract\"\n",
    "author_given_keywords_column = \"Author Keywords\"\n",
    "published_date = \"Publication Year\"\n",
    "\n",
    "#Define the filename of the original .bib file without its extension\n",
    "bib_file = \"Library\"\n",
    "\n",
    "#Define the folder in which the pdf files are stored. It does not matter if they are stored in subfolders.\n",
    "pdf_folder = \"C:/.../Zotero/storage\"\n",
    "\n",
    "#Define the filename in which to store the keywords\n",
    "keyword_list = \"keywords\"\n",
    "\n",
    "\n",
    "#Define output\n",
    "##Define the name for the list of tagged files\n",
    "tagged_files_csv = \"tagged_files\"\n",
    "\n",
    "##Define the name for the merged output csv\n",
    "output_csv = \"savedrecs_tagged\"\n",
    "#Define what delimiter the output should use. \";\" is advised due to the use of comma's in author lists, titles and abstracts\n",
    "separator = \";\"\n",
    "\n",
    "\n",
    "#Define the filename of a curated set of articles, otherwise leave equal to the tagged csv\n",
    "curated_csv = output_csv\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Derive some of the needed folder and file paths from the diorectory and names just set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_folder = f\"{working_directory}/input\"\n",
    "output_folder = f\"{working_directory}/output\"\n",
    "template_folder = f\"{input_folder}/templates\"\n",
    "\n",
    "input_csv_path = f\"{input_folder}/{input_csv}.csv\"\n",
    "bib_file_path = f\"{input_folder}/{bib_file}.bib\"\n",
    "keyword_list_path = f\"{input_folder}/{keyword_list}.csv\"\n",
    "curated_csv_path = f\"{input_folder}/{curated_csv}.csv\"\n",
    "\n",
    "tagged_files_csv_path = f\"{output_folder}/csv/{tagged_files_csv}.csv\"\n",
    "output_csv_path = f\"{output_folder}/csv/{output_csv}.csv\"\n",
    "output_csv_path_deduplicated = f\"{output_folder}/csv/{output_csv}_deduplicated.csv\"\n",
    "output_csv_path_clusters = f\"{output_folder}/csv/{output_csv}_clusters.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "share the working directory with the command line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['working_directory'] = working_directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect pdf files \n",
    "To scan the full text for tags, the pdf files are needed. Possibly you already downloaded them, alternatively there are two methods to find these files.\n",
    "\n",
    "##### Open access papers\n",
    "Download links for open access papers may be found, and exported to csv, using the following function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.semantic_scholar import batch_collect_paper_metadata\n",
    "\n",
    "batch_collect_paper_metadata(input=input_csv_path, output=f\"{input_folder}/{input_csv}_metadata.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### download specific articles using sci-hub\n",
    "The following function tries to find an article by name using scholarly and then downloads it, storing it using the title as its name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.download_pdf import scihub_download_pdf, get_article\n",
    "\n",
    "paper = get_article(title=\"The unsuccessful self-treatment of a case of \\\"writer's block\\\"\")\n",
    "scihub_download_pdf(paper=paper, output_folder = f\"{input_folder}/pdf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optional: One could loop over the title column of their article list to serve as input for the download function. However, it is not tested whether scihub will block such automated requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(input_csv_path)\n",
    "title_list = df[title_column]\n",
    "\n",
    "for title in title_list:\n",
    "    paper = get_article(title=title)\n",
    "    scihub_download_pdf(paper=paper, output_folder = f\"{input_folder}/pdf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Move to a working directory to store the input and output files\n",
    "Using a standard folder structure makes it easier to supply input files and to store output files. To generate this structure and to see whether the APARTS package is correctly configured, run the following commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C:\\Users\\boerlijs\\aparts\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\boerlijs\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\IPython\\core\\magics\\osm.py:417: UserWarning: using dhist requires you to install the `pickleshare` library.\n",
      "  self.shell.db['dhist'] = compress_dhist(dhist)[-100:]\n"
     ]
    }
   ],
   "source": [
    "cd $working_directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To generate the folder structure that will be used to take and store files within the working directory, use: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.construct_keylist import generate_folder_structure\n",
    "\n",
    "generate_folder_structure()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Store the article list (csv) under /input\n",
    "- If you want to add to an existing .bib file, add that to /input ass well.\n",
    "- Article pdfs will automatically be stored under /input/pdf as well during one of the later steps.\n",
    "\n",
    "\n",
    "# Generate keyword list\n",
    "To generate the keyword list from your csv of articles, run the following commands and change \"records\" to the name of your CSV. Optionally, a column with author given keywords and a bibfile with self-determined keywords may be supplied. If these fields are left empty, the package proceeds without them.\n",
    "This step calculates tags using 7 algorithms for both the title and abstract, which may take several minutes, especially when analysing the abstracts. It then includes all keywords that are returned by at least two and no more than 4 of the 14 lists to exclude unique word(combination)s and common English. The final file is saved as \"keyword_list\" (or your chosen alternative) in the input folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.construct_keylist import generate_keylist\n",
    "generate_keylist(input_folder = input_folder, records = input_csv, bibfile = bib_file, author_given_keywords=author_given_keywords_column, output_name=keyword_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the input folder. 15 files should have been created. two per algorithm, of which one for the abstracts and one for the titles, and one final list \"keyword_list.txt\" (or your chosen alternative). Check the generated keywords in keywords.txt and remove any artefacts. Optionally, manually curate keywords that have been missed when merging the different lists. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#open input folder\n",
    "os.startfile(input_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Optionally, to rerun the construction of the keyword list with different minimum and maximum overlap, change the respective value and run the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.construct_keylist import construct_keylist\n",
    "\n",
    "construct_keylist(input_folder = input_folder, libtex_csv = input_csv, output_name = \"keywords\", minimum = 2, maximum = 4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tag the articles\n",
    "Converts the articles from .pdf to utf8 encoded .txt, tags each file and stores the output as a csv and uses the filename (most likely equal to the article title) as key. Optionally, the tags are added to a .bib for use in a reference manager like Zotero and as markdown summaries to use in a editor like Obsidian.\n",
    "Weighted tagging gives more importance to tags mentioned certain sections i.e. the abstract and discussion, as compared to the introduction. Unweighted tagging weighs all sections equally.\n",
    "\n",
    ">If conversion is interrupted for any reason, run pdf2txtfolder again and it will pick up where it stopped previously.\n",
    "\n",
    "To run everything at once:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.APT import automated_pdf_tagging\n",
    "\n",
    "automated_pdf_tagging(keylist_path=keyword_list_path, source_folder=pdf_folder, bibfile=bib_file, alternate_lists=\"all\", weighted = True, treshold = 5, summaries = True, separator = separator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you may run the commands sequentially, for instance when filenames were not equal to the title. To run the pdf to txt conversion and tagging, use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.APT import pdf2txtfolder, tag_folder_weighted\n",
    "\n",
    "pdf2txtfolder(PDFfolder = f\"{input_folder}/pdf\", TXTfolder = f\"{input_folder}/pdf/docs\")\n",
    "tag_folder_weighted(input_path = f\"{input_folder}/pdf\", outputCSV=tagged_files_csv_path, alternate_lists = \"all\", separator = separator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To generate the .bib and .md files, use the following commands:\n",
    "> - If needed (manually) filter the files that have no metadata from the resulting csv. \n",
    "> - Make sure CSVtotal has a \"title\" and \"author\" column when generating the summaries. Change the column names if needed.\n",
    "> - If the templates are missing, you may download them [here](https://github.com/SamBoerlijst/aparts/tree/main/app/aparts/src/templates). Please store them under /input/templates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.APT import write_bib, create_summaries\n",
    "\n",
    "write_bib(output_csv_file = tagged_files_csv_path, libtex_csv = input_csv_path, bibfile = bib_file_path, bibfolder = f\"{output_folder}/bib\", CSVtotal = output_csv_path, separator = separator)\n",
    "#create_summaries(mdFolder = f\"{output_folder}/md/\", Article_template = f\"{template_folder}/Paper.md\", Author_template = f\"{template_folder}//Author.md\", Journal_template = f\"{template_folder}/Journal.md\", CSVtotal = output_csv_path, separator = separator, author_column=author_column, title_column=title_column)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optional: To merge synonyms from the resulting dataset, run the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main contributing tags for 50% explained variance: Aedes, Aedes/geminus, Aedes_(O.)_atactavittatustaxonomic, Anopheles, Anopheles/cinereus/hispaniola, Anopheles/flavirostris, Anopheles/punctimacula, Anopheles_atroparvus, Anopheles_cinereus, Anopheles_claviger, Anopheles_darlingi, Anopheles_fluviatilis, Anopheles_lesteri, Anopheles_melanoon, Anopheles_merus, Anopheles_multicolor, Anopheles_punctimacula, Anopheles_stephensi, Anti-predator_behavior, Antibiotics, Arbovirus, Argentina, BG_sentinel_trap, Colloids, Coquilettidia, Culex/modestus, Deinocerites, Diving_angle, Fungi, Horse, Ochlerotatus_pionips, Puerto_Rico, Thailand, Tripteroides, West Nile virus, aedes aegypti, buffer\n"
     ]
    }
   ],
   "source": [
    "from aparts.src.deduplication import merge_similar_tags_from_dataframe, plot_pca_tags\n",
    "\n",
    "Dataframe = merge_similar_tags_from_dataframe(input_file=output_csv_path, output=output_csv_path_deduplicated, variables=\"keywords\", id=\"title\", tag_length=4)\n",
    "plot_pca_tags(data=Dataframe, n_components_for_variance=50, show_plots=\"all\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization\n",
    "Generate an interactive note-graph for the papers, tags, authors and journals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "graph_view() takes 7 positional arguments but 13 were given",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[11], line 5\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01maparts\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msrc\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mgraph\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m graph_view\n\u001b[0;32m      3\u001b[0m palette \u001b[38;5;241m=\u001b[39m [\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m#87DE3C\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m#7029A6\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m#FF1288\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m#FFEB0A\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m#2B95FF\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m#894fc0\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m#FFFFFF\u001b[39m\u001b[38;5;124m\"\u001b[39m]\n\u001b[1;32m----> 5\u001b[0m \u001b[43mgraph_view\u001b[49m\u001b[43m(\u001b[49m\u001b[43moutput_csv_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43mf\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;132;43;01m{\u001b[39;49;00m\u001b[43minput_folder\u001b[49m\u001b[38;5;132;43;01m}\u001b[39;49;00m\u001b[38;5;124;43m/pdf/docs/\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m1080px\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m100\u001b[39;49m\u001b[38;5;124;43m%\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m3\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mpalette\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mnetwork\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutput_folder\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtitle_column\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mKeywords\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mYear\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mAuthors\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mJournal\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
      "\u001b[1;31mTypeError\u001b[0m: graph_view() takes 7 positional arguments but 13 were given"
     ]
    }
   ],
   "source": [
    "from aparts.src.graph import graph_view\n",
    "\n",
    "palette = [\"#87DE3C\", \"#7029A6\", \"#FF1288\", \"#FFEB0A\", \"#2B95FF\", \"#894fc0\", \"#FFFFFF\"]\n",
    "\n",
    "graph_view(output_csv_path, f\"{input_folder}/pdf/docs/\", \"1080px\", \"100%\", 3, palette, 'network', output_folder, title_column, \"Keywords\", \"Year\", \"Authors\", \"Journal\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Query expansion\n",
    "Optimize your query by proposing similar tags using a curated set of articles. Alternatively use the first n articles as trainingset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "np.nan is an invalid document, expected byte or unicode string.",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[7], line 3\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01maparts\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msrc\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mquery_expansion\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m iteratively_propose_query\n\u001b[1;32m----> 3\u001b[0m query_dict \u001b[38;5;241m=\u001b[39m \u001b[43miteratively_propose_query\u001b[49m\u001b[43m(\u001b[49m\u001b[43moriginal_query\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mquery\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minput_file\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moutput_csv_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtrainingset\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m20\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mn_tags\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m30\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mthreshold\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m0.2\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtag_column\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mkeywords\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtarget_file\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtarget_title_column\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtitle_column\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msource_title_column\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtitle_column\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msource_abstract_column\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mabstract_column\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_matches\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m30\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmethod\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mrandom\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msubsample_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m10\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[0;32m      4\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m i \u001b[38;5;129;01min\u001b[39;00m query_dict: \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mi\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m;\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mquery_dict[i][\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mquery\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m;\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mquery_dict[i][\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mscore\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m;\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mquery_dict[i][\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmatches\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\query_expansion.py:1071\u001b[0m, in \u001b[0;36miteratively_propose_query\u001b[1;34m(original_query, input_file, trainingset, n_tags, threshold, tag_column, target_file, target_title_column, source_title_column, source_abstract_column, max_matches, method, subsample_size, separator)\u001b[0m\n\u001b[0;32m   1069\u001b[0m merged_df \u001b[38;5;241m=\u001b[39m pd\u001b[38;5;241m.\u001b[39mmerge(df, sample, how\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mouter\u001b[39m\u001b[38;5;124m'\u001b[39m, indicator\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[0;32m   1070\u001b[0m outgroup \u001b[38;5;241m=\u001b[39m merged_df[merged_df[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m_merge\u001b[39m\u001b[38;5;124m'\u001b[39m] \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mleft_only\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;241m.\u001b[39mdrop(columns\u001b[38;5;241m=\u001b[39m[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m_merge\u001b[39m\u001b[38;5;124m'\u001b[39m])\n\u001b[1;32m-> 1071\u001b[0m best_query, highest_score, lowest_matches \u001b[38;5;241m=\u001b[39m \u001b[43mcalculate_best_query_by_subset\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m   1072\u001b[0m \u001b[43m    \u001b[49m\u001b[43moriginal_query\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moriginal_query\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minput_file\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minput_file\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mingroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msample\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutgroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moutgroup\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtag_column\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtag_column\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mn_tags\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mn_tags\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mthreshold\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mthreshold\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtarget_title_column\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtarget_title_column\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msource_title_column\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msource_title_column\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msource_abstract_column\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[43msource_abstract_column\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_matches\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msubsample_size\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   1073\u001b[0m query_dict[\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00min_len\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m_\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mi\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m {\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mquery\u001b[39m\u001b[38;5;124m\"\u001b[39m: best_query,\n\u001b[0;32m   1074\u001b[0m                     \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mscore\u001b[39m\u001b[38;5;124m\"\u001b[39m: highest_score, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmatches\u001b[39m\u001b[38;5;124m\"\u001b[39m: lowest_matches}\n\u001b[0;32m   1075\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mThe current best query for \u001b[39m\u001b[38;5;132;01m{\u001b[39;00min_len\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mmethod\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m items is: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mbest_query\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m with a score of \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mhighest_score\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m/\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mlowest_matches\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\query_expansion.py:1003\u001b[0m, in \u001b[0;36mcalculate_best_query_by_subset\u001b[1;34m(original_query, input_file, tag_column, n_tags, threshold, target_title_column, source_title_column, source_abstract_column, max_matches, ingroup, outgroup)\u001b[0m\n\u001b[0;32m    973\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcalculate_best_query_by_subset\u001b[39m(original_query: \u001b[38;5;28mstr\u001b[39m, input_file:\u001b[38;5;28mstr\u001b[39m, tag_column: \u001b[38;5;28mstr\u001b[39m, n_tags: \u001b[38;5;28mint\u001b[39m, threshold: \u001b[38;5;28mfloat\u001b[39m, target_title_column: \u001b[38;5;28mstr\u001b[39m, source_title_column: \u001b[38;5;28mstr\u001b[39m, source_abstract_column: \u001b[38;5;28mstr\u001b[39m, max_matches: \u001b[38;5;28mint\u001b[39m, ingroup:pd\u001b[38;5;241m.\u001b[39mDataFrame \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m, outgroup:pd\u001b[38;5;241m.\u001b[39mDataFrame \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m):\n\u001b[0;32m    974\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m    975\u001b[0m \u001b[38;5;124;03m    Description:\u001b[39;00m\n\u001b[0;32m    976\u001b[0m \u001b[38;5;124;03m    ------------\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   1001\u001b[0m \u001b[38;5;124;03m    lowest_matches (int): The number of matches of the optimized query.\u001b[39;00m\n\u001b[0;32m   1002\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[1;32m-> 1003\u001b[0m     merged_terms \u001b[38;5;241m=\u001b[39m \u001b[43mpropose_tags\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m   1004\u001b[0m \u001b[43m            \u001b[49m\u001b[43moriginal_query\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moriginal_query\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minput_file\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minput_file\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mkeyword_column\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtag_column\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mn_tags\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mn_tags\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mthreshold\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mthreshold\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mingroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mingroup\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutgroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moutgroup\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   1005\u001b[0m     best_query, highest_score, lowest_matches \u001b[38;5;241m=\u001b[39m auto_optimize_query(query\u001b[38;5;241m=\u001b[39moriginal_query, extra_tags\u001b[38;5;241m=\u001b[39mmerged_terms, target_title_column\u001b[38;5;241m=\u001b[39mtarget_title_column, source_title_column\u001b[38;5;241m=\u001b[39msource_title_column, \n\u001b[0;32m   1006\u001b[0m                                                                         source_abstract_column\u001b[38;5;241m=\u001b[39msource_abstract_column, max_matches\u001b[38;5;241m=\u001b[39mmax_matches, ingroup\u001b[38;5;241m=\u001b[39mingroup, outgroup\u001b[38;5;241m=\u001b[39moutgroup)\n\u001b[0;32m   1007\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m best_query, highest_score, lowest_matches\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\query_expansion.py:952\u001b[0m, in \u001b[0;36mpropose_tags\u001b[1;34m(original_query, keyword_column, n_tags, threshold, ingroup, outgroup, input_file)\u001b[0m\n\u001b[0;32m    926\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mpropose_tags\u001b[39m(original_query: \u001b[38;5;28mstr\u001b[39m, keyword_column: \u001b[38;5;28mstr\u001b[39m, n_tags: \u001b[38;5;28mint\u001b[39m, threshold: \u001b[38;5;28mfloat\u001b[39m, ingroup: pd\u001b[38;5;241m.\u001b[39mDataFrame \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m, outgroup: pd\u001b[38;5;241m.\u001b[39mDataFrame \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m, input_file: \u001b[38;5;28mstr\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m ):\n\u001b[0;32m    927\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m    928\u001b[0m \u001b[38;5;124;03m    Description:\u001b[39;00m\n\u001b[0;32m    929\u001b[0m \u001b[38;5;124;03m    ------------\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    950\u001b[0m \u001b[38;5;124;03m    merged_terms (list of str): List of proposed tags after merging similar terms.\u001b[39;00m\n\u001b[0;32m    951\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[1;32m--> 952\u001b[0m     expanded_query1, top_articles \u001b[38;5;241m=\u001b[39m \u001b[43mpseudo_relevance_feedback\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    953\u001b[0m \u001b[43m        \u001b[49m\u001b[43moriginal_query\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moriginal_query\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mn_tags\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mn_tags\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mingroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mingroup\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutgroup\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moutgroup\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    954\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m ingroup \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m    955\u001b[0m         expanded_query2 \u001b[38;5;241m=\u001b[39m similarity_feedback(\n\u001b[0;32m    956\u001b[0m             ingroup\u001b[38;5;241m=\u001b[39mingroup, original_query\u001b[38;5;241m=\u001b[39moriginal_query, keyword_column\u001b[38;5;241m=\u001b[39mkeyword_column, threshold\u001b[38;5;241m=\u001b[39mthreshold)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\query_expansion.py:461\u001b[0m, in \u001b[0;36mpseudo_relevance_feedback\u001b[1;34m(original_query, n_tags, n_articles, print_weights, input_file, ingroup, outgroup, batch_size, separator, tag_column)\u001b[0m\n\u001b[0;32m    458\u001b[0m df \u001b[38;5;241m=\u001b[39m pd\u001b[38;5;241m.\u001b[39mconcat([ingroup, outgroup], ignore_index\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m)\n\u001b[0;32m    460\u001b[0m tfidf_vectorizer \u001b[38;5;241m=\u001b[39m TfidfVectorizer()\n\u001b[1;32m--> 461\u001b[0m updated_query_array \u001b[38;5;241m=\u001b[39m \u001b[43mcalculate_updated_query\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    462\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtfidf_vectorizer\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moriginal_query\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mingroup\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutgroup\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    463\u001b[0m top_relevant_tags \u001b[38;5;241m=\u001b[39m get_top_relevant_tags(\n\u001b[0;32m    464\u001b[0m     tfidf_vectorizer, updated_query_array, n_tags, print_weights)\n\u001b[0;32m    465\u001b[0m df \u001b[38;5;241m=\u001b[39m batch_process(df, tfidf_vectorizer, updated_query_array, batch_size)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\query_expansion.py:417\u001b[0m, in \u001b[0;36mpseudo_relevance_feedback.<locals>.calculate_updated_query\u001b[1;34m(tfidf_vectorizer, original_query, ingroup, outgroup, tag_column)\u001b[0m\n\u001b[0;32m    414\u001b[0m beta \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0.75\u001b[39m\n\u001b[0;32m    415\u001b[0m gamma \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m0.25\u001b[39m\n\u001b[1;32m--> 417\u001b[0m tfidf_matrix \u001b[38;5;241m=\u001b[39m \u001b[43mtfidf_vectorizer\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit_transform\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    418\u001b[0m \u001b[43m    \u001b[49m\u001b[43mpd\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mconcat\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[43mingroup\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutgroup\u001b[49m\u001b[43m]\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mignore_index\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\u001b[43m[\u001b[49m\u001b[43mtag_column\u001b[49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    419\u001b[0m expanded_query_vector \u001b[38;5;241m=\u001b[39m tfidf_vectorizer\u001b[38;5;241m.\u001b[39mtransform([original_query])\n\u001b[0;32m    420\u001b[0m updated_query \u001b[38;5;241m=\u001b[39m alpha \u001b[38;5;241m*\u001b[39m expanded_query_vector \u001b[38;5;241m+\u001b[39m beta \u001b[38;5;241m*\u001b[39m \\\n\u001b[0;32m    421\u001b[0m     np\u001b[38;5;241m.\u001b[39mmean(tfidf_matrix[ingroup\u001b[38;5;241m.\u001b[39mindex, :], axis\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m0\u001b[39m) \u001b[38;5;241m-\u001b[39m \\\n\u001b[0;32m    422\u001b[0m     gamma \u001b[38;5;241m*\u001b[39m np\u001b[38;5;241m.\u001b[39mmean(tfidf_matrix[outgroup\u001b[38;5;241m.\u001b[39mindex, :], axis\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m0\u001b[39m)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\sklearn\\feature_extraction\\text.py:2104\u001b[0m, in \u001b[0;36mTfidfVectorizer.fit_transform\u001b[1;34m(self, raw_documents, y)\u001b[0m\n\u001b[0;32m   2097\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_params()\n\u001b[0;32m   2098\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_tfidf \u001b[38;5;241m=\u001b[39m TfidfTransformer(\n\u001b[0;32m   2099\u001b[0m     norm\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mnorm,\n\u001b[0;32m   2100\u001b[0m     use_idf\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39muse_idf,\n\u001b[0;32m   2101\u001b[0m     smooth_idf\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msmooth_idf,\n\u001b[0;32m   2102\u001b[0m     sublinear_tf\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msublinear_tf,\n\u001b[0;32m   2103\u001b[0m )\n\u001b[1;32m-> 2104\u001b[0m X \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit_transform\u001b[49m\u001b[43m(\u001b[49m\u001b[43mraw_documents\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   2105\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_tfidf\u001b[38;5;241m.\u001b[39mfit(X)\n\u001b[0;32m   2106\u001b[0m \u001b[38;5;66;03m# X is already a transformed view of raw_documents so\u001b[39;00m\n\u001b[0;32m   2107\u001b[0m \u001b[38;5;66;03m# we set copy to False\u001b[39;00m\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\sklearn\\base.py:1389\u001b[0m, in \u001b[0;36m_fit_context.<locals>.decorator.<locals>.wrapper\u001b[1;34m(estimator, *args, **kwargs)\u001b[0m\n\u001b[0;32m   1382\u001b[0m     estimator\u001b[38;5;241m.\u001b[39m_validate_params()\n\u001b[0;32m   1384\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m config_context(\n\u001b[0;32m   1385\u001b[0m     skip_parameter_validation\u001b[38;5;241m=\u001b[39m(\n\u001b[0;32m   1386\u001b[0m         prefer_skip_nested_validation \u001b[38;5;129;01mor\u001b[39;00m global_skip_validation\n\u001b[0;32m   1387\u001b[0m     )\n\u001b[0;32m   1388\u001b[0m ):\n\u001b[1;32m-> 1389\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m fit_method(estimator, \u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\sklearn\\feature_extraction\\text.py:1376\u001b[0m, in \u001b[0;36mCountVectorizer.fit_transform\u001b[1;34m(self, raw_documents, y)\u001b[0m\n\u001b[0;32m   1368\u001b[0m             warnings\u001b[38;5;241m.\u001b[39mwarn(\n\u001b[0;32m   1369\u001b[0m                 \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mUpper case characters found in\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   1370\u001b[0m                 \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m vocabulary while \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlowercase\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   1371\u001b[0m                 \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m is True. These entries will not\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   1372\u001b[0m                 \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m be matched with any documents\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m   1373\u001b[0m             )\n\u001b[0;32m   1374\u001b[0m             \u001b[38;5;28;01mbreak\u001b[39;00m\n\u001b[1;32m-> 1376\u001b[0m vocabulary, X \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_count_vocab\u001b[49m\u001b[43m(\u001b[49m\u001b[43mraw_documents\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfixed_vocabulary_\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   1378\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mbinary:\n\u001b[0;32m   1379\u001b[0m     X\u001b[38;5;241m.\u001b[39mdata\u001b[38;5;241m.\u001b[39mfill(\u001b[38;5;241m1\u001b[39m)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\sklearn\\feature_extraction\\text.py:1263\u001b[0m, in \u001b[0;36mCountVectorizer._count_vocab\u001b[1;34m(self, raw_documents, fixed_vocab)\u001b[0m\n\u001b[0;32m   1261\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m doc \u001b[38;5;129;01min\u001b[39;00m raw_documents:\n\u001b[0;32m   1262\u001b[0m     feature_counter \u001b[38;5;241m=\u001b[39m {}\n\u001b[1;32m-> 1263\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m feature \u001b[38;5;129;01min\u001b[39;00m \u001b[43manalyze\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdoc\u001b[49m\u001b[43m)\u001b[49m:\n\u001b[0;32m   1264\u001b[0m         \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m   1265\u001b[0m             feature_idx \u001b[38;5;241m=\u001b[39m vocabulary[feature]\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\sklearn\\feature_extraction\\text.py:99\u001b[0m, in \u001b[0;36m_analyze\u001b[1;34m(doc, analyzer, tokenizer, ngrams, preprocessor, decoder, stop_words)\u001b[0m\n\u001b[0;32m     77\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Chain together an optional series of text processing steps to go from\u001b[39;00m\n\u001b[0;32m     78\u001b[0m \u001b[38;5;124;03ma single document to ngrams, with or without tokenizing or preprocessing.\u001b[39;00m\n\u001b[0;32m     79\u001b[0m \n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m     95\u001b[0m \u001b[38;5;124;03m    A sequence of tokens, possibly with pairs, triples, etc.\u001b[39;00m\n\u001b[0;32m     96\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m     98\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m decoder \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m---> 99\u001b[0m     doc \u001b[38;5;241m=\u001b[39m \u001b[43mdecoder\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdoc\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    100\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m analyzer \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m    101\u001b[0m     doc \u001b[38;5;241m=\u001b[39m analyzer(doc)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\sklearn\\feature_extraction\\text.py:232\u001b[0m, in \u001b[0;36m_VectorizerMixin.decode\u001b[1;34m(self, doc)\u001b[0m\n\u001b[0;32m    229\u001b[0m     doc \u001b[38;5;241m=\u001b[39m doc\u001b[38;5;241m.\u001b[39mdecode(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mencoding, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdecode_error)\n\u001b[0;32m    231\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m doc \u001b[38;5;129;01mis\u001b[39;00m np\u001b[38;5;241m.\u001b[39mnan:\n\u001b[1;32m--> 232\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[0;32m    233\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnp.nan is an invalid document, expected byte or unicode string.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    234\u001b[0m     )\n\u001b[0;32m    236\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m doc\n",
      "\u001b[1;31mValueError\u001b[0m: np.nan is an invalid document, expected byte or unicode string."
     ]
    }
   ],
   "source": [
    "from aparts.src.query_expansion import iteratively_propose_query\n",
    "\n",
    "query_dict = iteratively_propose_query(original_query=query, input_file=output_csv_path, trainingset=20, n_tags=30, threshold=0.2, tag_column=\"keywords\", target_file=curated_csv_path, target_title_column=title_column, source_title_column=title_column, source_abstract_column=abstract_column, max_matches=30, method=\"random\", subsample_size=10)\n",
    "for i in query_dict: print(f\"{i};{query_dict[i]['query']};{query_dict[i]['score']};{query_dict[i]['matches']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Emulate the retrieved query\n",
    "The following code tests the amount of titles in the curated list that are captured by the new query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.query_expansion import emulate_query\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(output_csv_path)\n",
    "df1 = pd.read_csv(curated_csv_path)\n",
    "\n",
    "titles = emulate_query(query=query, df=df, title_column=title_column, abstract_column=abstract_column)\n",
    "selection = df1[df1[title_column].isin(titles)]\n",
    "\n",
    "print(selection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Articles by dissimilarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pictorial keys for the identification of mosquitoes (Diptera: Culicidae) associated with dengue virus transmission\n",
      "Comparison of transmission parameters between Anopheles argyritarsis and Anopheles pseudopunctipennis in two ecologically different localities of Bolivia\n",
      "Estimating population size by genotyping faeces\n",
      "The dominant Anopheles vectors of human malaria in Africa, Europe and the Middle East: occurrence data, distribution maps and bionomic précis.\n",
      "Thiacloprid-induced toxicity influenced by nutrients: Evidence from in situ bioassays in experimental ditches: Reduced toxicity under nutrient-enriched conditions\n",
      "Cutadapt removes adapter sequences from high-throughput sequencing reads\n",
      "Culiseta annulata : A New Mosquito For Kuwait\n",
      "Anopheles (Anopheles) pseudopunctipennis Theobald (Diptera: Culicidae): neotype designation and description.\n",
      "Environmental and socioeconomic effects of mosquito control in Europe using the biocide Bacillus thuringiensis subsp. israelensis (Bti)\n",
      "Mosquito studies (Diptera, Culicidae) {XXXI}. A revision of the subgenus Carollia of Culex.\n",
      "nan\n",
      "Geologic and Tectonic Background of the Lesser Antilles\n",
      "The effect of hairpin structure on {PCR} amplification efficiency\n",
      "Wing shape of dengue vectors from around the world.\n",
      "Mosquitoes of the Netherlands Antilles and their hygienic importance\n",
      "nan\n",
      "More than 75 percent decline over 27 years in total flying insect biomass in protected areas\n",
      "A two-year evaluation of elevated canopy trapping for Culex mosquitoes and West Nile virus in an operational surveillance program in the northeastern United States.\n",
      "Species diversity and distribution of mosquito vectors in coastal habitats of Samut Songkhram province, Thailand\n",
      "Mosquito studies (Diptera, Culicidae) {XXXVI}. Subgenera \\textit{Aedinus}, \\textit{Tinolestes} and \\textit{Anoedioporpa} of {\\textless}i{\\textgreater}Culex{\\textless}i{\\textgreater}\n",
      "nan\n",
      "nan\n",
      "Host Preferences of Mosquitoes (Diptera: Culicidae) in Suffolk County, New York\n",
      "Can tigers survive in human-dominated landscapes?\n",
      "Barcoding blood meals: New vertebrate-specific primer sets for assigning taxonomic identities to host {DNA} from mosquito blood meals\n",
      "Complexiteit van nutriëntenlimitaties in oppervlaktewateren\n",
      "Reactions of Mosquitoes to Chemicals in their Oviposition Sites\n",
      "Yeast-generated {CO} 2 : A convenient source of carbon dioxide for mosquito trapping using the {BG}-Sentinel ® traps\n",
      "The Culicidae (Diptera): a review of taxonomy, classification and phylogeny\n",
      "Environmental {DNA} – An emerging tool in conservation for monitoring past and present biodiversity\n",
      "Nectar thieves or invited pollinators? A case study of tansy flowers and common house mosquitoes\n",
      "nan\n",
      "Longitudinal Studies of Aedes aegypti (Diptera: Culicidae) in Thailand and Puerto Rico: Population Dynamics\n",
      "Selection of cyanobacteria isolated from mosquito breeding sites as a potential food source for mosquito larvae\n",
      "The diet of the spotted hyaenas Crocuta crocuta in Kruger National Park\n",
      "Genomic Microsatellites as Evolutionary Chronometers: A Test in Wild Cats\n",
      "Redescription of Cx. corniger Theobald and Elevation of Culex (Culex) Lactator Dyar and {KNAB} from Synonymy Based on Specimens from Central America (Diptera: Culicidae).\n",
      "Reproductive and genetic consequences of founding isolated lion populations\n",
      "Differences in the effects of salinity on larval growth and developmental programs of a freshwater and a euryhaline mosquito species (Insecta: Diptera,Culicidae)\n",
      "nan\n"
     ]
    }
   ],
   "source": [
    "from aparts.src.subsampling import subsample_from_csv\n",
    "from aparts.src.query_expansion import count_title_matches_from_list\n",
    "\n",
    "titles = subsample_from_csv(CSV_path=output_csv_path, y=\"keywords\", x=\"title\", n=40, distance_type=\"dissimilarity\")\n",
    "#count_title_matches_from_list(file=curated_csv_path, file1_column=title_column, selected_list=titles,show_score=True)\n",
    "\n",
    "for article in titles: print(article)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify tag clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from aparts.src.deduplication import retrieve_clusters\n",
    "from aparts.src.query_expansion import analyze_clusters\n",
    "\n",
    "title_column = \"title\"\n",
    "dataframe = retrieve_clusters(input_file=output_csv_path, output=output_csv_path_clusters, variables=\"keywords\", id=title_column, tag_length=4,  number_of_records=\"\", n_components_for_variance=0, show_plots=\"\", transpose=False, max_clusters=5, save_clusters = \"all\", visualize_clusters=True)\n",
    "query_dict = analyze_clusters(query=query, cluster_column=\"Cluster\", cluster_range=(1, 5),training_filepath=output_csv_path_clusters, test_filepath=curated_csv_path, title_column_training_file=title_column, title_column_test_file=title_column, abstract_column=abstract_column, keyword_column=\"keywords\", n_tags=30, threshold=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Find dominant tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[12], line 2\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01maparts\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msrc\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mdeduplication\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m retrieve_pca_components \n\u001b[1;32m----> 2\u001b[0m components \u001b[38;5;241m=\u001b[39m \u001b[43mretrieve_pca_components\u001b[49m\u001b[43m(\u001b[49m\u001b[43minput_file\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moutput_csv_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutput\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moutput_csv_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mvariables\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mkeywords\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mid\u001b[39;49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtitle\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtag_length\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m4\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mn_components_for_variance\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m80\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnumber_of_records\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m50\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mshow_plots\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mloading and scree and saturation\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\deduplication.py:516\u001b[0m, in \u001b[0;36mretrieve_pca_components\u001b[1;34m(input_file, output, variables, id, tag_length, number_of_records, n_components_for_variance, show_plots)\u001b[0m\n\u001b[0;32m    490\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mretrieve_pca_components\u001b[39m(input_file: \u001b[38;5;28mstr\u001b[39m, output: \u001b[38;5;28mstr\u001b[39m, variables: \u001b[38;5;28mstr\u001b[39m, \u001b[38;5;28mid\u001b[39m: \u001b[38;5;28mstr\u001b[39m, tag_length: \u001b[38;5;28mint\u001b[39m,  number_of_records: \u001b[38;5;28mint\u001b[39m, n_components_for_variance: \u001b[38;5;28mint\u001b[39m, show_plots: \u001b[38;5;28mstr\u001b[39m):\n\u001b[0;32m    491\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m    492\u001b[0m \u001b[38;5;124;03m    Retrieve principal components after merging similar tags, dropping unique columns, and performing PCA.\u001b[39;00m\n\u001b[0;32m    493\u001b[0m \n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    514\u001b[0m \u001b[38;5;124;03m    list: List of principal components.\u001b[39;00m\n\u001b[0;32m    515\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[1;32m--> 516\u001b[0m     Dataframe_merged \u001b[38;5;241m=\u001b[39m \u001b[43mmerge_similar_tags_from_dataframe\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    517\u001b[0m \u001b[43m        \u001b[49m\u001b[43minput_file\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moutput\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mvariables\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mid\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtag_length\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnumber_of_records\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    518\u001b[0m     Dataframe_filtered \u001b[38;5;241m=\u001b[39m drop_unique_columns(Dataframe_merged)\n\u001b[0;32m    519\u001b[0m     components \u001b[38;5;241m=\u001b[39m plot_pca_tags(\n\u001b[0;32m    520\u001b[0m         Dataframe_filtered, n_components_for_variance, show_plots)[\u001b[38;5;241m0\u001b[39m]\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\deduplication.py:245\u001b[0m, in \u001b[0;36mmerge_similar_tags_from_dataframe\u001b[1;34m(input_file, output, variables, id, tag_length, number_of_records, threshold, manual, show_output)\u001b[0m\n\u001b[0;32m    241\u001b[0m                 \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mInput invalid. Please answer with \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124my\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m or \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mn\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m    243\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m deduplicated_dataframe\n\u001b[1;32m--> 245\u001b[0m matrix \u001b[38;5;241m=\u001b[39m \u001b[43mgenerate_binary_item_matrix\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    246\u001b[0m \u001b[43m    \u001b[49m\u001b[43minput_file\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mvariables\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mid\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtag_length\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnumber_of_records\u001b[49m\u001b[43m)\u001b[49m[\u001b[38;5;241m0\u001b[39m]\n\u001b[0;32m    247\u001b[0m matrix \u001b[38;5;241m=\u001b[39m drop_0_columns(matrix)\n\u001b[0;32m    248\u001b[0m deduplicated_tags \u001b[38;5;241m=\u001b[39m calculate_tag_similarity(matrix, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpairs\u001b[39m\u001b[38;5;124m\"\u001b[39m, threshold)[\u001b[38;5;241m0\u001b[39m]\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\subsampling.py:49\u001b[0m, in \u001b[0;36mgenerate_binary_item_matrix\u001b[1;34m(CSV_path, y, x, keyword_length, number_of_records, delimiter, separator)\u001b[0m\n\u001b[0;32m     47\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m i, item \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(rows):\n\u001b[0;32m     48\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m j, dim \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(dimensions_filtered):\n\u001b[1;32m---> 49\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m dim \u001b[38;5;129;01min\u001b[39;00m \u001b[43mdimensions\u001b[49m\u001b[43m[\u001b[49m\u001b[43mi\u001b[49m\u001b[43m]\u001b[49m:\n\u001b[0;32m     50\u001b[0m             binary_matrix[i, j] \u001b[38;5;241m=\u001b[39m \u001b[38;5;241m1\u001b[39m\n\u001b[0;32m     52\u001b[0m \u001b[38;5;66;03m# Remove rows with only 0 values\u001b[39;00m\n",
      "\u001b[1;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "from aparts.src.deduplication import retrieve_pca_components \n",
    "components = retrieve_pca_components(input_file=output_csv_path, output=output_csv_path, variables=\"keywords\", id=\"title\", tag_length=4, n_components_for_variance=80, number_of_records=50, show_plots=\"loading and scree and saturation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Upcoming: text summarization\n",
    "Summarize all items in a csv by the top n sentences based on the tags they contain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "OSError",
     "evalue": "Cannot save file into a non-existent directory: 'C:\\NLPvenv\\NLP\\output\\csv'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mOSError\u001b[0m                                   Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[5], line 2\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01maparts\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msrc\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msummarization\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m summarize_csv\n\u001b[1;32m----> 2\u001b[0m \u001b[43msummarize_csv\u001b[49m\u001b[43m(\u001b[49m\u001b[43moutputCSV\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mC:/NLPvenv/NLP/output/csv/summarized.csv\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtxtfolder\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43mf\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;132;43;01m{\u001b[39;49;00m\u001b[38;5;28;43minput\u001b[39;49m\u001b[38;5;132;43;01m}\u001b[39;49;00m\u001b[38;5;124;43m/pdf/docs\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43msections\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mabstract\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mdiscussion\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mconclusion\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mamount\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m3\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moffset\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\summarization.py:171\u001b[0m, in \u001b[0;36msummarize_csv\u001b[1;34m(outputCSV, txtfolder, sections, amount, offset, separator)\u001b[0m\n\u001b[0;32m    151\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m    152\u001b[0m \u001b[38;5;124;03mGenerates a summary of the source text for all txt files in the given folder. The generated summary is derived from sentences from the listed sections.\u001b[39;00m\n\u001b[0;32m    153\u001b[0m \u001b[38;5;124;03m\u001b[39;00m\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    168\u001b[0m \u001b[38;5;124;03msummary (str): Summarized representation of the source file.\u001b[39;00m\n\u001b[0;32m    169\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m    170\u001b[0m header \u001b[38;5;241m=\u001b[39m pd\u001b[38;5;241m.\u001b[39mDataFrame(columns \u001b[38;5;241m=\u001b[39m [\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfile\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124msummary\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[1;32m--> 171\u001b[0m \u001b[43mguarantee_csv_exists\u001b[49m\u001b[43m(\u001b[49m\u001b[43moutputCSV\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mheader\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    172\u001b[0m file_list \u001b[38;5;241m=\u001b[39m list_filenames(txtfolder, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m*.txt\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m    173\u001b[0m previously_summarized \u001b[38;5;241m=\u001b[39m pd\u001b[38;5;241m.\u001b[39mread_csv(outputCSV, sep \u001b[38;5;241m=\u001b[39m separator)[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfile\u001b[39m\u001b[38;5;124m\"\u001b[39m]\u001b[38;5;241m.\u001b[39mtolist()\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\aparts\\src\\APT.py:677\u001b[0m, in \u001b[0;36mguarantee_csv_exists\u001b[1;34m(filename, df)\u001b[0m\n\u001b[0;32m    675\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Create csv if not yet present populated by the given dataframe.\"\"\"\u001b[39;00m\n\u001b[0;32m    676\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39misfile(filename):\n\u001b[1;32m--> 677\u001b[0m     \u001b[43mdf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mto_csv\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfilename\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mindex\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[0;32m    678\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\pandas\\util\\_decorators.py:333\u001b[0m, in \u001b[0;36mdeprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    327\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(args) \u001b[38;5;241m>\u001b[39m num_allow_args:\n\u001b[0;32m    328\u001b[0m     warnings\u001b[38;5;241m.\u001b[39mwarn(\n\u001b[0;32m    329\u001b[0m         msg\u001b[38;5;241m.\u001b[39mformat(arguments\u001b[38;5;241m=\u001b[39m_format_argument_list(allow_args)),\n\u001b[0;32m    330\u001b[0m         \u001b[38;5;167;01mFutureWarning\u001b[39;00m,\n\u001b[0;32m    331\u001b[0m         stacklevel\u001b[38;5;241m=\u001b[39mfind_stack_level(),\n\u001b[0;32m    332\u001b[0m     )\n\u001b[1;32m--> 333\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m func(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\pandas\\core\\generic.py:3967\u001b[0m, in \u001b[0;36mNDFrame.to_csv\u001b[1;34m(self, path_or_buf, sep, na_rep, float_format, columns, header, index, index_label, mode, encoding, compression, quoting, quotechar, lineterminator, chunksize, date_format, doublequote, escapechar, decimal, errors, storage_options)\u001b[0m\n\u001b[0;32m   3956\u001b[0m df \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(\u001b[38;5;28mself\u001b[39m, ABCDataFrame) \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mto_frame()\n\u001b[0;32m   3958\u001b[0m formatter \u001b[38;5;241m=\u001b[39m DataFrameFormatter(\n\u001b[0;32m   3959\u001b[0m     frame\u001b[38;5;241m=\u001b[39mdf,\n\u001b[0;32m   3960\u001b[0m     header\u001b[38;5;241m=\u001b[39mheader,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   3964\u001b[0m     decimal\u001b[38;5;241m=\u001b[39mdecimal,\n\u001b[0;32m   3965\u001b[0m )\n\u001b[1;32m-> 3967\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mDataFrameRenderer\u001b[49m\u001b[43m(\u001b[49m\u001b[43mformatter\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mto_csv\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m   3968\u001b[0m \u001b[43m    \u001b[49m\u001b[43mpath_or_buf\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3969\u001b[0m \u001b[43m    \u001b[49m\u001b[43mlineterminator\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlineterminator\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3970\u001b[0m \u001b[43m    \u001b[49m\u001b[43msep\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msep\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3971\u001b[0m \u001b[43m    \u001b[49m\u001b[43mencoding\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mencoding\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3972\u001b[0m \u001b[43m    \u001b[49m\u001b[43merrors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43merrors\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3973\u001b[0m \u001b[43m    \u001b[49m\u001b[43mcompression\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcompression\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3974\u001b[0m \u001b[43m    \u001b[49m\u001b[43mquoting\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mquoting\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3975\u001b[0m \u001b[43m    \u001b[49m\u001b[43mcolumns\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcolumns\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3976\u001b[0m \u001b[43m    \u001b[49m\u001b[43mindex_label\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mindex_label\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3977\u001b[0m \u001b[43m    \u001b[49m\u001b[43mmode\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmode\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3978\u001b[0m \u001b[43m    \u001b[49m\u001b[43mchunksize\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mchunksize\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3979\u001b[0m \u001b[43m    \u001b[49m\u001b[43mquotechar\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mquotechar\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3980\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdate_format\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdate_format\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3981\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdoublequote\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdoublequote\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3982\u001b[0m \u001b[43m    \u001b[49m\u001b[43mescapechar\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mescapechar\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3983\u001b[0m \u001b[43m    \u001b[49m\u001b[43mstorage_options\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstorage_options\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   3984\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\pandas\\io\\formats\\format.py:1014\u001b[0m, in \u001b[0;36mDataFrameRenderer.to_csv\u001b[1;34m(self, path_or_buf, encoding, sep, columns, index_label, mode, compression, quoting, quotechar, lineterminator, chunksize, date_format, doublequote, escapechar, errors, storage_options)\u001b[0m\n\u001b[0;32m    993\u001b[0m     created_buffer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m\n\u001b[0;32m    995\u001b[0m csv_formatter \u001b[38;5;241m=\u001b[39m CSVFormatter(\n\u001b[0;32m    996\u001b[0m     path_or_buf\u001b[38;5;241m=\u001b[39mpath_or_buf,\n\u001b[0;32m    997\u001b[0m     lineterminator\u001b[38;5;241m=\u001b[39mlineterminator,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   1012\u001b[0m     formatter\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfmt,\n\u001b[0;32m   1013\u001b[0m )\n\u001b[1;32m-> 1014\u001b[0m \u001b[43mcsv_formatter\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msave\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   1016\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m created_buffer:\n\u001b[0;32m   1017\u001b[0m     \u001b[38;5;28;01massert\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(path_or_buf, StringIO)\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\pandas\\io\\formats\\csvs.py:251\u001b[0m, in \u001b[0;36mCSVFormatter.save\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m    247\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m    248\u001b[0m \u001b[38;5;124;03mCreate the writer & save.\u001b[39;00m\n\u001b[0;32m    249\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m    250\u001b[0m \u001b[38;5;66;03m# apply compression and byte/text conversion\u001b[39;00m\n\u001b[1;32m--> 251\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[43mget_handle\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    252\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfilepath_or_buffer\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    253\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmode\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    254\u001b[0m \u001b[43m    \u001b[49m\u001b[43mencoding\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mencoding\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    255\u001b[0m \u001b[43m    \u001b[49m\u001b[43merrors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43merrors\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    256\u001b[0m \u001b[43m    \u001b[49m\u001b[43mcompression\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcompression\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    257\u001b[0m \u001b[43m    \u001b[49m\u001b[43mstorage_options\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mstorage_options\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    258\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m handles:\n\u001b[0;32m    259\u001b[0m     \u001b[38;5;66;03m# Note: self.encoding is irrelevant here\u001b[39;00m\n\u001b[0;32m    260\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mwriter \u001b[38;5;241m=\u001b[39m csvlib\u001b[38;5;241m.\u001b[39mwriter(\n\u001b[0;32m    261\u001b[0m         handles\u001b[38;5;241m.\u001b[39mhandle,\n\u001b[0;32m    262\u001b[0m         lineterminator\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mlineterminator,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    267\u001b[0m         quotechar\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mquotechar,\n\u001b[0;32m    268\u001b[0m     )\n\u001b[0;32m    270\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_save()\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\pandas\\io\\common.py:749\u001b[0m, in \u001b[0;36mget_handle\u001b[1;34m(path_or_buf, mode, encoding, compression, memory_map, is_text, errors, storage_options)\u001b[0m\n\u001b[0;32m    747\u001b[0m \u001b[38;5;66;03m# Only for write methods\u001b[39;00m\n\u001b[0;32m    748\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;129;01min\u001b[39;00m mode \u001b[38;5;129;01mand\u001b[39;00m is_path:\n\u001b[1;32m--> 749\u001b[0m     \u001b[43mcheck_parent_directory\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mstr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mhandle\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m    751\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m compression:\n\u001b[0;32m    752\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m compression \u001b[38;5;241m!=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mzstd\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[0;32m    753\u001b[0m         \u001b[38;5;66;03m# compression libraries do not like an explicit text-mode\u001b[39;00m\n",
      "File \u001b[1;32m~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\pandas\\io\\common.py:616\u001b[0m, in \u001b[0;36mcheck_parent_directory\u001b[1;34m(path)\u001b[0m\n\u001b[0;32m    614\u001b[0m parent \u001b[38;5;241m=\u001b[39m Path(path)\u001b[38;5;241m.\u001b[39mparent\n\u001b[0;32m    615\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m parent\u001b[38;5;241m.\u001b[39mis_dir():\n\u001b[1;32m--> 616\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mOSError\u001b[39;00m(\u001b[38;5;124mrf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCannot save file into a non-existent directory: \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mparent\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "\u001b[1;31mOSError\u001b[0m: Cannot save file into a non-existent directory: 'C:\\NLPvenv\\NLP\\output\\csv'"
     ]
    }
   ],
   "source": [
    "from aparts.src.summarization import summarize_csv\n",
    "summarize_csv(outputCSV=\"C:/NLPvenv/NLP/output/csv/summarized.csv\", txtfolder=f\"{input}/pdf/docs\", sections=['abstract', 'discussion', 'conclusion'], amount=3, offset=2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}