02_Project_data_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OhOD36r9EUdo"
   },
   "source": [
    "# 2 - Accessing Related Items through Relationships\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the libraries so that they can be used within the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ry6UzSk1Yx10"
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "from pandas import json_normalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the FAIRDOMHub directly for these examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RdO2Ow8UZVJf"
   },
   "outputs": [],
   "source": [
    "base_url = 'https://www.fairdomhub.org'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A helper method for receiving the JSON response for a give resource type and id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yo-j406CZU8f"
   },
   "outputs": [],
   "source": [
    "def json_for_resource(type, id):    \n",
    "\n",
    "  headers = {\"Accept\": \"application/vnd.api+json\",\n",
    "           \"Accept-Charset\": \"ISO-8859-1\"}\n",
    "  r = requests.get(base_url + \"/\" + type + \"/\" + str(id), headers=headers)\n",
    "  r.raise_for_status()\n",
    "  return r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EsPcgAu7EiOt"
   },
   "source": [
    "Fetches the JSON for Project 60\n",
    "\n",
    "Shows just the relationships block, rather than the full JSON, as this is what we are interested in here.\n",
    "\n",
    "Relationships are grouped by type. Each relationship has a '*data*' block, which provides an array of relationship resource.\n",
    "\n",
    "Each resource is described with an '*id*' and '*type*'.\n",
    "\n",
    "SEEK supports Restful Routes, each resource can be identified by:\n",
    "\n",
    "    https://<host>/<resource-type>/<id>\n",
    "\n",
    "https://fairdomhub.org/projects/60\n",
    "    \n",
    "so,  *{'id': '18', 'type': 'institutions'}* would be https://fairdomhub.org/institutions/18\n",
    "\n",
    "**Note** that only visible resources are listed, any related items that are not visible due to sharing permissions, are omitted. (more in accessing as an authorized user later)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rTgP55PoZavm"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project: Systems toxicology of Atlantic cod\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'project_administrators': {'data': [{'id': '847', 'type': 'people'},\n",
       "   {'id': '713', 'type': 'people'},\n",
       "   {'id': '853', 'type': 'people'},\n",
       "   {'id': '848', 'type': 'people'},\n",
       "   {'id': '849', 'type': 'people'},\n",
       "   {'id': '852', 'type': 'people'},\n",
       "   {'id': '1125', 'type': 'people'}]},\n",
       " 'pals': {'data': []},\n",
       " 'asset_housekeepers': {'data': []},\n",
       " 'asset_gatekeepers': {'data': []},\n",
       " 'organisms': {'data': [{'id': '1933753720', 'type': 'organisms'}]},\n",
       " 'human_diseases': {'data': []},\n",
       " 'people': {'data': [{'id': '697', 'type': 'people'},\n",
       "   {'id': '713', 'type': 'people'},\n",
       "   {'id': '847', 'type': 'people'},\n",
       "   {'id': '848', 'type': 'people'},\n",
       "   {'id': '849', 'type': 'people'},\n",
       "   {'id': '853', 'type': 'people'},\n",
       "   {'id': '904', 'type': 'people'},\n",
       "   {'id': '913', 'type': 'people'},\n",
       "   {'id': '914', 'type': 'people'},\n",
       "   {'id': '916', 'type': 'people'},\n",
       "   {'id': '1087', 'type': 'people'},\n",
       "   {'id': '1105', 'type': 'people'},\n",
       "   {'id': '1125', 'type': 'people'},\n",
       "   {'id': '1190', 'type': 'people'},\n",
       "   {'id': '1191', 'type': 'people'},\n",
       "   {'id': '1192', 'type': 'people'},\n",
       "   {'id': '1260', 'type': 'people'},\n",
       "   {'id': '1341', 'type': 'people'},\n",
       "   {'id': '1456', 'type': 'people'},\n",
       "   {'id': '1510', 'type': 'people'},\n",
       "   {'id': '1511', 'type': 'people'},\n",
       "   {'id': '1512', 'type': 'people'},\n",
       "   {'id': '852', 'type': 'people'},\n",
       "   {'id': '901', 'type': 'people'},\n",
       "   {'id': '902', 'type': 'people'},\n",
       "   {'id': '597', 'type': 'people'}]},\n",
       " 'institutions': {'data': [{'id': '23', 'type': 'institutions'},\n",
       "   {'id': '126', 'type': 'institutions'},\n",
       "   {'id': '18', 'type': 'institutions'},\n",
       "   {'id': '57', 'type': 'institutions'}]},\n",
       " 'programmes': {'data': [{'id': '22', 'type': 'programmes'}]},\n",
       " 'investigations': {'data': []},\n",
       " 'studies': {'data': [{'id': '217', 'type': 'studies'},\n",
       "   {'id': '361', 'type': 'studies'},\n",
       "   {'id': '424', 'type': 'studies'},\n",
       "   {'id': '593', 'type': 'studies'},\n",
       "   {'id': '697', 'type': 'studies'},\n",
       "   {'id': '753', 'type': 'studies'},\n",
       "   {'id': '802', 'type': 'studies'}]},\n",
       " 'assays': {'data': [{'id': '651', 'type': 'assays'},\n",
       "   {'id': '652', 'type': 'assays'},\n",
       "   {'id': '717', 'type': 'assays'},\n",
       "   {'id': '738', 'type': 'assays'},\n",
       "   {'id': '751', 'type': 'assays'},\n",
       "   {'id': '752', 'type': 'assays'},\n",
       "   {'id': '753', 'type': 'assays'},\n",
       "   {'id': '754', 'type': 'assays'},\n",
       "   {'id': '756', 'type': 'assays'},\n",
       "   {'id': '973', 'type': 'assays'},\n",
       "   {'id': '974', 'type': 'assays'},\n",
       "   {'id': '975', 'type': 'assays'},\n",
       "   {'id': '976', 'type': 'assays'},\n",
       "   {'id': '1010', 'type': 'assays'},\n",
       "   {'id': '1044', 'type': 'assays'},\n",
       "   {'id': '1046', 'type': 'assays'},\n",
       "   {'id': '1050', 'type': 'assays'},\n",
       "   {'id': '1051', 'type': 'assays'},\n",
       "   {'id': '1052', 'type': 'assays'},\n",
       "   {'id': '1203', 'type': 'assays'},\n",
       "   {'id': '1211', 'type': 'assays'},\n",
       "   {'id': '1292', 'type': 'assays'},\n",
       "   {'id': '1294', 'type': 'assays'},\n",
       "   {'id': '1309', 'type': 'assays'},\n",
       "   {'id': '1310', 'type': 'assays'},\n",
       "   {'id': '1365', 'type': 'assays'},\n",
       "   {'id': '1376', 'type': 'assays'},\n",
       "   {'id': '1377', 'type': 'assays'},\n",
       "   {'id': '1378', 'type': 'assays'},\n",
       "   {'id': '1379', 'type': 'assays'},\n",
       "   {'id': '1380', 'type': 'assays'},\n",
       "   {'id': '1644', 'type': 'assays'},\n",
       "   {'id': '1645', 'type': 'assays'},\n",
       "   {'id': '1646', 'type': 'assays'}]},\n",
       " 'data_files': {'data': [{'id': '1880', 'type': 'data_files'},\n",
       "   {'id': '1883', 'type': 'data_files'},\n",
       "   {'id': '1923', 'type': 'data_files'},\n",
       "   {'id': '1924', 'type': 'data_files'},\n",
       "   {'id': '1925', 'type': 'data_files'},\n",
       "   {'id': '2249', 'type': 'data_files'},\n",
       "   {'id': '2250', 'type': 'data_files'},\n",
       "   {'id': '2251', 'type': 'data_files'},\n",
       "   {'id': '2255', 'type': 'data_files'},\n",
       "   {'id': '2256', 'type': 'data_files'},\n",
       "   {'id': '2272', 'type': 'data_files'},\n",
       "   {'id': '2273', 'type': 'data_files'},\n",
       "   {'id': '2277', 'type': 'data_files'},\n",
       "   {'id': '2278', 'type': 'data_files'},\n",
       "   {'id': '2279', 'type': 'data_files'},\n",
       "   {'id': '2280', 'type': 'data_files'},\n",
       "   {'id': '2282', 'type': 'data_files'},\n",
       "   {'id': '2822', 'type': 'data_files'},\n",
       "   {'id': '2951', 'type': 'data_files'},\n",
       "   {'id': '2952', 'type': 'data_files'},\n",
       "   {'id': '2953', 'type': 'data_files'},\n",
       "   {'id': '2954', 'type': 'data_files'},\n",
       "   {'id': '2956', 'type': 'data_files'},\n",
       "   {'id': '2957', 'type': 'data_files'},\n",
       "   {'id': '3284', 'type': 'data_files'},\n",
       "   {'id': '3288', 'type': 'data_files'},\n",
       "   {'id': '3289', 'type': 'data_files'},\n",
       "   {'id': '3737', 'type': 'data_files'},\n",
       "   {'id': '3738', 'type': 'data_files'},\n",
       "   {'id': '3739', 'type': 'data_files'},\n",
       "   {'id': '3740', 'type': 'data_files'},\n",
       "   {'id': '3743', 'type': 'data_files'},\n",
       "   {'id': '3766', 'type': 'data_files'},\n",
       "   {'id': '3767', 'type': 'data_files'},\n",
       "   {'id': '3889', 'type': 'data_files'},\n",
       "   {'id': '3890', 'type': 'data_files'},\n",
       "   {'id': '3891', 'type': 'data_files'},\n",
       "   {'id': '3892', 'type': 'data_files'},\n",
       "   {'id': '3950', 'type': 'data_files'},\n",
       "   {'id': '3956', 'type': 'data_files'},\n",
       "   {'id': '3957', 'type': 'data_files'},\n",
       "   {'id': '3958', 'type': 'data_files'},\n",
       "   {'id': '3959', 'type': 'data_files'},\n",
       "   {'id': '3960', 'type': 'data_files'},\n",
       "   {'id': '3961', 'type': 'data_files'},\n",
       "   {'id': '4094', 'type': 'data_files'},\n",
       "   {'id': '4095', 'type': 'data_files'},\n",
       "   {'id': '4154', 'type': 'data_files'},\n",
       "   {'id': '4155', 'type': 'data_files'},\n",
       "   {'id': '4977', 'type': 'data_files'},\n",
       "   {'id': '4978', 'type': 'data_files'},\n",
       "   {'id': '4979', 'type': 'data_files'},\n",
       "   {'id': '4980', 'type': 'data_files'},\n",
       "   {'id': '4981', 'type': 'data_files'},\n",
       "   {'id': '4982', 'type': 'data_files'},\n",
       "   {'id': '6178', 'type': 'data_files'},\n",
       "   {'id': '6179', 'type': 'data_files'}]},\n",
       " 'file_templates': {'data': []},\n",
       " 'placeholders': {'data': []},\n",
       " 'models': {'data': [{'id': '756', 'type': 'models'},\n",
       "   {'id': '757', 'type': 'models'},\n",
       "   {'id': '758', 'type': 'models'}]},\n",
       " 'sops': {'data': []},\n",
       " 'publications': {'data': [{'id': '309', 'type': 'publications'},\n",
       "   {'id': '310', 'type': 'publications'},\n",
       "   {'id': '311', 'type': 'publications'},\n",
       "   {'id': '312', 'type': 'publications'},\n",
       "   {'id': '420', 'type': 'publications'},\n",
       "   {'id': '421', 'type': 'publications'},\n",
       "   {'id': '438', 'type': 'publications'},\n",
       "   {'id': '603', 'type': 'publications'},\n",
       "   {'id': '604', 'type': 'publications'},\n",
       "   {'id': '606', 'type': 'publications'},\n",
       "   {'id': '607', 'type': 'publications'}]},\n",
       " 'presentations': {'data': []},\n",
       " 'events': {'data': []},\n",
       " 'documents': {'data': [{'id': '872', 'type': 'documents'},\n",
       "   {'id': '922', 'type': 'documents'},\n",
       "   {'id': '923', 'type': 'documents'},\n",
       "   {'id': '925', 'type': 'documents'}]},\n",
       " 'workflows': {'data': []},\n",
       " 'collections': {'data': []}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project_id = 60\n",
    "\n",
    "result = json_for_resource('projects',project_id)\n",
    "\n",
    "\n",
    "print(\"Project: \" + result['data']['attributes']['title'] + \"\\n\")\n",
    "\n",
    "result['data']['relationships']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SKI4ufNfF2F1"
   },
   "source": [
    "Here we iterate over each related data_file, and GET the resource JSON, and display the resulting id and title in a table\n",
    "iterating over related data items and retrieving more detailed information about them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aXPRjrknab7P"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26 found\n"
     ]
    }
   ],
   "source": [
    "\n",
    "files = []\n",
    "type = 'people'\n",
    "\n",
    "for item in result['data']['relationships'][type]['data']:\n",
    "  j = json_for_resource(item['type'],item['id'])  \n",
    "  files.append({\n",
    "      'id':j['data']['id'],\n",
    "      'title':j['data']['attributes']['title'],      \n",
    "  })\n",
    "  \n",
    "print(str(len(files)) + \" found\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The json_normalize(files) function is used to convert the list of dictionaries (files) into a flat, tabular (DataFrame) format. In this case, files contains dictionaries with id and title as keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>697</td>\n",
       "      <td>Inge Jonassen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>713</td>\n",
       "      <td>Fatemeh Zamanzad Ghavidel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>847</td>\n",
       "      <td>Fekadu Yadetie</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>848</td>\n",
       "      <td>Karina Dale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>849</td>\n",
       "      <td>Nello Blaser</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>853</td>\n",
       "      <td>Anders Goksøyr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>904</td>\n",
       "      <td>Xiaokang Zhang</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>913</td>\n",
       "      <td>Morten Brun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>914</td>\n",
       "      <td>Dorothy Dankel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>916</td>\n",
       "      <td>Shirin Fallahi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1087</td>\n",
       "      <td>Marta Eide</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1105</td>\n",
       "      <td>Kidane Tekle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1125</td>\n",
       "      <td>Sofie Söderström</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1190</td>\n",
       "      <td>Libe Aranguren Abadía</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1191</td>\n",
       "      <td>Siri Øfsthus Goksøyr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1192</td>\n",
       "      <td>Odd André Karlsen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1260</td>\n",
       "      <td>Eileen Marie Hanna</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1341</td>\n",
       "      <td>Pål A. Olsvik</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1456</td>\n",
       "      <td>Korbinian Bösl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1510</td>\n",
       "      <td>Alejandra Gilabert</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1511</td>\n",
       "      <td>Anna Marqueño</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>1512</td>\n",
       "      <td>Cinta Porte</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>852</td>\n",
       "      <td>Zhanna Tairova</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>901</td>\n",
       "      <td>Ketil Hylland</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>902</td>\n",
       "      <td>Essa Ahsan Khan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>597</td>\n",
       "      <td>Jon Olav Vik</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id                      title\n",
       "0    697              Inge Jonassen\n",
       "1    713  Fatemeh Zamanzad Ghavidel\n",
       "2    847             Fekadu Yadetie\n",
       "3    848                Karina Dale\n",
       "4    849               Nello Blaser\n",
       "5    853             Anders Goksøyr\n",
       "6    904             Xiaokang Zhang\n",
       "7    913                Morten Brun\n",
       "8    914             Dorothy Dankel\n",
       "9    916             Shirin Fallahi\n",
       "10  1087                 Marta Eide\n",
       "11  1105               Kidane Tekle\n",
       "12  1125           Sofie Söderström\n",
       "13  1190      Libe Aranguren Abadía\n",
       "14  1191       Siri Øfsthus Goksøyr\n",
       "15  1192          Odd André Karlsen\n",
       "16  1260         Eileen Marie Hanna\n",
       "17  1341              Pål A. Olsvik\n",
       "18  1456             Korbinian Bösl\n",
       "19  1510         Alejandra Gilabert\n",
       "20  1511              Anna Marqueño\n",
       "21  1512                Cinta Porte\n",
       "22   852             Zhanna Tairova\n",
       "23   901              Ketil Hylland\n",
       "24   902            Essa Ahsan Khan\n",
       "25   597               Jon Olav Vik"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_normalize(files)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "02 Project data files.ipynb",
   "private_outputs": true,
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}