notebooks/ICESat-2_MODIS_Arctic_Sea_Ice/Customize and Access Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discover, Customize and Access NSIDC DAAC Data\n",
    "\n",
    "This notebook is based off of the [NSIDC-Data-Access-Notebook](https://github.com/nsidc/NSIDC-Data-Access-Notebook) provided through NSIDC's Github organization. \n",
    "\n",
    "Now that we've visualized our study areas, we will first explore data coverage, size, and customization (subsetting, reformatting, reprojection) service availability, and then access those associated files. \n",
    "\n",
    "___A note on data access options:___\n",
    "We will be pursuing data discovery and access \"programmatically\" using Application Programming Interfaces, or APIs. \n",
    "\n",
    "*What is an API?* You can think of an API as a middle man between an application or end-use (in this case, us) and a data provider. In this case the data provider is both the Common Metadata Repository (CMR) housing data information, and NSIDC as the data distributor. These APIs are generally structured as a URL with a base plus individual key-value-pairs separated by ‘&’.\n",
    "\n",
    "There are other discovery and access methods available from NSIDC listed on the data set landing page (e.g. [ATL07 Data Access](https://nsidc.org/data/atl07?qt-data_set_tabs=1#qt-data_set_tabs)) and [NASA Earthdata Search](https://search.earthdata.nasa.gov/). Programmatic API access is beneficial for those of you who want to incorporate data access into your visualization and analysis workflow. This method is also reproducible and documented to ensure data provenance. \n",
    "\n",
    "Here are the steps you will learn in this customize and access notebook:\n",
    "   \n",
    "1. Search for data programmatically using the Common Metadata Repository API by time and area of interest.\n",
    "2. Determine subsetting, reformatting, and reprojection capabilities for our data of interest.\n",
    "3. Access and customize data using NSIDC's data access and service API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import packages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import math\n",
    "import earthaccess \n",
    "\n",
    "# This is our functions module. We created several functions used in this notebook and the Visualize and Analyze notebook.\n",
    "import tutorial_helper_functions as fn "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explore data availability using the Common Metadata Repository \n",
    "\n",
    "The Common Metadata Repository (CMR) is a high-performance, high-quality, continuously evolving metadata system that catalogs Earth Science data and associated service metadata records. These metadata records are registered, modified, discovered, and accessed through programmatic interfaces leveraging standard protocols and APIs. Note that not all NSIDC data can be searched at the file level using CMR, particularly those outside of the NASA DAAC program. \n",
    "\n",
    "CMR API documentation: https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select data sets and determine version number\n",
    "\n",
    "Data sets are selected by data set IDs (e.g. ATL07, ATL10, and MOD29). In the CMR API documentation, a data set id is referred to as a \"short name\". These short names are located at the top of each NSIDC data set landing page in white underneath the full title, after 'DATA SET:'. \n",
    "\n",
    "We are using the Python Requests package to access the CMR. Data are then converted to [JSON](https://en.wikipedia.org/wiki/JSON) format; a language independant human-readable open-standard file format. More than one version can exist for a given data set: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create dictionary of data set parameters we'll use in our access API command below. We'll start with data set IDs (e.g. ATL07) of interest here, also known as \"short name\".\n",
    "data_dict = {\n",
    "    'sea_ice_fb' : {'short_name' : 'ATL10'},\n",
    "    'sea_ice_height' : {'short_name' : 'ATL07'},\n",
    "    'ist' : {'short_name' : 'MOD29'},\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get json response from CMR collection metadata to grab version numbers and add the most recent version number to data_dict\n",
    "\n",
    "for i in range(len(data_dict)):\n",
    "    cmr_collections_url = 'https://cmr.earthdata.nasa.gov/search/collections.json'\n",
    "    response = requests.get(cmr_collections_url, params=list(data_dict.values())[i])\n",
    "    results = json.loads(response.content) \n",
    "\n",
    "    # Find all instances of 'version_id' in metadata and print most recent version number\n",
    "    versions = [el['version_id'] for el in results['feed']['entry']]\n",
    "    versions = [i for i in versions if not any(c.isalpha() for c in i)]\n",
    "    data_dict[list(data_dict.keys())[i]]['version'] = max(versions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select time and area of interest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will add spatial and temporal filters to the data dictionary. The bounding box coordinates cover our region of interest over the East Siberian Sea and the temporal range covers March 23, 2019. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Bounding Box spatial parameter in 'W,S,E,N' decimal degrees format\n",
    "bounding_box = '140,72,153,80'\n",
    "\n",
    "# Each date in yyyy-MM-ddTHH:mm:ssZ format; date range in start,end format\n",
    "temporal = '2019-03-23T00:00:00Z,2019-03-23T23:59:59Z' \n",
    "\n",
    "#add bounding_box and temporal to each data set in the dictionary\n",
    "for k, v in data_dict.items(): \n",
    "    data_dict[k]['bounding_box'] = bounding_box\n",
    "    data_dict[k]['temporal'] = temporal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Determine how many files exist over this time and area of interest, as well as the average size and total volume of those granules\n",
    "\n",
    "We will use the `granule_info` function to query the CMR granule API. The function prints the number of granules, average size, and total volume of those granules. It returns the granule number value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for i in range(len(data_dict)):\n",
    "    gran_num = fn.granule_info(list(data_dict.values())[i])\n",
    "    list(data_dict.values())[i]['gran_num'] = gran_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that subsetting, reformatting, or reprojecting can alter the size of the granules if those services are applied to your request."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determine the subsetting, reformatting, and reprojection services enabled for your data set of interest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The NSIDC DAAC supports customization (subsetting, reformatting, reprojection) services on many of our NASA Earthdata mission collections. Let's discover whether or not our data set has these services available using the `print_service_options` function. If services are available, we will also determine the specific service options supported for this data set, which we will then add to our data dictionary. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input Earthdata Login credentials\n",
    "\n",
    "An Earthdata Login account is required to query data services and to access data from the NSIDC DAAC. If you do not already have an Earthdata Login account, visit http://urs.earthdata.nasa.gov to register. We are going to use the [earthaccess](https://github.com/nsidc/earthaccess) library to authenticate with our Earthdata Login credentials. We recommend using a netrc file to store your Earthdata Login credentials for authentication. Below are instructions for creating a netrc file. If we don't want to use a netrc file, skip these instructions and instead we will be prompted to enter our credentials below. \n",
    "\n",
    "#### Creating a netrc file \n",
    "1. In your home directory create a netrc file using a text editor. On a Mac or Linux Operating System the file should be named \".netrc\" and if on a Windows Operating System it should be named \"_netrc\".\n",
    "2. In the file add the following content:\n",
    "```bash \n",
    "machine urs.earthdata.nasa.gov login <USERNAME> password <PASSWORD>\n",
    "```\n",
    "where \\<USERNAME\\> and \\<PASSWORD\\> are replaced by your Earthdata Login username and password. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "auth = earthaccess.login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need to create an HTTP session in order to store cookies and pass our credentials to the data service URLs. The capability URL below is what we will query to determine service information. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.filterwarnings('ignore')\n",
    "# Query service capability URL \n",
    "for i in range(len(data_dict)):\n",
    "    capability_url = f\"https://n5eil02u.ecs.nsidc.org/egi/capabilities/{list(data_dict.values())[i]['short_name']}.{list(data_dict.values())[i]['version']}.xml\" \n",
    "    \n",
    "    # Create session to store cookie and pass credentials to capabilities url\n",
    "    session = earthaccess.get_requests_https_session()\n",
    "    s = session.get(capability_url)\n",
    "    response = session.get(s.url)\n",
    "    response.raise_for_status() # Raise bad request to check that Earthdata Login credentials were accepted \n",
    "    \n",
    "    #This function provides a list of all available services\n",
    "    fn.print_service_options(list(data_dict.values())[i], response) # not sure if the syntax is correct for a loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Populate data dictionary with services of interest\n",
    "\n",
    "We already added our CMR search keywords to our data dictionary, so now we need to add the service options we want to request. A list of all available service keywords for use with NSIDC's access and service API are available in our [Key-Value-Pair table](https://nsidc.org/support/tool/table-key-value-pair-kvp-operands-subsetting-reformatting-and-reprojection-services), as a part of our [Programmatic access guide](https://nsidc.org/support/how/how-do-i-programmatically-request-data-services). For our ATL10 and ATL07 request, we are interested in bounding box, temporal subsetting. For MOD29 we are interested in spatial subsetting. These options crop the data values to the specified ranges and variables of interest. We will enter those values into our data dictionary below.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Spatial subsetting:__ Output files are cropped to the specified bounding box extent.\n",
    "\n",
    "__Temporal subsetting:__ Output files are cropped to the specified temporal range extent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Spatial and temporal subsetting for ATL10\n",
    "\n",
    "data_dict['sea_ice_fb']['bbox'] = bounding_box\n",
    "data_dict['sea_ice_fb']['time'] = '2019-03-23T00:00:00,2019-03-23T23:59:59'\n",
    "\n",
    "# Spatial and temporal subsetting for ATL07\n",
    "\n",
    "data_dict['sea_ice_height']['bbox'] = bounding_box\n",
    "data_dict['sea_ice_height']['time'] = '2019-03-23T00:00:00,2019-03-23T23:59:59'\n",
    "\n",
    "# Spatial subsetting and polar stereographic reprojection for MOD29\n",
    "\n",
    "data_dict['ist']['bbox'] = bounding_box"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ATL07, we are also interested in variable subsetting. \n",
    "\n",
    "__Variable subsetting:__ Subsets the data set variable or group of variables. For hierarchical data, all lower level variables are returned if a variable group or subgroup is specified. \n",
    "\n",
    "For ATL07, we will use only strong beams since these groups contain higher coverage and resolution due to higher surface returns. According to the user guide, the spacecraft was in the backwards orientation during our day of interest, setting the `gt*l` beams as the strong beams. \n",
    "\n",
    "We'll use these primary geolocation, height and quality variables of interest for each of the three strong beams. The following descriptions are provided in the [ATL07 Data Dictionary](https://nsidc.org/sites/nsidc.org/files/technical-references/ATL07-data-dictionary-v001.pdf), with additional information on the algorithm and variable descriptions in the [ATBD (Algorithm Theoretical Basis Document)](https://icesat-2.gsfc.nasa.gov/sites/default/files/page_files/ICESat2_ATL07_ATL10_ATBD_r002.pdf).\n",
    "\n",
    "`delta_time`: Number of GPS seconds since the ATLAS SDP epoch. \n",
    "\n",
    "`latitude`: Latitude, WGS84, North=+, Lat of segment center\n",
    "\n",
    "`longitude`: Longitude, WGS84, East=+,Lon of segment center\n",
    "\n",
    "`height_segment_height`: Mean height from along-track segment fit determined by the sea ice algorithm\n",
    "\n",
    "`height_segment_confidence`: Confidence level in the surface height estimate based on the number of photons; the background noise rate; and the error\n",
    "analysis\n",
    "\n",
    "`height_segment_quality`: Height segment quality flag, 1 is good quality, 0 is bad\n",
    "\n",
    "`height_segment_surface_error_est`: Error estimate of the surface height (reported in meters)\n",
    "\n",
    "`height_segment_length_seg`: along-track length of segment containing n_photons_actual\n",
    "\n",
    "We will enter these variables into the data dictionary below "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_dict['sea_ice_height']['coverage'] = '/gt1l/sea_ice_segments/delta_time,\\\n",
    "/gt1l/sea_ice_segments/latitude,\\\n",
    "/gt1l/sea_ice_segments/longitude,\\\n",
    "/gt1l/sea_ice_segments/heights/height_segment_confidence,\\\n",
    "/gt1l/sea_ice_segments/heights/height_segment_height,\\\n",
    "/gt1l/sea_ice_segments/heights/height_segment_quality,\\\n",
    "/gt1l/sea_ice_segments/heights/height_segment_surface_error_est,\\\n",
    "/gt1l/sea_ice_segments/heights/height_segment_length_seg,\\\n",
    "/gt2l/sea_ice_segments/delta_time,\\\n",
    "/gt2l/sea_ice_segments/latitude,\\\n",
    "/gt2l/sea_ice_segments/longitude,\\\n",
    "/gt2l/sea_ice_segments/heights/height_segment_confidence,\\\n",
    "/gt2l/sea_ice_segments/heights/height_segment_height,\\\n",
    "/gt2l/sea_ice_segments/heights/height_segment_quality,\\\n",
    "/gt2l/sea_ice_segments/heights/height_segment_surface_error_est,\\\n",
    "/gt2l/sea_ice_segments/heights/height_segment_length_seg,\\\n",
    "/gt3l/sea_ice_segments/delta_time,\\\n",
    "/gt3l/sea_ice_segments/latitude,\\\n",
    "/gt3l/sea_ice_segments/longitude,\\\n",
    "/gt3l/sea_ice_segments/heights/height_segment_confidence,\\\n",
    "/gt3l/sea_ice_segments/heights/height_segment_height,\\\n",
    "/gt3l/sea_ice_segments/heights/height_segment_quality,\\\n",
    "/gt3l/sea_ice_segments/heights/height_segment_surface_error_est,\\\n",
    "/gt3l/sea_ice_segments/heights/height_segment_length_seg'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select data access configurations\n",
    "\n",
    "The data request can be accessed asynchronously or synchronously. The asynchronous option will allow concurrent requests to be queued and processed as orders. Those requested orders will be delivered to the specified email address, or they can be accessed programmatically as shown below. Synchronous requests will automatically download the data as soon as processing is complete. For this tutorial, we will be selecting the asynchronous method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Set NSIDC data access base URL\n",
    "base_url = 'https://n5eil02u.ecs.nsidc.org/egi/request'\n",
    "\n",
    "for k, v in data_dict.items():\n",
    "    \n",
    "    #Set the request mode to asynchronous\n",
    "    data_dict[k]['request_mode'] = 'async'\n",
    "\n",
    "    #Set the page size to the maximum for asynchronous requests \n",
    "    page_size = 2000\n",
    "    data_dict[k]['page_size'] = page_size\n",
    "\n",
    "    #Determine number of orders needed for requests over 2000 granules. \n",
    "    page_num = math.ceil(data_dict[k]['gran_num']/page_size)\n",
    "    data_dict[k]['page_num'] = page_num\n",
    "    del data_dict[k]['gran_num']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the data request API endpoint \n",
    "Programmatic API requests are formatted as HTTPS URLs that contain key-value-pairs specifying the service operations that we specified above. We will first create a string of key-value-pairs from our data dictionary and we'll feed those into our API endpoint. This API endpoint can be executed via command line, a web browser, or in Python below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "endpoint_list = [] \n",
    "for k, v in data_dict.items():\n",
    "    param_string = '&'.join(\"{!s}={!r}\".format(k,v) for (k,v) in v.items()) # Convert data_dict to string\n",
    "    param_string = param_string.replace(\"'\",\"\") # Remove quotes\n",
    "    \n",
    "    #Print API base URL + request parameters\n",
    "    API_request = api_request = f'{base_url}?{param_string}'\n",
    "    endpoint_list.append(API_request)\n",
    "    if data_dict[k]['page_num'] > 1:\n",
    "        for i in range(data_dict[k]['page_num']):\n",
    "            page_val = i + 2\n",
    "            data_dict[k]['page_num'] = page_val\n",
    "            API_request = api_request = f'{base_url}?{param_string}'\n",
    "            endpoint_list.append(API_request)\n",
    "\n",
    "print(\"\\n\".join(\"\\n\"+s for s in endpoint_list))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Request data and clean up Output folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now download data using the `request_data` function, which utilizes the Python requests library. Our param_dict and HTTP session will be passed to the function to allow Earthdata Login access. The data will be downloaded directly to this notebook directory in a new Outputs folder. The progress of the order will be reported. The data are returned in separate files, so we'll use the `clean_folder` function to remove those individual folders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Need to remove the page_num parameter from data_dict\n",
    "for k, v in data_dict.items():\n",
    "    del data_dict[k]['page_num']\n",
    "    \n",
    "#Now use the request_data function to request the data\n",
    "\n",
    "for i in range(len(data_dict)):\n",
    "    # Create session to store cookie and pass credentials to capabilities url\n",
    "    session = earthaccess.get_requests_https_session() \n",
    "    s = session.get(capability_url)\n",
    "    response = session.get(s.url)\n",
    "    response.raise_for_status() # Raise bad request to check that Earthdata Login credentials were accepted\n",
    "    \n",
    "    fn.request_data(list(data_dict.values())[i],session)\n",
    "    \n",
    "#clean up output folder\n",
    "fn.clean_folder()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To review, we have explored data availability and volume over a region and time of interest, discovered and selected data customization options, constructed API endpoints for our requests, and downloaded data. Let's move on to the analysis portion of the tutorial."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}