Squashed commit of the following:

commit a93b7e7c060cb49d3070c9f32209f1b9e7788ab1 Author: Heather Cronk <[email protected]> Date: Fri Jul 16 16:37:37 2021 -0600 Added netcdf4, owslib, and pillow to support updates. Updated miniconda image and versions of h5py, pandas, and cartopy for environment upgrade and compatability commit 30fc534176b941ef221da818948ac1dd64b14915 Merge: 757ed99 1eabfcf Author: Heather Cronk <[email protected]> Date: Fri Jul 16 14:58:51 2021 -0600 Testing PR #14: Cartopy Method Merge branch 'cartopy_method' of https://github.com/tkach940/oco2_modis_vistool into tkach940-cartopy_method commit 1eabfcf Merge: 4f7d55d 42a900c Author: tkach940 <[email protected]> Date: Thu Jul 15 20:24:12 2021 +0300 Merge pull request #3 from aronnem/cartopy_method Cartopy method commit 42a900c Author: Aronne Merrelli <[email protected]> Date: Thu Jul 15 15:55:40 2021 +0000 update the JSON input config files so that they both directly work with the repository contents added a sample OCO-2 Lite CO2 netCDF file, extracted from real sounding locations commit 2202e1f Author: Aronne Merrelli <[email protected]> Date: Thu Jul 15 15:54:36 2021 +0000 bugfix for the auto output name, it was creating a double underscore if Region was not set commit c52e439 Author: Aronne Merrelli <[email protected]> Date: Thu Jul 15 15:49:16 2021 +0000 rearrange the Encodings, so that the most likely used layers occur first. In particular, made the original hardcoded layer the first one (MODIS Aqua RGB) commit 4f7d55d Merge: 7fcab2e f23acf4 Author: tkach940 <[email protected]> Date: Tue Jul 13 17:17:30 2021 +0300 Merge pull request #2 from aronnem/cartopy_method Cartopy method commit f23acf4 Author: Aronne Merrelli <[email protected]> Date: Mon Jul 12 22:01:52 2021 +0000 bugfix: only retrieve the layer name and URL when Worldview is selected commit 9b07d47 Author: Aronne Merrelli <[email protected]> Date: Mon Jul 12 21:37:50 2021 +0000 a few more replacements of the module name - replace oco2_modis_vistool with oco_vistool commit bcc70b6 Author: Aronne Merrelli <[email protected]> Date: Mon Jul 12 21:35:30 2021 +0000 update documentation for the new default filenames and input JSON field for background image commit 8323964 Author: Aronne Merrelli <[email protected]> Date: Mon Jul 12 21:34:08 2021 +0000 update the creation of default filenames. added a new JSON field to optionally control the name of the background only plot image file simplified the default filename tags commit 7fcab2e Author: tkach940 <[email protected]> Date: Thu Jun 24 22:32:56 2021 +0000 Fixed the bug because of which the layer colorbar wasn't shown on the layer-only image. commit 85d0820 Author: tkach940 <[email protected]> Date: Thu Jun 24 20:50:50 2021 +0000 Removed all output files. Deleted all mentions of MODIS and adapted the vistool to the sensor being a required field in the config JSON file. Updated ReadMe according to all updates that have been made in that branch. Moved the second colorbar part of the script to the helper method. commit d3d7108 Merge: cbb5ff5 0378dce Author: tkach940 <[email protected]> Date: Thu Jun 24 19:46:57 2021 +0300 Merge pull request #1 from aronnem/cartopy_method remove ipython/jupyter dependency commit 0378dce Author: Aronne Merrelli <[email protected]> Date: Thu Jun 24 14:32:03 2021 +0000 remove ipython/jupyter dependency copied the contents of the jupyter notebook to a plain python script, added some comments removed the ipython checkpoint directory commit cbb5ff5 Author: Matvey Tkachev <[email protected]> Date: Tue Jun 22 15:49:35 2021 +0000 Removed unneeded and repetitive output files. commit a22821f Author: Matvey Tkachev <[email protected]> Date: Tue Jun 22 15:44:48 2021 +0000 The Cartopy method has been functionally integrated (despite possible future changes in styling and plotting parameters). Unneeded changes to the OCO-3 parts have been removed. The documentations have been updated accordingly to the new method. commit 85743c6 Author: Matvey Tkachev <[email protected]> Date: Mon Jun 21 15:20:01 2021 +0000 Removed unneeded and repetitive output files. commit ff44978 Author: Matvey Tkachev <[email protected]> Date: Mon Jun 21 15:16:09 2021 +0000 The script parsed the whole wmts XML (capabilities) file to create the relation: layer name (by NASA Worldview standards) -- layer XML url. Script, input, and output are included in the repo. The functionality is added to the script so that it can now create the second colorbar (when needed). commit dbc1ba6 Author: Matvey Tkachev <[email protected]> Date: Fri Jun 11 23:50:07 2021 +0000 Deleted unneeded output files. Just execute the vistool with current default params to get these output files back. Still no solution to the layer colorbar problem. commit f6cf384 Author: Matvey Tkachev <[email protected]> Date: Fri Jun 11 17:21:04 2021 +0000 Removed all XML and GDAL dependents from the code and repo. Next step is to research into the possibility of having the second colorbar (for the chosen layer). commit 2850ddb Author: Matvey Tkachev <[email protected]> Date: Fri Jun 11 17:15:41 2021 +0000 The overlayed image has appeared and behaves as expected. Need to investigate the second colorbar options and the get rid of the GDAL and XML dependents. commit 9ffe49a Author: Matvey Tkachev <[email protected]> Date: Thu Jun 10 22:47:09 2021 +0000 Continued the Cartopy transformation. Handled all bugs so that the script compiles and has output now while having the new method integrated. The layer imagery doesn't appear for some reason yet. However, everything else is as expected. commit 92a1975 Author: Matvey Tkachev <[email protected]> Date: Wed Jun 9 21:17:17 2021 +0000 Started the XML --> Cartopy transformation in the overlay function. commit e2a1fab Author: Matvey Tkachev <[email protected]> Date: Wed Jun 9 19:48:23 2021 +0000 Started Cartopy transformation. The layer processing (from the dict and in the script) and csv encoding are integrated at this stage. Haven't transformed the XML method into Cartopy yet.
hcronk · Jul 17, 2021 · 32a4c44 · 32a4c44
1 parent 757ed99
commit 32a4c44
Show file tree

Hide file tree

Showing 14 changed files with 28,911 additions and 248 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -1,4 +1,4 @@
-FROM continuumio/miniconda3:4.6.14
+FROM continuumio/miniconda3:4.9.2
 
-RUN conda install -y -c conda-forge h5py=2.9.0 \
-    pandas=0.25.0 cartopy=0.17.0 gdal=3.0.1
+RUN conda install -y -c conda-forge netcdf4=1.5.7 h5py=2.10.0 \
+    pandas=1.1.4 cartopy=0.18.0 owslib=0.24.1 pillow=7.2.0
diff --git a/Encoding.csv b/Encoding.csv
@@ -0,0 +1,17 @@
+Name,Code
+MODIS_Aqua_CorrectedReflectance_TrueColor,0
+MODIS_Terra_CorrectedReflectance_TrueColor,1
+VIIRS_SNPP_CorrectedReflectance_TrueColor,2
+MODIS_Terra_AOD_Deep_Blue_Land,3
+MODIS_Aqua_AOD_Deep_Blue_Land,4
+MODIS_Terra_AOD_Deep_Blue_Combined,5
+MODIS_Aqua_AOD_Deep_Blue_Combined,6
+MODIS_Combined_MAIAC_L2G_AerosolOpticalDepth,7
+MODIS_Combined_MAIAC_L2G_BidirectionalReflectance_Bands143,8
+OMPS_SO2_Lower_Troposphere,9
+MODIS_Aqua_Cloud_Top_Height_Day,10
+MODIS_Terra_Cloud_Top_Height_Day,11
+VIIRS_SNPP_Cloud_Top_Height_Day,12
+GOES-East_ABI_Band2_Red_Visible_1km,13
+GOES-West_ABI_Band2_Red_Visible_1km,14
+Himawari_AHI_Band3_Red_Visible_1km,15
diff --git a/GIBS_Aqua_MODIS_truecolor.xml b/GIBS_Aqua_MODIS_truecolor.xml
diff --git a/Layer_xml.csv b/Layer_xml.csv
diff --git a/README.md b/README.md
@@ -1,8 +1,7 @@
-# oco2_modis_vistool.py
+# oco_vistool.py
 
-The purpose of this script is to pull Aqua-MODIS RGB images from Worldview
-using the NASA GIBS API and overlay various OCO-2 data fields for case study 
-analysis in support of OCO-2 cloud and aerosol screening validation.
+The purpose of this script is to pull different RGB images (depending on the user's choice from Encoding.csv) from Worldview using the NASA GIBS API and overlay various OCO-2 data fields for 
+case study analysis in support of OCO-2 cloud and aerosol screening validation.
 
 It can be used as a command line tool to plot any data in the OCO-2 CO2 or SIF Lite file, filtered by warn level, quality flag, and/or footprint number where applicable.  
 It can also be called as a function from within another Python program for expanded use with other datasets.
@@ -19,17 +18,6 @@ It can also be called as a function from within another Python program for expan
 
 2) Make sure you have the necessary system and Python requirements.
     #### Option 1: DIY   
-
-    ##### System Requirements:
-    GDAL version 1.9.1 or greater with cURL support enabled.  
-
-    To check if cURL is enabled for GDAL, use the command `gdalinfo --format WMS`  
-    If cURL is enabled you will see information about the WMS format, if not, 
-    you will get an error message and you will need to reconfigure GDAL to support cURL.  
-    **NOTE:** If this comes with your Anaconda distribution of Python, you may need
-    to set an environmental variable called GDAL_DATA in your bash profile that points to
-    gdal within your Anaconda folder.
-    (Ex: `export GDAL_DATA="/local/home/hcronk/anaconda/share/gdal"`)
 
     ##### Python requirements:
     Python 2.7 or higher
@@ -41,7 +29,6 @@ It can also be called as a function from within another Python program for expan
 
     The conda installs required on top of what comes with Anaconda 5.2 for both Python 2.7 and Python 3.6 are as follows:
     + conda install -c conda-forge cartopy
-    + conda install -c conda-forge gdal=2.2.4
     + conda install -c conda-forge ncurses
     + conda install -c conda-forge hdf5
     + conda install -c conda-forge future
@@ -50,7 +37,7 @@ It can also be called as a function from within another Python program for expan
     Dockerfiles and docker-compose files to set up an appropriate Python 2.7 or Python 3.6 container are included in the repo    
 
 3) Test run from the command line  
-   `python oco2_modis_vistool.py`
+   `python oco_vistool.py`
 
 
 4) Look for the output PNG and HDF5 in the code directory
@@ -67,6 +54,15 @@ It can also be called as a function from within another Python program for expan
      **geo_lower_right** (list, format:[lat, lon]):  
          The latitude and longitude coordinates of the lower right hand corner of your area of 
          interest.
+     **sensor** (string):
+         The sensor name that is desired to be used for the background image. Supported sensors at the moment are: 
+         "Worldview", "GOES16_ABI_C", "GOES16_ABI_F", "GOES17_ABI_C", "GOES17_ABI_F".
+   #### Required if the sensor field, required above, is "Worldview" (otherwise, not needed):
+     **layer** (integer, format: integer between 0 and maximum code number in Encoding.csv):
+         The code of the background layer to be used by the vistool. Each layer is given its code (encoded) 
+         in the Encoding.csv file in the code directory. The file contains the relation: layer name (by the
+         NASA Worldview standards) - code (generated unique number). So, when adding new
+         layers to the CSV, use the above standards.
 
    #### Optional:
 	**region** (string, default: none):  
@@ -81,7 +77,7 @@ It can also be called as a function from within another Python program for expan
        http://matplotlib.org/examples/color/named_colors.html
 
      **oco2_overlay_info** (dict, default: none):  
-         The information for the OCO-2 variable to overlay on the MODIS RGB.  
+         The information for the OCO-2 variable to overlay on the Background image.  
 		**{**  
 		**file** (string):  
         		This is *required* for a variable overlay.  
@@ -135,35 +131,42 @@ It can also be called as a function from within another Python program for expan
 	**out_plot_dir** (string, default: the directory that you are running the code in):  
     	The full output directory path for the output plot.
 
-	**out_plot_name** (string, default example: xco2_Amazon_20150701_all_quality_WL_0to20.png):  
+	**out_plot_name** (string, default example: MODIS_Aqua_CorrectedReflectance_TrueColor_xco2_Amazon_20150701_B7302b_QF_good_WL_0to20.png):  
     	The name of the output plot.
 
 	**out_data_dir** (string, default: the directory that you are running the code in):  
     	The full output directory path for the output data file.
 
-	**out_data_name** (string, default example: xco2_Amazon_20150701_all_quality_WL_0to20.h5):  
+	**out_data_name** (string, default example: xco2_Amazon_20150701_B7302b_QF_good_WL_0to20.h5):  
     	The name of the output data file.
 
+	**out_background_name** (string, default example: MODIS_Aqua_CorrectedReflectance_TrueColor_Amazon_20150701.png):  
+    	The name of the output background file, created if no overlay data is present or make_background_image is enabled.
+
 
 ## Other Information
 
 GIBS developer documentation:  https://wiki.earthdata.nasa.gov/display/GIBS/GIBS+API+for+Developers
 
+#### WMTS Capabilities File and Its Parser
+There is a script in the code directory called "parse_wmts.py" which parses the wmts.xml capabilities file. The
+reason of this Python parser is to get the relation, layer name - layer specifications XML url, for all NASA Worldview
+layers (as of June 2021) into a separate CSV file. This relation CSV file is used for plotting and styling the output image.
+The parser used a copy of the wmts capabilities file which was downloaded into the code directory from: https://gibs.earthdata.nasa.gov/wmts/epsg4326/best/wmts.cgi?SERVICE=WMTS&REQUEST=GetCapabilities
 
 Minimum command line call:  
-`python oco2_modis_vistool.py`
+`python oco_vistool.py`
 
 
 Minimum function call:  
 ```
-from oco2_modis_vistool import do_modis_overlay_plot
+from oco_vistool import do_overlay_plot
 
-do_modis_overlay_plot([lat_of_upper_lefthand_corner, lon_of_upper_lefthand_corner],
-                      [lat_of_lower_righthand_corner, lon_of_lower_righthand_corner], 
-		      date, lat_data, lon_data, variable_data)
+do_overlay_plot([lat_of_upper_lefthand_corner, lon_of_upper_lefthand_corner],
+                [lat_of_lower_righthand_corner, lon_of_lower_righthand_corner], 
+                date, lat_data, lon_data, variable_data)
 ```  
 
-
 ## A Note about Questions, Suggestions, and Issues
 
 For the benefit of the entire user community, it is requested that you submit questions, suggestions, and issues via the GitHub issue tracker so that everyone is able to see the work that is in progress. 

diff --git a/annotated_json_format.txt b/annotated_json_format.txt
@@ -18,6 +18,7 @@ geo_upper_left: two element lat, lon values for upper left corner of
     background image (MODIS or GOES-ABI)
 geo_lower_right: two element lat, lon values for lower right corner of
     background image (MODIS or GOES-ABI)
+layer: layer code (from Encoding.csv) of the desired background layer
 
 
 Optional Fields
@@ -48,6 +49,10 @@ out_data_name: set to a specific filename to control the output h5 data
     filename. Skipping this field is equivalent to setting it to an empty
     string, which means a default file name is constructed, matching the
     default plot name.
+out_background_name: set to a specific filename to control the output
+    background image filename, if make_background_image is enabled.
+    Skipping this field is equivalent to setting it to an empty string,
+    which means a default filename is constructed.
 
 Overlay specification:
 

diff --git a/oco2_LtCO2_160727_B99999r_sampledata.nc4 b/oco2_LtCO2_160727_B99999r_sampledata.nc4
diff --git a/oco2_LtCO2_20160727_B7302b_r01_COoffline.nc4 b/oco2_LtCO2_20160727_B7302b_r01_COoffline.nc4
diff --git a/oco_goes-abi_vistool_config.json b/oco_goes-abi_vistool_config.json
@@ -1,17 +1,17 @@
 {
 	"date": "2019-10-08",
 	"sensor":"GOES16_ABI_C",
-        "data_home":"/home/merrelli/data_dir/goes16",
+        "data_home":"goes16_data_dir",
 	"geo_upper_left": [47.0, -92.5],
 	"geo_lower_right": [45.0, -89.0],
-	"region": "ParkFalls",
+	"region": "",
 	"oco2_overlay_info": {
 		"file": "oco_L1bScSC_191008_B99999_sampledata.h5",
 		"variable": "SoundingMeasurements/rad_continuum_o2",
 		"variable_plot_lims":[0.8e20, 1.2e20],
 		"lat_name": "/FootprintGeometry/footprint_vertex_latitude",
 		"lon_name": "/FootprintGeometry/footprint_vertex_longitude",
-		"orbit": 11111,
+		"orbit": 10201,
 		"make_background_image":1
 	},
 	"ground_site": "",