{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "# Subsetting ICESat-2 Data\n",
    "This notebook ({nb-download}`download <IS2_data_access2-subsetting.ipynb>`) illustrates the use of icepyx for subsetting ICESat-2 data ordered through the NASA NSIDC DAAC. We'll show how to find out what subsetting options are available and how to specify the subsetting options for your order.\n",
    "\n",
    "For more information on using icepyx to find, order, and download data, see our complimentary [ICESat-2 Data Access Notebook](https://icepyx.readthedocs.io/en/latest/example_notebooks/IS2_data_access.html).\n",
    "\n",
    "Questions? Be sure to check out the FAQs throughout this notebook, indicated as italic headings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _What is SUBSETTING anyway?_\n",
    "\n",
    "_Anyone who's worked with geospatial data has probably encountered subsetting. Typically, we search for data wherever it is stored and download the chunks (granules, scenes, passes, swaths, etc.) that contain something we are interested in. Then, we have to extract from each chunk the pieces we actually want to analyze. Those pieces might be geospatial (i.e., an area of interest) or temporal (i.e., certain months of a time series). This process of extracting the data we are going to use is called subsetting._\n",
    "\n",
    "_In the case of ICESat-2 data from the NASA NSIDC DAAC, we can do this subsetting step on the data prior to download, reducing our number of data processing steps and resulting in smaller, faster downloads and storage._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import packages, including icepyx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import icepyx as ipx\n",
    "\n",
    "import numpy as np\n",
    "import xarray as xr\n",
    "import pandas as pd\n",
    "\n",
    "import h5py\n",
    "import os,json\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "Create a query object and log in to Earthdata\n",
    "\n",
    "For this example, we'll be working with a sea ice product (ATL07) for an area along West Greenland (Disko Bay)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a = ipx.Query('ATL07',[-55, 68, -48, 71],['2019-02-22','2019-02-28'], \\\n",
    "                           start_time='00:00:00', end_time='23:59:59')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "## Discover customization options\n",
    "\n",
    "You can see the customization options for a given product by calling `show_custom_options()`. The options are presented as a dictionary of key-value pairs. Three options are currently available:\n",
    "\n",
    "* `bboxSubset`: bounding box subsetting\n",
    "* `shapeSubset`: polygon subsetting\n",
    "* `temporalSubset`: temporal subsetting\n",
    "\n",
    "`outputFormats` indicates that only HDF5 is a supported output format. `variableSubset`, `concatenate`,  and `reproject` are currently unavailable (set to false).\n",
    "\n",
    "Note that these subsetting options are available for all L2-L3A products. Subsetting options are not currently supported for L3B or Quick Looks products."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "region_a.show_custom_options()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "By default, spatial and temporal subsetting based on your initial inputs is applied to your order. This will be true no matter if you  use the `order_granules()` function or the `download_granules()` function (which calls `.order_granules under()` the hood if you have not already placed your order). If you don't want your order to be spatially subset, you can use the `subset=False` argument in either `.order_granules()` or `.download_granules()`.\n",
    "Additional subsetting options must be specified as keyword arguments to the order/download functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "### _Why do I have to provide spatial bounds to icepyx even if I don't use them to subset my data order?_\n",
    "\n",
    "_Because they're still needed for the granule level search._\n",
    "_Spatial inputs are usually required for any data search, on any platform, even if your search parameters cover the entire globe._\n",
    "\n",
    "_The spatial information you provide is used to search the data repository and determine which granules might contain data over your area of interest._\n",
    "_When you use that spatial information for subsetting, it's actually asking the NASA Harmony subsetter to extract the appropriate data from each granule._\n",
    "_Thus, even if you set `subset=False` and download entire granules, you still need to provide some inputs on what geographic area you'd like data for._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "## About data variables in a query object\n",
    "\n",
    "A given ICESat-2 product may have over 200 variable + path combinations.\n",
    "icepyx includes a custom `Variables` module that is \"aware\" of the ATLAS sensor and how the ICESat-2 data products are stored.\n",
    "The [ICESat-2 Data Variables Example](https://icepyx.readthedocs.io/en/latest/example_notebooks/IS2_data_variables.html) provides a detailed set of examples on how to use icepyx's built in `Variables` module.\n",
    "\n",
    "While variable subsetting is not supported for ICESat-2 data, you can refer to the aforementioned Jupyter Notebook to learn how to interact with ICESat-2 variables after requesting your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## _Why not just download all the data and subset locally? What if I need more granules?_\n",
    "\n",
    "_Taking advantage of the NASA Harmony subsetting service is a great way to reduce your download size and thus your download time and the amount of storage required, especially if you're storing your data locally during analysis. By downloading your data using icepyx, it is easy to go back and get additional data with the same, similar, or different parameters. Related tools (e.g., [`captoolkit`](https://github.com/fspaolo/captoolkit)) will let you easily merge files if you're uncomfortable merging them during read-in for processing._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "short_name = 'ATL06'\n",
    "spatial_extent = './supporting_files/simple_test_poly.gpkg'\n",
    "date_range = ['2019-10-01','2019-10-05']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a = ipx.Query(short_name, spatial_extent\n",
    ", \n",
    "   cycles=['03','04','05','06'], tracks=['0849','0902'])\n",
    "\n",
    "print(region_a.product)\n",
    "print(region_a.product_version)\n",
    "print(region_a.cycles)\n",
    "print(region_a.tracks)\n",
    "print(region_a.spatial_extent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.visualize_spatial_extent()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also print a list of available granules for our query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.avail_granules(cloud=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying granule subsetting to your order and downloading the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order = region_a.order_granules(subset=True) \n",
    "order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking an order status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "order.status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading subsetted granules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = order.download_granules(\"./data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### _Why does the subsetter say no matching data was found?_\n",
    "_Sometimes, granules (\"files\") returned in our initial search end up not containing any data in our specified area of interest._\n",
    "_This is because the initial search is completed using summary metadata for a granule._\n",
    "_You've likely encountered this before when viewing available imagery online: your spatial search turns up a bunch of images with only a few border or corner pixels, maybe even in no data regions, in your area of interest._\n",
    "_Thus, when you go to extract the data from the area you want (i.e., spatially subset it), you don't get any usable data from that image._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling large orders\n",
    "\n",
    "By default, the Harmony subsetter will only process the first 300 granules for large orders, placing them into a \"previewing\" status. This allows users to check that results look correct. Once the job has completed its preview, which includes the first 100 granules, then we can resume the order if we are satisfied that our request is correct. The following guidance is commented out by default but can be uncommented to test this large order behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# short_name = 'ATL06'\n",
    "# spatial_extent = './supporting_files/simple_test_poly.gpkg'\n",
    "# date_range = ['2018-10-01','2020-02-05']\n",
    "\n",
    "# region_a = ipx.Query(short_name, spatial_extent, date_range)\n",
    "\n",
    "# order = region_a.order_granules(subset=True) \n",
    "# order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This order includes 311 input granules, and therefore it is automatically placed into a previewing state. We can inspect the status of this order and wait until it moves to a \"paused\" state, once the initial 100 granules are complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# order.status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are satisfied with the order, then we can resume processing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# order.resume()\n",
    "# order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with the downloaded data\n",
    "\n",
    "Now that the subsetted files have been downloaded, we can now work with them using the `icepyx` [Read](https://icepyx.readthedocs.io/en/latest/user_guide/documentation/read.html) class. See the [Reading ICESat-2 Data in for Analysis](https://icepyx.readthedocs.io/en/latest/example_notebooks/IS2_data_read-in.html#) notebook for more information. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Credits\n",
    "* notebook contributors: Zheng Liu, Jessica Scheick, Amy Steiker, and Theresa Andersen\n",
    "* some source material: [NSIDC Data Access Notebook](https://github.com/ICESAT-2HackWeek/ICESat2_hackweek_tutorials/tree/main/03_NSIDCDataAccess_Steiker) by Amy Steiker and Bruce Wallin"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}