Accessing ICESat-2 Data
This notebook (
download) illustrates the use of icepyx for programmatic ICESat-2 data query and download from the NASA NSIDC DAAC (NASA National Snow and Ice Data Center Distributed Active Archive Center).
A complimentary notebook demonstrates in greater detail the subsetting options available when ordering data.
Import packages, including icepyx
import icepyx as ipx import os import shutil %matplotlib inline
The entire process of getting ICESat-2 data (from query to download) can ultimately be accomplished in three minimal lines of code:
region_a = ipx.Query(short_name, spatial_extent, date_range)
where the function inputs are described in more detail below.
The rest of this notebook explains the required inputs used above, optional inputs not available in the minimal example, and the other data search and visualization tools built in to icepyx that make it easier for the user to find, explore, and download ICESat-2 data programmatically from NSIDC. The detailed steps outlined and the methods showcased below are meant to give the user more control over the data they find and download (including options to order/download only the relevant portions of a data granule), some of which are called using default values behind the scenes if the user simply skips to the
Key Steps for Programmatic Data Access
There are several key steps for accessing data from the NSIDC API:
Define your parameters (spatial, temporal, dataset, etc.)
Query the NSIDC API to find out more information about the dataset
Log in to NASA Earthdata
Define additional parameters (e.g. subsetting/customization options)
Order your data
Download your data
icepyx streamlines this process into a minimal number of lines of code.
Create an ICESat-2 data object with the desired search parameters
There are three required inputs, depending on how you want to search for data. Two are required in all cases:
short_name= the data product of interest, known as its “short name”. See https://nsidc.org/data/icesat-2/data-sets for a list of the available data products.
spatial extent= a region of interest to search within. This can be entered as a bounding box, polygon vertex coordinate pairs, or a polygon geospatial file (currently shp, kml, and gpkg are supported).
bounding box: Given in decimal degrees for the lower left longitude, lower left latitude, upper right longitude, and upper right latitude
polygon vertices: Given as longitude, latitude coordinate pairs of decimal degrees with the last entry a repeat of the first.
polygon file: A string containing the full file path and name.
NOTE: The input keyword for
short_name was updated in the code from
product to match common usage.
This should not affect users providing positional inputs as demonstrated in this tutorial.
Then, for all non-gridded products (ATL<=13), you must include AT LEAST one of the following inputs (temporal or orbital constraints):
date_range= the date range for which you would like to search for results. Must be formatted as a set of ‘YYYY-MM-DD’ strings separated by a comma.
cycles= Which orbital cycle to use, input as a numerical string or a list of strings. If no input is given, this value defaults to all available cycles within the search parameters. An orbital cycle refers to the 91-day repeat period of the ICESat-2 orbit.
tracks= Which Reference Ground Track (RGT) to use, input as a numerical string or a list of strings. If no input is given, this value defaults to all available RGTs within the spatial and temporal search parameters.
Below are examples of each type of spatial extent input using a date range and an example using orbital parameters. Please choose and run only one of the next three cells to set your spatial parameters.
# bounding box short_name = 'ATL06' spatial_extent = [-55, 68, -48, 71] date_range = ['2019-02-20','2019-02-28']
# polygon vertices (here equivalent to the bounding box, above) short_name = 'ATL06' spatial_extent = [(-55, 68), (-55, 71), (-48, 71), (-48, 68), (-55, 68)] date_range = ['2019-02-20','2019-02-28']
# polygon geospatial file (metadata match but no subset match) # short_name = 'ATL06' # spatial_extent = './supporting_files/data-access_PineIsland/glims_polygons.kml' # date_range = ['2019-02-22','2019-02-28'] # #polygon geospatial file (subset and metadata match) # short_name = 'ATL06' # spatial_extent = './supporting_files/data-access_PineIsland/glims_polygons.shp' # date_range = ['2019-10-01','2019-10-05'] #polygon geospatial file (same area as other examples; subset and metadata match) short_name = 'ATL06' spatial_extent = './supporting_files/simple_test_poly.gpkg' date_range = ['2019-10-01','2019-10-05']
Create the data object using our inputs
region_a = ipx.Query(short_name, spatial_extent, date_range)
# using orbital parameters with one of the above data products + spatial parameters region_a = ipx.Query(short_name, spatial_extent, cycles=['03','04','05','06','07'], tracks=['0849','0902']) print(region_a.product) print(region_a.product_version) print(region_a.cycles) print(region_a.tracks)
These properties include visualization of the spatial extent on a map. The style of map you will see depends on whether or not you have a certain library,
geoviews, installed. Under the hood, this is because the
proj library must be installed with conda (it is not available from PyPI) to support some
geoviews dependencies. With
geoviews, this plotting function returns an interactive map. Otherwise, your spatial extent will plot on a static map using
# print(region_a.spatial_extent) region_a.visualize_spatial_extent()
Formatted parameters and function calls allow us to see the the properties of the data object we have created.
print(region_a.product) print(region_a.dates) print(region_a.start_time) print(region_a.end_time) print(region_a.cycles) print(region_a.tracks) print(region_a.product_version) # print(region_a.spatial_extent) region_a.visualize_spatial_extent()
There are also several optional inputs to allow the user finer control over their search. Start and end time are only valid inputs on a temporally limited search.
start_time= start time to search for data on the start date. If no input is given, this defaults to 00:00:00.
end_time= end time for the end date of the temporal search parameter. If no input is given, this defaults to 23:59:59. Times must be input as ‘HH:mm:ss’ strings.
version= What version of the data product to use, input as a numerical string. If no input is given, this value defaults to the most recent version of the product specified in
NOTE Version 002 is used as an example in the below cell. However, using it will cause ‘no results’ errors in granule ordering for some search parameters. These issues have been resolved in later versions of the data products, so it is best to use the most recent version where possible.
Similarly, if you try to order/download too old a version (such that it is no longer hosted by NSIDC), you will get a “no data matched your request” error.
Thus, you will need to update the version associated with
region_a and rerun the next cell for the rest of this notebook to run.
region_a = ipx.Query(short_name, spatial_extent, date_range, \ start_time='03:30:00', end_time='21:30:00', version='002') print(region_a.product) print(region_a.dates) print(region_a.start_time) print(region_a.end_time) print(region_a.product_version) # print(region_a.spatial_extent)
Alternatively, you can also just create the query object without creating named variables first:
# region_a = ipx.Query('ATL06',[-55, 68, -48, 71],['2019-02-01','2019-02-28'], # start_time='00:00:00', end_time='23:59:59', version='002')
More information about your query object
In addition to viewing the stored object information shown above (e.g. product short name, start and end date and time, version, etc.), we can also request summary information about the data product itself or confirm that we have manually specified the latest version.
If the summary does not provide all of the information you are looking for, or you would like to see information for previous versions of the data product, all available metadata for the collection product is available in a readable format.
Querying a data product
In order to search the product collection for available data granules, we need to build our search parameters. This is done automatically behind the scenes when you run
region_a.avail_granules(), but you can also build and view them by calling
region_a.CMRparams. These are formatted as a dictionary of key:value pairs according to the CMR documentation.
#build and view the parameters that will be submitted in our query region_a.CMRparams
Now that our parameter dictionary is constructed, we can search the CMR database for the available granules. Granules returned by the CMR metadata search are automatically stored within the data object. The search completed at this level relies completely on the granules’ metadata. As a result, some (and in rare cases all) of the granules returned may not actually contain data in your specified region, particularly if the region is small or located near the boundaries of a given granule. If this is the case, the subsetter will not return any data when you actually place the order. A warning message will be issued during ordering for each granule to which this applies (but no message is output for successfully subsetted granules, so don’t worry!)
#search for available granules and provide basic summary info about them region_a.avail_granules()
#get a list of granule IDs for the available granules region_a.avail_granules(ids=True)
#print detailed information about the returned search results region_a.granules.avail
Log in to NASA Earthdata
In order to download any data from NSIDC, we must first authenticate ourselves using a valid Earthdata login. This will create a valid token to interface with the DAAC as well as start an active logged-in session to enable data download. Once you have successfully logged in for a given query instance, the token and session will be passed behind the scenes as needed for you to order and download data. Passwords are entered but not shown or stored in plain text by the system.
There are multiple ways to provide your Earthdata credentials via icepyx. The in-notebook login is used below, and other methods will soon be detailed in the documentation (as of September 2021, as they are still in development).
earthdata_uid = 'icepyx_devteam' email = 'email@example.com' region_a.earthdata_login(earthdata_uid, email)
Additional Parameters and Subsetting
Once we have generated our session, we must build the required configuration parameters needed to actually download data. These will tell the system how we want to download the data. As with the CMR search parameters, these will be built automatically when you run
region_a.order_granules(), but you can also create and view them with
region_a.reqparams. The default parameters, given below, should work for most users.
page_size= 2000. This is the number of granules we will request per order.
page_num= 1. Determine the number of pages based on page size and the number of granules available. If no page_num is specified, this calculation is done automatically to set page_num, which then provides the number of individual orders we will request given the number of granules.
More details about the configuration parameters
request_mode is “asynchronous” by default, which allows concurrent requests to be queued and processed without the need for a continuous connection between you and the API endpoint.
In contrast, using a “synchronous”
request_mode means that the request relies on a direct, continous connection between you and the API endpoint.
Outputs are directly downloaded, or “streamed”, to your working directory.
For this tutorial, we will set the request mode to asynchronous.
Use the streaming
request_mode with caution: While it can be beneficial to stream outputs directly to your local directory, note that timeout errors can result depending on the size of the request, and your request will not be queued in the system if NSIDC is experiencing high request volume. For best performance, NSIDC recommends setting
page_size=1 to download individual outputs, which will eliminate extra time needed to zip outputs and will ensure faster processing times per request.
Recall that we queried the total number and volume of granules prior to applying customization services.
page_num can be used to adjust the number of granules per request up to a limit of 2000 granules for asynchronous, and 100 granules for synchronous (streaming). For now, let’s select 9 granules to be processed in each zipped request. For ATL06, the granule size can exceed 100 MB so we want to choose a granule count that provides us with a reasonable zipped download size.
print(region_a.reqparams) # region_a.reqparams['page_size'] = 9 # print(region_a.reqparams)
In addition to the required parameters (CMRparams and reqparams) that are submitted with our order, for ICESat-2 data products we can also submit subsetting parameters to NSIDC. For a deeper dive into subsetting, please see our Subsetting Tutorial Notebook, which covers subsetting in more detail, including how to get a list of subsetting options, how to build your list of subsetting parameters, and how to generate a list of desired variables (most datasets have more than 200 variable fields!), including using pre-built default lists (these lists are still in progress and we welcome contributions!).
Subsetting utilizes the NSIDC’s built in subsetter to extract only the data you are interested (spatially, temporally, variables of interest, etc.). The advantages of using the NSIDC’s subsetter include:
easily reproducible downloads, particularly when coupled with an icepyx query object
smaller file size, meaning faster downloads, less storage required, and no need to subset the data on your own
still easy to go back and order more data/variables with the same or similar search parameters
no extraneous data means you can move directly to analysis and easily navigate your dataset
Certain subset parameters are specified by default unless
subset=False is included as an input to
download_granules() (which calls
order_granules() under the hood). A separate, companion notebook tutorial covers subsetting in more detail, including how to get a list of subsetting options, how to build your list of subsetting parameters, and how to generate a list of desired variables (most products have more than 200 variable fields!), including using pre-built default lists (these lists are still in progress and we welcome contributions!).
As for the CMR and required parameters, default subset parameters can be built and viewed using
subsetparams. Where an input spatial file is used, rather than a bounding box or manually entered polygon, the spatial file will be used for subsetting (unless subset is set to False) but not show up in the
icepyx also makes it easy to take advantage of the reformatting (e.g. file format conversion) options offered by NSIDC. These are covered in more detail in the Subsetting Tutorial Notebook.
Place the order
Then, we can send the order to NSIDC using the order_granules function. Information about the granules ordered and their status will be printed automatically. Status information can also be emailed to the address provided when the
True. Additional information on the order, including request URLs, can be viewed by setting the optional keyword input ‘verbose’ to True.
region_a.order_granules() # region_a.order_granules(verbose=True, subset=False, email=False)
#view a short list of order IDs region_a.granules.orderIDs
Download the order
Finally, we can download our order to a specified directory (which needs to have a full path but doesn’t have to point to an existing directory) and the download status will be printed as the program runs. Additional information is again available by using the optional boolean keyword
path = './download' region_a.download_granules(path)