HARP manual

What is HARP?

HARP is a toolkit for reading, processing and inter-comparing satellite remote sensing data, model data, in-situ data, and ground based remote sensing data. The toolkit is composed of:

A set of command line tools
A library of analysis functions

The main goal of HARP is to assist in the inter-comparison of data sets. By appropriately chaining calls to HARP command line tools one can pre-process data sets such that two datasets that need to be compared end up having the same temporal/spatial grid, same data format/structure, and same physical unit. At the end of the toolchain you will have a set of data files that can be directly compared in e.g. Python, IDL or MATLAB.

In order for the HARP command line tools to handle each others output the toolkit uses its own data format conventions for intermediate files. The use of this harmonized formatting means that satellite products need to be converted first to HARP compliant files. Once in this format, further processing can be performed.

Data formats

In order for HARP command line tools to handle each others output the toolkit makes uses of its own data format conventions.

There is one standard for representing and storing measurement data, which only prescribes a specific set of constraints to the data format and is still flexible enough to allow data storage using either netCDF, HDF4, or HDF5 and inclusion of metadata from other standards such as netCDF-CF.

The other data format convention is a standard for storing information on collocations, the Collocation Result File, which is based on CSV.

Both data formats are described in more detail in the Data formats section of the documentation.

Algorithms

HARP can perform a wide variety of operations on the data. A description of the algorithms as used in HARP can be found in the Algorithms section of the documentation.

Operations

Several HARP functions and tools can take a sequence of operations, such as filters or regridding operations, which are then performed on a product. A description of the operations expressions is provided in the Operations section of the documentation.

C library

Common parts that are used by the various HARP command line tools are gathered in a single HARP C library. This library can be used to build custom applications or libraries that work with HARP compliant data products.

IDL interface

The HARP IDL interface provides a set of functions to import and export HARP products, and to ingest non-HARP products of a type supported by HARP from IDL.

MATLAB interface

The HARP MATLAB interface provides a set of functions to import and export HARP products, and to ingest non-HARP products of a type supported by HARP from MATLAB.

Python interface

The HARP Python interface provides a set of functions to import and export HARP products, and to ingest non-HARP products of a type supported by HARP from Python. The Python interface depends on the _cffi_backend module, which is part of the C foreign function interface (cffi) package. This package must be installed in order to be able to use the Python interface. See the cffi documentation for details on how to install the cffi package.

R interface

The HARP R interface provides a set of functions to import and export HARP products, and to ingest non-HARP products of a type supported by HARP from R.

Command line tools

The HARP command line tools perform various operations on HARP compliant data products. All command line tools except harpconvert expect one or more HARP data products as input. The harpconvert tool can be used to convert non-HARP products into HARP products.

The available command line tools are described in more detail in the Command line tools section of the documentation.

A typical inter-comparison

Typical sequence when comparing two datasets of measurements:

harpconvert …: Convert all products that do not already follow the HARP format conventions to a common data format. Filter/convert the data in each file: remove unneeded parameters/measurements, add derived physical parameters, perform unit conversion, regrid dimensions, etc.
harpcollocate matchup …: Find the measurement pairs that match within the specified collocation criteria and save the result into a collocation result file.
harpcollocate resample …: Apply resampling to the collocation result file.
harpconvert -a ‘collocate-left(<csvfile>)’ …: Apply the collocation filter for each file in the primary dataset
harpconvert -a ‘collocate-right(<csvfile>)’ …: Apply the collocation filter for each file in the secondary dataset

Input files can be verified with

harpcheck …

and after each stage, the contents of each file can be inspected from the command line using:

harpdump …