CODA IDL

The CODA IDL interface consists of just a handful of IDL 'named structures' and functions. A primary design goal was to simplify, as much as possible, the number of different types and functions a user would have to master in order to effectively use the library.

Contents

CODA Definition Path

Note that in order to access products whose formats are defined using .codadef files, you should let CODA now where these .codadef files are stored. By default the CODA IDL interface will look for .codadef files in a directory relative to the location of the CODA IDL DLM file (../../../share/coda/definitions; note that software that embed CODA may sometimes override this default location). However, this will only work if you have set the IDL_DLM_PATH environment variable as specified in the CODA installation instructions for the IDL interface (otherwise no default location will be set).

You can override the default location by setting the CODA_DEFINITION environment variable. This environment variable should be a ':' separated (';' on Windows) list of absolute paths to directories containing .codadef files or absolute paths to .codadef files themselves (or a mix of those).

Named Structures

The CODA IDL interface defines three 'named structures'. This section discusses them in detail. For reference, the three types are shown in the table below.

namefieldstypeusage
CODA_DATAHANDLEPF_ID
OPAQUE		ULONG64
not user-accessible		represents a data-item in a product-file
CODA_ERRORERRNO
MESSAGE		INT
STRING		CODA call return status (can also indicate success)
CODA_NO_DATAOPAQUEBYTErepresents an empty data item; the OPAQUE field is a dummy value and should not be used

The CODA_DATAHANDLE named structure

The CODA_DATAHANDLE type represents a piece of data within a currently opened product-file; internally, it maintains a pointer to a certain offset within the data product, as well as information on the type of expression residing at that offset.

CODA_DATAHANDLE variables are used extensively 'under the hood' of the CODA IDL interface; however, there are two closely related cases where this type will also be visible at the IDL user level.

If a call to CODA_FETCH is made that should return an 'array of array' or 'array of record' type, the call will return an 'array of CODA_DATAHANDLE' instead.

The reason for this behavior is that the sophisticated type system supported by CODA cannot be mapped in full generality onto the IDL type system from within IDL DLMs. Within a DLM it is not posible to create IDL arrays where different elements have different storage sizes (and DLMs do not have public access to the 'pointer interface' that can be used from within IDL code). Due to the generality of the CODA interface it is currently also not possible to make an exception for the possible cases where the records/arrays might per-chance have the same storage size for each array element. For consistency, we have therefore chosen to handle all these cases uniformly, always returning an 'array of CODA_DATAHANDLE' when an 'array of array' or an 'array of record' is fetched. You can then manually traverse the 'array of CODA_DATAHANDLE', and use CODA_FETCH (providing a CODA_DATAHANDLE as first argument) to obtain the individual records or arrays represented by the CODA_DATAHANDLE elements.

Although the proper use of CODA_DATAHANDLE-arrays is probably the most subtle and difficult aspect of using the CODA IDL interface, it has proved to be quite easy to get used to. In fact, when used properly, the use of an 'array of CODA_DATAHANDLE' often improves performance as the CODA_DATAHANDLE elements effectively cache the information needed to access the data, thereby speeding up subsequent accesses.

The CODA_DATAHANDLE structure has two fields:

The meaning of a CODA_DATAHANDLE becomes undefined after the product to which it is associated is closed; You should not use it in subsequent CODA IDL interface operations. Doing so will generate an error.

The CODA_ERROR named structure

The CODA_ERROR structure may be returned by any CODA IDL interface function, usually indicating that the operation failed for some reason. Despite the name, you should think of a CODA_ERROR as a 'status report' of the last CODA IDL interface call; the status may, in fact, indicate successful completion.

For example, the CODA_CLOSE function will always return a value of type CODA_ERROR, even when the operation succeeds.

The CODA_ERROR has two fields. The ERRNO field gives a numerical error code. This code is guaranteed to be either 0 (zero), indicating success, or a negative number, indicating failure.

The MESSAGE field is a STRING that provides a human-readable description corresponding to the ERRNO field. Some errors originate from the underlying C library that the CODA IDL interface uses, others are specific to IDL. For example, if one of the parameters you pass to a CODA IDL interface function is incorrect you will get a CODA_ERROR indicating the type of error.

The CODA_NO_DATA named structure

The CODA_NO_DATA is used to represent unavailable record fields or zero-element arrays in IDL. CODA is capable of representing arrays that have zero indices in any dimension but such a data structure cannot be represented in IDL. Whenever CODA needs to represent a zero-element array, an instance of type CODA_NO_DATA is used instead.

The CODA_NO_DATA has one field. The OPAQUE field is present only to assure that the structure has a field (which is required by IDL). It does not contain any useful data.

Functions and Procedures

Function CODA_VERSION(): STRING

The CODA_VERSION function returns a string containing the current version number of CODA. The version number is always of the format 'x.y.z', i.e., major, minor, and revision numbers, separated by dots.

Function CODA_GETOPT(STRING option_name): INT

The CODA_GETOPT function returns the numerical value of a CODA IDL interface option. The following options are supported:

option namepossible valuesdefault valuemeaning
FilterRecordFields 0 or 1 1 if set, the CODA IDL interface will skip spare fields and fields that have a fixed value (such as header labels) when retrieving records, avoiding unnecessary clutter.
PerformConversions 0 or 1 1 if set, CODA will perform unit/value conversions for fields that have a 'conversion' defined in the CODA product format definitions.
PerformBoundaryChecks 0 or 1 1 if set, CODA will perform boundary checking on array accesses. This makes CODA more robust, at the cost of some performance. It is recommended to disable boundary checks only for thoroughly tested programs; disabling boundary checks can lead to wrong results and/or IDL crashes if arrays are accessed beyond their boundaries.
SwapDimensions 0 or 1 1 if set, CODA will swap the dimensions of multi-dimensional arrays (i.e. performing a multi-dimensional transpose on the data) so the array dimensions that are found in e.g. the CODA format definitions can be used as-is on the data. This distinction is needed because IDL uses Fortran-style ordering for array indices whereas CODA uses C-style array dimensioning ordering as the standard. If you disable this option, CODA will no longer transpose the data itself, but will invert the ordering of the array indices you pass to e.g. CODA_FETCH and the dimension sizes that are returned by CODA_SIZE.
TimeUnitDays 0 or 1 0 if set, the CODA IDL interface will represent all time values as days since 1-1-2000 instead of seconds since 1-1-2000. This is true both for fetched values and for values passed to CODA_TIME_TO_STRING. Use of this functionality is discouraged, as this will make data exchange between programs more difficult.
UseMMap 0 or 1 1 By default CODA uses a technique called 'memory mapping' to open and access data from product files. Using mmap greatly outperforms the default approach of reading data using the open()/read() combination. The downside of mapping a file into memory is that it takes away valuable address space. When you run a 32-bit Operating System your maximum addressable memory range is 4GB and if you simultaneously try to keep a few large product files open your memory space can quickly become full. Opening additional files will then produce 'out of memory' errors. Note that this 'out of memory' situation has nothing to do with the amount of RAM you have installed in your computer. It is only related to the size of a memory pointer on your system, which is limited to 4GB. If you are using CODA in a situation where you need to have multiple large product files open at the same time you can turn of the use of memory mapping by disabling this option. If you change the memory mapping option, the new setting will only be applicable for files that will be opened after you changed the option. Any files that were already open will keep using the mechanism with which they were opened.
UseSpecialTypes 0 or 1 1 if you disable this option, CODA will use the base type of a special type (and not the special type itself) when reading data or retrieving information about a data item. See the CODA Product Format Definitions documentation for more information about special types.
Verbose 0 or 1 1 if set, the CODA IDL interface will echo 'error' return values to the IDL command line just before returning them to the IDL user level. This can be helpful in detecting problems.

Function CODA_SETOPT(STRING option_name, INT new_value): INT

The CODA_SETOPT function is used to set the CODA IDL interface option to a new value. See the CODA_GETOPT description above for a list of allowed options and values, and their associated meanings.

The CODA_SETOPT function will return the previous value of the option.

Function CODA_OPEN(STRING filename): ULONG64

CODA_OPEN will attempt to open the file (which may fail, for example, if you specify a file that does not exist).

If CODA_OPEN succeeds, it returns a unique positive 64-bit unsigned integer that can subsequently be used to identifty the file in CODA_FETCH and CODA_CLOSE calls. If it fails a CODA_ERROR structure is returned giving the reason for failure.

It is perfectly OK to have multiple files open at the same time. Still, you are advised to close files as soon as you are done with them, since each of them consumes a vital system resource: address space. Note that this is not the same as 'memory': a data product is not read in its entirety into memory. A typical opened product file should consume a couple of tens of kilobytes of memory. However, the entire product file is virtually mapped into memory (a well-known technique that provides high-performance I/O access to a file). On a modern 32-bit computer (e.g., an Intel x86-class computer running Linux or Windows), up to 3 gigabytes worth of files may be mapped into memory at any one point in time.

In the CODA IDL interface, you can have up to 100 files open simultaneously. An attempt to open more files will yield an error.

Function CODA_OPEN_AS(STRING filename, STRING product_class, STRING product_type, INT version): ULONG64

CODA_OPEN_AS will try to open the specified file for reading similar to CODA_OPEN, but instead of trying to automatically recognise the applicable product class/type/version as CODA_OPEN does, this function will impose the format definition that is associated with the given product_class, product_type, and version parameters.

Note that you normally won't need this function as CODA will be able to recognize which format definition to use automatically. However, for the rare occasions where CODA_OPEN is not sufficient, you can use this function to force the use of a specific format definition.

You can specify -1 for the version to request the latest available version of the format definition.

Function CODA_CLOSE(ULONG64 pf_id): CODA_ERROR

The CODA_CLOSE function is used to release a previously-opened product file. Its one argument should be a 64-bit unsigned integer as returned by a previously executed successful CODA_OPEN call.

Function CODA_PRODUCT_CLASS(ULONG64 pf_id): STRING

The CODA_PRODUCT_CLASS function returns a string containing the product class of a product file if it has one (otherwise an empty string is returned).

Function CODA_PRODUCT_TYPE(ULONG64 pf_id): STRING

The CODA_PRODUCT_TYPE function returns a string containing the product type of a product file if it has one (otherwise an empty string is returned).

Function CODA_PRODUCT_VERSION(ULONG64 pf_id): INT

The CODA_PRODUCT_VERSION function returns an integer denoting the product format version. If the product does not have a format version number this functiion will return -1.

Function CODA_FETCH(...)

The CODA_FETCH function is used to fetch data from somewhere in an open product-file and turn it into an IDL variable.

CODA_FETCH: the first argument

Several forms of CODA_FETCH exist that are distinguished by the first argument argument. Depending on the form of the call, CODA_FETCH will try to read data from different header-parts or datasets in the product files. This is summarized below.

callFetch-root
CODA_FETCH(pf_id)Entire product
CODA_FETCH(pf_id, [, what, ...])Data (from product root)
CODA_FETCH(data_handle [, what, ...])Data described by handle

As noted before, in some cases the CODA_FETCH call may return an array of CODA_DATAHANDLE structures. The last form of the CODA_FETCH call allows you to traverse such an array and fetch the data described by each of the data-handles in turn.

CODA_FETCH: data-item specification arguments

After the first argument you can specify more arguments that tell CODA_FETCH to retrieve only a specific part of the data pointed to by the first argument. It is possible to have more than one such 'specification argument' in case of nested data-types.

Three forms of 'specification arguments' exist:

An example will help to understand all this. Suppose we have a product that contains a dataset called 'MEASUREMENTS', having 1000 records. Each of these records contains a field called 'samples' that is a 2-dimensional 101x101 array of doubles.

Now suppose we want to fetch the central 'sample' contained in record #10. The following CODA_FETCH would do the trick:

result = CODA_FETCH(pf_id, 'MEASUREMENTS', [10], 'samples', [50,50])

Alternatively, since the '[10]' specifies the index of a one-dimensional array, we could also write

result = CODA_FETCH(pf, 'MEASUREMENTS', 10, 'samples', [50,50])

Or we could use a single string parameter to provide the full path

result = CODA_FETCH(pf, 'MEASUREMENTS[10]/samples[5100]')

For the CODA_FETCH and CODA_FETCH_DATAHANDLE functions (but not for the other CODA IDL interface functions that use data-item specification arguments) there is an additional feature. If you provide a -1 for one or more of the dimensions of an array you will fetch all elements in this dimension. For example, with

result = CODA_FETCH(pf, 'MEASUREMENTS', -1, 'samples', [50,50])

you can fetch all central samples of all measurements into a single array. Similarly, you can use

result = CODA_FETCH(pf, 'MEASUREMENTS', 0, 'samples', [-1,0])

to fetch the first samples column of the first measurement.

The -1 parameter may only be used for one array in a fetch statement (e.g. CODA_FETCH(pf, 'MEASUREMENTS', 0, 'samples', [-1,-1]) is allowed, but CODA_FETCH(pf, 'MEASUREMENTS', -1, 'samples', [-1,0]) is not). Also, the -1 index only works when passing it as an explicit argument (i.e. calling coda_fetch(pf, "dataset[-1]/dsr_time[5100]"), where the -1 index is part of a string argument, will not work; calling coda_fetch(pf, "dataset", -1, "dsr_time[5100]") will work).

CODA_FETCH return type

The IDL type of the CODA_FETCH return-value depends on the CODA data type of the data item being fetched. Three cases are distinguished:

Return types for simple (non-composite) types

If the data item has a basic type the return type is determined according to the following table.

CODA read typeIDL Typeremark
int8INT 
uint8BYTEa BYTE is unsigned in IDL
int16INT 
uint16UINT 
int32LONG 
uint32ULONG 
int64LONG64 
uint64ULONG64 
floatFLOAT 
doubleDOUBLE 
charSTRING 
stringSTRING 
bytesBYTEThis will actually translate into an array of type BYTE

CODA_FETCH return types for basic data types

Return types for Record types

Record types will be returned as anonymous (unnamed) IDL structures, where each field will correspond to a field in the original data set. If a field has the hidden property and if the FilterRecordFields option is set then the field will not be copied into the IDL structure.

For basic types, the IDL field type will be determined according to the 'basic data types' table given above. Fields that are themselves records are expanded as 'nested records'. Fields that are themselves arrays are expanded as 'array types', as discussed below.

If a record contains no fields (i.e. an empty record) then CODA will return a CODA_NO_DATA named structure instead of an empty anonymous struct.

Return types for Array types

Array types will be returned as multi-dimensional IDL arrays, with the IDL array base-type derived from the base-type of the CODA type according to the 'basic data types' table given above. There are, however, some complications that IDL programmers need to be aware of when the base type itself is either an array or a record.

For array and record base-types (i.e., 'array of array', 'array of record'), there is, in general, no equivalent type in IDL. As explained in the section about the CODA_DATAHANDLE structure above, both these cases will yield an 'array of CODA_DATAHANDLE' instead.

If an array is empty (i.e. the size of one of its dimensions is 0) then CODA will return a CODA_NO_DATA named structure instead of a multi-dimensional IDL array.

Function CODA_FETCH_DATAHANDLE(...): CODA_DATAHANDLE

This function takes the same kind of argument as CODA_FETCH; however, instead of returning the specified data from the product, it always returns a datahandle to the specified data (even if it would be easily representable in IDL).

Function CODA_EVAL(STRING expression, ...)

This function takes a CODA expression in the form of a string and an optional product location. The product location is the same kind of argument (list) as used for CODA_FETCH. If the CODA expression can be evaluated statically, then the product location argument is not required, otherwise the product location argument is mandatory.

The syntax for the CODA expression language can be found in the CODA Expressions documentation. Note that evaluation of 'void' expressions is not supported.

Function CODA_ATTRIBUTES(...)

This function takes the same kind of argument as CODA_FETCH; however, instead of returning the specified data from the product, it returns a record containing all attributes of the specified data item. The record can be empty if no attributes are available.

Function CODA_FIELDAVAILABLE(...): INT

This function takes the same kind of argument as CODA_FETCH; however, instead of returning the specified data from the product, it returns the availabillity of a record field. It returns 1 if it is available and 0 if it is not.

If the arguments to CODA_FIELDAVAILABLE do not point to a record, a CODA_ERROR structure is returned.

Function CODA_FIELDCOUNT(...): INT

This function takes the same kind of argument as CODA_FETCH; however, instead of returning the specified data from the product, it returns the number of fields in a record.

If the arguments to CODA_FIELDCOUNT do not point to a record, a CODA_ERROR structure is returned.

Function CODA_FIELDNAMES(...): STRING-array

This function takes the same kind of argument as CODA_FETCH; however, instead of returning the specified data from the product, it returns the field-names of the fields in a record.

If the arguments to CODA_FIELDNAMES do not point to a record, a CODA_ERROR structure is returned.

Function CODA_SIZE(...): INT-array

This function takes the same kind of argument as CODA_FETCH; however, instead of returning the specified data from the product, it returns the actual dimensionality of a multi-dimensional array.

If the arguments to CODA_SIZE do not point to an array, a CODA_ERROR structure is returned.

Function CODA_DESCRIPTION(...): STRING

This function takes the same kind of argument as CODA_FETCH; however, instead of returning the specified data from the product, it returns the description of a data item.

Function CODA_UNIT(...): STRING

This function takes the same kind of argument as CODA_FETCH; however, instead of returning the specified data from the product, it returns the unit of a data item.

Function CODA_TIME_TO_STRING(numerical argument): STRING

CODA by default stores time values as a double denoting the number of seconds since Midnight, January 1st, 2000 UTC.

The CODA_TIME_TO_STRING function takes a DOUBLE-encoded time value and converts it to a human-readable string of the form "2000-01-01 00:00:00.000000"

Function CODA_IS_NO_DATA(...): INT

The CODA_IS_NO_DATA function takes one argument; it returns a value of TRUE (1) if and only if the argument is a CODA_EMPTY_ARRAY structure.

Function CODA_IS_ERROR(...): INT

The CODA_IS_ERROR function takes one argument; it returns a value of TRUE (1) if and only if

If either of these is not true, CODA_IS_ERROR returns FALSE (0).

Using CODA_IS_ERROR, you can make your CODA-based IDL programs robust (i.e., resistant to failures). If you test the result of any CODA IDLinterface call with CODA_IS_ERROR, you should be able to intercept any error.

Procedure CODA_UNLOAD

The CODA_UNLOAD procedure will close all opened product files and unload any chached CODA product format definitions from memory.

The CODA product format definitions are loaded on-demand if any CODA function that needs it is invoked.

This function may be (slightly) useful on systems with little memory. You could open a product file, fetch the important data, close the file, and then unload the product format definitions to free extra memory before starting data-processing. Other than that, this function is of little practical use.

Copyright © 2007-2022 s[&]t, The Netherlands.