ncgen(1) UNIDATA UTILITIES ncgen(1)
NAME
ncgen - From a CDL file generate a netCDF-3 file, a netCDF-4 file or a C
program
SYNOPSIS
ncgen [-format_code] [-1|3|4|5|6|7] [-b] [-c] [-d] [-D debuglevel] [-f]
[-h] [-H] [-k format_name] [-l b|c|f77|java] [-L loglevel] [-M
name] [-n] [-N datasetname] [-o netcdf_filename] [-P] [-x]
DESCRIPTION
ncgen generates either a netCDF-3 (i.e. classic) binary .nc file, a
netCDF-4 (i.e. enhanced) binary .nc file or a file in some source
language that when executed will construct the corresponding binary .nc
file. The input to ncgen is a description of a netCDF file in a small
language known as CDL (network Common Data form Language), described
below. Input is read from standard input if no input_file is specified.
If no options are specified in invoking ncgen, it merely checks the
syntax of the input CDL file, producing error messages for any violations
of CDL syntax. Other options can be used, for example, to create the
corresponding netCDF file, or to generate a C program that uses the
netCDF C interface to create the netCDF file.
Note that this version of ncgen was originally called ncgen4. The older
ncgen program has been renamed to ncgen3.
ncgen may be used with the companion program ncdump to perform some
simple operations on netCDF files. For example, to rename a dimension in
a netCDF file, use ncdump to get a CDL version of the netCDF file, edit
the CDL file to change the name of the dimensions, and use ncgen to
generate the corresponding netCDF file from the edited CDL file.
OPTIONS
-1|3|4|5|6|7
Alternate method to specify the format.
3 => netcdf classic format
4 => netCDF-4 format (enhanced data model)
5 => netcdf 5 format
6 => netCDF 64-bit format
7 => netCDF-4 classic model format (3+4 == 7)
See the -k flag.
-b Create a (binary) netCDF file. If the -o option is absent, a
default file name will be constructed from the basename of the CDL
file, with any suffix replaced by the `.nc' extension. If a file
already exists with the specified name, it will be overwritten.
-c Generate C source code that will create a netCDF file matching the
netCDF specification. The C source code is written to standard
output; equivalent to -lc.
-d Same as -D1.
-D debuglevel
Set the level of debug output.
-f Generate FORTRAN 77 source code that will create a netCDF file
matching the netCDF specification. The source code is written to
standard output; equivalent to -lf77.
-h Output help information.
-H Output the header only; ignore the data section.
-k format_name
-format_code
The -k flag specifies the format of the file to be created and, by
inference, the data model accepted by ncgen (i.e. netcdf-3
(classic) versus netcdf-4 vs netcdf-5). As a shortcut, a numeric
format_code may be specified instead. The possible format_name
values for the -k option are:
'classic' or 'nc3' => netCDF classic format
'64-bit offset' or 'nc6' => netCDF 64-bit format
'64-bit data or 'nc5' => netCDF-5 (64-bit data) format
'netCDF-4' 0r 'nc4' => netCDF-4 format (enhanced data
model)
'netCDF-4 classic model' or 'nc7' => netCDF-4 classic model
format
Accepted format_code numeric arguments, just shortcuts for format_names,
are:
3 => netcdf classic format
5 => netcdf 5 format
6 => netCDF 64-bit format
4 => netCDF-4 format (enhanced data model)
7 => netCDF-4 classic model format
The numeric code "7" is used because "7=3+4", a mnemonic for the format
that uses the netCDF-3 data model for compatibility with the netCDF-4
storage format for performance. Credit is due to NCO for use of these
numeric codes instead of the old and confusing format numbers.
Note: The old version format numbers '1', '2', '3', '4', equivalent to
the format names 'nc3', 'nc6', 'nc4', or 'nc7' respectively, are also
still accepted but deprecated, due to easy confusion between format
numbers and format names. Various old format name aliases are also
accepted but deprecated, e.g. 'hdf5', 'enhanced-nc3', etc. Also, note
that -v is accepted to mean the same thing as -k for backward
compatibility.
-l b|c|f77|java
The -l flag specifies the output language to use when generating
source code that will create or define a netCDF file matching the
netCDF specification. The output is written to standard output.
The currently supported languages have the following flags.
c|C' => C language output.
f77|fortran77' => FORTRAN 77 language output
; note that currently only the classic model is
supported.
j|java' => (experimental) Java language output
; targets the existing Unidata Java interface, which
means that only the classic model is supported.
Choosing the output format
The choice of output format is determined by three flags.
-k flag.
_Format attribute (see below).
Occurrence of CDF-5 (64-bit data) or
netcdf-4 constructs in the input CDL." The term "netCDF-4
constructs" means constructs from the enhanced data model, not
just special performance-related attributes such as
_ChunkSizes, _DeflateLevel, _Endianness, etc. The term "CDF-5
constructs" means extended unsigned integer types allowed in the
64-bit data model.
Note that there is an ambiguity between the netCDF-4 case and the CDF-5
case is only an unsigned type is seen in the input.
The rules are as follows, in order of application.
1. If either Fortran or Java output is specified, then -k flag value
of 1 (classic model) will be used. Conflicts with the use of
enhanced constructs in the CDL will report an error.
2. If both the -k flag and _Format attribute are specified, the
_Format flag will be ignored. If no -k flag is specified, and a
_Format attribute value is specified, then the -k flag value will
be set to that of the _Format attribute. Otherwise the -k flag is
undefined.
3. If the -k option is defined and is consistent with the CDL, ncgen
will output a file in the requested form, else an error will be
reported.
4. If the -k flag is undefined, and if there are CDF-5 constructs,
only, in the CDL, a -k flag value of 5 (64-bit data model) will be
used. If there are true netCDF-4 constructs in the CDL, a -k flag
value of 3 (enhanced model) will be used.
5. If special performance-related attributes are specified in the
CDL, a -k flag value of 4 (netCDF-4 classic model) will be used.
6. Otherwise ncgen will set the -k flag to 1 (classic model).
-L loglevel
-M name
Specify the name for the main function for C, F77, or Java.
-n
-N datasetname
-o netcdf_file
Name of the file to pass to calls to "nc_create()". If this
option is specified it implies (in the absence of any explicit -l
flag) the "-b" option. This option is necessary because netCDF
files cannot be written directly to standard output, since
standard output is not seekable.
-P Use NC_DISKLESS mode to create the file totally in memory before
persisting it to disk.
-W maxwholevarsize
Set wholevarsizem where if total number of elements is less than
maxwholevarsize then updata a variable using a single nc_put_var.
Requires that the variable has no unlimited dimensions.
-x Don't initialize data with fill values. This can speed up
creation of large netCDF files greatly, but later attempts to read
unwritten data from the generated file will not be easily
detectable.
EXAMPLES
Check the syntax of the CDL file `foo.cdl':
ncgen foo.cdl
From the CDL file `foo.cdl', generate an equivalent binary netCDF file
named `x.nc':
ncgen -o x.nc foo.cdl
From the CDL file `foo.cdl', generate a C program containing the netCDF
function invocations necessary to create an equivalent binary netCDF file
named `x.nc':
ncgen -lc foo.cdl >x.c
USAGE
CDL Syntax Overview
Below is an example of CDL syntax, describing a netCDF file with several
named dimensions (lat, lon, and time), variables (Z, t, p, rh, lat, lon,
time), variable attributes (units, long_name, valid_range, _FillValue),
and some data. CDL keywords are in boldface. (This example is intended
to illustrate the syntax; a real CDL file would have a more complete set
of attributes so that the data would be more completely self-describing.)
netcdf foo { // an example netCDF specification in CDL
types:
ubyte enum enum_t {Clear = 0, Cumulonimbus = 1, Stratus = 2};
opaque(11) opaque_t;
int(*) vlen_t;
dimensions:
lat = 10, lon = 5, time = unlimited ;
variables:
long lat(lat), lon(lon), time(time);
float Z(time,lat,lon), t(time,lat,lon);
double p(time,lat,lon);
long rh(time,lat,lon);
string country(time,lat,lon);
ubyte tag;
// variable attributes
lat:long_name = "latitude";
lat:units = "degrees_north";
lon:long_name = "longitude";
lon:units = "degrees_east";
time:units = "seconds since 1992-1-1 00:00:00";
// typed variable attributes
string Z:units = "geopotential meters";
float Z:valid_range = 0., 5000.;
double p:_FillValue = -9999.;
long rh:_FillValue = -1;
vlen_t :globalatt = {17, 18, 19};
data:
lat = 0, 10, 20, 30, 40, 50, 60, 70, 80, 90;
lon = -140, -118, -96, -84, -52;
group: g {
types:
compound cmpd_t { vlen_t f1; enum_t f2;};
} // group g
group: h {
variables:
/g/cmpd_t compoundvar;
data:
compoundvar = { {3,4,5}, enum_t.Stratus } ;
} // group h
}
All CDL statements are terminated by a semicolon. Spaces, tabs, and
newlines can be used freely for readability. Comments may follow the
characters `//' on any line.
A CDL description consists of five optional parts: types, dimensions,
variables, data, beginning with the keyword `types:', `dimensions:',
`variables:', and `data:', respectively. Note several things: (1) the
keyword includes the trailing colon, so there must not be any space
before the colon character, and (2) the keywords are required to be lower
case.
The variables: section may contain variable declarations and attribute
assignments. All sections may contain global attribute assignments.
In addition, after the data: section, the user may define a series of
groups (see the example above). Groups themselves can contain types,
dimensions, variables, data, and other (nested) groups.
The netCDF types: section declares the user defined types. These may be
constructed using any of the following types: enum, vlen, opaque, or
compound.
A netCDF dimension is used to define the shape of one or more of the
multidimensional variables contained in the netCDF file. A netCDF
dimension has a name and a size. A dimension can have the unlimited
size, which means a variable using this dimension can grow to any length
in that dimension.
A variable represents a multidimensional array of values of the same
type. A variable has a name, a data type, and a shape described by its
list of dimensions. Each variable may also have associated attributes
(see below) as well as data values. The name, data type, and shape of a
variable are specified by its declaration in the variable section of a
CDL description. A variable may have the same name as a dimension; by
convention such a variable is one-dimensional and contains coordinates of
the dimension it names. Dimensions need not have corresponding
variables.
A netCDF attribute contains information about a netCDF variable or about
the whole netCDF dataset. Attributes are used to specify such properties
as units, special values, maximum and minimum valid values, scaling
factors, offsets, and parameters. Attribute information is represented
by single values or arrays of values. For example, "units" is an
attribute represented by a character array such as "celsius". An
attribute has an associated variable, a name, a data type, a length, and
a value. In contrast to variables that are intended for data, attributes
are intended for metadata (data about data). Unlike netCDF-3, attribute
types can be any user defined type as well as the usual built-in types.
In CDL, an attribute is designated by a a type, a variable, a ':', and
then an attribute name. The type is optional and if missing, it will be
inferred from the values assigned to the attribute. It is possible to
assign global attributes not associated with any variable to the netCDF
as a whole by omitting the variable name in the attribute declaration.
Notice that there is a potential ambiguity in a specification such as
x : a = ...
In this situation, x could be either a type for a global attribute, or
the variable name for an attribute. Since there could both be a type
named x and a variable named x, there is an ambiguity. The rule is that
in this situation, x will be interpreted as a type if possible, and
otherwise as a variable.
If not specified, the data type of an attribute in CDL is derived from
the type of the value(s) assigned to it. The length of an attribute is
the number of data values assigned to it, or the number of characters in
the character string assigned to it. Multiple values are assigned to
non-character attributes by separating the values with commas. All
values assigned to an attribute must be of the same type.
The names for CDL dimensions, variables, attributes, types, and groups
may contain any non-control utf-8 character except the forward slash
character (`/'). However, certain characters must escaped if they are
used in a name, where the escape character is the backward slash `\'. In
particular, if the leading character off the name is a digit (0-9), then
it must be preceded by the escape character. In addition, the characters
` !"#$%&()*,:;<=>?[]^`'{}|~\' must be escaped if they occur anywhere in a
name. Note also that attribute names that begin with an underscore (`_')
are reserved for the use of Unidata and should not be used in user
defined attributes.
Note also that the words `variables', `dimensions', `data', `group', and
`types' are legal CDL names, but be careful that there is a space between
them and any following colon character when used as a variable name.
This is mostly an issue with attribute declarations. For example,
consider this.
netcdf ... {
...
variables:
int dimensions;
dimensions: attribute=0 ; // this will cause an error
dimensions : attribute=0 ; // this is ok.
...
}
The optional data: section of a CDL specification is where netCDF
variables may be initialized. The syntax of an initialization is simple:
a variable name, an equals sign, and a comma-delimited list of constants
(possibly separated by spaces, tabs and newlines) terminated with a
semicolon. For multi-dimensional arrays, the last dimension varies
fastest. Thus row-order rather than column order is used for matrices.
If fewer values are supplied than are needed to fill a variable, it is
extended with a type-dependent `fill value', which can be overridden by
supplying a value for a distinguished variable attribute named
`_FillValue'. The types of constants need not match the type declared
for a variable; coercions are done to convert integers to floating point,
for example. The constant `_' can be used to designate the fill value
for a variable. If the type of the variable is explicitly `string', then
the special constant `NIL` can be used to represent a nil string, which
is not the same as a zero length string.
Primitive Data Types
char characters
byte 8-bit data
short 16-bit signed integers
int 32-bit signed integers
long (synonymous with int)
int64 64-bit signed integers
float IEEE single precision floating point (32 bits)
real (synonymous with float)
double IEEE double precision floating point (64 bits)
ubyte unsigned 8-bit data
ushort 16-bit unsigned integers
uint 32-bit unsigned integers
uint64 64-bit unsigned integers
string arbitrary length strings
CDL supports a superset of the primitive data types of C. The names for
the primitive data types are reserved words in CDL, so the names of
variables, dimensions, and attributes must not be primitive type names.
In declarations, type names may be specified in either upper or lower
case.
Bytes are intended to hold a full eight bits of data, and the zero byte
has no special significance, as it mays for character data. ncgen
converts byte declarations to char declarations in the output C code and
to the nonstandard BYTE declaration in output Fortran code.
Shorts can hold values between -32768 and 32767. ncgen converts short
declarations to short declarations in the output C code and to the
nonstandard INTEGER*2 declaration in output Fortran code.
Ints can hold values between -2147483648 and 2147483647. ncgen converts
int declarations to int declarations in the output C code and to INTEGER
declarations in output Fortran code. long is accepted as a synonym for
int in CDL declarations, but is deprecated since there are now platforms
with 64-bit representations for C longs.
Int64 can hold values between -9223372036854775808 and
9223372036854775807. ncgen converts int64 declarations to longlong
declarations in the output C code.
Floats can hold values between about -3.4+38 and 3.4+38. Their external
representation is as 32-bit IEEE normalized single-precision floating
point numbers. ncgen converts float declarations to float declarations
in the output C code and to REAL declarations in output Fortran code.
real is accepted as a synonym for float in CDL declarations.
Doubles can hold values between about -1.7+308 and 1.7+308. Their
external representation is as 64-bit IEEE standard normalized double-
precision floating point numbers. ncgen converts double declarations to
double declarations in the output C code and to DOUBLE PRECISION
declarations in output Fortran code.
The unsigned counterparts of the above integer types are mapped to the
corresponding unsigned C types. Their ranges are suitably modified to
start at zero.
The technical interpretation of the char type is that it is an unsigned
8-bit value. The encoding of the 256 possible values is unspecified by
default. A variable of char type may be marked with an "_Encoding"
attribute to indicate the character set to be used: US-ASCII, ISO-8859-1,
etc. Note that specifying the encoding of UTF-8 is equivalent to
specifying US-ASCII This is because multi-byte UTF-8 characters cannot be
stored in an 8-bit character. The only legal single byte UTF-8 values are
by definition the 7-bit US-ASCII encoding with the top bit set to zero.
Strings are assumed by default to be encoded using UTF-8. Note that this
means that multi-byte UTF-8 encodings may be present in the string, so it
is possible that the number of distinct UTF-8 characters in a string is
smaller than the number of 8-bit bytes used to store the string.
CDL Constants
Constants assigned to attributes or variables may be of any of the basic
netCDF types. The syntax for constants is similar to C syntax, except
that type suffixes must be appended to shorts and floats to distinguish
them from longs and doubles.
A byte constant is represented by an integer constant with a `b' (or `B')
appended. In the old netCDF-2 API, byte constants could also be
represented using single characters or standard C character escape
sequences such as `a' or `0. This is still supported for backward
compatibility, but deprecated to make the distinction clear between the
numeric byte type and the textual char type. Example byte constants
include:
0b // a zero byte
-1b // -1 as an 8-bit byte
255b // also -1 as a signed 8-bit byte
short integer constants are intended for representing 16-bit signed
quantities. The form of a short constant is an integer constant with an
`s' or `S' appended. If a short constant begins with `0', it is
interpreted as octal, except that if it begins with `0x', it is
interpreted as a hexadecimal constant. For example:
-2s // a short -2
0123s // octal
0x7ffs //hexadecimal
int integer constants are intended for representing 32-bit signed
quantities. The form of an int constant is an ordinary integer constant,
although it is acceptable to optionally append a single `l' or `L'
(again, deprecated). Be careful, though, the L suffix is interpreted as a
32 bit integer, and never as a 64 bit integer. This can be confusing
since the C long type can ambigously be either 32 bit or 64 bit.
If an int constant begins with `0', it is interpreted as octal, except
that if it begins with `0x', it is interpreted as a hexadecimal constant
(but see opaque constants below). Examples of valid int constants
include:
-2
1234567890L
0123 // octal
0x7ff // hexadecimal
int64 integer constants are intended for representing 64-bit signed
quantities. The form of an int64 constant is an integer constant with an
`ll' or `LL' appended. If an int64 constant begins with `0', it is
interpreted as octal, except that if it begins with `0x', it is
interpreted as a hexadecimal constant. For example:
-2ll // an unsigned -2
0123LL // octal
0x7ffLL //hexadecimal
Floating point constants of type float are appropriate for representing
floating point data with about seven significant digits of precision.
The form of a float constant is the same as a C floating point constant
with an `f' or `F' appended. For example the following are all
acceptable float constants:
-2.0f
3.14159265358979f // will be truncated to less precision
1.f
Floating point constants of type double are appropriate for representing
floating point data with about sixteen significant digits of precision.
The form of a double constant is the same as a C floating point constant.
An optional `d' or `D' may be appended. For example the following are
all acceptable double constants:
-2.0
3.141592653589793
1.0e-20
1.d
Unsigned integer constants can be created by appending the character 'U'
or 'u' between the constant and any trailing size specifier, or
immediately at the end of the size specifier. Thus one could say 10U,
100su, 100000ul, or 1000000llu, for example.
Single character constants may be enclosed in single quotes. If a
sequence of one or more characters is enclosed in double quotes, then its
interpretation must be inferred from the context. If the dataset is
created using the netCDF classic model, then all such constants are
interpreted as a character array, so each character in the constant is
interpreted as if it were a single character. If the dataset is netCDF
extended, then the constant may be interpreted as for the classic model
or as a true string (see below) depending on the type of the attribute or
variable into which the string is contained.
The interpretation of char constants is that those that are in the
printable ASCII range (' '..'~') are assumed to be encoded as the 1-byte
subset ofUTF-8, which is equivalent to US-ASCII. In all cases, the usual
C string escape conventions are honored for values from 0 thru 127.
Values greater than 127 are allowed, but their encoding is undefined.
For netCDF extended, the use of the char type is deprecated in favor of
the string type.
Some character constant examples are as follows.
'a' // ASCII `a'
"a" // equivalent to 'a'
"Two\nlines\n" // a 10-character string with two embedded newlines
"a bell:\007" // a string containing an ASCII bell
Note that the netCDF character array "a" would fit in a one-element
variable, since no terminating NULL character is assumed. However, a
zero byte in a character array is interpreted as the end of the
significant characters by the ncdump program, following the C convention.
Therefore, a NULL byte should not be embedded in a character string
unless at the end: use the byte data type instead for byte arrays that
contain the zero byte.
String constants are, like character constants, represented using double
quotes. This represents a potential ambiguity since a multi-character
string may also indicate a dimensioned character value. Disambiguation
usually occurs by context, but care should be taken to specify thestring
type to ensure the proper choice. String constants are assumed to always
be UTF-8 encoded. This specifically means that the string constant may
actually contain multi-byte UTF-8 characters. The special constant `NIL`
can be used to represent a nil string, which is not the same as a zero
length string.
Opaque constants are represented as sequences of hexadecimal digits
preceded by 0X or 0x: 0xaa34ffff, for example. These constants can still
be used as integer constants and will be either truncated or extended as
necessary.
Compound Constant Expressions
In order to assign values to variables (or attributes) whose type is
user-defined type, the constant notation has been extended to include
sequences of constants enclosed in curly brackets (e.g. "{"..."}"). Such
a constant is called a compound constant, and compound constants can be
nested.
Given a type "T(*) vlen_t", where T is some other arbitrary base type,
constants for this should be specified as follows.
vlen_t var[2] = {t11,t12,...t1N}, {t21,t22,...t2m};
The values tij, are assumed to be constants of type T.
Given a type "compound cmpd_t {T1 f1; T2 f2...Tn fn}", where the Ti are
other arbitrary base types, constants for this should be specified as
follows.
cmpd_t var[2] = {t11,t12,...t1N}, {t21,t22,...t2n};
The values tij, are assumed to be constants of type Ti. If the fields
are missing, then they will be set using any specified or default fill
value for the field's base type.
The general set of rules for using braces are defined in the Specifying
Datalists section below.
Scoping Rules
With the addition of groups, the name space for defined objects is no
longer flat. References (names) of any type, dimension, or variable may
be prefixed with the absolute path specifying a specific declaration.
Thus one might say
variables:
/g1/g2/t1 v1;
The type being referenced (t1) is the one within group g2, which in turn
is nested in group g1. The similarity of this notation to Unix file
paths is deliberate, and one can consider groups as a form of directory
structure.
When name is not prefixed, then scope rules are applied to locate the
specified declaration. Currently, there are three rules: one for
dimensions, one for types and enumeration constants, and one for all
others.
When an unprefixed name of a dimension is used (as in a variable
declaration), ncgen first looks in the immediately enclosing group
for the dimension. If it is not found there, then it looks in the
group enclosing this group. This continues up the group hierarchy
until the dimension is found, or there are no more groups to
search.
2. When an unprefixed name of a type or an enumeration constant is used,
ncgen searches the group tree using a pre-order depth-first
search. This essentially means that it will find the matching
declaration that precedes the reference textually in the cdl file
and that is "highest" in the group hierarchy.
3. For all other names, only the immediately enclosing group is searched.
One final note. Forward references are not allowed. This means that
specifying, for example, /g1/g2/t1 will fail if this reference occurs
before g1 and/or g2 are defined.
Specifying Enumeration Constants
References to Enumeration constants (in data lists) can be ambiguous
since the same enumeration constant name can be defined in more than one
enumeration. If a cdl file specified an ambiguous constant, then ncgen
will signal an error. Such constants can be disambiguated in two ways.
1. Prefix the enumeration constant with the name of the enumeration
separated by a dot: enum.econst, for example.
2. If case one is not sufficient to disambiguate the enumeration
constant, then one must specify the precise enumeration type using
a group path: /g1/g2/enum.econst, for example.
Special Attributes
Special, virtual, attributes can be specified to provide performance-
related information about the file format and about variable properties.
The file must be a netCDF-4 file for these to take effect.
These special virtual attributes are not actually part of the file, they
are merely a convenient way to set miscellaneous properties of the data
in CDL
The special attributes currently supported are as follows: `_Format',
`_Fletcher32, `_ChunkSizes', `_Endianness', `_DeflateLevel', `_Shuffle',
and `_Storage'.
`_Format' is a global attribute specifying the netCDF format variant. Its
value must be a single string matching one of `classic', `64-bit offset',
`64-bit data', `netCDF-4', or `netCDF-4 classic model'.
The rest of the special attributes are all variable attributes.
Essentially all of then map to some corresponding `nc_def_var_XXX'
function as defined in the netCDF-4 API. For the attributes that are
essentially boolean (_Fletcher32, _Shuffle, and _NOFILL), the value true
can be specified by using the strings `true' or `1', or by using the
integer 1. The value false expects either `false', `0', or the integer
0. The actions associated with these attributes are as follows.
1. `_Fletcher32 sets the `fletcher32' property for a variable.
2. `_Endianness' is either `little' or `big', depending on how the
variable is stored when first written.
3. `_DeflateLevel' is an integer between 0 and 9 inclusive if compression
has been specified for the variable.
4. `_Shuffle' specifies if the the shuffle filter should be used.
5. `_Storage' is `contiguous' or `compact` or `chunked'.
6. `_ChunkSizes' is a list of chunk sizes for each dimension of the
variable
Note that attributes such as "add_offset" or "scale_factor" have no
special meaning to ncgen. These attributes are currently conventions,
handled above the library layer by other utility packages, for example
NCO.
Specifying Datalists
Specifying datalists for variables in the `data:` section can be somewhat
complicated. There are some rules that must be followed to ensure that
datalists are parsed correctly by ncgen.
First, the top level is automatically assumed to be a list of items, so
it should not be inside {...}. That means that if the variable is a
scalar, there will be a single top-level element and if the variable is
an array, there will be N top-level elements. For each element of the
top level list, the following rules should be applied.
1. Instances of UNLIMITED dimensions (other than the first dimension)
must be surrounded by {...} in order to specify the size.
2. Compound instances must be embedded in {...}
3. Non-scalar fields of compound instances must be embedded in {...}.
4. Instances of vlens must be surrounded by {...} in order to specify the
size.
Datalists associated with attributes are implicitly a vector (i.e., a
list) of values of the type of the attribute and the above rules must
apply with that in mind.
7. No other use of braces is allowed.
Note that one consequence of these rules is that arrays of values cannot
have subarrays within braces. Consider, for example, int
var(d1)(d2)...(dn), where none of d2...dn are unlimited. A datalist for
this variable must be a single list of integers, where the number of
integers is no more than D=d1*d2*...dn values; note that the list can be
less than D, in which case fill values will be used to pad the list.
Rule 6 about attribute datalist has the following consequence. If the
type of the attribute is a compound (or vlen) type, and if the number of
entries in the list is one, then the compound instances must be enclosed
in braces.
Specifying Character Datalists
Specifying datalists for variables of type char also has some
complications. consider, for example
dimensions: u=UNLIMITED; d1=1; d2=2; d3=3;
d4=4; d5=5; u2=UNLIMITED;
variables: char var(d4,d5);
datalist: var="1", "two", "three";
We have twenty elements of var to fill (d5 X d4) and we have three
strings of length 1, 3, 5. How do we assign the characters in the
strings to the twenty elements?
This is challenging because it is desirable to mimic the original ncgen
(ncgen3). The core algorithm is notionally as follows.
1. Assume we have a set of dimensions D1..Dn, where D1 may optionally be
an Unlimited dimension. It is assumed that the sizes of the Di are
all known (including unlimited dimensions).
2. Given a sequence of string or character constants C1..Cm, our goal is
to construct a single string whose length is the cross product of D1
thru Dn. Note that for purposes of this algorithm, character
constants are treated as strings of size 1.
3. Construct Dx = cross product of D1 thru D(n-1).
4. For each constant Ci, add fill characters as needed so that its length
is a multiple of Dn.
5. Concatenate the modified C1..Cm to produce string S.
6. Add fill characters to S to make its length be a multiple of Dn.
8. If S is longer than the Dx * Dn, then truncate and generate a warning.
There are three other cases of note.
1. If there is only a single, unlimited dimension, then all of the
constants are concatenated and fill characters are added to the end of
the resulting string to make its length be that of the unlimited
dimension. If the length is larger than the unlimited dimension, then
it is truncated with a warning.
2. For the case of character typed vlen, "char(*) vlen_t" for example.
we simply concatenate all the constants with no filling at all.
3. For the case of a character typed attribute, we simply concatenate all
the constants.
In netcdf-4, dimensions other than the first can be unlimited. Of course
by the rules above, the interior unlimited instances must be delimited by
{...}. For example.
variables: char var(u,u2);
datalist: var={"1", "two"}, {"three"};
In this case u will have the effective length of two. Within each
instance of u2, the rules above will apply, leading to this.
datalist: var={"1","t","w","o"}, {"t","h","r","e","e"};
The effective size of u2 will be the max of the two instance lengths
(five in this case) and the shorter will be padded to produce this.
datalist: var={"1","t","w","o","\0"}, {"t","h","r","e","e"};
Consider an even more complicated case.
variables: char var(u,u2,u3);
datalist: var={{"1", "two"}}, {{"three"},{"four","xy"}};
In this case u again will have the effective length of two. The u2
dimensions will have a size = max(1,2) = 2; Within each instance of u2,
the rules above will apply, leading to this.
datalist: var={{"1","t","w","o"}}, {{"t","h","r","e","e"},{"f","o","u","r","x","y"}};
The effective size of u3 will be the max of the two instance lengths
(six in this case) and the shorter ones will be padded to produce this.
datalist: var={{"1","t","w","o"," "," "}}, {{"t","h","r","e","e"," "},{"f","o","u","r","x","y"}};
Note however that the first instance of u2 is less than the max length of
u2, so we need to add a filler for another instance of u2, producing
this.
datalist: var={{"1","t","w","o"," "," "},{" "," "," "," "," "," "}}, {{"t","h","r","e","e"," "},{"f","o","u","r","x","y"}};
BUGS
The programs generated by ncgen when using the -c flag use initialization
statements to store data in variables, and will fail to produce
compilable programs if you try to use them for large datasets, since the
resulting statements may exceed the line length or number of continuation
statements permitted by the compiler.
The CDL syntax makes it easy to assign what looks like an array of
variable-length strings to a netCDF variable, but the strings may simply
be concatenated into a single array of characters. Specific use of the
string type specifier may solve the problem
Identifiers and Keywords
Under certain conditions, some keywords can be used as identifiers.
1. If a type keyword is not a type supported by the format of the
.cdl file, then it can be used as an identifier. So, for example,
when translating a .cdl file as a netCDF-3 file, then "string" or
"uint64" can be used as identifiers.
2. The keyword "data" can be used as an identifier because it can be
tested in a context sensitive fashion to see if "data" is a
keyword versus an identifier.
CDL Grammar
The file ncgen.y is the definitive grammar for CDL, but a stripped down
version is included here for completeness.
ncdesc: NETCDF
datasetid
rootgroup
;
datasetid: DATASETID
rootgroup: '{'
groupbody
subgrouplist
'}';
groupbody:
attrdecllist
typesection
dimsection
vasection
datasection
;
subgrouplist:
/*empty*/
| subgrouplist namedgroup
;
namedgroup: GROUP ident '{'
groupbody
subgrouplist
'}'
attrdecllist
;
typesection: /* empty */
| TYPES
| TYPES typedecls
;
typedecls:
type_or_attr_decl
| typedecls type_or_attr_decl
;
typename: ident ;
type_or_attr_decl:
typedecl
| attrdecl ';'
;
typedecl:
enumdecl optsemicolon
| compounddecl optsemicolon
| vlendecl optsemicolon
| opaquedecl optsemicolon
;
optsemicolon:
/*empty*/
| ';'
;
enumdecl: primtype ENUM typename ;
enumidlist: enumid
| enumidlist ',' enumid
;
enumid: ident '=' constint ;
opaquedecl: OPAQUE '(' INT_CONST ')' typename ;
vlendecl: typeref '(' '*' ')' typename ;
compounddecl: COMPOUND typename '{' fields '}' ;
fields: field ';'
| fields field ';'
;
field: typeref fieldlist ;
primtype: CHAR_K
| BYTE_K
| SHORT_K
| INT_K
| FLOAT_K
| DOUBLE_K
| UBYTE_K
| USHORT_K
| UINT_K
| INT64_K
| UINT64_K
;
dimsection: /* empty */
| DIMENSIONS
| DIMENSIONS dimdecls
;
dimdecls: dim_or_attr_decl ';'
| dimdecls dim_or_attr_decl ';'
;
dim_or_attr_decl: dimdeclist | attrdecl ;
dimdeclist: dimdecl
| dimdeclist ',' dimdecl
;
dimdecl:
dimd '=' UINT_CONST
| dimd '=' INT_CONST
| dimd '=' DOUBLE_CONST
| dimd '=' NC_UNLIMITED_K
;
dimd: ident ;
vasection: /* empty */
| VARIABLES
| VARIABLES vadecls
;
vadecls: vadecl_or_attr ';'
| vadecls vadecl_or_attr ';'
;
vadecl_or_attr: vardecl | attrdecl ;
vardecl: typeref varlist ;
varlist: varspec
| varlist ',' varspec
;
varspec: ident dimspec ;
dimspec: /* empty */
| '(' dimlist ')'
;
dimlist: dimref
| dimlist ',' dimref
;
dimref: path ;
fieldlist:
fieldspec
| fieldlist ',' fieldspec
;
fieldspec: ident fielddimspec ;
fielddimspec: /* empty */
| '(' fielddimlist ')'
;
fielddimlist:
fielddim
| fielddimlist ',' fielddim
;
fielddim:
UINT_CONST
| INT_CONST
;
/* Use this when referencing defined objects */
varref: type_var_ref ;
typeref: type_var_ref ;
type_var_ref:
path
| primtype
;
/* Use this for all attribute decls */
/* Watch out; this is left recursive */
attrdecllist: /*empty*/ | attrdecl ';' attrdecllist ;
attrdecl:
':' ident '=' datalist
| typeref type_var_ref ':' ident '=' datalist
| type_var_ref ':' ident '=' datalist
| type_var_ref ':' _FILLVALUE '=' datalist
| typeref type_var_ref ':' _FILLVALUE '=' datalist
| type_var_ref ':' _STORAGE '=' conststring
| type_var_ref ':' _CHUNKSIZES '=' intlist
| type_var_ref ':' _FLETCHER32 '=' constbool
| type_var_ref ':' _DEFLATELEVEL '=' constint
| type_var_ref ':' _SHUFFLE '=' constbool
| type_var_ref ':' _ENDIANNESS '=' conststring
| type_var_ref ':' _NOFILL '=' constbool
| ':' _FORMAT '=' conststring
;
path:
ident
| PATH
;
datasection: /* empty */
| DATA
| DATA datadecls
;
datadecls:
datadecl ';'
| datadecls datadecl ';'
;
datadecl: varref '=' datalist ;
datalist:
datalist0
| datalist1
;
datalist0:
/*empty*/
;
/* Must have at least 1 element */
datalist1:
dataitem
| datalist ',' dataitem
;
dataitem:
constdata
| '{' datalist '}'
;
constdata:
simpleconstant
| OPAQUESTRING
| FILLMARKER
| NIL
| econstref
| function
;
econstref: path ;
function: ident '(' arglist ')' ;
arglist:
simpleconstant
| arglist ',' simpleconstant
;
simpleconstant:
CHAR_CONST /* never used apparently*/
| BYTE_CONST
| SHORT_CONST
| INT_CONST
| INT64_CONST
| UBYTE_CONST
| USHORT_CONST
| UINT_CONST
| UINT64_CONST
| FLOAT_CONST
| DOUBLE_CONST
| TERMSTRING
;
intlist:
constint
| intlist ',' constint
;
constint:
INT_CONST
| UINT_CONST
| INT64_CONST
| UINT64_CONST
;
conststring: TERMSTRING ;
constbool:
conststring
| constint
;
/* Push all idents thru here for tracking */
ident: IDENT ;
Printed: 0-0-0 $Date: 2010/04/29 16:38:55 $ ncgen(1)
netcdf 4.9.0 - Generated Fri Jun 24 05:48:20 CDT 2022