mgd77sniffer(1) GMT mgd77sniffer(1)
NAME
mgd77sniffer - Along-track quality control of MGD77 cruises
SYNOPSIS
mgd77sniffer NGDC-ids [ -Afieldabbrev,scale,offset ] [ -Cmaxspd ] [
-Dd|e|E|f|l|m|s|v[r] ] [ -E ] [ -Gfieldabbrev,imggrid,scale,mode or
-Gfieldabbrev,grid ] [ -H ] [ -Ifieldabbrev,rec1,recN ] [ -Lcus-
tom-limits-file ] [ -M ] [ -N ] [ -Rregion ] [ -Sd|s|t ] [ -Tgap ]
[ -V[level] ] [ -Wc|g|o|s|t|v|x ] [ -Zlevel ] [ -bobinary ] [ -dono-
data ] [ -nflags ]
Note: No space is allowed between the option flag and the associated
arguments.
DESCRIPTION
mgd77sniffer scans old (pre-Y2K) and new format ASCII MGD77 files for
errors using point-by-point sanity checking, along-track detection of
excessive slopes, and optional comparison of cruise data with global
gravity and predicted bathymetry grids. Detected data problems are out-
put by default as verbose descriptions of each detected error, often
resulting in multiple messages per scanned record. Data problems are
optionally output (-De option) using a computer-parseable format (see
E77 ERROR FORMAT description below). Default error thresholds are
derived from histograms of all MGD77 geophysical data collected between
1952 and January, 2006. Thresholds are adjustable with the -L option.
REQUIREMENTS
The mgd77sniffer links with Generic Mapping Tools 4.0 or later along
with the supplemental GMT packages x2sys and mgd77. See
http://gmt.soest.hawaii.edu for GMT details. Grids for comparison with
cruise data may be downloaded via the web.
REQUIRED ARGUMENTS
NGDC-ids
Can be one or more of five kinds of specifiers:
1. 8-character NGDC IDs, e.g., 01010083, JA010010etc., etc.
2. 2-character agency codes which will return all cruises from
each agency.
3. 4-character <agency><vessel> codes, which will return all
cruises from those vessels.
4. =list, where list is a table with NGDC IDs, one per line.
5. If nothing is specified we return all cruises in the data
base.
(See mgd77info -L for agency and vessel codes). If no file
extension is given then we search for files with one of the four
known extensions. The search order (and the extensions) tried
is MGD77+ (a.nca), MGD77T (a.m77ta), MGD77 (a.mgd77a ) and plain
text file (a.data). Use -I to ignore one or more of these file
types). Cruise files will be looked for first in the current
directory and second in all directories listed in
$MGD77_HOME/mgd77_paths.txt [If $MGD77_HOME is not set it will
default to $GMT_SHAREDIR/mgd77].
OPTIONAL ARGUMENTS
-Afieldabbrev,scale,offset
Apply scale factor and DC adjustment to specified data field.
Allows adjustment of cruise data prior to along-track analysis.
CAUTION: data must be thoroughly examined before applying these
global data adjustments. May not be used for multiple cruises.
-Cmaxspd
Set maximum ship speed in m/s, or knots with -N option. Ship
speeds exceeding 10 m/s (~20 knots) are flagged as excessive by
default.
-Dd|e|E|f|l|m|s|v[r]
Suppress default warning output and only dump cruise data
row-by-row such as values, gradients, grid-cruise differences,
E77 error summaries for each record, re-created MGD77 records or
sniffer limits. Append r to include all records (default omits
records where navigation errors were detected).
-Dd output differences between cruise and grid data. Requires -G
option. Output columns include:
lat lon dist cruiseZ gridZ diff [cruiseZ2 gridZ2 diff2 ^<i>a|]
Note: grid values are subtracted from cruise data so a positive
difference implies cruise > grid. For multiple grid comparison,
cruiseZ gridZ diff are repeated for each grid comparison in com-
mand line order.
-De output E77 error classification format. Error output is
divided into (1) a header containing information globally appli-
cable to the cruise and (2) individual error records summarizing
all errors encountered in each cruise record. mgd77sniffer
writes E77 directly to <ngdc_id.e77> file handle. See E77 ERROR
FORMAT below for additional details.
-DE Same as -De but no regression tests will be carried out.
-Df output delta Z (change in geophysical field) column and
delta S (change in distance) for each geophysical field. Dis-
tance between observations often differ for different fields
depending on instrument sampling rate, so ds is included for
each geophysical observation. Output columns include:
d[twt] ds d[depth] ds d[mtf1] ds d[mtf2] ds d[mag] ds d[diur] ds
d[msd] ds d[gobs] ds d[eot] ds d[faa] ds
-Dl display mgd77sniffer limits. Customize this output to create
a custom limits file for the -L option. No additional arguments
are required. Output columns include:
fieldabbrev min max maxSlope maxArea
-Dm output MGD77 format records in Y2K-compliant MGD77 format
-Dn output distance to coast for each record. Requires the -Gnav
option. Output columns include:
lat lon dist distToCoast
-Ds output calculated gradients for speed and geophysical
fields. Gradients correspond to the gradient type selected in
the -S option (spatial derivatives by default). Output columns
include:
speed d[twt] d[depth] d[mtf1] d[mtf2] d[mag] d[diur] d[msd]
d[gobs] d[eot] d[faa]
See MGD77 FIELD INFO below for field and abbreviations descrip-
tions.
-Dv display values for the twelve position and geophysical
fields for each MGD77 data record (in this order):
lat lon twt depth mtf1 mtf2 mag diur msens gobs eot faa
See below for MGD77 FIELD INFO.
-E Reverse navigation quality flags (good to bad and vice versa).
May be necessary when a majority of navigation fixes are erro-
neously flagged bad, which can happen when a cruiseas first nav-
igation fix is extremely erroneous. Caution! This will affect
sniffer output and should only be attempted after careful manual
navigation review.
-Ginformation
Compare cruise data to GMT or IMG grids. Use one of the formats
below. -Gfieldabbrev,imggrid,scale,mode Compare cruise data to
the specified grid in Sandwell/Smith Mercator format. Requires a
valid MGD77 field abbreviation (see MGD77 FIELD INFO below) fol-
lowed by a comma, the path (if not in current directory) and
grid filename, a scale to multiply the data (1 or 0.1), and mode
which stand for the following: (0) Img files with no constraint
code, returns data at all points, (1) Img file with constraints
coded, return data at all points, (2) Img file with constraints
coded, return data only at constrained points and NaN elsewhere,
and (3) Img file with constraints coded, return 1 at constraints
and 0 elsewhere. -Gfieldabbrev,grid Compare cruise data to the
specified grid. Requires a valid MGD77 field abbreviation (see
MGD77 FIELD INFO below) followed by a comma, then the path (if
not in current directory) and grid filename. Multiple grid com-
parison is supported by using separate -G calls for each grid.
See GRID FILE INFO below.
Grid comparison activates several additional error checks. (1)
Re-weighted Least Squares Regression of ship versus grid data
determines slope and DC shift, which when differing from
expected 1 and 0, respectively, may indicate incorrectly scaled
ship data, including incorrect units or instrument drift as well
as erroneous gravity tie-in. (2) Accumulated ship grid offsets
are computed along-track and excessive offsets are flagged
according to maxArea threshold (use -L option to adjust
maxArea). Warning: predicted bathymetry grids are constrained by
cruise data so grids and cruise data are not always independent.
Comparison of cruise bathymetry with predicted bathymetry grids
also activates a anavigation crossing over landa check.
-H (with -G|g only) disable (or force) decimation during RLS analy-
sis of ship and gridded data. By default mgd77sniffer analyses
both the full and decimated data sets then reports RLS statis-
tics for the higher correlation regression.
-Hb analyze both (default), report better of two.
-Hd to disable data decimation (equivalent to -H with no argu-
ment).
-Hf to force data decimation.
-Ifieldabbrev,rec1,recN
Append a field abbreviation and the first and last record in a
range of records that should be flagged as bad (and set to NaN
prior to the analysis). Repeat as many times as needed. May not
be used for multiple cruises.
-Lcustom-limits-file
Override mgd77sniffer default error detection limits. Supply
path and filename to the custom limits file. Rows not beginning
with a valid MGD77 field abbreviation are ignored. Field abbre-
viations are listed below in exact form under MGD77 FIELD INFO.
Multiple field limits may be modified using one default file,
one field per line. Field min, max, max slope and max area may
be changed for each field. Max slope pertains to the gradient
type selected using the -S option. Max area is used by the -G
option as the threshold for flagging excessive offsets from the
specified grid. Dump defaults -Dl to view syntax or to quickly
create an editable custom limits file.
Example custom default file contents (see below for units):
+---------+------+-------+----------+---------+
|# abbrev | min | max | maxSlope | maxArea |
+---------+------+-------+----------+---------+
|twt | 0 | 15 | 1 | 0 |
+---------+------+-------+----------+---------+
|depth | 0 | 11000 | 500 | 5000 |
+---------+------+-------+----------+---------+
|mag | -800 | 800 | a | a |
+---------+------+-------+----------+---------+
|faa | -300 | 300 | 100 | 2500 |
+---------+------+-------+----------+---------+
Use a dash a-a to retain a default limit. Hint: to test your
custom limits, try: mgd77sniffer -Dl -L<yourlimitsfile>
-M Adjust navigation on land threshold (meters inland) [100].
-N Use nautical units.
-Rwest/east/south/north[/zmin/zmax][+r][+uunit]
west, east, south, and north specify the region of interest, and
you may specify them in decimal degrees or in
[A+-]dd:mm[:ss.xxx][W|E|S|N] format Append +r if lower left and
upper right map coordinates are given instead of w/e/s/n. The
two shorthands -Rg and -Rd stand for global domain (0/360 and
-180/+180 in longitude respectively, with -90/+90 in latitude).
Alternatively for grid creation, give Rcodelon/lat/nx/ny, where
code is a 2-character combination of L, C, R (for left, center,
or right) and T, M, B for top, middle, or bottom. e.g., BL for
lower left. This indicates which point on a rectangular region
the lon/lat coordinate refers to, and the grid dimensions nx and
ny with grid spacings via -I is used to create the corresponding
region. Alternatively, specify the name of an existing grid
file and the -R settings (and grid spacing, if applicable) are
copied from the grid. Appending +uunit expects projected (Carte-
sian) coordinates compatible with chosen -J and we inversely
project to determine actual rectangular geographic region. For
perspective view (-p), optionally append /zmin/zmax. In case of
perspective view (-p), a z-range (zmin, zmax) can be appended to
indicate the third dimension. This needs to be done only when
using the -Jz option, not when using only the -p option. In the
latter case a perspective view of the plane is plotted, with no
third dimension.
-Sd|s|t
Specify gradient type for along-track excessive slope checking.
-Sd Calculate change in z values along track (dz). Output is
given in geophysical units, e.g., mGal. -Ss Calculate spatial
gradients (dz/ds). Output is given in geophysical units per km
along the survey track, e.g., mGal/km. -St Calculate time gra-
dients (dz/dt) [default]. Output is given in geophysical units
per second along the survey track, e.g., mGal/sec.
-Tgap Adjusts mgd77sniffer gap handling. By default, data gaps greater
than 5 km are skipped. Set to zero to de-activate gap skipping.
-Wc|g|o|s|t|v|x
Print out only certain warning types for verbose error messages.
Comma delimit any combination of c|g|o|s|t|v|x: where (c) type
code warnings, (g)radient out of range, (o)ffsets from grid
(requires -G|g), (s)peed out of range, (t)ime warnings, (v)alue
out of range, (x) warning summaries. By default ALL warning mes-
sages are printed.Not compatible with any -D options.
-Z Flag regression statistics that are outside the specified confi-
dence level. (i.e., -Z5 flags coefficients m, b, rms, and r that
fall outside 95%.)
-V[level] (more a|)
Select verbosity level [c].
-bo[ncols][type] (more a|)
Select native binary output. Output binary data for -Dd|f|s|v
option.
-donodata (more a|)
Replace output columns that equal NaN with nodata.
-n[b|c|l|n][+a][+bBC][+c][+tthreshold] (more a|)
Select interpolation mode for grids.
-^ or just -
Print a short message about the syntax of the command, then
exits (NOTE: on Windows just use -).
-+ or just +
Print an extensive usage (help) message, including the explana-
tion of any module-specific option (but not the GMT common
options), then exits.
-? or no arguments
Print a complete usage (help) message, including the explanation
of all options, then exits.
CONSEQUENCES OF GRID RESAMPLING
Resample or sampling of grids will use various algorithms (see -n) that
may lead to possible distortions or unexpected results in the resampled
values. One expected effect of resampling with splines is the tendency
for the new resampled values to slightly exceed the global min/max lim-
its of the original grid. If this is unacceptable, you can impose
clipping of the resampled values values so they do not exceed the input
min/max values by adding +c to your -n option.
MGD77 FIELD INFO
+----------+--------------+-----------+
|Field | Abbreviation | Units |
+----------+--------------+-----------+
|Two-way | Travel | Time |
+----------+--------------+-----------+
|Corrected | Depth | depth |
+----------+--------------+-----------+
|Mag | Total | Field1 |
+----------+--------------+-----------+
|Mag | Total | Field2 |
+----------+--------------+-----------+
|Residual | Magnetic | mag |
+----------+--------------+-----------+
|Diurnal | Correction | diur |
+----------+--------------+-----------+
|Mag | Sensor | Depth/Alt |
+----------+--------------+-----------+
|Observed | Gravity | gobs |
+----------+--------------+-----------+
|Eotvos | Correction | eot |
+----------+--------------+-----------+
|Free | Air | Anomaly |
+----------+--------------+-----------+
GRID FILE INFO
For -G the grids must eitehr be in the format used by Sandwell &
Smith, which is a spherical Mercator 2-byte grid with no header, or any
grid type supported by GMT and therefore must contain a GMT header. A
correctly formatted *.i2 grid file can be generated using grdraster as
shown below.
gmtset GRIDFILE_SHORTHAND TRUE
Create/edit .gmt_io file to include the following rows:
# GMT I/O shorthand file
# suffix format_id scale offset NaN
grd 0 - - - -
i2 2 - - 32767
gmt grdraster 1 -R0/359:55/-90/90 -Getopo5_hdr.i2
The new grid, etopo5_hdr.i2 in this example, contains a GMT header and
can be used in the -G option to compare cruise depth with grid values.
E77 ERROR FORMAT
Header Information pertaining to an entire cruise, such as NGDC and
survey institution identification codes, cruise examination
time, two-way travel time corrector information, data precision
warnings, as well as systematic scales, DC shifts and correla-
tion coefficients from global grid comparisons are reported as
E77 header information.
Sample
# Cruise 08010039 ID 74010908 MGD77 FILE VERSION: 19801230
N_RECS: 3066
# Examined: Wed Oct 3 16:30:13 2007 by mtchandl
# Arguments: -De -Gdepth,/data/GRIDS/etopo5_hdr.i2
N Errata table verification status
# mgd77manage applies corrections if the errata table is veri-
fied (toggle aNa above to aYa after review)
# For instructions on E77 format and usage, see
http://gmt.soest.hawaii.edu/mgd77/errata.php
# Verified by:
# Comments:
# Errata: Header
Y-E-08010039-H13-02: Invalid Magnetics Sampling Rate: (99) [ ]
Y-W-08010039-H13-10: Survey year (1975) outside magnetic refer-
ence field IGRF 1965 time range (1965-1970)
Y-I-08010039-depth-00: RLS m: 1.00053 b: 0 rms: 127.851 r:
0.973422 significant: 1 decimation: 0
Y-W-08010039-twt-09: More recent bathymetry correction table
available
Y-W-08010039-mtf1-10: Integer precision
Y-W-08010039-mag-10: Integer precision
Error Record
Individual error records have strict format. Included is a time
or distance column followed by record number, a formatted error
code string, and finally a verbose description of errors
detected in the record. Three error classes are encoded into the
error code string with different alphabetic characters repre-
senting unique error types. See below for error code format
description.
Format <time/distance> <record number> <error code string>
<description>
Sample
# Errata: Data
Y 08010039 1975-05-10T22:16:05.88 74 C-0-0 NAV: excessive
speed
Error Code Description
Each of the three error classes is separated by a dash - and
described by a combination of alphabetic characters or 0 signi-
fying no detected problems.
Error classes: NAV-VAL-GRAD
Error Class Descriptions
NAV (navigation):
0 - fine
A - time out of range
B - time decreasing
C - excessive speed
D - above sea level
E - lat undefined
F - lon undefined
VAL (value):
0 - fine
K - twt invalid
L - depth invalid
O - mtf1 invalid
etc.
GRAD (gradient):
0 - fine
K - d[twt] excessive
L - d[depth] excessive
O - d[mtf1] excessive
etc.
The NAV error class has unique cases while VAL and GRAD classes
are described by alphabetic characters for each of the 24
numeric fields in MGD77 format order.
MGD77 bit-pattern w/ E77 alpha characters
|aaaaaaaaaaaaaaaa-|aaa-|
| X W V U T S R Q P O N M L K J I H G F E D C B A | E77 Code |
| - - - - - - - - - - - - - - - - - - - - - - - - | - - - - -|
| n f e g m d m m m m b b d t p l l m h d m y t d | F I |
| q a o o s i s a t t t c e w t o a i o a o e z r | i D |
| c a t b d u e g f f c c p t c n t n u y n a t | e |
| s r n 2 1 t r t r | l |
| s h h | d |
| - - - - - - - - - - - - - - - - - - - - - - - - | - - - - -|
| 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 | Bit place|
| - G C G C C - G G G - - G G - - - T T T T T - - | Bit type |
|aaaaaaaaaaaaaaaa-|aaa-|
Bit types: (G)eophysical, (C)orrection, (T)ime
EXAMPLES
To scan for excessive values or gradients, try
gmt mgd77sniffer 08010001
To dump cruise gradients, try
gmt mgd77sniffer 08010001 -Ds
To compare cruise depth with ETOPO5 bathymetry and gravity with
Sandwell/Smith 2 min gravity version 11, try
mgd77sniffer 08010001 -Gdepth,/data/GRIDS/etopo5_hdr.i2 \
-Gfaa,/data/GRIDS/grav.11.2.img,0.1,1
SEE ALSO
grdraster(1), mgd77list(1), mgd77track(1), x2sys_init(1)
REFERENCES
The Marine Geophysical Data Exchange Format - MGD77, see
http://www.ngdc.noaa.gov/mgg/dat/geodas/docs/mgd77.txt.
COPYRIGHT
2017, P. Wessel, W. H. F. Smith, R. Scharroo, J. Luis, and F. Wobbe
5.4.2 Jun 24, 2017 mgd77sniffer(1)
gmt5 5.4.2 - Generated Thu Jun 29 14:50:36 CDT 2017