-
Notifications
You must be signed in to change notification settings - Fork 24
Conversion Tools (CDF)
Original Author: R. J. Barnes (JHU/APL)
A CDF file is divided up into rVariables and zVariables. All rVariables share the same dimensionality. zVariables can have different dimensionality from each other. A CDF file can contain multiple records but each record must have the same rVariables and zVariables.
In translating dmap files, scalars are stored in rVariables and arrays are stored in zVariables.
An CDF file can be created from a specially formatted text file called a skeleton table. The utility skeletoncdf produces the empty CDF file from the skeleton table.
The tool dmaptoskeleton first analyses a dmap file to determine the variables requires and produces an outline skeleton table. The user can edit this file to correct any problems and add any meta-data required. The user then creates an empty CDF file using skeletoncdf. The final step is to populate the CDF file using dmaptocdf.
Let's go through this process step by step:
The first step is to analyse the data file using dmaptoskeleton:
dmaptoskeleton 20021219.kap.fitacf fitacf.skt fitacf.cdfmap
The program takes one input file (20021219.kap.fitacf in this example), and produces two output files (fitacf.skt and fitacf.cdfmap). The first of the output files is the skeleton table for the dataset. The second file maps the dmap variables to their CDF equivalent in the skeleton table. For the example above, part of the skeleton table looks like this:
cat fitacf.skt
#header
CDF NAME: dmapcdf
DATA ENCODING: NETWORK
MAJORITY: ROW
FORMAT: SINGLE
! Variables G.Attributes V.Attributes Records Dims Sizes
! --------- ------------ ------------ ------- ---- -----
45/40 0 0 23035/z 0
#GLOBALattributes
#VARIABLEattributes
#variables
! Variable Data Number Record Dimension
! Name Type Elements Variance Variances
! -------- ---- -------- -------- ---------
"radar_revision_major" CDF_CHAR 1 T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
! Variable Data Number Record Dimension
! Name Type Elements Variance Variances
! -------- ---- -------- -------- ---------
"radar_revision_minor" CDF_CHAR 1 T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
.
.
.
#zVariables
! Variable Data Number Record Dimension
! Name Type Elements Dims Sizes Variance Variances
! -------- ---- -------- ---- ----- -------- ---------
"ptab" CDF_INT2 1 1 7 T T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
! Variable Data Number Record Dimension
! Name Type Elements Dims Sizes Variance Variances
! -------- ---- -------- ---- ----- -------- ---------
"ltab" CDF_INT2 1 2 2 18 T T T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
! Variable Data Number Record Dimension
! Name Type Elements Dims Sizes Variance Variances
! -------- ---- -------- ---- ----- -------- ---------
"pwr0" CDF_FLOAT 1 1 70 T T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
! Variable Data Number Record Dimension
! Name Type Elements Dims Sizes Variance Variances
! -------- ---- -------- ---- ----- -------- ---------
"slist" CDF_INT2 1 1 67 T T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
.
.
.
#end
The converter has done the best job it can, but the skeleton table needs some work. The major problem is that in this particular fitacf file, the maximum number of stored ranges is 69, when in fact the theoretical maximum number is 75. For consistency, the dimensions of the pulse table and lag table have also been adjusted.
Below is part of the corrected skeleton table:
cat fitacf.skt
#header
CDF NAME: dmapcdf
DATA ENCODING: NETWORK
MAJORITY: ROW
FORMAT: SINGLE
! Variables G.Attributes V.Attributes Records Dims Sizes
! --------- ------------ ------------ ------- ---- -----
45/40 0 0 23035/z 0
#GLOBALattributes
#VARIABLEattributes
#variables
! Variable Data Number Record Dimension
! Name Type Elements Variance Variances
! -------- ---- -------- -------- ---------
"radar_revision_major" CDF_CHAR 1 T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
! Variable Data Number Record Dimension
! Name Type Elements Variance Variances
! -------- ---- -------- -------- ---------
"radar_revision_minor" CDF_CHAR 1 T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
.
.
.
#zVariables
! Variable Data Number Record Dimension
! Name Type Elements Dims Sizes Variance Variances
! -------- ---- -------- ---- ----- -------- ---------
"ptab" CDF_INT2 1 1 16 T T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
! Variable Data Number Record Dimension
! Name Type Elements Dims Sizes Variance Variances
! -------- ---- -------- ---- ----- -------- ---------
"ltab" CDF_INT2 1 2 2 48 T T T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
! Variable Data Number Record Dimension
! Name Type Elements Dims Sizes Variance Variances
! -------- ---- -------- ---- ----- -------- ---------
"pwr0" CDF_FLOAT 1 1 75 T T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
! Variable Data Number Record Dimension
! Name Type Elements Dims Sizes Variance Variances
! -------- ---- -------- ---- ----- -------- ---------
"slist" CDF_INT2 1 1 75 T T
! Attribute Data Value
! Name Type
! -------- ---- --------
.
.
.
.
#end
If we wanted to we could also define attributes for the variables in the skeleton table.
The corresponding cdfmap file looks like this:
cat fitacf.cdfmap
char 0 "radar.revision.major"=radar_revision_major;
char 0 "radar.revision.minor"=radar_revision_minor;
short 0 "cp"=cp;
short 0 "stid"=stid;
short 0 "time.yr"=time_yr;
short 0 "time.mo"=time_mo;
short 0 "time.dy"=time_dy;
short 0 "time.hr"=time_hr;
short 0 "time.mt"=time_mt;
short 0 "time.sc"=time_sc;
short 0 "time.us"=time_us;
short 0 "txpow"=txpow;
short 0 "nave"=nave;
short 0 "atten"=atten;
short 0 "lagfr"=lagfr;
short 0 "smsep"=smsep;
short 0 "ercod"=ercod;
short 0 "stat.agc"=stat_agc;
short 0 "stat.lopwr"=stat_lopwr;
float 0 "noise.search"=noise_search;
float 0 "noise.mean"=noise_mean;
short 0 "channel"=channel;
short 0 "bmnum"=bmnum;
short 0 "scan"=scan;
short 0 "offset"=offset;
short 0 "rxrise"=rxrise;
short 0 "intt.sc"=intt_sc;
short 0 "intt.us"=intt_us;
short 0 "txpl"=txpl;
short 0 "mpinc"=mpinc;
short 0 "mppul"=mppul;
short 0 "mplgs"=mplgs;
short 0 "nrang"=nrang;
short 0 "frang"=frang;
short 0 "rsep"=rsep;
short 0 "xcf"=xcf;
short 0 "tfreq"=tfreq;
int 0 "mxpwr"=mxpwr;
int 0 "lvmax"=lvmax;
int 0 "fitacf.revision.major"=fitacf_revision_major;
int 0 "fitacf.revision.minor"=fitacf_revision_minor;
string 0 "combf"=combf;
float 0 "noise.sky"=noise_sky;
float 0 "noise.lag0"=noise_lag0;
float 0 "noise.vel"=noise_vel;
short 1 "ptab"=ptab;
short 2 "ltab"=ltab;
float 1 "pwr0"=pwr0;
short 1 "slist"=slist;
short 1 "nlag"=nlag;
char 1 "qflg"=qflg;
char 1 "gflg"=gflg;
float 1 "p_l"=p_l;
float 1 "p_l_e"=p_l_e;
float 1 "p_s"=p_s;
float 1 "p_s_e"=p_s_e;
float 1 "v"=v;
float 1 "v_e"=v_e;
float 1 "w_l"=w_l;
float 1 "w_l_e"=w_l_e;
float 1 "w_s"=w_s;
float 1 "w_s_e"=w_s_e;
float 1 "sd_l"=sd_l;
float 1 "sd_s"=sd_s;
float 1 "sd_phi"=sd_phi;
char 1 "x_qflg"=x_qflg;
char 1 "x_gflg"=x_gflg;
float 1 "x_p_l"=x_p_l;
float 1 "x_p_l_e"=x_p_l_e;
float 1 "x_p_s"=x_p_s;
float 1 "x_p_s_e"=x_p_s_e;
float 1 "x_v"=x_v;
float 1 "x_v_e"=x_v_e;
float 1 "x_w_l"=x_w_l;
float 1 "x_w_l_e"=x_w_l_e;
float 1 "x_w_s"=x_w_s;
float 1 "x_w_s_e"=x_w_s_e;
float 1 "phi0"=phi0;
float 1 "phi0_e"=phi0_e;
float 1 "elv"=elv;
float 1 "elv_low"=elv_low;
float 1 "elv_high"=elv_high;
float 1 "x_sd_l"=x_sd_l;
float 1 "x_sd_s"=x_sd_s;
float 1 "x_sd_phi"=x_sd_phi;
The only thing to note here is that dmap files do not have any restrictions on the characters that can be contained in a variable name, so here the period "." in the dmap variable names has been converted to an underscore "_". If you look at the skeleton table the same has been done.
The next step is to generate an empty CDF file using skeletoncdf:
mv fitacf.skt.corrected fitacf.skt
skeletoncdf -cdf 20021219.kap fitacf.skt
The final step is to populate the CDF file using dmaptocdf:
dmaptocdf 20021219.kap.fitacf fitacf.cdfmap 20021219.kap.cdf
It should be obvious that the skeleton table and the cdfmap file only need to be generated once and can be re-used for any other fitacf file, providing that the file format is not changed.
Hopefully the SuperDARN community will agree on "standard" skeleton table and cdfmap files so that we can provide standard CDF versions of our fitacf and rawacf files.
The dmap file format was designed to be very efficient in terms of disk usage. However, the current version of the software is not optimized for speed and reading and writing dmap files is somewhat slow (Converting one day of data on a 3.2GHz P4 Linux machine takes roughly a minute).
The CDF format was designed for fast access and incorporates many features missing from the dmap format, but is also much less efficient. (For the example shown, the day of data occupied 84Mb when stored as a dmap file but required 243Mb when converted to netCDF).