manpagez: man pages & more
info ffe
Home | html | info | man
[ << ] [ < ] [ Up ] [ > ] [ >> ]         [Top] [Contents] [Index] [ ? ]

4 How ffe works

Following examples use two different input files:

Fixed length example

Fixed length personnel file with header and trailer, line (record) is identified by the first byte (H = Header, E = Employee, B = Boss, T = trailer).

$cat personnel.fix
H2006-02-25
EJohn     Ripper       23
BScott    Tiger        45
EMary     Moore        41
ERidge    Forrester    31
T0004
$

Structure for reading file above. Note that record ‘boss’ reuses fields from ‘employee’.

structure personel_fix {
    type fixed
    record header {
        id 1 H
        field type 1
        field date 10
    }
    record employee {
        id 1 E
        field EmpType 1
        field FirstName 9
        field LastName  13
        field Age 2
    }
    record boss {
        id 1 B
        fields-from employee
    }
    record trailer {
        id 1 T
        field type 1
        field count 4
    }
}

Separated example

Same file as above, but now separated by comma.

$cat personnel.sep
H,2006-02-25
E,john,Ripper,23
B,Scott,Tiger,45
E,Mary,Moore,41
E,Ridge,Forrester,31
T,0004
$

Structure for reading file above. Note that the field lengths are not needed in separated format. Length is need if the separated data is to be printed in fixed length format.

structure personel_sep {
    type separated ,
    record header {
        id 1 H
        field type 
        field date 
    }
    record employee {
        id 1 E
        field type 
        field FirstName 
        field LastName 
        field Age 
    }
    record boss {
        id 1 B
        fields-from employee
    }
    record trailer {
        id 1 T
        field type 
        field count
    }
}

Printing in XML format

Data in examples above can be printed in XML using output definition like:

output xml {
    file_header "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
    data "<%n>%t</%n>\n"
    record_header "<%r>\n"
    record_trailer "</%r>\n"
    indent " "
}

Example output using command (assuming definitions above are saved in ~/.fferc)

ffe -p xml personnel.sep

<?xml version="1.0" encoding="UTF-8"?>
 <header>
  <type>H</type>
  <date>2006-02-25</date>
 </header>
 <employee>
  <type>E</type>
  <FirstName>john</FirstName>
  <LastName>Ripper</LastName>
  <Age>23</Age>
 </employee>
 <boss>
  <type>B</type>
  <FirstName>Scott</FirstName>
  <LastName>Tiger</LastName>
  <Age>45</Age>
 </boss>
 <employee>
  <type>E</type>
  <FirstName>Mary</FirstName>
  <LastName>Moore</LastName>
  <Age>41</Age>
 </employee>
 <employee>
  <type>E</type>
  <FirstName>Ridge</FirstName>
  <LastName>Forrester</LastName>
  <Age>31</Age>
 </employee>
 <trailer>
  <type>T</type>
  <count>0004</count>
 </trailer>

Printing sql commands

Data in examples above can be loaded to database by generated sql commands. Note that the header and trailer are not loaded, because only fields ‘FirstName’,‘LastName’ and ‘Age’ are printed and ‘no-data-print’ is set as no. This prevents the ‘record_header’ and ‘record_trailer’ to be printed for file header and trailer.

output sql {
    file_header "delete table boss;\ndelete table employee;\n"
    record_header "insert into %r values("
    data "'%t'"
    separator ","
    record_trailer ");\n"
    file_trailer "commit\nquit\n"
    no-data-print no
    field-list FirstName,LastName,Age
}

Output from command

ffe -p sql personnel.sep

delete table boss;
delete table employee;
insert into employee values('john','Ripper','23');
insert into boss values('Scott','Tiger','45');
insert into employee values('Mary','Moore','41');
insert into employee values('Ridge','Forrester','31');
commit
quit

Human readable output

This output format shows the fields in format suitable for displaying in screen or printing.

output nice {
    record_header "%s - %r - %f - %o\n"
    data "%n=%t\n"
    justify =
    indent " "
}

Output from command

ffe -p nice personnel.fix

 personel - header - personnel.fix - 1
  type=H
  date=2006-02-25
 
 personel - employee - personnel.fix - 2
    EmpType=E
  FirstName=John
   LastName=Ripper
        Age=23
 
 personel - boss - personnel.fix - 3
    EmpType=B
  FirstName=Scott
   LastName=Tiger
        Age=45
 
 personel - employee - personnel.fix - 4
    EmpType=E
  FirstName=Mary
   LastName=Moore
        Age=41
 
 personel - employee - personnel.fix - 5
    EmpType=E
  FirstName=Ridge
   LastName=Forrester
        Age=31
 
 personel - trailer - personnel.fix - 6
   type=T
  count=0004

HTML table

Personnel data can be displayed as HTML table using output like:

output html {
    file_header "<html>\n<head>\n</head>\n<body>\n<table border=\"1\">\n<tr>\n"
    header "<th>%n</th>\n"
    record_header "<tr>\n"
    data "<td>%t</td>\n"
    file_trailer "</table>\n</body>\n</html>\n"
    no-data-print no
}

Output from command

ffe -p html -f FirstName,LastName,Age personnel.fix

<html>
<head>
</head>
<body>
<table border="1">
<tr>
<th>FirstName</th>
<th>LastName</th>
<th>Age</th>

<tr>
<td>John</td>
<td>Ripper</td>
<td>23</td>

<tr>
<td>Scott</td>
<td>Tiger</td>
<td>45</td>

<tr>
<td>Mary</td>
<td>Moore</td>
<td>41</td>

<tr>
<td>Ridge</td>
<td>Forrester</td>
<td>31</td>

</table>
</body>
</html>

Using expression

Printing only Scott’s record using expression with previous example:

ffe -p html -f FirstName,LastName,Age -e FirstName^Scott personnel.fix

<html>
<head>
</head>
<body>
<table border="1">
<tr>
<th>FirstName</th>
<th>LastName</th>
<th>Age</th>

<tr>
<td>Scott</td>
<td>Tiger</td>
<td>45</td>

</table>
</body>
</html>

Using replace

Make all bosses and write a new personnel file printing the fields in fixed length format using directive %D:

Output definition:

output fixed 
{
    data "%D"
}

Write a new file:

$ffe -p fixed -r EmpType=B -o personnel.fix.new personnel.fix
$cat personnel.fix.new
H2006-02-25
BJohn     Ripper       23
BScott    Tiger        45
BMary     Moore        41
BRidge    Forrester    31
T0004
$

Using constant

The length of the fields FirstName and LastName in fixed length format will be made two bytes longer. This will be done by printing a constant after those two fields. We use dots instead of spaces in order to make change more visible.

Because we do not want to change header and trailer we need specially crafted configuration file. Employee and boss records will be printed using new output fixed2 and other records will be printed using output default.

New definition file ‘new_fixed.rc’:

const 2dots ".."

structure personel_fix {
    type fixed
    record header {
        id 1 H 
        field type 1 
        field date 10
    }
    record employee {
        id 1 E
        field EmpType 1 
        field FirstName 9
        field LastName  13
        field Age 2
        output fixed2
    }
    record boss {
        id 1 B
        fields-from employee
        output fixed2
    }
    record trailer {
        id 1 T
        field type 1 
        field count 4
    }
}

output default
{
    data "%D"
}

output fixed2
{
    data "%D"
    field-list Emptype,FirstName,2dots,LastName,2dots,Age
}

Print new flat file:

$ ffe -c new_fixed.rc personel_fix
H2006-02-25
EJohn     ..Ripper       ..23
BScott    ..Tiger        ..45
EMary     ..Moore        ..41
ERidge    ..Forrester    ..31
T0004
$

Using lookup table

Lookup table is used to explain the EmpTypes contents in output format nice:

Lookup definition:

lookup Type
{
    search exact
    pair H Header
    pair B "He is a Boss!"
    pair E "Not a Boss!"
    pair T Trailer
    default-value "Unknown record type!"
}   

Mapping the EmpType field to lookup:

structure personel_fix {
    type fixed
    record header {
        id 1 H
        field type 1
        field date 10
    }
    record employee {
        id 1 E
        field EmpType 1 Type
        field FirstName 9
        field LastName  13
        field Age 2
    }
    record boss {
        id 1 B
        fields-from employee
    }
    record trailer {
        id 1 T
        field type 1
        field count 4
    }
}

Adding the lookup option to output definition nice.

output nice {
    record_header "%s - %r - %f - %o\n"
    data "%n=%t\n"
    lookup "%n=%t (%l)\n"
    justify =
    indent " "
}

Running ffe:

 $ffe -p nice personnel.fix
 personel_fix - header - personel_fix - 1
  type=H
  date=2006-02-25
 
 personel_fix - employee - personel_fix - 2
    EmpType=E (Not a Boss!)
  FirstName=John
   LastName=Ripper
        Age=23
 
 personel_fix - boss - personel_fix - 3
    EmpType=B (He is a Boss!)
  FirstName=Scott
   LastName=Tiger
        Age=45
 
 personel_fix - employee - personel_fix - 4
    EmpType=E (Not a Boss!)
  FirstName=Mary
   LastName=Moore
        Age=41
 
 personel_fix - employee - personel_fix - 5
    EmpType=E (Not a Boss!)
  FirstName=Ridge
   LastName=Forrester
        Age=31
 
 personel_fix - trailer - personel_fix - 6
   type=T
  count=0004

External lookup file

In previous example the lookup data could be read from external file like:

$cat lookupdata
H;Header
B;He is a Boss!
E;Not a Boss!
T;Trailer
$

Lookup definition using file above:

lookup Type
{
    search exact
    file lookupdata
    default-value "Unknown record type!"
}

Making universal csv reader using command substitution

Command substitution can be used to make a configuration for reading any csv file. The number of fields will be read from the first file using awk. Input file names and date are printed in the file header:

structure csv {
    type separated ,
    header first
    record csv {
        field-count `awk "-F," 'FNR == 1 {print NF;exit;}' $FFE_FIRST_FILE`
    }
}

output default {
    file_header "Files: `echo $FFE_FILES`\n`date`\n"
    data "%n=%d\n"
    justify =
}

Reading binary data

A binary block having a 3 byte text (ABC) in 5 bytes long space, one byte integer (35), a 32 bit integer (12345678), a double (345.385), a 3 byte bcd number (45112) and a 4 byte hexadecimal data (f15a9188) can be read using following configuration:

structure bin_data
{
    type binary
    record b
    {
        field text 5
        field byte_int int8
        field integer int
        field number double
        field bcd_number bcd_be_3
        field hex hex_be_4
    }
}

output default
{
    data "%n = %d (%h)\n"
}

The %h directive gives a hex dump of the input data.

Hexadecimal dump of the data:

$ od -t x1 example_bin
0000000 41 42 43 00 08 23 4e 61 bc 00 5c 8f c2 f5 28 96
0000020 75 40 45 11 2f f1 5a 91 88
0000031

Using ffe:

$ffe -c example_bin.fferc -s bin_data example_bin
text = ABC (x41x42x43x00x08)
byte_int = 35 (x23)
integer = 12345678 (x4ex61xbcx00)
number = 345.385000 (x5cx8fxc2xf5x28x96x75x40)
bcd_number = 45112 (x45x11x2f)
hex = f15a9188 (xf1x5ax91x88)

Note that the text has only 3 characters before NULL byte. Because this example was made in little endian machine, same result can be achieved with different configuration:

structure bin_data
{
    type binary
    record b
    {
        field text 5
        field byte_int int8
        field integer int32_le
        field number double_le
        field bcd_number bcd_be_3
        field hex hex_be_4
    }
}

This configuration is more portable in case the same data is to be read in a different architecture because endianess of integer and double are explicit given.

If the bcd number is read with bcd_le_3 it would look as

bcd_number = 5411 (x45x11x2f)

Note that nybbles are swapped and last byte is handled as f2 (f stops the printing) causing only first two bytes to be printed.

and if hexadecimal data is read with hex_le_4 it would look as

hex = 88915af1 (xf1x5ax91x88)

Bytes are printed starting from the end of the data.

Printing nested XML

The keyword level in record definition can be used to print data in multi-level nested form. In this example a parent row is in level one and a child row is in level two. Children after a parent row belongs to the parent before child rows, so they are enclosed in a parent element.

Example data:

P,John Smith,3 
C,Kathren,6,Blue 
C,Jimmy,4,Red
C,Peter,2,Green
P,Margaret Eelers,2
C,Aden,16,White
C,Amanda,20,Black

A parent row consistd of ID (P), parent name, and the count of the children. A child row consists of id (C), child name, age and favorite color.

This can be printed in nested XML using rc file:

structure family
{
    type separated ,
    record parent
    {
        id 1 P
        field FILLER
        field Name
        field Child_count
        level 1 parent
    }

    record child
    {
        id 1 C
        field FILLER
        field Name
        field Age
        field FavoriteColor
        level 2 child children
    }
}

output nested_xml
{
    file_header "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
    data "<%n>%t</%n>\n"
    indent " "
    record_trailer ""
    group_header "<%g>\n"
    group_trailer "</%g>\n"
    element_header "<%m>\n"
    element_trailer "</%m>\n"
}

Output:

<?xml version="1.0" encoding="UTF-8"?>
 <parent>
  <Name>John Smith</Name>
  <Child_count>3</Child_count>
  <children>
   <child>
    <Name>Kathren</Name>
    <Age>6</Age>
    <FavoriteColor>Blue</FavoriteColor>
   </child>
   <child>
    <Name>Jimmy</Name>
    <Age>4</Age>
    <FavoriteColor>Red</FavoriteColor>
   </child>
   <child>
    <Name>Peter</Name>
    <Age>2</Age>
    <FavoriteColor>Green</FavoriteColor>
   </child>
  </children>
 </parent>
 <parent>
  <Name>Margaret Eelers</Name>
  <Child_count>2</Child_count>
  <children>
   <child>
    <Name>Aden</Name>
    <Age>16</Age>
    <FavoriteColor>White</FavoriteColor>
   </child>
   <child>
    <Name>Amanda</Name>
    <Age>20</Age>
    <FavoriteColor>Black</FavoriteColor>
   </child>
  </children>
 </parent>

Some examples put in a single file

structure personel_fix {
    type fixed
    record header {
        id 1 H
        field type 1
        field date 10
    }
    record employee {
        id 1 E
        field EmpType 1 Type
        field FirstName 9
        field LastName  13
        field Age 2
    }
    record boss {
        id 1 B
        fields-from employee
    }
    record trailer {
        id 1 T
        field type 1
        field count 4
    }
}

structure personel_sep {
    type separated ,
    record header {
        id 1 H
        field type 
        field date 
    }
    record employee {
        id 1 E
        field type 
        field FirstName 
        field LastName  
        field Age 
    }
    record boss {
        id 1 B
        fields-from employee
    }
        record trailer {
        id 1 T
        field type 
        field count
    }
}

structure bin_data
{
    type binary
    record b
    {
        field text 5
        field byte_int int8
        field integer int32_le
        field number double_le
        field bcd_number bcd_be_3
        field hex hex_be_4
    }
}

output xml {
    file_header "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
    data "<%n>%t</%n>\n"
    record_header "<%r>\n"
    record_trailer "</%r>\n"
    indent " "
}

output sql {
    file_header "delete table boss;\ndelete table employee;\n"
    record_header "insert into %r values("
    data "'%t'"
    separator ","
    record_trailer ");\n"
    file_trailer "commit\nquit\n"
    no-data-print no
    field-list FirstName,LastName,Age
}

output nice {
    record_header "%s - %r - %f - %o\n"
    data "%n=%t\n"
    lookup "%n=%t (%l)\n"
    justify =
    indent " "
}

output html {
    file_header "<html>\n<head>\n</head>\n<body>\n<table border=\"1\">\n<tr>\n"
    header "<th>%n</th>\n"
    record_header "<tr>\n"
    data "<td>%t</td>\n"
    file_trailer "</table>\n</body>\n</html>\n"
    no-data-print no
}

output fixed 
{
    data "%D"
}

lookup Type
{
    search exact
    pair H Header
    pair B "He is a Boss!"
    pair E "Not a Boss!"
    pair T Trailer
    default-value "Unknown record type!"
}   

Using ffe to test file integrity

ffe can be used to check flat file integrity, because ffe checks for all lines the line length and id’s for fixed length structure and field count and id’s for separated structure.

Integrity can be checked using command

ffe -p no -l inputfiles…

Because option ‘-p’ has value no nothing is printed to output except the error messages. Option ‘-l’ causes all erroneous lines to be reported, not just the first one.

Example output:

ffe: Invalid input line in file 'inputfileB', line 14550
ffe: Invalid input line in file 'inputfileD', line 12

[ << ] [ < ] [ Up ] [ > ] [ >> ]         [Top] [Contents] [Index] [ ? ]

This document was generated on March 30, 2014 using texi2html 5.0.

© manpagez.com 2000-2024
Individual documents may contain additional copyright information.