Parsing by_format

One of the aims of Data::Tubes is to provide a toolset that, in the author's wildest dreams, might be used by people with little to no knowledge of Perl. Hence, instead of requiring to know all the details about how to open a file, iterate through it, etc. etc., you just take a look at the toolset.

One of the most challenging parts is how to parse the inputs. Depending on what you have to read, different strategies might apply. Here we will explore one of the simplest, i.e. when you can describe your inputs with some kind of "format".

What's a Format Anyway?

Every time you have data that fits in some columns it is almost surely eligible to be handled with a format. What's a format, you might ask? It's a sequence like this:

<key1><separator1><key2><separator2>...<keyn>

i.e. a sequence of some keys, interlaced with a sequence of separators. All keys MUST be different from one another and composed of word-characters only; all separators can be whatever sequence of non-word characters. For example, the following is an example of a valid format, in which we are using two different separators (the colon and the comma, in specific positions):

name:address:mobile,fixed,fax:email:website

The following records apply to the format above:

Foo Bar:Via Baz, 10:+39 348 1234567,,:foo@example.com:
John Doe::1234,456,7890::http://www.example.com

As you can see, the last entry includes a semicolon in the website, but this appears after the last separator so it's fine to have it there.

What's in for Me?

If you happen to have a format, by_format is the tool for you. Let's see it at work in an example:

#!/usr/bin/env perl
# vim: sts=3 ts=3 sw=3 et ai :
use strict;
use warnings;
use Data::Dumper; $Data::Dumper::Indent = 1;
use Data::Tubes qw< pipeline >;

my $input = <<'END';
Foo Bar:Via Baz, 10:+39 348 1234567,,:foo@example.com:
John Doe::1234,456,7890::http://www.example.com
END

pipeline(
   'Source::open_file',
   'Reader::by_line',
   ['Parser::by_format' => 'name:address:mobile,fixed,fax:email:website' ],
   sub {
      $_[0]{rendered} = Dumper($_[0]{structured});
      return $_[0];
   },
   'Writer::to_files',
   {tap => 'sink'},
)->(\$input);

The rendering is a simple wrapper around Data::Dumper, just to show you what's going on:

shell$ ./format-00 
$VAR1 = {
  'name' => 'Foo Bar',
  'website' => '',
  'email' => 'foo@example.com',
  'mobile' => '+39 348 1234567',
  'address' => 'Via Baz, 10',
  'fax' => '',
  'fixed' => ''
};
$VAR1 = {
  'name' => 'John Doe',
  'website' => 'http://www.example.com',
  'fixed' => '456',
  'fax' => '7890',
  'address' => '',
  'mobile' => '1234',
  'email' => ''
};

Any Separator?

As long as a separator is composed of a sequence of non-word characters, it is considered a valid separator. In this context, non-word character means anything that a Perl regular expression would not consider a word character, alias \W.

The other implicit constraint in a format is that the format is provided as a single string carrying both the name of the extracted keys and the separators. So, you can only work with fixed separators, and you cannot pass regular expressions for them.

Trimming

When your input comes from some machine-generated data, it's not difficult to have something that is already well-formatted. As humans, though, we like to see things clearly, and we might want to add some spacing here and there just to understand things better.

As of release 0.730 you can get rid of leading and trailing whitespaces by using option trim. Hence, the following example gives the equivalent output of the one provided before, even though the input is different because of added spacing:

#!/usr/bin/env perl
# vim: sts=3 ts=3 sw=3 et ai :
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 1;
use Data::Tubes qw< pipeline >;

my $input = <<'END';
Foo Bar  :Via Baz, 10:+39 348 1234567,   ,    :foo@example.com:
John Doe :           :1234           ,456,7890::http://www.example.com
END

pipeline(
   'Source::open_file',
   'Reader::by_line',
   [
      'Parser::by_format' => 'name:address:mobile,fixed,fax:email:website',
      trim                => 1
   ],
   sub {
      $_[0]{rendered} = Dumper($_[0]{structured});
      return $_[0];
   },
   'Writer::to_files',
   {tap => 'sink'},
)->(\$input);

The output is equivalent because Data::Dumper does not provide a specific ordering in the serialization of hashes; here's the output from one run:

shell$ ./format-01 
$VAR1 = {
  'fixed' => '',
  'name' => 'Foo Bar',
  'website' => '',
  'address' => 'Via Baz, 10',
  'email' => 'foo@example.com',
  'mobile' => '+39 348 1234567',
  'fax' => ''
};
$VAR1 = {
  'name' => 'John Doe',
  'fixed' => '456',
  'website' => 'http://www.example.com',
  'address' => '',
  'email' => '',
  'mobile' => '1234',
  'fax' => '7890'
};

If you have a previous version of Data::Tubes... you will get spaces instead. Time to ask to upgrade!

Value Specification

By default, values are whatever is found between two successive separators. It's really as easy as this, but this also comes at a price: what if you need to fit a separator inside a value?

This is, admittedly, something that you might be able to avoid most of the times, especially if the data you're dealing with is simple, like numbers or text made only of letters. Other times, you might have to allow for that happening; this is where value comes handy.

Let's say that you decide to go for a simple escaping mechanism, using the backslash: every time there's a backslash, the following character has to be taken verbatim as part of a value, even if it would be considered a separator otherwise. Later, of course, you want that escape character to just disappear, because it's just something that is needed to encode the data in the right way. The input we consider is the following:

Foo Barious:Baz\: The Definitive Guide:2016

and the format we would like to use is:

Author:Title:Year

Let's see a couple of ways to cope with this.

Regex

The first thing you can set for the value is a regular expression. By default, a value is matched by .*?, meaning "whatever comes from two consecutive separators" (note that the match is non-greedy). You can pass a regular expression to match something different though, as in the following example:

#!/usr/bin/env perl
# vim: sts=3 ts=3 sw=3 et ai :
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 1;
use Data::Tubes qw< pipeline >;
use Log::Log4perl::Tiny qw< :easy LOGLEVEL >;

my $input = <<'END';
Foo Barious:Baz\: The Definitive Guide:2016
END

pipeline(
   'Source::open_file',
   'Reader::by_line',
   [
      'Parser::by_format' => 'Author:Title:Year',
      value => qr{(?mxs: [^\\] | \\. )*?},
   ],
   sub {
      $_[0]{rendered} = Dumper($_[0]{structured});
      return $_[0];
   },
   'Writer::to_files',
   {tap => 'sink'},
)->(\$input);

This mostly works:

shell$ ./format-03 
$VAR1 = {
  'Author' => 'Foo Barious',
  'Title' => 'Baz\\: The Definitive Guide',
  'Year' => '2016'
};

Our only problem is that the colon is still escaped, and we want to get rid of the stray backslash. It's easy to do with another tube, as in the following example:

#!/usr/bin/env perl
# vim: sts=3 ts=3 sw=3 et ai :
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 1;
use Data::Tubes qw< pipeline >;
use Log::Log4perl::Tiny qw< :easy LOGLEVEL >;

my $input = <<'END';
Foo Barious:Baz\: The Definitive Guide:2016
END

pipeline(
   'Source::open_file',
   'Reader::by_line',
   [
      'Parser::by_format' => 'Author:Title:Year',
      value => qr{(?mxs: [^\\] | \\. )*?},
   ],
   sub {
      s{\\(.)}{$1}gmxs for values %{$_[0]{structured}};
      return $_[0];
   },
   sub {
      $_[0]{rendered} = Dumper($_[0]{structured});
      return $_[0];
   },
   'Writer::to_files',
   {tap => 'sink'},
)->(\$input);

The result is as expected:

shell$ ./format-03-01
$VAR1 = {
  'Author' => 'Foo Barious',
  'Title' => 'Baz: The Definitive Guide',
  'Year' => '2016'
};

It turns out that you can pass filtering functions too, namely decode and decode_values. The first one receives a single encoded value and is supposed to return the decoded value, like this example:

#!/usr/bin/env perl
# vim: sts=3 ts=3 sw=3 et ai :
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 1;
use Data::Tubes qw< pipeline >;
use Log::Log4perl::Tiny qw< :easy LOGLEVEL >;

my $input = <<'END';
Foo Barious:Baz\: The Definitive Guide:2016
END

pipeline(
   'Source::open_file',
   'Reader::by_line',
   [
      'Parser::by_format' => 'Author:Title:Year',
      value => qr{(?mxs: [^\\] | \\. )*?},
      decode => sub { $_[0] =~ s{\\(.)}{$1}gmxs; return $_[0]; },
   ],
   sub {
      $_[0]{rendered} = Dumper($_[0]{structured});
      return $_[0];
   },
   'Writer::to_files',
   {tap => 'sink'},
)->(\$input);

Again, the result is as expected:

shell$ ./format-03-02
$VAR1 = {
  'Title' => 'Baz: The Definitive Guide',
  'Year' => '2016',
  'Author' => 'Foo Barious'
};

The second alternative, i.e. decode_values, allows you to pass a function that will do the decoding for all parsed values in a single call. Hence, you are supposed to do the iteration by yourself, but this will save you a lot of sub calls:

#!/usr/bin/env perl
# vim: sts=3 ts=3 sw=3 et ai :
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 1;
use Data::Tubes qw< pipeline >;
use Log::Log4perl::Tiny qw< :easy LOGLEVEL >;

my $input = <<'END';
Foo Barious:Baz\: The Definitive Guide:2016
END

pipeline(
   'Source::open_file',
   'Reader::by_line',
   [
      'Parser::by_format' => 'Author:Title:Year',
      value => qr{(?mxs: [^\\] | \\. )*?},
      decode_values => sub { s{\\(.)}{$1}gmxs for @{$_[0]}; return $_[0] },
   ],
   sub {
      $_[0]{rendered} = Dumper($_[0]{structured});
      return $_[0];
   },
   'Writer::to_files',
   {tap => 'sink'},
)->(\$input);

As you can see, it receives as input an array reference with the values, and is supposed to return back another array reference. This allows you to do fancy things like removing or adding elements!

Batteries Included

Using the backslash for escaping is a somehow diffused convention. Also, de-escaping a string does make sense indeed after you have isolated it. Guess what? You already have them available out of the box, as the following example demonstrates:

#!/usr/bin/env perl
# vim: sts=3 ts=3 sw=3 et ai :
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 1;
use Data::Tubes qw< pipeline >;
use Log::Log4perl::Tiny qw< :easy LOGLEVEL >;

my $input = <<'END';
Foo Barious:Baz\: The Definitive Guide:2016
END

pipeline(
   'Source::open_file',
   'Reader::by_line',
   ['Parser::by_format' => 'Author:Title:Year', value => 'escaped'],
   sub {
      $_[0]{rendered} = Dumper($_[0]{structured});
      return $_[0];
   },
   'Writer::to_files',
   {tap => 'sink'},
)->(\$input);

Providing the string escaped does what we saw in the previous section automatically: sets the right regular expression and the right decoding function too. You can still pass a different decode, of course, but at this point it will be up to you to do all the heavy lifting... Do that with care!

In addition to escaped, you have some more possibilities:

single-quoted (or single_quoted), to extract single-quoted strings. Rules are easy in this case, because there is no escaping and whatever is found in a pair of single quotes is taken verbatim (this is different from what Perl does, more similar to the shell's behaviour);
double-quoted/double_quoted, to extract double-quoted strings. The backslash allows escaping characters inside, which turns out to be useful if you want to put a double quote inside your string;
quoted, to set both single-quoted and double-quoted as valid;
specials, to set single-quoted, double-quoted and escaped in one single sweep;
whatever, to use the default extraction mechanism of whatever comes between two consecutive separators.

You can actually mix and match these alternatives, together with regular expressions, as long as you use an array reference for the value argument as in the following example:

#!/usr/bin/env perl
# vim: sts=3 ts=3 sw=3 et ai :
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 1;
use Data::Tubes qw< pipeline >;
use Log::Log4perl::Tiny qw< :easy LOGLEVEL >;

my $input = <<'END';
'Foo::Barious':Baz\: The Definitive Guide:2016
END

pipeline(
   'Source::open_file',
   'Reader::by_line',
   [
      'Parser::by_format' => 'Author:Title:Year',
      value               => [qw< escaped single-quoted >]
   ],
   sub {
      $_[0]{rendered} = Dumper($_[0]{structured});
      return $_[0];
   },
   'Writer::to_files',
   {tap => 'sink'},
)->(\$input);

Note that the name of the author changed to Foo::Barious, so we found it easier to use quotes to preserve the two colons:

shell$ ./format-04-01
$VAR1 = {
  'Title' => 'Baz: The Definitive Guide',
  'Year' => '2016',
  'Author' => 'Foo::Barious'
};

Flexible Format

Sometimes you might have a flexible format, in that only some of the fields are always present, while other ones might be missing instead. By default, by_format is picky in these cases, requiring each input record to strictly adhere to the format, but you can be forgiving if one or more fields at the end of the format are missing through option allow_missing.

For example, suppose to have the following input, in which you're actually only interested into the first two fields:

John:Doe:44:married
Jeah:Doe

A suitable format for this case might be the following:

first:last:rest

Here's an example:

#!/usr/bin/env perl
# vim: sts=3 ts=3 sw=3 et ai :
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 1;
use Data::Tubes qw< pipeline >;

my $input = <<'END';
John:Doe:44:married
Jean:Doe
END

pipeline(
   'Source::open_file',
   'Reader::by_line',
   [
      'Parser::by_format' => 'first:last:rest',
      allow_missing => 1,
   ],
   sub {
      $_[0]{rendered} = Dumper($_[0]{structured});
      return $_[0];
   },
   'Writer::to_files',
   {tap => 'sink'},
)->(\$input);

Output:

shell$ ./format-02 
$VAR1 = {
  'first' => 'John',
  'last' => 'Doe',
  'rest' => '44:married'
};
$VAR1 = {
  'rest' => undef,
  'last' => 'Doe',
  'first' => 'Jean'
};

Here, we can just disregard rest and focus on first/last.

Main Site - Manual - Tubergen (download) - Wiki Home - MetaCPAN

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parsing by_format

What's a Format Anyway?

What's in for Me?

Any Separator?

Trimming

Value Specification

Regex

Batteries Included

Flexible Format

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally