Reading Text Inputs

There might be an intermediate step between managing your sources and parsing them, especially if you have multiple items inside a single source (e.g. each record is a line in a file).

The available readers are pretty basic but effective, allowing you to read by line, paragraph or based on a separator. At the very basics, it's just a way to set the input record separator and read one record at a time.

Reading Alternatives

Reading line by line is as easy as using by_line. As you will probably use it with pipeline, you will most probably just put the string Reader::by_line at the right place, and let Data::Tubes do the heavy lifting. As you might have guessed by now, it draws its input from something opened by [source handlers](Handling Sources) and provides something useable by the available parsers.

pipeline(
   'Source::iterate_files`,
   `Reader::by_line`,
   sub {
      my $record = shift;
      $record->{rendered} = "Line: [$record->{raw}]\n";
      return $record;
   },
   `Writer::to_files`,
   {tap => 'sink'},
)->(\@ARGV);

If your input happens to be by the paragraph, e.g. a bunch of HTTP headers or an LDAP record inside an LDIF file, you can read it using by_paragraph. We'll copy the example above for this one:

pipeline(
   'Source::iterate_files`,
   `Reader::by_paragraph`,
   sub {
      my $record = shift;
      # Indent the paragraph to make it more "visible"
      (my $para = $record->{raw}) =~ s{^}{   }gmxs;
      $record->{rendered} = "Paragraph:\n$para\n\n";
      return $record;
   },
   `Writer::to_files`,
   {tap => 'sink'},
)->(\@ARGV);

As already said, we're just leveraging standard input handling by Perl, and this is done by setting the right input record separator. This is set automatically in the case of by_line and by_paragraph, but you can set yours too:

pipeline(
   'Source::iterate_files`,
   [`Reader::by_separator`, "\n---\n"],
   sub {
      my $record = shift;
      # Indent the paragraph to make it more "visible"
      (my $para = $record->{raw}) =~ s{^}{   }gmxs;
      $record->{rendered} = "Paragraph:\n$para\n\n";
      return $record;
   },
   `Writer::to_files`,
   {tap => 'sink'},
)->(\@ARGV);

As of now, you cannot set the separator to be a regular expression. In the future, maybe...

Chomping

By default, the readers in Reading Alternatives remove the separator before returning the read data, like when you use CORE::chomp. You can turn this off by setting a false value for parameter chomp:

pipeline(
   'Source::iterate_files`,
   [`Reader::by_line`, chomp => 0],
   sub {
      my $record = shift;
      $record->{rendered} = "Line:\n>>> $record->{raw}\n";
      return $record;
   },
   `Writer::to_files`,
   {tap => 'sink'},
)->(\@ARGV);

End of File Records

Sometimes, you might need to know when you hit the end-of-file marker for a source, and act accordingly.

Option emit_eof allows you to receive a fake record each time the end-of-file is hit. By default, when this happens you don't get any record; if you set the option to a true value, anyway, you will get a record with the output field (i.e. raw by default) set to undef:

pipeline(
   'Source::iterate_files`,
   [`Reader::by_line`, emit_eof => 1],
   sub {
      my $record = shift;
      if (defined($record->{raw})) {
         $record->{rendered} = "Line:\n>>> $$record->{raw}\n";
      }
      else {
         $record->{rendered} = "END-OF-FILE!!!\n";
      }
      return $record;
   },
   `Writer::to_files`,
   {tap => 'sink'},
)->(\@ARGV);

How can this be useful? As an example, suppose that you want to produce an output file matching each input file, and you also want to make sure that output files are closed as soon as possible (e.g. because you have a long-running process and you can't wait to auto-close them at the end of the process):

use feature 'state';
use Data::Tubes::Util::Output;
pipeline(
   'Source::iterate_files`,
   [`Reader::by_line`, emit_eof => 1],
   sub {
      my $record = shift;
      if (defined($record->{raw})) {
         $record->{rendered} = "Line:\n>>> $$record->{raw}\n";
      }
      else {
         $record->{rendered} = "END-OF-FILE!!!\n";
      }
      return $record;
   },

   # A custom writer!
   sub {
      my $record = shift;

      # output channel, open if necessary
      # set output filename according to input filename
      state $output;
      $output //= Data::Tubes::Util::Output->new(
         header    => "--- this is the header  ---\n",
         interlude => "--- just a little pause ---\n",
         footer    => "--- this is the footer  ---\n",
         output    => $record->{source}{name} . '.out',
      );

      if (defined $record->{rendered}) {
         $output->print($record->{rendered});
      }
      else {
         $output = undef; # this closes the file too
      }

      return $record; # just if you want to chain something else...
   },

   {tap => 'sink'},
)->(\@ARGV);

Reading via a record reader

If you already have a function that does the reading, you probably already have your tube. Well, most of it: to take advantage of the other goodies in the toolkit, you have to make sure not only to adhere to the tube interface (which is admittedly very easy to do), but also put what you read in the right place assuming that the record is a hash reference. Oh, by the way, you also have to know where to get data from!

If this all sounds too complicated, and you just want to provide a function that:

• takes a filehandle in
• provides some data out

then by_record_reader is the tool for you. For example, suppose that you want to read inputs that are normally by the line, but you also want to allow to terminate a line with a backslash and continue on the following line (like the shell). If you want to print out normalized lines (i.e. only one line per record, each with a terminator) you might do it like this:

pipeline(
   'Source::iterate_files`,
   [
      `Reader::by_record_reader`,
      sub {
         my $fh = shift;
         my $line;
         local $/ = "\n";
         while (<$fh>) {
            chomp();
            $line .= $_;
            last if (!length($line))
                 || (substr($line, -1, 1) ne '\\');
            chop $line; # remove backslash at end
         }
         return $line . "\n";
      }
   ],
   ['Writer::to_files', input => 'raw'],
   {tap => 'sink'},
)->(\@ARGV);

Purists might object that this allows having the last line with a backslash only and no following line, but we'll assume that you're liberal enough to allow this in your inputs.

Reading by_record_reader does not give you access to the chomp capabilities like described above, of course, because it's up to you to decide what you want to provide back (in our example, we're providing out a line with a terminator). On the other hand, it gives you emit_eof as described in End of File Records, in case you need it.