Handling Sources

Handling your sources is probably something that will be very specific to your case. Reading from some web site? You'll need a user agent (and you have plenty to choose from: LWP::UserAgent, WWW::Mechanize, HTTP::Tiny, Mojo::UserAgent to name a few). If your needs are confined to handling arrays or files... we have something for you in Data::Tubes::Plugin::Source!

Iterating over arrays

One thing you might want to consider is to get your input records from an array. It might be a list of URIs you want to process with an ad-hoc tube with a User Agent. A list of numbers. Your shopping list. OK, you get the idea!

If this is the case, iterate_array is what you need.

There are two ways you can provide items to iterate over: while creating the tube, and while invoking the tube. You can pass arrays in both occasions, in which case the items passed at creations will be iterated first, then the one at invocation. Yes, this means that if you invoke the tube a second time, you will iterate over the creation-time items again.

So, suppose you want a tube to provide you the first few prime numbers, you can do like this:

my $src = iterate_array([2, 3, 5, 7, 11, 13, 17, 19]);

The funny thing about this tube is that you might actually not provide an input record when invoking it. Well, I hope you set something upon creation anyway, otherwise your tube will do little.

my (undef, $iterator) = $src->();

As you might have guessed, it always returns an iterator. At each iteration, an element is provided back, whatever comes next in the array. Hence, this tube actually does not stick to any specific output formatting:

my $two = $iterator->();
my $three = $iterator->(); # and so on...

The other alternative is to create an empty tube and pass the array as its input record, like this:

my $src = iterate_array();
my (undef, $iterator) = $src->([2, 3, 5, 7, 11, 13, 17, 19]);
my $two = $iterator->();
my $three = $iterator->(); # and so on...

For example, suppose you want to print the release year of a few movies from OMDb via their API:

use strict;
use warnings;
use HTTP::Tiny;
use Data::Tubes qw< pipeline >;
use JSON::PP qw< decode_json >;
my $ua = HTTP::Tiny->new();
my @movies = ('Pulp Fiction', 'Fight Club', 'Febbre da cavallo');
pipeline(
   'Source::iterate_array',
   sub {
      my $title = shift;
      my $record = {};
      $title =~ s{\ }{%20}gmxs;    # poor man's escaping
      $record->{get} = $ua->get("http://www.omdbapi.com/?t=$title");
      $record->{structured} = decode_json($get->{content});
      return $record;
   },
   [
      'Renderer::with_template_perlish',
      "[% Title %] was released in year [% Year %]\n"
   ],
   'Writer::to_files',
   {tap => 'sink'}
)->(\@movies);

Getting inputs from a file

The next step is: what if my inputs should be read from a file? This is where open_file comes to the rescue, paired with some reading function from Data::Tubes::Plugin::Reader. So, now we will assume our titles are in a file:

shell$ cat movies.txt
Pulp Fiction
Fight Club
Febbre da cavallo
Back to the future

We will reuse most of the code from the previous run, with just a few slight modifications when reading the title:

use strict;
use warnings;
use HTTP::Tiny;
use Data::Tubes qw< pipeline >;
use JSON::PP qw< decode_json >;
my $ua = HTTP::Tiny->new();
pipeline(
   'Source::open_file',
   'Reader::by_line',
   sub {
      my $record = shift;
      my $title = $record->{raw};
      $title =~ s{\ }{%20}gmxs;    # poor man's escaping
      $record->{get} = $ua->get("http://www.omdbapi.com/?t=$title");
      $record->{structured} = decode_json($record->{get}{content});
      return $record;
   },
   [
      'Renderer::with_template_perlish',
      "[% Title %] was released in year [% Year %]\n"
   ],
   'Writer::to_files',
   {tap => 'sink'}
)->('movies.txt');

Mix and match

What if you have multiple files? You can mix the two techniques, iterating over the file names and opening each file:

use strict;
use warnings;
use HTTP::Tiny;
use Data::Tubes qw< pipeline >;
use JSON::PP qw< decode_json >;
my $ua = HTTP::Tiny->new();
pipeline(
   'Source::iterate_array',
   'Source::open_file',
   'Reader::by_line',
   sub {
      my $record = shift;
      my $title = $record->{raw};
      $title =~ s{\ }{%20}gmxs;    # poor man's escaping
      $record->{get} = $ua->get("http://www.omdbapi.com/?t=$title");
      $record->{structured} = decode_json($record->{get}{content});
      return $record;
   },
   [
      'Renderer::with_template_perlish',
      "[% Title %] was released in year [% Year %]\n"
   ],
   'Writer::to_files',
   {tap => 'sink'}
)->(['movies.txt', 'other-movies.txt']);

Yes, it's really as simple as adding Source::iterate_array at the beginning of the pipeline, and providing the list of filenames in an array reference. It turns out that it can be simpler too:

use strict;
use warnings;
use HTTP::Tiny;
use Data::Tubes qw< pipeline >;
use JSON::PP qw< decode_json >;
my $ua = HTTP::Tiny->new();
pipeline(
   'Source::iterate_files,
   'Reader::by_line',
   sub {
      my $record = shift;
      my $title = $record->{raw};
      $title =~ s{\ }{%20}gmxs;    # poor man's escaping
      $record->{get} = $ua->get("http://www.omdbapi.com/?t=$title");
      $record->{structured} = decode_json($record->{get}{content});
      return $record;
   },
   [
      'Renderer::with_template_perlish',
      "[% Title %] was released in year [% Year %]\n"
   ],
   'Writer::to_files',
   {tap => 'sink'}
)->(['movies.txt', 'other-movies.txt']);

This has the added benefit to emit a log line each time a new file is read, if you enable logs! How do you do this? Simple, just use Log::Log4perl::Tiny:

use strict;
use warnings;
use HTTP::Tiny;
use Data::Tubes qw< pipeline >;
use JSON::PP qw< decode_json >;
use Log::Log4perl::Tiny; # just using it suffices
my $ua = HTTP::Tiny->new();
pipeline(
   'Source::iterate_files,
   'Reader::by_line',
   sub {
      my $record = shift;
      my $title = $record->{raw};
      $title =~ s{\ }{%20}gmxs;    # poor man's escaping
      $record->{get} = $ua->get("http://www.omdbapi.com/?t=$title");
      $record->{structured} = decode_json($record->{get}{content});
      return $record;
   },
   [
      'Renderer::with_template_perlish',
      "[% Title %] was released in year [% Year %]\n"
   ],
   'Writer::to_files',
   {tap => 'sink'}
)->(['movies.txt', 'other-movies.txt']);