Writing to_files
The manual provides an introduction to to_files, a factory function that generates tubes useful for... writing to files. Here you can find addiitonal details, let's first restart from the example in the manual:
pipeline(
['Source::iterate_files', open_file_args => {binmode => ':raw'}],
'Reader::by_line',
['Parser::by_format', 'name;nick;age'],
['Renderer::with_template_perlish', <<'END' ],
-
name: [% name %]
nick: [% nick %]
age: [% age %]
END
# Printing, gets `rendered`, returns input unchanged. Goes
# to standard output by default
'Writer::to_files',
# Options, just flush the output to the sink
{ tap => 'sink' },
)->([qw< mydata-04.txt >]);The writer tools rely upon Data::Tubes::Util::Output, that is a smart wrapper that allows you to handle multiple cases. For example, suppose that instead of YAML you want to output JSON; you might start with this:
pipeline(
['Source::iterate_files', open_file_args => {binmode => ':raw'}],
'Reader::by_line',
['Parser::by_format', 'name;nick;age'],
['Renderer::with_template_perlish', <<'END' ],
{"name":"[% name %];"nick":"[% nick %]";"age":[% age %]}
END
# Printing, gets `rendered`, returns input unchanged
'Writer::to_files',
# Options, just flush the output to the sink
{ tap => 'sink' },
)->([qw< mydata-04.txt >]);There's a problem though: for more than one input record, the overall output is not valid JSON:
{"name":"Foo","nick":"foorious","age":32}
{"name":"Bar","nick":"barious","age":19}
{"name":"Baz","nick":"bazzecolas","age":44}
We should put this in an array, and we should separate the objects with a comma.
The first thing might be naively solved like this:
# DO NOT USE THIS SOLUTION! THIS IS WRONG!
print "[\n";
pipeline(
['Source::iterate_files', open_file_args => {binmode => ':raw'}],
'Reader::by_line',
['Parser::by_format', format => 'name;nick;age'],
['Renderer::with_template_perlish', format => <<'END' ],
{"name":"[% name %];"nick":"[% nick %]";"age":[% age %]}
END
# Printing, gets `rendered`, returns input unchanged
['Writer::to_files', filename => \*STDOUT],
# Options, just flush the output to the sink
{ tap => 'sink' },
)->([qw< mydata-04.txt >]);
# DO NOT USE THIS SOLUTION! THIS IS WRONG!
print "]\n"This has two problems:
- first of all, it does not allow you to change the output channel. What if you are writing to a file instead? You need something that is able to write the array-opener and the array-closer from within the pipeline;
- you still have to figure out how to print out the separator comma!
An additional pitfall to consider is that JSON is quite picky in the presence of separator commas, in that it does not allow you to have a trailing one, so the following would be incorrect:
[
{"name":"Foo","nick":"foorious","age":32},
{"name":"Bar","nick":"barious","age":19},
{"name":"Baz","nick":"bazzecolas","age":44},
]
because the last comma would be out of place.
Fortunately, to_files can help you with header,
interlude and footer:
pipeline(
['Source::iterate_files', open_file_args => {binmode => ':raw'}],
'Reader::by_line',
['Parser::by_format', format => 'name;nick;age'],
['Renderer::with_template_perlish', format => <<'END' ],
{"name":"[% name %];"nick":"[% nick %]";"age":[% age %]}
END
# Printing, gets `rendered`, returns input unchanged
[
'Writer::to_files',
header => "[\n",
footer => "]\n",
interlude => ','
],
# Options, just flush the output to the sink
{ tap => 'sink' },
)->([qw< mydata-04.txt >]);The above code produces:
[
{"name":"Foo","nick":"foorious","age":32}
,{"name":"Bar","nick":"barious","age":19}
,{"name":"Baz","nick":"bazzecolas","age":44}
]
that is probably not very good-looking, but at least correct and also working correctly whatever the output filename you set. You can change things to make it better looking, though; just get rid of the newline in the template, add it after the comma and put some indentation:
pipeline(
['Source::iterate_files', open_file_args => {binmode => ':raw'}],
'Reader::by_line',
['Parser::by_format', 'name;nick;age'],
['Renderer::with_template_perlish',
' {"name":"[% name %];"nick":"[% nick %]";"age":[% age %]}'],
# Printing, gets `rendered`, returns input unchanged
[
'Writer::to_files',
header => "[\n",
footer => "\n]\n",
interlude => ",\n"
],
# Options, just flush the output to the sink
{ tap => 'sink' },
)->([qw< mydata-04.txt >]);This now prints:
[
{"name":"Foo","nick":"foorious","age":32},
{"name":"Bar","nick":"barious","age":19},
{"name":"Baz","nick":"bazzecolas","age":44}
]
which you might like more.
The three arguments header, interlude and footer can either be
strings, or array references that will be turned into the argument list
for read_file.
If you're handling a lot of input records, you might want to segment the output in order to distribute the records into multiple files, instead of having only one single file. It turns out that to_files gets you covered also in this case!
The basic thing that you can do is to set a policy object, where you
have two knobs: records_threshold and characters_threshold. They set
a threshold that will close the output channel when overcome, and open
a new one when needed.
We will assume that we're writing to files here:
pipeline(
['Source::iterate_files', open_file_args => {binmode => ':raw'}],
'Reader::by_line',
['Parser::by_format', 'name;nick;age'],
['Renderer::with_template_perlish',
' {"name":"[% name %];"nick":"[% nick %]";"age":[% age %]}'],
# Printing, gets `rendered`, returns input unchanged
[
'Writer::to_files',
header => "[\n",
footer => "\n]\n",
interlude => ",\n",
# writing to files instead of STDOUT, setting a threshold
filename => 'output-01.json',
policy => {records_threshold => 2},
],
# Options, just flush the output to the sink
{ tap => 'sink' },
)->([qw< mydata-04.txt >]);The example above produces two output files:
$ cat output-01.json
[
{"name":"Foo","nick":"foorious","age":32},
{"name":"Bar","nick":"barious","age":19}
]
$ cat output-01.json_1
[
{"name":"Baz","nick":"bazzecolas","age":44}
]
Again, valid JSON files with the correct number of (maximum) records. You might have some issues with the file names, though, especially if you rely on the file extension to do... anything.
to_files allows you to set a filename template, using sprintf-like
sequences using %n, like this:
pipeline(
['Source::iterate_files', open_file_args => {binmode => ':raw'}],
'Reader::by_line',
['Parser::by_format', format => 'name;nick;age'],
['Renderer::with_template_perlish', format =>
' {"name":"[% name %];"nick":"[% nick %]";"age":[% age %]}'],
# Printing, gets `rendered`, returns input unchanged
[
'Writer::to_files',
header => "[\n",
footer => "\n]\n",
interlude => ",\n",
# writing to files instead of STDOUT, setting a threshold
filename => 'output-01-%03n.json',
policy => {records_threshold => 2},
],
# Options, just flush the output to the sink
{ tap => 'sink' },
)->([qw< mydata-04.txt >]);The example above produces two output files:
$ cat output-02-000.json
[
{"name":"Foo","nick":"foorious","age":32},
{"name":"Bar","nick":"barious","age":19}
]
$ cat output-02-001.json
[
{"name":"Baz","nick":"bazzecolas","age":44}
]
This should make you happy, at least!
Last thing you need to know about to_files is that you can
set the encoding too, just set
a CORE::binmode
compatible string using the binmode argument, that is set to
:encoding(UTF-8) by default.
use Data::Tubes qw< pipeline >;
pipeline(
['Source::iterate_files', open_file_args => {binmode => ':raw'}],
'Reader::by_line',
['Parser::by_format', format => 'name;nick;age'],
['Renderer::with_template_perlish', format =>
' {"name":"[% name %];"nick":"[% nick %]";"age":[% age %]}'],
# Printing, gets `rendered`, returns input unchanged
['Writer::to_files', filename => 'output-02-%03n.json',
header => "[\n", footer => "\n]\n", interlude => ",\n",
policy => {records_threshold => 2},
binmode => ':raw',
],
# Options, just flush the output to the sink
{ tap => 'sink' },
)->([qw< mydata-04.txt >]);Now, you have full control over your output. Or have you? Maybe you can find some additional trick in the article about dispatch_to_files.
