Create an Arrow File format

[asfimport]

File layout:
(DictionaryBatch, RecordBatch, Schema as defined in Message.fbs)

MAGIC:   ARROW1
(
DictionaryBatch:  DictionaryBatch Header (FlatBuffer)
DictionaryBatch: DictionaryBatch Body (buffers concatenated)
)*
(
RecordBacth: RecordBatch Header (FlatBuffer)
RecordBacth: RecordBatch Body (buffers concatenated)
)+
Footer: Flatbuffer
Footer length: int (4 bytes unsigned LE)
MAGIC: ARROW1

Footer definition:

table Footer {
  schema: org.apache.arrow.flatbuf.Schema;
  dictionaries: [ Block ];
  recordBatches: [ Block ];
}
struct Block {
  offset: long;
  metaDataLength: int;
  bodyLength: long;
}

Reporter: Julien Le Dem / @julienledem
Assignee: Julien Le Dem / @julienledem

Related issues:

• [C++] C++ implementation of file-like layout for RPC / IPC (is depended upon by)

PRs and other links:

• PR

_{Note: This issue was originally created as ARROW-264. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Looks like a good start to me. We should add some minor internal details, like padding all byte buffers to start and end on 8-byte boundaries (according to the Arrow spec memory will already be aligned and padded, but the serialized metadata may require padding bytes). This is a similar, but much more general version of a file layout compared with what we did in Feather (which has a schema and record batch headers in a single metadata chunk, but only a single record batch and no dictionaries – https://github.com/wesm/feather/blob/master/doc/FORMAT.md).

[asfimport]

Julien Le Dem / @julienledem:
PR: #123
(work in progress)

[asfimport]

Julien Le Dem / @julienledem:
@wesm this sounds good. I'm considering reworking the RecordBatch message. I find the flattened FieldNode and Buffer lists very error prone since there are various factors that influence the number of buffers returned for a FieldNode. I'd rather have a FieldNode tree mimicking the Schema and have a list of Buffers in each. That would be more explicit and easier to manipulate. It would also make bugs related to reading the buffers a lot more obvious. Right now it is easy to misattribute the buffers and not know about it.

[asfimport]

Julien Le Dem / @julienledem:
@wesm Here is a PR that reflects my previous comment: #124

[asfimport]

Julien Le Dem / @julienledem:
Issue resolved by pull request 123
#123