restructure

Author: Karsten1987

articles/rosbags.md

@@ -27,80 +27,48 @@ The technical challenges shall be examined within this article.
 We shall the discuss the requirements on such a tool, the technical challenges and changes to be made to the current ROS2.0 system.
 
 
-## Requirements
+## Requirements for storage format
 
-### API for recording and replaying
+To be chosen from section "Relaxed or dismissed requirements" in the "Alternatives" section
 
-The most important requirement for rosbags is being capable to record all available topics with minimal deserializing overhead.
-We shall therefore implement a set of functions in the rmw layer, which allows users to take messages raw, meaning in a serialized form object to the underlying middleware.
-In the case of DDS, such a raw message shall correspond to the CDR data being sent over the wire.
-Simultaneously, there shall be an API to convert a ROS2.0 message into its binary representation, such as CDR, in order to record data which is not being sent over the wire but created manually.
 
-The same requirements are set for publishing stored data, where already serialized data shall be transmitted over the wire without the need of serializing.
-Analog to taking a message in its raw format, we shall implement a rmw function which allows publishing a raw message on a topic.
-In order to read a serialized message from a rosbag, we shall have a function which converts a serialized binary representation in its corresponding ROS2.0 message.
+## Proposal for data storage format
 
-Given the requirements above, we propose the following rmw API:
+To be chosen from section "Dismissed data storage formats" in the "Alternatives" section
 
-```c
-rmw_ret_t
-rmw_take_raw(const rmw_subscription_t * subscription, rmw_message_raw_t * raw_message, bool * taken);
 
-rmw_ret_t
-rmw_to_raw_message(const void * ros_message, const rosidl_message_type_support_t * type_support, rmw_message_raw_t * raw_message);
+## Alternatives
 
-rmw_ret_t
-rmw_publish_raw(const rmw_publisher_t * publisher, const rmw_message_raw_t * raw_message);
+### Relaxed or dismissed requirements
+We need a data storage format which allows to sufficiently store and replay transmitted data with the least possible cost overhead.
+There are a few requirements for writing and reading to such a data storage format:
 
-rmw_ret_t
-rmw_from_raw_message(const void * rmw_message_raw_t, const rosidl_message_type_support_t * type_support, void * ros_message);
-```
+#### Scalability
+The chosen format has to scale up to multiple terabytes of file size.
 
-The in the code snippet mentioned raw message shall be defined as follows:
+#### Parallelism
+It must be possible to write to the file from multiple instances at the same time.
 
-```c
-typedef struct rmw_message_raw_t
-{
-  unsigned int length;
-  char * raw_data;
-} rmw_message_raw_t;
-```
+#### Compression
+When the file size becomes larger, it should be possible to compress the file for long time archiving.
 
-The `raw_data` field shall contain all message data needed to extract a ROS message given a respective type support, which contains all necessary information on how to concert the raw data into its corresponding ros message type.
-An example for CDR data in case of DDS:
+#### Random access
+It must be possible to grant random read access to the file and extract specific individual messages.
+Random access further means that it should be possible to extract the n-th message of one topic.
 
-The ROS message string
-```
-std_msgs::msg::String msg;
-msg.data = "hello world 42";
-```
-translates into a rmw_message_raw_t
-```
-length: 24
-data (in hex): 0x00 0x01 0x00 0x00 0x0f 0x00 0x00 0x00 0x68 0x65 0x6c 0x6c 0x6f 0x20 0x77 0x6f 0x72 0x6c 0x64 0x20 0x34 0x32 0x00 0x00
-```
-
-
-### Requirements for storage format
-
-Given the API proposed in the above section, we need a data storage format which allows to sufficiently store and replay this serialized data.
-There are a few requirements for writing to such a data storage format:
-
-- Scalability: The chosen format has to scale up to multiple terabytes of file size.
-- Parallelism: It must be possible to write to the file from multiple instances at the same time.
-- Compression (might be relaxed): Ideally, when the file size becomes larger, it should be possible to compress the file for long time archiving.
+#### Range access
+It further should be possible to only replay/read a section of the record, specified by a range of time.
+It should be possible to access a range of messages in terms of timestamps from `tx` to `tx+n`.
 
-In terms of reading from the file, there are similar requirements:
-
-- Random access: It must be possible to grant random read access to the file and extract specific individual messages.
-- Range access: It further should be possible to only replay/read a section of the record, specified by a range of time.
-- Chunk sizes: The chunk sizes must be configurable to fit various large message types.
+#### Variable Chunk sizes
+The chunk sizes must be configurable to fit various large message types in order to stay performant even with large data messages.
 
+#### Backwards compatible with ROS1
 A general requirement is to be backwards compatible with existing ROS1 bags.
 This compatibility can either be via a conversion script, which permanently converts ROS1 bags into ROS2 bags or a bridge API which allows to manually open existing ROS1 bags and publish them into the ROS2 system.
 
 
-## Evaluation of possible data storage formats
+### Dismissed data storage formats
 
 In the following, we are iterating over a couple of data formats, which may be suitable for the underlying ROSbag implementation.
 We hereby iterate over existing third party software as well as examining of maintaining a self-made format.
@@ -150,3 +118,55 @@ The Gazebo team created a nicely composed [comparison](https://osrfoundation.atl
 - Ability to query the tables with classical relational SQL syntax.
 
 #### Cons
+
+
+## API for recording and replaying
+
+The most important requirement for rosbags is being capable to record all available topics with minimal deserializing overhead.
+We shall therefore implement a set of functions in the rmw layer, which allows users to take messages raw, meaning in a serialized form object to the underlying middleware.
+In the case of DDS, such a raw message shall correspond to the CDR data being sent over the wire.
+Simultaneously, there shall be an API to convert a ROS2.0 message into its binary representation, such as CDR, in order to record data which is not being sent over the wire but created manually.
+
+The same requirements are set for publishing stored data, where already serialized data shall be transmitted over the wire without the need of serializing.
+Analog to taking a message in its raw format, we shall implement a rmw function which allows publishing a raw message on a topic.
+In order to read a serialized message from a rosbag, we shall have a function which converts a serialized binary representation in its corresponding ROS2.0 message.
+
+Given the requirements above, we propose the following rmw API:
+
+```c
+rmw_ret_t
+rmw_take_raw(const rmw_subscription_t * subscription, rmw_message_raw_t * raw_message, bool * taken);
+
+rmw_ret_t
+rmw_to_raw_message(const void * ros_message, const rosidl_message_type_support_t * type_support, rmw_message_raw_t * raw_message);
+
+rmw_ret_t
+rmw_publish_raw(const rmw_publisher_t * publisher, const rmw_message_raw_t * raw_message);
+
+rmw_ret_t
+rmw_from_raw_message(const void * rmw_message_raw_t, const rosidl_message_type_support_t * type_support, void * ros_message);
+```
+
+The in the code snippet mentioned raw message shall be defined as follows:
+
+```c
+typedef struct rmw_message_raw_t
+{
+  unsigned int length;
+  char * raw_data;
+} rmw_message_raw_t;
+```
+
+The `raw_data` field shall contain all message data needed to extract a ROS message given a respective type support, which contains all necessary information on how to concert the raw data into its corresponding ros message type.
+An example for CDR data in case of DDS:
+
+The ROS message string
+```
+std_msgs::msg::String msg;
+msg.data = "hello world 42";
+```
+translates into a rmw_message_raw_t
+```
+length: 24
+data (in hex): 0x00 0x01 0x00 0x00 0x0f 0x00 0x00 0x00 0x68 0x65 0x6c 0x6c 0x6f 0x20 0x77 0x6f 0x72 0x6c 0x64 0x20 0x34 0x32 0x00 0x00
+```