Skip to content

Commit ccdaf17

Browse files
authored
rosbag2_cpp: move local message definition source out of MCAP plugin (ros2#1265)
The intention of this PR is to move the message-definition-finding capability outside of rosbag2_storage_mcap, and allow any rosbag2 storage plugin to store message definitions.
1 parent 7e91733 commit ccdaf17

File tree

53 files changed

+767
-382
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+767
-382
lines changed

.github/workflows/test.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -31,9 +31,9 @@ jobs:
3131
rosbag2_py
3232
rosbag2_storage
3333
rosbag2_storage_mcap
34-
rosbag2_storage_mcap_testdata
3534
rosbag2_storage_sqlite3
3635
rosbag2_test_common
36+
rosbag2_test_msgdefs
3737
rosbag2_tests
3838
rosbag2_transport
3939
shared_queues_vendor

docs/message_definition_encoding.md

+96
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Message Definition Encoding
2+
3+
Message definitions can be stored in ROS 2 bags to facilitate decoding messages. This document
4+
describes how message definitions can be encoded within a bag.
5+
6+
## Background
7+
8+
The message definitions in the bag are not used for bag playback. However, some
9+
bag analysis tools are not aware of all message definitions ahead of time, and need to
10+
generate a deserializer using the message definitions at run-time.
11+
12+
When deserializing a message at run-time, the deserializer needs the definition of that message's
13+
type, along with the definitions for all fields of that type (and fields of field types, etc.).
14+
This set of definitions with all field types recursively included can be called a
15+
"complete message definition".
16+
17+
## `ros2msg` encoding
18+
19+
This encoding consists of definitions in [.msg](https://docs.ros.org/en/rolling/Concepts/About-ROS-Interfaces.html#message-description-specification) format, concatenated together in human-readable form with
20+
a delimiter.
21+
22+
The top-level message definition is present first, with no delimiter. All dependent .msg definitions are preceded by a two-line delimiter:
23+
24+
* One line containing exactly 80 `=` characters
25+
* One line containing `MSG: <package resource name>` for that type. The space between MSG: and the
26+
package resource name is mandatory. The package resource name does not include a file extension.
27+
28+
### `ros2msg` example
29+
30+
For example, the complete message definition for `my_msgs/msg/ExampleMsg` in `ros2msg` form is:
31+
32+
```
33+
# defines a message that includes a field of a custom message type
34+
my_msgs/BasicMsg my_basic_field
35+
================================================================================
36+
MSG: my_msgs/msg/BasicMsg
37+
# defines a message with a primitive type field
38+
float32 my_float
39+
```
40+
41+
## `ros2idl` encoding
42+
43+
The IDL definition of the type specified by name along with all dependent types are stored together. The IDL definitions can be stored in any order. Every definition is preceded by a two-line delimiter:
44+
45+
* a line containing exactly 80 `=` characters, then
46+
* A line containing only `IDL: <package resource name>` for that definition. The space between IDL: and the package resource name is mandatory. The package resource name does not include a file extension.
47+
48+
### `ros2idl` example
49+
50+
For example, the complete message definition for `my_msgs/msg/ComplexMsg` in `ros2idl` form is:
51+
52+
```
53+
================================================================================
54+
IDL: my_msgs/msg/ComplexMsg
55+
// generated from rosidl_adapter/resource/msg.idl.em
56+
// with input from my_msgs/msg/ComplexMsg.msg
57+
// generated code does not contain a copyright notice
58+
59+
#include "my_msgs/msg/BasicMsg.idl"
60+
61+
module my_msgs {
62+
module msg {
63+
struct ComplexMsg {
64+
my_msgs::msg::BasicMsg my_basic_field;
65+
};
66+
};
67+
};
68+
================================================================================
69+
IDL: my_msgs/msg/BasicMsg
70+
// generated from rosidl_adapter/resource/msg.idl.em
71+
// with input from my_msgs/msg/BasicMsg.msg
72+
// generated code does not contain a copyright notice
73+
74+
75+
module my_msgs {
76+
module msg {
77+
struct BasicMsg {
78+
float my_float;
79+
};
80+
};
81+
};
82+
```
83+
84+
## FAQ
85+
86+
### Why store message definitions in human-readable form?
87+
88+
These formats are designed to be updated as rarely as possible. Definitions in bags recorded now
89+
should ideally be parseable years in the future with no format migrations. MSG and OMG IDL 4.2 are
90+
both well-known formats for ROS 2 message definitions, and no additional knowledge besides their
91+
specifications are required for readers to deserialize messages.
92+
93+
### Why do these examples include comments?
94+
95+
The `.msg` and `.idl` message definitions are encoded according to their specifications, which
96+
include comments as valid syntax. Bag writers may strip comments out or leave them in.

rosbag2_compression/CMakeLists.txt

+1
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ if(BUILD_TESTING)
7272
find_package(ament_lint_auto REQUIRED)
7373
find_package(rclcpp REQUIRED)
7474
find_package(rosbag2_test_common REQUIRED)
75+
find_package(test_msgs REQUIRED)
7576
ament_lint_auto_find_test_dependencies()
7677

7778
add_library(fake_plugin SHARED

rosbag2_compression/include/rosbag2_compression/sequential_compression_writer.hpp

+12
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,18 @@ class ROSBAG2_COMPRESSION_PUBLIC SequentialCompressionWriter
8080
*/
8181
void create_topic(const rosbag2_storage::TopicMetadata & topic_with_type) override;
8282

83+
/**
84+
* Create a new topic in the underlying storage. Needs to be called for every topic used within
85+
* a message which is passed to write(...).
86+
*
87+
* \param topic_with_type name and type identifier of topic to be created
88+
* \param message_definition definition of topic_with_type.type
89+
* \throws runtime_error if the Writer is not open.
90+
*/
91+
void create_topic(
92+
const rosbag2_storage::TopicMetadata & topic_with_type,
93+
const rosbag2_storage::MessageDefinition & message_definition) override;
94+
8395
/**
8496
* Remove a new topic in the underlying storage.
8597
* If creation of subscription fails remove the topic

rosbag2_compression/package.xml

+1
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
<test_depend>ament_lint_auto</test_depend>
2323
<test_depend>ament_lint_common</test_depend>
2424
<test_depend>rclcpp</test_depend>
25+
<test_depend>test_msgs</test_depend>
2526
<test_depend>rosbag2_test_common</test_depend>
2627

2728
<export>

rosbag2_compression/src/rosbag2_compression/sequential_compression_writer.cpp

+8
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,14 @@ void SequentialCompressionWriter::create_topic(
227227
SequentialWriter::create_topic(topic_with_type);
228228
}
229229

230+
void SequentialCompressionWriter::create_topic(
231+
const rosbag2_storage::TopicMetadata & topic_with_type,
232+
const rosbag2_storage::MessageDefinition & message_definition)
233+
{
234+
std::lock_guard<std::recursive_mutex> lock(storage_mutex_);
235+
SequentialWriter::create_topic(topic_with_type, message_definition);
236+
}
237+
230238
void SequentialCompressionWriter::remove_topic(
231239
const rosbag2_storage::TopicMetadata & topic_with_type)
232240
{

rosbag2_compression/test/rosbag2_compression/mock_storage.hpp

+3-1
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,9 @@ class MockStorage : public rosbag2_storage::storage_interfaces::ReadWriteInterfa
3434
open,
3535
void(const rosbag2_storage::StorageOptions &, rosbag2_storage::storage_interfaces::IOFlag));
3636
MOCK_METHOD1(update_metadata, void(const rosbag2_storage::BagMetadata &));
37-
MOCK_METHOD1(create_topic, void(const rosbag2_storage::TopicMetadata &));
37+
MOCK_METHOD2(
38+
create_topic, void(const rosbag2_storage::TopicMetadata &,
39+
const rosbag2_storage::MessageDefinition &));
3840
MOCK_METHOD1(remove_topic, void(const rosbag2_storage::TopicMetadata &));
3941
MOCK_METHOD1(set_read_order, bool(const rosbag2_storage::ReadOrder &));
4042
MOCK_METHOD0(has_next, bool());

rosbag2_cpp/CMakeLists.txt

+8
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ add_library(${PROJECT_NAME} SHARED
6262
src/rosbag2_cpp/clocks/time_controller_clock.cpp
6363
src/rosbag2_cpp/converter.cpp
6464
src/rosbag2_cpp/info.cpp
65+
src/rosbag2_cpp/message_definitions/local_message_definition_source.cpp
6566
src/rosbag2_cpp/reader.cpp
6667
src/rosbag2_cpp/readers/sequential_reader.cpp
6768
src/rosbag2_cpp/rmw_implemented_serialization_format_converter.cpp
@@ -133,6 +134,7 @@ if(BUILD_TESTING)
133134
find_package(ament_cmake_gmock REQUIRED)
134135
find_package(ament_lint_auto REQUIRED)
135136
find_package(test_msgs REQUIRED)
137+
find_package(rosbag2_test_msgdefs REQUIRED)
136138
ament_lint_auto_find_test_dependencies()
137139

138140
add_library(
@@ -186,6 +188,12 @@ if(BUILD_TESTING)
186188
target_link_libraries(test_storage_without_metadata_file ${PROJECT_NAME})
187189
endif()
188190

191+
ament_add_gmock(test_local_message_definition_source
192+
test/rosbag2_cpp/test_local_message_definition_source.cpp)
193+
if(TARGET test_local_message_definition_source)
194+
target_link_libraries(test_local_message_definition_source ${PROJECT_NAME})
195+
endif()
196+
189197
ament_add_gmock(test_message_cache
190198
test/rosbag2_cpp/test_message_cache.cpp)
191199
if(TARGET test_message_cache)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
// Copyright 2022, Foxglove Technologies. All rights reserved.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
#ifndef ROSBAG2_CPP__MESSAGE_DEFINITIONS__LOCAL_MESSAGE_DEFINITION_SOURCE_HPP_
16+
#define ROSBAG2_CPP__MESSAGE_DEFINITIONS__LOCAL_MESSAGE_DEFINITION_SOURCE_HPP_
17+
18+
#ifndef ROSBAG2_CPP_LOCAL_MESSAGE_DEFINITION_SOURCE_MAX_RECURSION_DEPTH
19+
#define ROSBAG2_CPP_LOCAL_MESSAGE_DEFINITION_SOURCE_MAX_RECURSION_DEPTH 50
20+
#endif
21+
22+
#include <set>
23+
#include <string>
24+
#include <unordered_map>
25+
#include <unordered_set>
26+
#include <utility>
27+
28+
#include "rosbag2_cpp/visibility_control.hpp"
29+
#include "rosbag2_storage/message_definition.hpp"
30+
31+
// This is necessary because of using stl types here. It is completely safe, because
32+
// a) the member is not accessible from the outside
33+
// b) there are no inline functions.
34+
#ifdef _WIN32
35+
# pragma warning(push)
36+
# pragma warning(disable:4251)
37+
#endif
38+
39+
namespace rosbag2_cpp
40+
{
41+
42+
class DefinitionNotFoundError : public std::exception
43+
{
44+
private:
45+
std::string name_;
46+
47+
public:
48+
explicit DefinitionNotFoundError(std::string name)
49+
: name_(std::move(name))
50+
{
51+
}
52+
53+
const char * what() const noexcept override
54+
{
55+
return name_.c_str();
56+
}
57+
};
58+
59+
class ROSBAG2_CPP_PUBLIC LocalMessageDefinitionSource final
60+
{
61+
public:
62+
/**
63+
* Concatenate the message definition with its dependencies into a self-contained schema.
64+
* The format is different for MSG and IDL definitions, and is described fully in
65+
* docs/message_definition_encoding.md
66+
* Throws DefinitionNotFoundError if one or more definition files are missing for the given
67+
* package resource name.
68+
*/
69+
rosbag2_storage::MessageDefinition get_full_text(const std::string & root_topic_type);
70+
71+
enum struct Format
72+
{
73+
UNKNOWN = 0,
74+
MSG = 1,
75+
IDL = 2,
76+
};
77+
78+
explicit LocalMessageDefinitionSource() = default;
79+
80+
LocalMessageDefinitionSource(const LocalMessageDefinitionSource &) = delete;
81+
LocalMessageDefinitionSource(const LocalMessageDefinitionSource &&) = delete;
82+
83+
private:
84+
struct MessageSpec
85+
{
86+
MessageSpec(Format format, std::string text, const std::string & package_context);
87+
const std::set<std::string> dependencies;
88+
const std::string text;
89+
Format format{Format::UNKNOWN};
90+
};
91+
92+
struct DefinitionIdentifier
93+
{
94+
DefinitionIdentifier() = delete;
95+
DefinitionIdentifier(const std::string & topic_type, Format format)
96+
: topic_type_(topic_type)
97+
, format_(format)
98+
{
99+
size_t h1 = std::hash<Format>()(format_);
100+
size_t h2 = std::hash<std::string>()(topic_type_);
101+
hash_ = h1 ^ h2;
102+
}
103+
bool operator==(const DefinitionIdentifier & di) const
104+
{
105+
return (format_ == di.format_) && (topic_type_ == di.topic_type_);
106+
}
107+
108+
size_t hash() const
109+
{
110+
return hash_;
111+
}
112+
113+
Format format() const
114+
{
115+
return format_;
116+
}
117+
118+
std::string topic_type() const
119+
{
120+
return topic_type_;
121+
}
122+
123+
private:
124+
std::string topic_type_;
125+
Format format_;
126+
size_t hash_;
127+
};
128+
129+
struct DefinitionIdentifierHash
130+
{
131+
size_t operator()(const DefinitionIdentifier & di) const
132+
{
133+
return di.hash();
134+
}
135+
};
136+
137+
/**
138+
* Load and parse the message file referenced by the given datatype, or return it from
139+
* msg_specs_by_datatype
140+
*/
141+
const MessageSpec & load_message_spec(const DefinitionIdentifier & definition_identifier);
142+
143+
static std::string delimiter(const DefinitionIdentifier & definition_identifier);
144+
145+
std::unordered_map<DefinitionIdentifier,
146+
MessageSpec, DefinitionIdentifierHash> msg_specs_by_definition_identifier_;
147+
};
148+
149+
ROSBAG2_CPP_PUBLIC
150+
std::set<std::string> parse_definition_dependencies(
151+
LocalMessageDefinitionSource::Format format,
152+
const std::string & text,
153+
const std::string & package_context);
154+
155+
} // namespace rosbag2_cpp
156+
157+
#ifdef _WIN32
158+
# pragma warning(pop)
159+
#endif
160+
161+
#endif // ROSBAG2_CPP__MESSAGE_DEFINITIONS__LOCAL_MESSAGE_DEFINITION_SOURCE_HPP_

rosbag2_cpp/include/rosbag2_cpp/writer.hpp

+12
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,18 @@ class ROSBAG2_CPP_PUBLIC Writer
9797
*/
9898
void create_topic(const rosbag2_storage::TopicMetadata & topic_with_type);
9999

100+
/**
101+
* Create a new topic in the underlying storage. Needs to be called for every topic used within
102+
* a message which is passed to write(...).
103+
*
104+
* \param topic_with_type name and type identifier of topic to be created
105+
* \param message_definition message definition content for this topic's type
106+
* \throws runtime_error if the Writer is not open.
107+
*/
108+
void create_topic(
109+
const rosbag2_storage::TopicMetadata & topic_with_type,
110+
const rosbag2_storage::MessageDefinition & message_definition);
111+
100112
/**
101113
* Trigger a snapshot when snapshot mode is enabled.
102114
* \returns true if snapshot is successful, false if snapshot fails or is not supported

rosbag2_cpp/include/rosbag2_cpp/writer_interfaces/base_writer_interface.hpp

+5
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#include "rosbag2_cpp/visibility_control.hpp"
2323

2424
#include "rosbag2_storage/serialized_bag_message.hpp"
25+
#include "rosbag2_storage/message_definition.hpp"
2526
#include "rosbag2_storage/storage_options.hpp"
2627
#include "rosbag2_storage/topic_metadata.hpp"
2728

@@ -43,6 +44,10 @@ class ROSBAG2_CPP_PUBLIC BaseWriterInterface
4344

4445
virtual void create_topic(const rosbag2_storage::TopicMetadata & topic_with_type) = 0;
4546

47+
virtual void create_topic(
48+
const rosbag2_storage::TopicMetadata & topic_with_type,
49+
const rosbag2_storage::MessageDefinition & message_definition) = 0;
50+
4651
virtual void remove_topic(const rosbag2_storage::TopicMetadata & topic_with_type) = 0;
4752

4853
virtual void write(std::shared_ptr<const rosbag2_storage::SerializedBagMessage> message) = 0;

0 commit comments

Comments
 (0)