feat(grpc): Add `protobuf` codegen #2320

arjan-bal · 2025-07-02T18:22:41Z

This PR includes the following:

A tonic codec that uses protobuf-rust.
A protoc plugin that generates tonic compatible client code.
A grpc-build crate that helps generate code during cargo builds.
Interop tests that use the new client codgen running against a Go server. This is used as an integration test for the new codegen.

To keep the CI fast, the protoc plugin binary for each OS is cached. The plugin binaries are re-built only if the plugin's code or build files are updated.

Notes

The codegen is generating code for tonic. It will be changed when the gRPC API design is finalized and the channel implementation is complete.

compiler/README.md

interop/test.sh

LucioFranco

Overall LGTM, lets get these last few things addressed and CI passing then lets merge.

LucioFranco · 2025-07-16T12:03:58Z

.github/workflows/CI.yml

+          # Share repository cache between workflows.
+          repository-cache: true
+          module-root: ./protoc-gen-rust-grpc
+      - name: Build protoc plugin


Does this step take a long time? Generally, speaking I would rather that we build the plugin in each test job to ensure we are building the latest code and there is no caching issues. Could you explain the reasoning for breaking it up like this?

Added a comment in the yaml. Building the protoc plugin from scratch takes 6–14 minutes, depending on the OS. This delays the execution of workflows that use the plugin in build.rs files.

Example workflow execution: https://github.com/hyperium/tonic/actions/runs/16217230891/job/45789317103

Okay yeah if this works its fine for now, we probably want to improve this in the future..

grpc/Cargo.toml

grpc/src/macros.rs

interop/Cargo.toml

interop/src/bin/client_new_codegen.rs

LucioFranco · 2025-07-16T12:40:07Z

tonic-protobuf-build/src/lib.rs

+                .unwrap();
+        }
+        let crate_mapping_path = if self.generate_message_code {
+            self.output_dir.join("crate_mapping.txt")


is it possible to have multiple codegens override the same file? Do we need to add a warning incase say you do that by accident and its surprising?

is it possible to have multiple codegens override the same file?

Yes it's possible. A bigger problem when having multiple protoc invocations using the same output directory is the generated.rs file getting overwritten. The generated.rs file exports the generated message symbols for all the proto files that were part of the input protos list.

Do we need to add a warning incase say you do that by accident and its surprising?

@acozzette wanted to get your views on this. Have you considered this issue?

LucioFranco · 2025-07-16T12:40:33Z

tonic-protobuf-build/src/lib.rs

+        };
+
+        // Generate the service code.
+        let mut cmd = std::process::Command::new("protoc");


We probably want to allow users to set the path to protoc via an env var as well. We do this in the prost code so we should follow how it does that.

Here are the relevant prost docs: https://docs.rs/prost-build/latest/prost_build/index.html#sourcing-protoc

The message codegen also doesn't support setting the path to protoc using an env var. I think protobuf-codegen should support this before gRPC. Do you want me to file a feature request for protobuf?

Yeah, that would be good I think it shouldn't be too contentious.

Filed protocolbuffers/protobuf#22718

tonic-protobuf/Cargo.toml

LucioFranco · 2025-07-16T12:43:17Z

tonic-protobuf/src/lib.rs

+    type Error = Status;
+
+    fn encode(&mut self, item: Self::Item, buf: &mut EncodeBuf<'_>) -> Result<(), Self::Error> {
+        let serialized = item.serialize().map_err(from_decode_error)?;


We should add a todo here mentioning that we want to figure out how to remove this extra copy that happens.

Added a comment. Do you want me to file an issue for the protobuf-rust?

That might be good, we discussed this with @acozzette and co last week but probably good to track it somewhere as well. I would be okay if this was also just a tonic issue.

Opened a tonic issue with context about why protobuf doesn't provide the required API: #2345

LucioFranco · 2025-07-16T12:44:19Z

tonic-protobuf/src/lib.rs

+        let item = U::parse(slice).map_err(from_decode_error)?;
+        buf.advance(slice.len());


I think its possible that the incoming buf slice is larger than the actual message, we should see if we can pull the len from the decoded amount of bytes.

From my understanding, each field in an encoded proto message either has a length prefix or a fixed length. The parser should continue reading fields until it reaches the end of the buffer. Fields not present in the proto descriptor should be ignored to ensure forward compatibility—for example, when a new field is added to the proto, but the receiver is using an older version. A parsing error should be returned if the bytes cannot be parsed. Since there is no length prefix for the entire message, the parser must consume all the bytes it is given.

I think its possible that the incoming buf slice is larger than the actual message

Based on the above, I believe it may be incorrect to pass a slice to a parser but not consume the entire buffer. Can you clarify when this situation arises?

Hmm that is a good question, I will look into it, maybe we also want to actually provide some type safe wrapper that enforces this invariant. I wouldn't worry about this for now I think your assumption is probably right on what we actually do.

Co-authored-by: Lucio Franco <[email protected]>

LucioFranco

LGTM, I tested this locally, the absolute paths is what solved it for me. We can merge this but I want to merge #2321 first and then rebase this ontop of that before we merge this.

This check was meant to enforce that the version of the protobuf crate exactly matches the version of the codegen crate, but this is not strictly necessary. Tonic would also like to re-export our crate from the `tonic-protobuf` crate, and the `DEP_UPB_VERSION` check is causing problems with that since the environment variable is set only for crates that directly depend on the `protobuf` crate (see [discussion](hyperium/tonic#2320 (comment))). While I was at it I also removed the code in the protobuf build script that sets the `DEP_UPB_INCLUDE` variable, which is unnecessary now that we no longer generate any C code which would need to know where to find upb's headers. PiperOrigin-RevId: 786745312

This check was meant to enforce that the version of the protobuf crate exactly matches the version of the codegen crate, but this is not strictly necessary. Tonic would also like to re-export our crate from the `tonic-protobuf` crate, and the `DEP_UPB_VERSION` check is causing problems with that since the environment variable is set only for crates that directly depend on the `protobuf` crate (see [discussion](hyperium/tonic#2320 (comment))). While I was at it I also removed the code in the protobuf build script that sets the `DEP_UPB_INCLUDE` variable, which is unnecessary now that we no longer generate any C code which would need to know where to find upb's headers. PiperOrigin-RevId: 786760264

arjan-bal · 2025-07-29T19:45:16Z

Hi @dfawley, @LucioFranco can you please have another look?

arjan-bal added 2 commits July 2, 2025 20:57

protobuf codec, codegen, interop test

1f78401

CI fixes

6a379ea

arjan-bal changed the title ~~gRPC client codegen~~ [WIP] gRPC client codegen Jul 2, 2025

LucioFranco reviewed Jul 2, 2025

View reviewed changes

compiler/README.md Outdated Show resolved Hide resolved

arjan-bal changed the title ~~[WIP] gRPC client codegen~~ gRPC client codegen Jul 2, 2025

arjan-bal marked this pull request as draft July 2, 2025 18:33

LucioFranco reviewed Jul 2, 2025

View reviewed changes

interop/test.sh Outdated Show resolved Hide resolved

LucioFranco changed the title ~~gRPC client codegen~~ feat(grpc): Add protobuf codegen Jul 2, 2025

arjan-bal force-pushed the grpc-codegen branch 4 times, most recently from d39e0f5 to de384b4 Compare July 4, 2025 08:54

Enforce c++17

f3f9446

arjan-bal force-pushed the grpc-codegen branch 17 times, most recently from 9e45ab4 to dfc5213 Compare July 4, 2025 22:43

LucioFranco requested changes Jul 16, 2025

View reviewed changes

Address review

9f06a59

arjan-bal force-pushed the grpc-codegen branch from 869d9ee to 9f06a59 Compare July 17, 2025 13:36

arjan-bal and others added 2 commits July 17, 2025 19:12

Apply suggestions from code review

5cd495c

Co-authored-by: Lucio Franco <[email protected]>

Combine client binaries, use flag for codec

c3e03f9

arjan-bal requested a review from LucioFranco July 17, 2025 18:00

arjan-bal added 2 commits July 21, 2025 15:11

Merge remote-tracking branch 'source/master' into grpc-codegen

e8aaa51

Fix cpp function name casing

726d227

arjan-bal mentioned this pull request Jul 22, 2025

Allow specifying protoc location using an env var for rust codegen protocolbuffers/protobuf#22718

Open

arjan-bal added 2 commits July 23, 2025 13:53

Address review

d13eed5

Align tonic-* crate versions in README

c33f380

LucioFranco reviewed Jul 24, 2025

View reviewed changes

copybara-service bot mentioned this pull request Jul 24, 2025

Rust: remove DEP_UPB_VERSION check in codegen crate protocolbuffers/protobuf#22762

Merged

acozzette mentioned this pull request Jul 24, 2025

Cherry-pick: Rust: remove DEP_UPB_VERSION check in codegen crate protocolbuffers/protobuf#22764

Merged

arjan-bal added 3 commits July 25, 2025 15:16

Use include! instead of path

00b6d63

Merge remote-tracking branch 'source/master' into grpc-codegen

e35ddb9

Export protobuf from tonic-protobuf

4b150c1

arjan-bal force-pushed the grpc-codegen branch from bc810a4 to 4b150c1 Compare July 29, 2025 19:36

arjan-bal requested review from LucioFranco and dfawley July 29, 2025 19:42

acozzette mentioned this pull request Jul 30, 2025

Rust protoc to support generate service? protocolbuffers/protobuf#22746

Open

LucioFranco approved these changes Jul 30, 2025

View reviewed changes

dfawley approved these changes Jul 30, 2025

View reviewed changes

dfawley merged commit d244567 into hyperium:master Jul 30, 2025
20 checks passed

		let item = U::parse(slice).map_err(from_decode_error)?;
		buf.advance(slice.len());

feat(grpc): Add protobuf codegen #2320

feat(grpc): Add protobuf codegen #2320

Uh oh!

Conversation

arjan-bal commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

Uh oh!

Uh oh!

Uh oh!

LucioFranco left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LucioFranco Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LucioFranco left a comment

Choose a reason for hiding this comment

Uh oh!

arjan-bal commented Jul 29, 2025

Uh oh!

Uh oh!

Uh oh!

feat(grpc): Add `protobuf` codegen #2320

feat(grpc): Add `protobuf` codegen #2320

arjan-bal commented Jul 2, 2025 •

edited

Loading

LucioFranco Jul 21, 2025 •

edited

Loading