protoc-gen-openapiv2/options/*.proto paths don't match package name

[micolous]

🐛 Bug Report

protoc-gen-openapiv2's Protobuf annotations file paths don't match the package name (protoc-gen-openapiv2/options/*.proto vs grpc.gateway.protoc_gen_openapiv2.options).

The present behaviour conflicts with the Protobuf style guide (emphasis added):

Package names should have unique names based on the project name, and possibly based on the path of the file containing the protocol buffer type definitions.

This also would have been flagged by the Buf lint rules PACKAGE_DIRECTORY_MATCH and PACKAGE_LOWER_SNAKE_CASE, but this project currently disables them in its buf.yaml.

Background / expected behaviour

As an example, the google.api and google.protobuf types are in google/api/*.proto and google/protobuf/*.proto respectively, and can be used in your own descriptors with:

import "google/api/annotations.proto";

service MyService {
  rpc ExampleMethod(ExampleMethodRequest) returns (ExampleMethodResponse) {
    option (google.api.http) = {
      post: "/v1/example/method";
    };
  }
}

The .proto file paths are consistent with their package names, and it's possible to ship .proto files along side other built artefacts in Java, JavaScript, Python and probably some other languages.

It's also possible to reflect these annotations at runtime from your application code.

protoc-gen-openapiv2 annotations should be usable in the same way, with an import that matches the complete package name:

import "grpc/gateway/protoc_gen_openapiv2/options/annotations.proto";

option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_swagger) = {
  info: {
    name: "My API";
  };
};

But this is not possible due to this inconsistency.

Existing behaviour

protoc requires that all imports and types resolve properly, including annotations, when generating either client or server-side stubs, even they are never referenced at run-time. This also constrains protoc-gen-openapiv2's annotations to the intersection of all language-specific Protobuf rules, not just Golang's.

Some tooling (like hatch-protobuf) will recursively enumerate all .proto files in a directory and attempt to build stubs, with no way to selectively include/exclude files. When applied to this repository, this means including a bunch of examples and internal implementation details.

The existing behaviour forces downstream consumers to keep a locally-modified copy of the descriptors with corrected imports and folder layout. However, this risks version skew, and all downstream consumers need to apply similar fixes.

Suggested fixes

• Move annotations.proto and openapiv2.proto into a new directory: protoc-gen-openapiv2/proto/grpc/gateway/protoc_gen_openapiv2/options/

• Make annotations.proto's an import of openapiv2.proto relative to //, rather than //grpc/gateway

• Re-enable PACKAGE_DIRECTORY_MATCH and PACKAGE_LOWER_SNAKE_CASE buf linter rules for annotations.proto and openapiv2.proto.

This makes everything consistent with the defined package name, and also means the file paths use hyphens, rather than underscores (which is nice for Java, Python, Rust and probably others).

This also makes it possible to include grpc-gateway as a git submodule, and have a --proto_path flag like third_party/grpc-gateway/protoc-gen-openapiv2/proto/, and only pick up the public protoc-gen-openapiv2 Protobuf descriptors.

It would also make it easier to make a regular Python package with something like hatch-protobuf, and take advantage of consistent import paths between Python and protoc, like you can with googleapis-common-protos and base google.protobuf types.

Alternatives considered

Using buf's schema registry to pull in protoc-gen-openapiv2 descriptors side-steps the path issues, but has a very low rate limit (10 requests/hour).

I'm aware of the py_protoc_gen_openapiv2 package – but I don't like the approach as it builds buf from source (so depends on a Golang toolchain), it doesn't include .proto files as artefacts that other tooling can use, and its releases don't match with grpc-gateway versions. There are other forks which use buf in different ways, but they all have similar problems.

I've got an alternative local package for Python which has corrected import paths and don't depend on any of the buf tooling, and it's just a pyproject.toml file that uses hatch-protobuf – there's no need to manually install protoc (as grpc-tools includes protoc), and everything just works.

[johanbrandhorst]

Hi, thanks for your issue. This sounds like a dupe of #2173, but I appreciate the amount of detail you've added and since stalebot closed the other one (we've since removed stalebot), lets keep this open. I have a few thoughts, but mostly they echo what I said in the original issue - this is a very unfortunate consequence of the original path to these files and the way python creates its imports. We cannot move these files as it would break a lot of existing users. As far as I can tell, the suggested fix hasn't considered the consequences of this breaking change. Am I missing something?

I'd be happy to accept any PRs that modify our bazel setup to make this work better for users, but we cannot move the protobuf files, sorry.