Skip to content

update pulsar-function proto to support batching configs #812

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .dockerignore.protogen
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Git files
.git
.gitignore

# Documentation
*.md
docs/

# Build artifacts
bin/
artifacts/
dist/

# IDE files
.vscode/
.idea/
*.swp
*.swo

# OS files
.DS_Store
Thumbs.db

# Docker files (except the ones we need)
Dockerfile
*.dockerfile
.dockerignore

# Test files
*_test.go
testdata/

# CI/CD
.github/
.travis.yml
.circleci/

# Other unnecessary files for protobuf generation
charts/
config/
hack/
images/
manifests/
mesh-worker-service/
tools/
LICENSE
install.sh
bump_version.sh
*.sh
!controllers/proto/generate.sh
8 changes: 4 additions & 4 deletions .github/workflows/bundle-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,10 @@ jobs:
username: ${{ secrets.DOCKER_USER }}
password: ${{ secrets.DOCKER_PASSWORD }}

- name: Set up GO 1.24.2
- name: Set up GO 1.24.4
uses: actions/setup-go@v1
with:
go-version: 1.24.2
go-version: 1.24.4
id: go

- name: InstallKubebuilder
Expand Down Expand Up @@ -180,10 +180,10 @@ jobs:
username: ${{ secrets.DOCKER_USER }}
password: ${{ secrets.DOCKER_PASSWORD }}

- name: Set up GO 1.24.2
- name: Set up GO 1.24.4
uses: actions/setup-go@v1
with:
go-version: 1.24.2
go-version: 1.24.4
id: go

- name: InstallKubebuilder
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/olm-verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,10 @@ jobs:
- name: checkout
uses: actions/checkout@v2

- name: Set up GO 1.24.2
- name: Set up GO 1.24.4
uses: actions/setup-go@v1
with:
go-version: 1.24.2
go-version: 1.24.4
id: go

- name: InstallKubebuilder
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
strategy:
fail-fast: false
matrix:
go-version: [1.22.12, 1.24.2]
go-version: [1.22.12, 1.24.4]
steps:
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/[email protected]
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,10 @@ jobs:
username: ${{ secrets.DOCKER_USER }}
password: ${{ secrets.DOCKER_PASSWORD }}

- name: Set up GO 1.24.2
- name: Set up GO 1.24.4
uses: actions/setup-go@v1
with:
go-version: 1.24.2
go-version: 1.24.4
id: go

- name: InstallKubebuilder
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/test-helm-charts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -83,11 +83,11 @@ jobs:
run: hack/kind-cluster-build.sh --name chart-testing -c 1 -v 10 --k8sVersion v1.23.17
if: steps.list-changed.outputs.changed == 'true'

- name: Set up GO 1.24.2
- name: Set up GO 1.24.4
if: steps.list-changed.outputs.changed == 'true'
uses: actions/setup-go@v1
with:
go-version: 1.24.2
go-version: 1.24.4
id: go

- name: setup kubebuilder 3.6.0
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/trivy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,10 @@ jobs:
repository: ${{github.event.pull_request.head.repo.full_name}}
ref: ${{ github.event.pull_request.head.sha }}

- name: Set up GO 1.24.2
- name: Set up GO 1.24.4
uses: actions/setup-go@v1
with:
go-version: 1.24.2
go-version: 1.24.4
id: go

- name: InstallKubebuilder
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/trivy_scheduled_master.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,10 @@ jobs:
repository: ${{github.event.pull_request.head.repo.full_name}}
ref: ${{ github.event.pull_request.head.sha }}

- name: Set up GO 1.24.2
- name: Set up GO 1.24.4
uses: actions/setup-go@v1
with:
go-version: 1.24.2
go-version: 1.24.4
id: go

- name: InstallKubebuilder
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Build the manager binary
FROM golang:1.24.2-bullseye as builder
FROM golang:1.24.4-bullseye as builder

WORKDIR /workspace/api
COPY api/ .
Expand Down
42 changes: 42 additions & 0 deletions Dockerfile.protogen
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Build environment for protobuf generation
# Requirements: protoc 3.17.3, protoc-gen-go v1.3.5
# Using Debian base to avoid musl/glibc compatibility issues with protoc x86_64 binaries
FROM golang:1.24.4

# Install system dependencies
RUN apt-get update && apt-get install -y \
bash \
curl \
unzip \
git \
make \
&& rm -rf /var/lib/apt/lists/*

# Set environment variables
ENV PROTOC_VERSION=3.17.3
ENV PROTOC_GEN_GO_VERSION=v1.3.5

# Install protoc 3.17.3
RUN set -eux; \
PROTOC_ZIP="protoc-${PROTOC_VERSION}-linux-x86_64.zip"; \
curl -OL "https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/${PROTOC_ZIP}"; \
unzip -o "$PROTOC_ZIP" -d /usr/local; \
rm -f "$PROTOC_ZIP"

# Install protoc-gen-go 1.25.0
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

# Verify installations
RUN protoc --version

# Set working directory
WORKDIR /workspace

# Copy the entire project
COPY . .

# Make sure generate.sh is executable
RUN chmod +x controllers/proto/generate.sh

# Default command: run the generate script
CMD ["/workspace/controllers/proto/generate.sh", "/workspace"]
71 changes: 71 additions & 0 deletions README-protogen.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Protobuf Generation Docker Environment

This Docker environment contains the tools and environment required to generate function-mesh protobuf Go definitions.

## Requirements

- protoc: v3.17.3
- protoc-gen-go: v1.25.0
- protoc-gen-go-grpc: v1.0.0
- Go: 1.24.4+

**Note**: This image uses a Debian base (not Alpine) to ensure compatibility with the protoc x86_64 binary, which is compiled for glibc rather than musl libc.

## Build and Usage

### 1. Build Docker Image

```bash
docker build -f Dockerfile.protogen -t function-mesh-protogen .
```

### 2. Run Protobuf Generation

```bash
# Run container and generate protobuf files
docker run --rm -v $(pwd):/workspace function-mesh-protogen

# Or run interactively for debugging
docker run --rm -it -v $(pwd):/workspace function-mesh-protogen bash
```

### 3. Manual Script Execution

If you need manual control over the generation process:

```bash
# Enter container
docker run --rm -it -v $(pwd):/workspace function-mesh-protogen bash

# Execute manually inside container
cd /workspace
./controllers/proto/generate.sh /workspace
```

## Generated Files

The script will update the following files:
- `controllers/proto/Function.pb.go`

## Environment Verification

You can verify tool versions inside the container:

```bash
protoc --version # Should show libprotoc 3.17.3
go version # Should show go1.24.4+
```

## Troubleshooting

If you encounter permission issues, ensure the generate.sh script has execute permissions:

```bash
chmod +x controllers/proto/generate.sh
```

If you encounter protoc-gen-go not found issues, ensure $GOPATH/bin is in PATH:

```bash
export PATH=$PATH:$(go env GOPATH)/bin
```
10 changes: 10 additions & 0 deletions api/compute/v1alpha1/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,16 @@ type ProducerConfig struct {
CryptoConfig *CryptoConfig `json:"cryptoConfig,omitempty"`
BatchBuilder string `json:"batchBuilder,omitempty"`
CompressionType CompressionType `json:"compressionType,omitempty"`
BatchingConfig *BatchingConfig `json:"batchingConfig,omitempty"`
}

type BatchingConfig struct {
Enabled bool `json:"enabled,omitempty"`
BatchingMaxPublishDelayMs int32 `json:"batchingMaxPublishDelayMs,omitempty"`
RoundRobinRouterBatchingPartitionSwitchFrequency int32 `json:"roundRobinRouterBatchingPartitionSwitchFrequency,omitempty"`
BatchingMaxMessages int32 `json:"batchingMaxMessages,omitempty"`
BatchingMaxBytes int32 `json:"batchingMaxBytes,omitempty"`
BatchBuilder string `json:"batchBuilder,omitempty"`
}

type CryptoConfig struct {
Expand Down
2 changes: 1 addition & 1 deletion api/go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/streamnative/function-mesh/api

go 1.24.2
go 1.24.4

require (
k8s.io/api v0.30.9
Expand Down
Loading
Loading