New Kafka Endpoint #4774

fpetrini15 · 2022-08-16T18:28:23Z

Changes are made in conjunction with changes made to the third_party repo: triton-inference-server/third_party#21

* Allow jetson test to confiure default shm size for python backend * Update jetson docs for 22.03

* Add test case for shutdown * Address comment * Fix up * Address comment

* Add metric API unit test to L0_metrics * Fix up * Fix shared library path

* Set address for metrics when http service is disable * Update comment * Update documentation

* Add testing for the lifetime of BLS tensors * Review edit

* Add test for optional input in Python backend * Review edit * Review edit * Reduce the default backend shared memory usage * Reduce the tensor size of non-contiguous test

* Add Perf Analyzer unit tests CI test * Addressed comments

* Add CLI test for MLFlow plugin * Add Python test. Fix up * Add newline

* Build pa with tfs and ts * Update flag names * Install cmake and rapidjson on sdk for test * Add l0 client build variants * Move installation of build deps to test * Add indent * Build without TFS/TS for most variants * Remove cmake 18.04 version

Co-authored-by: GuanLuo <[email protected]>

- CPU only builds need deps from GPU enabled container

* Update build.py to generate scripts containing build commands * Add --enable-all to build.py * Update docs for new build.py behavior * Don't include build dir in copyright check * Format

* Fix compose.py to allow configurable gpu-min container * Fix missing gpu-min in allowed image names

* updated qa tests to expect the new max_batch_size default-max-batch-size is now a global backend config documented default-max-batch-size feature * parsing backend config options is now global backend aware

Small fix to render test documentation correctly.

* Fix L0_backend_python * Review edit

rmccorm4 · 2022-09-09T21:35:31Z

build.py

+# apt get update
+# apt install -y install default-jre
+# apt install -y doctest-dev
+# apt install -y libboost-program-options-dev
+# LD_LIBRARY_PATH=/opt/tritonserver/lib:$LD_LIBRARY_PATH
+# LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH


Also removing "libboost-program-options-dev " from build.py. It used to be a dependency but after removing many of the unnecessary directories from the modern-cpp-kafka directory in the third_party repo, I don't think it's necessary anymore. Mentioning it just in case.

rmccorm4 · 2022-09-09T21:40:56Z

CMakeLists.txt

 # Endpoints
 option(TRITON_ENABLE_HTTP "Include HTTP API in server" ON)
 option(TRITON_ENABLE_GRPC "Include GRPC API in server" ON)
+option(TRITON_ENABLE_KAFKA "Include Kafka API in server" ON)


should this be OFF by default? @GuanLuo

and if so, we might need to update some gitlab-ci.yaml or build.py arg to set it to ON explicitly for CI.

qa/L0_kafka/test.sh

rmccorm4 · 2022-09-09T21:44:42Z

qa/L0_kafka/test.sh

+# On windows the paths invoked by the script (running in WSL) must use
+# /mnt/c when needed but the paths on the tritonserver command-line
+# must be C:/ style.
+if [[ "$(< /proc/sys/kernel/osrelease)" == *microsoft* ]]; then
+    MODELDIR=${MODELDIR:=C:/models}
+    DATADIR=${DATADIR:="/mnt/c/data/inferenceserver/${REPO_VERSION}"}
+    BACKEND_DIR=${BACKEND_DIR:=C:/tritonserver/backends}
+    SERVER=${SERVER:=/mnt/c/tritonserver/bin/tritonserver.exe}
+    export WSLENV=$WSLENV:TRITONSERVER_DELAY_SCHEDULER


is this test going to be run on windows?

rmccorm4 · 2022-09-09T21:46:11Z

qa/L0_kafka/kafka_test.py

+                            auto_offset_reset='latest',
+                            enable_auto_commit=True,
+                            bootstrap_servers=f"{_tritonserver_ipaddr}:9092")
+    KafkaTest.consumer_.poll()


Add comment with link to what this "dummy poll" is for

rmccorm4 · 2022-09-09T21:48:19Z

qa/L0_kafka/kafka_test.py

+        # Brand new topic partition
+        print(end)
+        if(end != 0):
+            self.consumer_.seek(partition, end-1)


plus link if relevant

Suggested change

self.consumer_.seek(partition, end-1)

# seek to latest message (end-1)

self.consumer_.seek(partition, end-1)

This is mentioned in the comment block just above

rmccorm4 · 2022-09-09T21:52:56Z

src/kafka_endpoint.cc

+    // Permitted b/c strings are stored as contiguous memory in c++11
+    char* start = &binary_data[binary_data_offset];
+    // TODO: This should not be necessary. Retry using memcpy.
+    unsigned char* base = (unsigned char*)malloc(payload_length + 1);


use c++ style cast

rmccorm4 · 2022-09-09T21:54:28Z

src/kafka_endpoint.cc

+    DisplayConsumerTopics();
+
+    while (consumer_active_) {
+      auto records = consumer_->poll(std::chrono::milliseconds(100));


Is there an event based system where consumer wakes up when there's a request, or does it have to be polled? Is the 100ms arbitrary or a best practice?

Should add this to the doc

manhtd98 · 2022-10-01T10:51:52Z

Did we got anything there? Kafka is nice idea for multi triton server

pthalasta · 2024-04-26T20:48:36Z

Just curious, are there any updates when this will be merged? Looking for kafka endpoint support as well!

jbkyang-nvi and others added 30 commits April 4, 2022 13:08

remove java client build from sdk

e1fea85

Remove java client examples install

bd40000

Update jetson docs for Jetpack 22.03 (#4120)

2bbec79

* Allow jetson test to confiure default shm size for python backend * Update jetson docs for 22.03

Add test case for shutdown (#4143)

095a44e

* Add test case for shutdown * Address comment * Fix up * Address comment

Enable capi for pa (#4161)

0d87c41

Use correct error message in L0_lifecycle(#4156)

43e5076

Update docs for ORT support on Jetson (#4157) (#4167)

6f3519a

Update for 22.03 model gen (#4173)

b7561f9

Move Java client examples to build (#4116)

daf1c8f

Add metric API unit test to L0_metrics (#4163)

aa73af1

* Add metric API unit test to L0_metrics * Fix up * Fix shared library path

Set http address for metrics when http service is disable (#4175)

25c6fec

* Set address for metrics when http service is disable * Update comment * Update documentation

Record INFER_RESPONSE_COMPLETE only at the first callback (#4176)

721ecb5

Add testing for the lifetime of BLS tensors (#4147)

464629e

* Add testing for the lifetime of BLS tensors * Review edit

Add pointer to 'raw binary data' documentation (#4182)

3145219

Add test for optional input in Python backend (#4164)

77b7614

* Add test for optional input in Python backend * Review edit * Review edit * Reduce the default backend shared memory usage * Reduce the tensor size of non-contiguous test

Add Perf Analyzer unit tests CI test (#4155)

17aa8ca

* Add Perf Analyzer unit tests CI test * Addressed comments

Fix MLFlow plugin. Add test (#4186)

312840d

* Add CLI test for MLFlow plugin * Add Python test. Fix up * Add newline

Update test to use new option (#4190)

93ab8f7

Add Model Register API test (#4191)

3fb8d41

Co-authored-by: GuanLuo <[email protected]>

Fix build.py to accept gpu-base image for CPU only (#4208)

3c89895

- CPU only builds need deps from GPU enabled container

Update build docs for --image,gpu-base (#4209)

a48f47e

Update build.py to generate scripts containing build commands (#4174)

9278504

* Update build.py to generate scripts containing build commands * Add --enable-all to build.py * Update docs for new build.py behavior * Don't include build dir in copyright check * Format

Fix compose.py to allow configurable gpu-min container (#4210)

905c5f1

* Fix compose.py to allow configurable gpu-min container * Fix missing gpu-min in allowed image names

updated qa tests to expect the new max_batch_size (#4092)

22c40ce

* updated qa tests to expect the new max_batch_size default-max-batch-size is now a global backend config documented default-max-batch-size feature * parsing backend config options is now global backend aware

Test Documentation Rendering (#4218)

127df4f

Small fix to render test documentation correctly.

Add testing for incorrect shape argument in PA (#4192)

3a11151

Fix L0_backend_python (#4207)

3c535ef

* Fix L0_backend_python * Review edit

Update master to track development for 2.22.0 / r22.05

c45d3b3

Add documentation for incremental builds (#4222)

f80f203

rmccorm4 reviewed Sep 9, 2022

View reviewed changes

qa/L0_kafka/test.sh Show resolved Hide resolved

rmccorm4 reviewed Sep 9, 2022

View reviewed changes

fpetrini15 force-pushed the fpetrini-kafka branch from 4f65cf1 to 3344f35 Compare September 9, 2022 23:18

mc-nv and others added 16 commits September 9, 2022 16:22

Update master to track development for 2.27.0 / r22.10

0e1257c

Uplift GO version

c787771

Install kafka lib

a68c52d

Adding boost library

85c8661

New kafka files

b8763a4

Intermediate Checkin

13ee61e

Add Kakfa Option

ec3004b

Container and compile changes

6859117

Fix gitlab warnings

b100633

Build kafka

7a51e50

Cleanup

8a4e337

Restore http_server.cc

6cbc2de

Adding comments

55b6378

Manually adding back CPU Metrics flags

2abddda

Review comments

ee6d02e

Pushing in case credentials expire

cfe9209

fpetrini15 force-pushed the fpetrini-kafka branch from 3344f35 to 1143ec9 Compare September 10, 2022 00:58

the-david-oy mentioned this pull request May 15, 2024

Support for Kafka endpoints #3131

Open

pvijayakrish force-pushed the fpetrini-kafka branch from 1143ec9 to cfe9209 Compare January 15, 2025 17:13

	self.consumer_.seek(partition, end-1)
	# seek to latest message (end-1)
	self.consumer_.seek(partition, end-1)

New Kafka Endpoint #4774

Are you sure you want to change the base?

New Kafka Endpoint #4774

Uh oh!

Conversation

fpetrini15 commented Aug 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rmccorm4 Sep 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rmccorm4 Sep 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

manhtd98 commented Oct 1, 2022

Uh oh!

pthalasta commented Apr 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

20 participants

fpetrini15 commented Aug 16, 2022 •

edited

Loading

rmccorm4 Sep 9, 2022 •

edited

Loading

rmccorm4 Sep 9, 2022 •

edited

Loading