Use Async functions for OpenAi functions #24
Conversation
|
|
||
| public FunctionExecutor(FunctionContext context, String functionName) { | ||
| this.metricTracker = new FunctionMetricTracker(context, functionName); | ||
| this.executorService = Executors.newFixedThreadPool(10); |
There was a problem hiding this comment.
Unfortunately, I couldn’t find a way to share the executor. The most optimal approach would be to borrow one from the context, but as far as I know, such a feature doesn’t exist. Would it make sense to make the pool size configurable via an environment variable, specific to the function? For instance, something like FUNCTION_NAME.toUpperCase() + "_POOL_SIZE".
There was a problem hiding this comment.
That's fine, we can have one threadpool per function and use a single environment variable ASYNC_FUNCTION_THREAD_POOL_SIZE to control it across all functions. We can later decide if we need to make it function specific.
mbroecheler
left a comment
mbroecheler
left a comment
There was a problem hiding this comment.
Nice! Two smallish changes, please
|
|
||
| public FunctionExecutor(FunctionContext context, String functionName) { | ||
| this.metricTracker = new FunctionMetricTracker(context, functionName); | ||
| this.executorService = Executors.newFixedThreadPool(10); |
There was a problem hiding this comment.
That's fine, we can have one threadpool per function and use a single environment variable ASYNC_FUNCTION_THREAD_POOL_SIZE to control it across all functions. We can later decide if we need to make it function specific.
| metricTracker.increaseCallCount(); | ||
| final long start = System.nanoTime(); | ||
|
|
||
| T result = executeWithRetry( |
There was a problem hiding this comment.
Flink async functions have retry build-in now:
https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/config/#table-exec-async-scalar-retry-strategy
Please remove the retry logic and just keep the metrics piece.
… Adding ASYNC_FUNCTION_THREAD_POOL_SIZE environment variable to control function threadpool size.
nandorsoma
deleted the
23-refactor-openai-functions-to-use-async-functions-interface-for-improved-performance
branch
No description provided.