enh: replace local input file copy with S3 ranged gets by acardace · Pull Request #554 · llm-d/llm-d-batch-gateway

acardace · 2026-07-06T15:24:59Z

Summary

The preprocessor streams the input file from S3 and writes a full copy to local
disk so the executor can do random-access reads by byte offset. With 20 workers
at the OpenAI max of 200 MB per file, this consumes up to 4 GB of emptyDir for
input copies alone, plus the output/error files growing during execution.

This PR adds a RetrieveRange method to the BatchFilesClient interface that
fetches a byte range from storage (using the S3 Range header), and replaces all
local input file reads in the executor with ranged storage reads.

Changes

Interface & implementations

Add RetrieveRange(ctx, fileName, folderName, offset, length) to BatchFilesClient
Implement in S3 (Range header), FS (ReadAt), retryclient, tracing, and mock

Preprocessor

Remove local input.jsonl write — the preprocessor still streams the full file
for validation and plan building, but no longer writes a local copy
Plan entry offsets are unchanged — they track byte positions in the stream,
which match the remote file layout

Executor

Add inputFileRef struct holding remote storage coordinates (storageName, folderName)
Add resolveInputFileCoords — resolves an inputFileID to storage coordinates via DB lookup (once per job)
Convert readRequestLine from package-level function to Processor method that calls RetrieveRange internally
Replace *os.File with *inputFileRef throughout executeJob, processModel, processModelAsync, executeOneRequest, drainAndFinalize, drainUnprocessedRequests
drainUnprocessedRequests takes a separate storageCtx (mainCtx) for S3 reads since progressCtx (requestAbortCtx) may be cancelled during drain

Removed

createLocalInputFile, jobInputFilePath, inputFileName constant

What stays on local disk

Plan files (small, sequential reads)
output.jsonl and error.jsonl (could be streamed to S3 as a follow-up)

Testing

All 1051 tests pass
Regression tests pass
New TestRetrieveRange tests for S3, FS, and retryclient implementations

Fixes #561

github-actions · 2026-07-06T15:25:14Z

Unsigned commits detected! Please sign your commits.

For instructions on how to set up GPG/SSH signing and verify your commits, please see GitHub Documentation.

acardace · 2026-07-06T15:26:29Z

@wseaton @vMaroon @evacchi

yizhaodev · 2026-07-06T16:22:34Z

+	defer file.Close()
+
+	buf := make([]byte, length)
+	if _, err := file.ReadAt(buf, offset); err != nil {


there miight be an issue
func readNormalizedLine in preprocessor.go in will add \n, so the file length is N, but plan is N+1
then file.ReadAt(buf, N+1) will return EOF error?

I think you're right, thank you! Will fix this.

This is now fixed.

lioraron · 2026-07-06T16:32:47Z

Thanks for looking into disk usage optimization, Antonio. Before we proceed with the review, I want to share some concerns about the approach and suggest a different workflow.

Process suggestion: For changes of this scope (18 files, new interface method, rearchitecting the executor's I/O path), it would be better to start with an issue describing the problem and proposed design. That gives us a chance to discuss trade-offs and agree on an approach before investing in implementation. Happy to collaborate on the design there.

On the problem statement:

The PR describes a worst-case of 20 workers × 200MB = 4GB of emptyDir pressure. Looking at the current code:

NumWorkers defaults to 1 — the 4GB scenario requires non-default configuration with 20 concurrent max-size batch jobs
cleanupJobArtifacts() calls os.RemoveAll(jobDir) on every terminal path (success, cancel, expiry, error, recovery) — files don't accumulate; each worker holds at most one input file at a time
Realistic disk pressure at defaults is ~200MB, not 4GB

The problem is real in theory at extreme scale, but it's not clear this is hitting anyone today.

On the proposed solution:

Replacing local ReadAt with per-request S3 range-GETs has significant trade-offs:

Latency: Local ReadAt with OS page cache is <0.01ms. S3 range-GET is ~5–50ms. This latency is added to every request on the critical dispatch path
New failure mode: Every batch request now depends on S3 being reachable at dispatch time. Network blips or S3 throttling become per-request failures instead of a one-time download at preprocessing
Scale: A 1000-request batch = 1000 S3 GETs per job. With Global: 100 concurrency, that's up to 100 concurrent S3 calls per worker. At scale this adds cost and pressure on S3 rate limits
Complexity: Threading inputFileRef + storageCtx through the executor, drain, and finalize paths adds surface area for bugs (e.g., the separate storageCtx for drain)

Simpler alternatives worth discussing:

If disk pressure does become a real concern at high worker counts, there are lighter-weight options:

Kubernetes emptyDir.sizeLimit — let operators control disk allocation natively, no code change
In-memory index — the preprocessor already streams the full file for validation; it could retain parsed lines in a [][]byte keyed by plan entry, eliminating both disk and S3 calls
Memory-mapped file — same random-access pattern as today, slightly lower memory overhead
Configurable max concurrent input files — a simple semaphore that bounds how many input copies exist at once, independent of NumWorkers

I'd suggest we close this PR, open an issue with the problem statement and these alternatives, and converge on a design before writing code. What do you think?

acardace · 2026-07-07T11:27:58Z

Thanks for looking into disk usage optimization, Antonio. Before we proceed with the review, I want to share some concerns about the approach and suggest a different workflow.

Process suggestion: For changes of this scope (18 files, new interface method, rearchitecting the executor's I/O path), it would be better to start with an issue describing the problem and proposed design. That gives us a chance to discuss trade-offs and agree on an approach before investing in implementation. Happy to collaborate on the design there.

On the problem statement:

The PR describes a worst-case of 20 workers × 200MB = 4GB of emptyDir pressure. Looking at the current code:

NumWorkers defaults to 1 — the 4GB scenario requires non-default configuration with 20 concurrent max-size batch jobs

cleanupJobArtifacts() calls os.RemoveAll(jobDir) on every terminal path (success, cancel, expiry, error, recovery) — files don't accumulate; each worker holds at most one input file at a time

Realistic disk pressure at defaults is ~200MB, not 4GB

NumWorkers defaults to 1 in the Go config code (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/config/config.go#L300), but both the Helm chart (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/charts/batch-gateway/values.yaml#L307) and the shipped config file (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/cmd/batch-processor/config.yaml#L4) override it to 20, which is the operational default that all deployments use. So the 20 × 200 MB = 4 GB calculation applies to any default deployment with max-size files. The emptyDir sizeLimit defaults to 10Gi (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/charts/batch-gateway/values.yaml#L237), and input copies alone can consume 40% of that budget before accounting for output/error files growing during execution.

The problem is real in theory at extreme scale, but it's not clear this is hitting anyone today.

On the proposed solution:

Replacing local ReadAt with per-request S3 range-GETs has significant trade-offs:

Latency: Local ReadAt with OS page cache is <0.01ms. S3 range-GET is ~5–50ms. This latency is added to every request on the critical dispatch path

Per-request latency overhead of S3 range GETs (5–50ms) is irrelevant for the batch gateway. Inference latency is hundreds of ms to seconds — the S3 overhead is noise. This is a batch system, not a real-time serving path.

New failure mode: Every batch request now depends on S3 being reachable at dispatch time. Network blips or S3 throttling become per-request failures instead of a one-time download at preprocessing

S3 is already a required dependency. A network blip during execution that prevents S3 range reads would equally prevent finalization from uploading results. The failure mode isn't new, it's the same dependency the system already has. Additionally, the retryclient wrapper handles transient errors with exponential backoff.

Scale: A 1000-request batch = 1000 S3 GETs per job. With Global: 100 concurrency, that's up to 100 concurrent S3 calls per worker. At scale this adds cost and pressure on S3 rate limits

100 concurrent S3 GET requests is well within S3's capabilities. AWS documents 5,500 GET requests per second per prefix (https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html), and S3-compatible stores (MinIO, Ceph) handle similar throughput. 100 concurrent requests is not a concern.

Complexity: Threading inputFileRef + storageCtx through the executor, drain, and finalize paths adds surface area for bugs (e.g., the separate storageCtx for drain)

The multiple-context pattern is already used extensively throughout the executor. On main, there are 10 functions that take 2+ context parameters:

executeJob (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/executor.go#L182) — 4 contexts (ctx, sloCtx, userCancelCtx, requestAbortCtx)
processModel (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/executor.go#L405) — 4 contexts
processModelAsync (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/executor.go#L546) - 4 contexts
drainAndFinalize (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/executor.go#L704) - 4 contexts
executeOneRequest (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/executor.go#L916) - 2 contexts
writeResult (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/executor.go#L989) - 3 contexts
preProcessJob (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/preprocessor.go#L47) - 3 contetxs
finalizeJob (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/finalizer.go#L90) - 2 contexts
runPollingLoop (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/worker.go#L243) - 2 contexts
Adding storageCtx to drainUnprocessedRequests is consistent with this existing pattern, not a new category of complexity.

Simpler alternatives worth discussing:

If disk pressure does become a real concern at high worker counts, there are lighter-weight options:

Kubernetes emptyDir.sizeLimit — let operators control disk allocation natively, no code change

This is already set (10Gi default (https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/charts/batch-gateway/values.yaml#L237)). It doesn't solve the problem, it just makes the pod get evicted when it exceeds the limit. The point is to not need the disk space at all.

In-memory index — the preprocessor already streams the full file for validation; it could retain parsed lines in a [][]byte keyed by plan entry, eliminating both disk and S3 calls

This moves the same 200 MB × 20 workers = 4 GB problem from disk to RAM, which is worse, RAM is more expensive, and OOM kills are harder to recover from than disk pressure.

Memory-mapped file — same random-access pattern as today, slightly lower memory overhead

Still requires the file on disk. Doesn't reduce disk usage at all, it just changes how the kernel pages it.

Configurable max concurrent input files — a simple semaphore that bounds how many input copies exist at once, independent of NumWorkers

Adds throttling complexity to work around a resource problem that this PR eliminates entirely.

wseaton · 2026-07-07T13:31:28Z

+1 on the direction. I view this type of change necessary as a crash safety improvement, not just a disk usage optimization. Running out of emptyDir can cause pod eviction and takes out every in-flight job on that worker. Because startup recovery is workdir-based, eviction destroys the partial outputs too, so those jobs land as failed with nothing salvageable whenever the orphan reconciler notices. Dropping the biggest unbounded consumer of that disk budget, and reading from the source of truth instead of a local copy, is actually less moving parts from a correctness PoV.

This is also a step on the rung to more stateless workers in general. The less we can use temporary scratch space the better.

wseaton · 2026-07-07T13:40:00Z

-			if err := json.Unmarshal(bytes.TrimSuffix(buf[:entry.Length], []byte{'\n'}), &req); err == nil {
-				customID = req.CustomID
-			}
+		req, _, _, readErr := p.readRequestLine(storageCtx, entry, inputRef, logger)


If entries are known apriori contiguous here, we can batch them together and do one combined read as an optimization during the drain, correct?

Yes, very good point!

Looking at this more I think we might be able to skip this read entirely if we start storing the custom_id in the plan entry and make it available here, since it's pre-calculated and then thrown away

An issue is that the the plan entry is a fixed 16 bytes binary record, adding custom_id would break the fixed-size invariant. Honestly the plan files could be just moved to memory and custom_id be added. Maybe as a follow up?

and tbh we can do batched reads anyway (with a fixed-size in-memory buffer) for multiple entries even in the normal path of a job execution.

lioraron · 2026-07-07T14:23:19Z

This discussion is a good example of why design issues before PRs matter — we're iterating on fundamental trade-offs in PR comments, which is harder to follow and means implementation work may need to be redone.

The PR does one S3 range GET per request in the batch. For Global: 100 concurrency, that's up to 100 concurrent S3 calls sustained throughout execution. This applies not only to the drain path but also to processModel and processModelAsync.

There can be options to reduce the S3 calls volume by chunking and using buffers, e.g. for contiguous ranges or per model.

wseaton · 2026-07-07T15:27:48Z

The PR does one S3 range GET per request in the batch. For Global: 100 concurrency, that's up to 100 concurrent S3 calls sustained throughout execution. This applies not only to the drain path but also to processModel and processModelAsync.

100 sustained concurrent s3 calls is impossible in practice, no? There is a global rate limiter that affects the entire request lifecycle which is high hundreds of milliseconds to seconds of latency. The inference throughput would need to be several orders of magnitude faster to cause that level of S3 read contention

There can be options to reduce the S3 calls volume by chunking and using buffers, e.g. for contiguous ranges or per model.

I think these optimizations are usually non-obvious without some initial implementation/PoC work, up front design is usually not the best place to hash those specifics out, experimentation and benchmarking will be required.

lioraron · 2026-07-07T16:15:07Z

I think these optimizations are usually non-obvious without some initial implementation/PoC work, up front design is usually not the best place to hash those specifics out, experimentation and benchmarking will be required.

I agree, however these types of large refactoring projects that have high level trade-offs and options, should be documented and discussed first in an organized proposal, rather than immediate PR.

acardace · 2026-07-07T16:27:10Z

I think these optimizations are usually non-obvious without some initial implementation/PoC work, up front design is usually not the best place to hash those specifics out, experimentation and benchmarking will be required.

I agree, however these types of large refactoring projects that have high level trade-offs and options, should be documented and discussed first in an organized proposal, rather than immediate PR.

I can create a github issue but I don't think this deserves a design document or anything like that. I think this PR solves an obvious issue. I've already answered point by point in #554 (comment).

Also PRs about implementation specific things like these is usually where discussions happen.

lioraron

@acardace Please create an issue with the details of the problem, solution options, and proposed solution?

lioraron · 2026-07-07T16:40:00Z

-	buf := make([]byte, entry.Length)
-	if _, err := inputFile.ReadAt(buf, entry.Offset); err != nil {
+func (p *Processor) readRequestLine(ctx context.Context, entry planEntry, inputRef *inputFileRef, logger logr.Logger) (*batch_types.Request, string, *outputLine, error) {
+	rc, err := p.files.storage.RetrieveRange(ctx, inputRef.storageName, inputRef.folderName, entry.Offset, int64(entry.Length))


Performance: every call to readRequestLine issues a separate S3 ranged GET. This affects all code paths — the main async dispatch loop (processModelAsync), the sync path (processModel/executeOneRequest), and the drain path (drainUnprocessedRequests).

For a 100K-request batch, that's 100K sequential S3 round-trips in the normal execution path alone, plus potentially thousands more during drain on cancellation/expiry.

This should use batched/chunked reads — fetch a block of consecutive entries in one ranged GET into a buffer, then serve individual readRequestLine calls from that buffer. The old code's local ReadAt was effectively free; the S3 replacement needs to amortize network overhead.

How are you getting these numbers?

Right now the code has semaphores which bound the total number of concurrent inference requests. There is a 100 cap on the number of requests that can happen in parallel at any given time across all workers, this means there can only be 100 S3 ranged Gets at any given time, this is far more than acceptable for me. Arguably we could push it further without issues.

https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/charts/batch-gateway/values.yaml#L313
https://github.com/llm-d/llm-d-batch-gateway/blob/cc34fa9/internal/processor/worker/executor.go#L466

Also the goroutine making the inference request will spend only a handful of milliseconds on the S3 Gets, the actual inference request will take multiple seconds in the best case making this time (compared to doing FS reads) really negligible.

In theory there can be only 100 Gets in parallel, in practice I expect there's going to be ~10 at most at any time given that these are really quick compared to everything else the job's goroutine have to do to complete.

One more thing, the async path is completely sequential in that regard, there's 1 S3 get per worker at any given time, so even less of an issue.

@acardace I agree the overhead is currently small relative to inference time. My concern is about the design pattern itself.

Per-line network I/O is a universally avoided pattern, even when each individual call is fast. Read-ahead is universally used when sequential access is known, not only because of the per-line-read cost, but also because per-line-read cost is unnecessary. The reason is that network round-trips carry fixed overhead (HTTP request/response framing, server-side request processing, S3 API call billing) that doesn't exist with local file I/O. The previous ReadAt on a local file got OS page-cache buffering for free — sequential reads were effectively memory reads. S3 has no equivalent; each ranged GET is a full HTTP request regardless of how small the payload is.

Also, "inference dominates" holds today but isn't guaranteed in the future — faster / smaller models, better hardware, or larger batch sizes can modify the ratio. I/O buffering removes the dependency on that assumption.

The fix is also not complex — fetching chunks of consecutive entries in a single ranged GET into a local buffer, then serving individual readRequestLine calls from that buffer. This brings the I/O cost profile back in line with the local-file behavior the code replaced.

This is about not introducing an unnecessary per-record network I/O pattern when a standard buffering approach avoids it.

@acardace I agree the overhead is currently small relative to inference time. My concern is about the design pattern itself.

Per-line network I/O is a universally avoided pattern, even when each individual call is fast. Read-ahead is universally used when sequential access is known, not only because of the per-line-read cost, but also because per-line-read cost is unnecessary. The reason is that network round-trips carry fixed overhead (HTTP request/response framing, server-side request processing, S3 API call billing) that doesn't exist with local file I/O. The previous ReadAt on a local file got OS page-cache buffering for free — sequential reads were effectively memory reads. S3 has no equivalent; each ranged GET is a full HTTP request regardless of how small the payload is.

You can't do read-ahead right now with the current code design, entries are sorted by prefix-hash not by offset. I think we have to argue with what we have right now, this is a batch system, latency doesn't matter that much here, disruption of live requests and throughput does.

This change betters the crash resiliency story so IMHO it's much better than having huge fs consumption, especially when we're going to crank up the limits allowing more parallel jobs to be handled.

Read-ahead can be implemented in the future with a bigger design refactor.

Also, "inference dominates" holds today but isn't guaranteed in the future — faster / smaller models, better hardware, or larger batch sizes can modify the ratio. I/O buffering removes the dependency on that assumption.

Again, let's deal with what we have right now, I'm not sure we'll quickly see (if ever) see LLM inference that takes ~10ms so that an S3 Get becomes the bottleneck. We're really arguing about something that likely takes 1% of the time slot of a request.

The fix is also not complex — fetching chunks of consecutive entries in a single ranged GET into a local buffer, then serving individual readRequestLine calls from that buffer. This brings the I/O cost profile back in line with the local-file behavior the code replaced.

This is about not introducing an unnecessary per-record network I/O pattern when a standard buffering approach avoids it.

Again I think doing better in regards to crash resiliency and recovery is more important, using S3 as the single source of truth can help us resume jobs much more easily as well.

acardace · 2026-07-09T09:46:35Z

@wseaton @lioraron just added a section in #561 on solving the concurrent S3 calls potential issue (even if I don't think there is one right now).

Add RetrieveRange(ctx, fileName, folderName, offset, length) to the BatchFilesClient interface for fetching byte ranges from stored files. Implementations: - S3: uses GetObjectInput.Range header (bytes=start-end) - FS: opens file, ReadAt into buffer, returns io.NopCloser - retryclient: wraps with retry logic (fresh GET per attempt) - tracing: adds OTel span with offset/length attributes - mock: file-backed ReadAt for testing This enables the executor to read individual request lines on demand from shared storage instead of requiring a full local copy. Signed-off-by: Antonio Cardace <acardace@redhat.com>

Remove the local input.jsonl copy from the preprocessor and executor. The preprocessor still streams the full file for validation and plan building, but no longer writes a local copy. The executor fetches each request line on demand from shared storage via RetrieveRange. This eliminates per-job emptyDir usage for input data (up to 200 MB per job), leaving only output.jsonl and error.jsonl on local disk. Signed-off-by: Antonio Cardace <acardace@redhat.com>

Adapt all worker tests to use inputFileRef instead of *os.File. setupExecutionJob and setupAsyncExecutionJob now seed the mock files client and FileDB so resolveInputFileCoords works. Add testInputFileRef helper for tests calling executeOneRequest/processModel directly. Remove local input.jsonl assertions from preprocessor tests. Signed-off-by: Antonio Cardace <acardace@redhat.com>

j-mok-dev · 2026-07-09T17:09:25Z

I took a close look at the behavior changes here. The plan-file path still builds plans by streaming the input once, so removing the local input copy by itself looks fine and I did not find a separate correctness issue there.

However, I do think the inline issue around RetrieveRange(context.Background()) should be fixed before this lands, because it changes the cancellation / SLO expiry / SIGTERM behavior in a worse direction than the old local ReadAt path.

I also share Lior's concern that moving per-request input access from local random reads to remote ranged reads introduces a potentially expensive hot-path trade-off. I understand the argument that this PR is primarily about reducing local disk pressure / crash risk, and I’m okay treating the performance side as follow-up work, but I do think that concern should be explicitly validated, either in this PR or in the follow-up issue.

j-mok-dev · 2026-07-09T17:17:23Z

-func readRequestLine(inputFile *os.File, entry planEntry, logger logr.Logger) (*batch_types.Request, string, *outputLine, error) {
-	buf := make([]byte, entry.Length)
-	if _, err := inputFile.ReadAt(buf, entry.Offset); err != nil {
+func (p *Processor) readRequestLine(entry planEntry, inputRef *inputFileRef, logger logr.Logger) (*batch_types.Request, string, *outputLine, error) {


RetrieveRange is now remote I/O, but this is using context.Background() instead of a request-scoped context. That means cancel / SLO expiry / SIGTERM cannot interrupt a slow or stuck range read, so dispatched goroutines can stay blocked here holding semaphores and delay the errCancelled / errExpired / errShutdown paths. Before this change the input access was local ReadAt, so this can be a behavior regression rather than just an optimization concern.

That's a fair point, what about using the abortCtx to detect cancellation and have the worker go through draining in case there's a network error? Would that be ok?

or a simple

ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() rc, err := p.files.storage.RetrieveRange(ctx, inputRef.storageName, inputRef.folderName, entry.Offset, int64(entry.Length))

might be enough?

requestAbortCtx so cancel / expiry / shutdown can interrupt a slow read.

acardace · 2026-07-10T07:56:16Z

I also share Lior's concern that moving per-request input access from local random reads to remote ranged reads introduces a potentially expensive hot-path trade-off.

I don't agree with this, as I said before this is not a component delivering real-time requests, the batch gateway most important metrics are throughput and the ability to not disrupt live traffic, having ranged S3 gets before dispatching an inference request doesn't degrade these key metrics, and it improves the resiliency of the component.

Also the time taken by an S3 request is orders of magnitude smaller than the inference request (we're talking about ~10-50ms vs seconds or even minutes) so I don't think the performance argument stands here, if this would be handling live traffic then it would be a different story altogether. I honestly don't think performance is impacted at all by this patch.

j-mok-dev · 2026-07-10T14:09:41Z

I also share Lior's concern that moving per-request input access from local random reads to remote ranged reads introduces a potentially expensive hot-path trade-off.

I don't agree with this, as I said before this is not a component delivering real-time requests, the batch gateway most important metrics are throughput and the ability to not disrupt live traffic, having ranged S3 gets before dispatching an inference request doesn't degrade these key metrics, and it improves the resiliency of the component.

Also the time taken by an S3 request is orders of magnitude smaller than the inference request (we're talking about ~10-50ms vs seconds or even minutes) so I don't think the performance argument stands here, if this would be handling live traffic then it would be a different story altogether. I honestly don't think performance is impacted at all by this patch.

That makes sense, and I agree this is not really a single-request latency concern. I’m more worried about the cost / request-volume trade-off of moving from one remote fetch plus local ReadAt to one remote ranged read per batch request, especially for very large batches. I’m okay not treating that as a blocker here, but I think it would be good to track and validate in a follow-up issue.

github-actions · 2026-08-01T01:31:00Z

This PR is marked as stale after 21d of inactivity. After an additional 14d of inactivity (7d to become rotten, then 7d more), it will be closed. To prevent this PR from being closed, add a comment or remove the lifecycle/stale label.

acardace requested review from j-mok-dev, lioraron, vishbhat and yizhaodev as code owners July 6, 2026 15:25

acardace force-pushed the feat/ranged-gets branch from 9ac5299 to 5cde401 Compare July 6, 2026 15:54

yizhaodev reviewed Jul 6, 2026

View reviewed changes

wseaton reviewed Jul 7, 2026

View reviewed changes

lioraron reviewed Jul 7, 2026

View reviewed changes

Comment thread internal/processor/worker/executor.go

Comment thread internal/files_store/retryclient/client.go

Comment thread internal/files_store/fs/client.go

Comment thread internal/processor/worker/executor.go Outdated

Comment thread internal/files_store/tracing/tracing.go Outdated

lioraron reviewed Jul 7, 2026

View reviewed changes

acardace force-pushed the feat/ranged-gets branch 2 times, most recently from a9a32e4 to 398f1d0 Compare July 8, 2026 10:11

acardace changed the title ~~worker: replace local input file copy with S3 ranged gets~~ enh: replace local input file copy with S3 ranged gets Jul 8, 2026

github-actions Bot added the enhancement New user-facing capability label Jul 8, 2026

acardace force-pushed the feat/ranged-gets branch 2 times, most recently from 08d95f6 to b917fc6 Compare July 9, 2026 07:38

acardace force-pushed the feat/ranged-gets branch from b917fc6 to 7e41e0b Compare July 9, 2026 15:29

acardace added 3 commits July 9, 2026 17:38

acardace force-pushed the feat/ranged-gets branch from 7e41e0b to b01fe15 Compare July 9, 2026 15:38

j-mok-dev reviewed Jul 9, 2026

View reviewed changes

github-actions Bot added the lifecycle/stale label Aug 1, 2026

github-actions Bot added lifecycle/rotten and removed lifecycle/stale labels Aug 9, 2026

Uh oh!

Conversation

acardace commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Interface & implementations

Preprocessor

Executor

Removed

What stays on local disk

Testing

Uh oh!

github-actions Bot commented Jul 6, 2026

Uh oh!

acardace commented Jul 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lioraron commented Jul 6, 2026

Uh oh!

acardace commented Jul 7, 2026

Uh oh!

wseaton commented Jul 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lioraron commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wseaton commented Jul 7, 2026

Uh oh!

lioraron commented Jul 7, 2026

Uh oh!

acardace commented Jul 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lioraron left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

acardace commented Jul 9, 2026

Uh oh!

j-mok-dev commented Jul 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

acardace Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

acardace commented Jul 10, 2026

Uh oh!

acardace commented Jul 6, 2026 •

edited

Loading

lioraron commented Jul 7, 2026 •

edited

Loading

acardace Jul 10, 2026 •

edited

Loading