chunked: use temporary file for tar-split data #2312
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: giuseppe The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
giuseppe
force-pushed
the
chunked-read-tarsplit-sequentially
branch
2 times, most recently
from
cebc9ff to
13cbaef
Compare
Luap99
reviewed
Apr 15, 2025
giuseppe
force-pushed
the
chunked-read-tarsplit-sequentially
branch
2 times, most recently
from
94a8a23 to
b0fd638
Compare
Luap99
reviewed
Apr 15, 2025
giuseppe
force-pushed
the
chunked-read-tarsplit-sequentially
branch
2 times, most recently
from
4f0de4e to
6342175
Compare
cgwalters
reviewed
Apr 15, 2025
giuseppe
force-pushed
the
chunked-read-tarsplit-sequentially
branch
3 times, most recently
from
f04c803 to
187c56b
Compare
liamstask
reviewed
Apr 23, 2025
giuseppe
force-pushed
the
chunked-read-tarsplit-sequentially
branch
from
187c56b to
d70254a
Compare
Luap99
reviewed
Apr 28, 2025
kolyshkin
reviewed
Apr 28, 2025
kolyshkin
reviewed
Apr 28, 2025
extract the blob checksum validation logic from decodeAndValidateBlob into a separate function. Signed-off-by: Giuseppe Scrivano <[email protected]>
Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe
force-pushed
the
chunked-read-tarsplit-sequentially
branch
from
d70254a to
0084bc1
Compare
kolyshkin
reviewed
Apr 28, 2025
giuseppe
force-pushed
the
chunked-read-tarsplit-sequentially
branch
2 times, most recently
from
6472d74 to
dfd505c
Compare
Replace the in-memory buffer with a O_TMPFILE file. This reduces the memory requirements for a partial pull since the tar-split data can be written to disk. Signed-off-by: Giuseppe Scrivano <[email protected]>
Replace the direct call to unix.Open with the O_TMPFILE flag with the dedicated openTmpFile helper function. Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe
force-pushed
the
chunked-read-tarsplit-sequentially
branch
from
dfd505c to
729821c
Compare
|
/lgtm |
mtrmac
reviewed
Apr 29, 2025
| @@ -333,13 +339,16 @@ func makeConvertFromRawDiffer(store storage.Store, blobDigest digest.Digest, blo | |||
| // makeZstdChunkedDiffer sets up a chunkedDiffer for a zstd:chunked layer. | |||
| // It may return an error matching ErrFallbackToOrdinaryLayerDownload / errFallbackCanConvert. | |||
| func makeZstdChunkedDiffer(store storage.Store, blobSize int64, tocDigest digest.Digest, annotations map[string]string, iss ImageSourceSeekable, pullOptions pullOptions) (*chunkedDiffer, error) { | |||
| manifest, toc, tarSplit, tocOffset, err := readZstdChunkedManifest(iss, tocDigest, annotations) | |||
| manifest, toc, tarSplit, tocOffset, err := readZstdChunkedManifest(store.RunRoot(), iss, tocDigest, annotations) | |||
There was a problem hiding this comment.
There are several error code paths; tarSplit doesn’t get closed?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replace the in-memory buffer with a O_TMPFILE file. This reduces the memory requirements for a partial pull since the tar-split data can be written to disk.
c/image needs the following patch: