Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds new handle to object stores that support S3 API #107

Draft
wants to merge 97 commits into
base: develop
Choose a base branch
from
+3,101 −3
Draft

Adds new handle to object stores that support S3 API #107

Changes from 1 commit
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
c70f2ce
feat(S3): initial commit for AWS S3 support
mcakircali Jan 31, 2024
c182574
refactor(S3): removed UPtr and SPtr
mcakircali Feb 1, 2024
e92e14e
feat(S3)!: adds Config, Session, Client, Context
mcakircali Feb 6, 2024
5fbf932
feat(S3): add create,delete,list for bucket and put,list object
mcakircali Feb 7, 2024
4aaec56
feat(S3): adds delete,list object
mcakircali Feb 7, 2024
eda4bfd
fix(S3): throw on unkown S3 client type
mcakircali Feb 7, 2024
3fb4e4d
feat(S3): adds credentials to S3Config
mcakircali Feb 7, 2024
408d9c2
refactor(S3): throw instead of return boolean
mcakircali Feb 8, 2024
028a4ee
chore(S3): remove advanced attribute from AWS_S3 cmake option
mcakircali Feb 8, 2024
7423b5c
style(S3): fix todo style comment
mcakircali Feb 8, 2024
c3e93a2
refactor(S3): namedspace -> unnamed namespace
mcakircali Feb 8, 2024
1aab48f
refactor(S3): cleanup S3Session
mcakircali Feb 8, 2024
d775bff
fix(S3): add missing pragma once
mcakircali Feb 8, 2024
620fd18
refactor(S3): cleanup S3Handle
mcakircali Feb 8, 2024
de556ff
refactor(S3): more cleanup for S3Handle, S3Name
mcakircali Feb 8, 2024
ed4ed4b
fix(S3): aws error parsing
mcakircali Feb 8, 2024
7eaa9be
feat(S3): rework credentials
mcakircali Feb 8, 2024
545f8c6
Add some URI infrastructure, and tweak some tests
simondsmart Feb 8, 2024
e8bcf92
refactor(S3): rework S3Config
mcakircali Feb 9, 2024
c84f046
feat(S3): start S3Name and S3URIManager
mcakircali Feb 9, 2024
a7bdb15
Merge remote-tracking branch 'refs/remotes/origin/feature/backend-s3'…
mcakircali Feb 9, 2024
563679e
fix(S3): authority must be true
mcakircali Feb 9, 2024
2003a5c
feat(Endpoint): construct from a URI
mcakircali Feb 9, 2024
ff342d1
fix(S3): client missing error msg
mcakircali Feb 9, 2024
4fbcea1
feat(S3): S3Name: parse bucket/object, exists, size
mcakircali Feb 9, 2024
929c0f2
feat(S3): add bucket/object exists, object size
mcakircali Feb 9, 2024
611742a
refactor(S3): session cleanup
mcakircali Feb 9, 2024
894bbae
feat(S3): adds open and size to S3Handle
mcakircali Feb 9, 2024
6a95ecb
fix(S3): adapt s3handle test
mcakircali Feb 9, 2024
140d75d
refactor(S3): cleanup and copyright fix
mcakircali Feb 9, 2024
e63bd18
refactor(S3): client and config cleanup
mcakircali Feb 10, 2024
75bb909
feat(S3): adds putObject
mcakircali Feb 10, 2024
fc359d9
feat(S3): adds emptyBucket
mcakircali Feb 10, 2024
eb09b13
refactor(S3): cleanup S3Client and S3Handle
mcakircali Feb 11, 2024
a4a6835
feat(S3): structure for get object (WIP)
mcakircali Feb 11, 2024
3650136
feat(S3): add S3Name asString
mcakircali Feb 11, 2024
f08c0b2
fix(S3): S3Name ref member in S3Handle
mcakircali Feb 11, 2024
833f88c
feat(S3): adds S3Handle read and getObject
mcakircali Feb 11, 2024
ef48592
feat(S3): fix tests3client and add print to S3Client
mcakircali Feb 11, 2024
709e51b
feat(S3): rework s3 config
mcakircali Feb 11, 2024
6b9f2a9
feat(S3): adds range get object and S3Handle fixes (WIP)
mcakircali Feb 13, 2024
e19d671
chore(S3): issue with S3Handle saveInto a MemoryHandle
mcakircali Feb 13, 2024
a0cc783
refactor(S3): correction throw messages
mcakircali Feb 13, 2024
61c0f89
refactor(S3): small cleanup S3Handle and S3Client
mcakircali Feb 14, 2024
d3d3ebe
feat(S3): adds S3Bucket.
nicolau-manubens Feb 14, 2024
26bb157
Merge branch 'feature/backend-s3' of https://github.com/ecmwf/eckit i…
nicolau-manubens Feb 14, 2024
4cfdb36
fix(S3): bug fixes and test for S3Handle, S3Client, S3Name
mcakircali Feb 14, 2024
986ac03
Merge remote-tracking branch 'refs/remotes/origin/feature/backend-s3'…
mcakircali Feb 14, 2024
bd245ef
refactor(S3): remove comment
mcakircali Feb 14, 2024
0585ad7
feat(S3): adds S3Bucket::ensureCreated and ensureDestroyed.
nicolau-manubens Feb 14, 2024
9b9ef61
fix(S3): solve merge conflict.
nicolau-manubens Feb 14, 2024
ce3b2c7
refactor(S3): cleanup test s3handle
mcakircali Feb 15, 2024
e7abb31
Merge remote-tracking branch 'refs/remotes/origin/feature/backend-s3'…
mcakircali Feb 15, 2024
5670806
fix(S3)!: client return type long long
mcakircali Feb 15, 2024
077e64a
fix(S3): assert fix and tidy up S3Handle, fix S3Name Length type
mcakircali Feb 15, 2024
b4cc9de
feat(S3): improve s3handle tests
mcakircali Feb 15, 2024
bec47da
feat(S3): adds a few S3Name features and minor API modifications.
nicolau-manubens Feb 16, 2024
6d01bf8
fix(S3): removed unnecessary stuff from S3Handle
mcakircali Feb 16, 2024
34b8f2c
fix(S3): write throws if object exists
mcakircali Feb 16, 2024
64c15a0
feat(S3): added test for no bucket s3handle
mcakircali Feb 16, 2024
2ffe322
fix(S3): commented expensive code in openForWrite
mcakircali Feb 16, 2024
e17e438
refactor(S3): removed eckit namespaces
mcakircali Feb 16, 2024
ff3342e
refactor(S3)!: S3BucketName
mcakircali Feb 16, 2024
13a3d01
refactor(S3)!: S3ClientAWS throws
mcakircali Feb 16, 2024
b4ae6a1
refactor(S3)!: S3Exception
mcakircali Feb 16, 2024
0c84e60
feat(S3)!: added S3ObjectName
mcakircali Feb 16, 2024
48797f6
fix(S3): memory leak S3Name
mcakircali Feb 17, 2024
f938234
feat(S3): added S3Name accessors
mcakircali Feb 17, 2024
7a47f99
refactor(S3): various log debug
mcakircali Feb 17, 2024
227db9f
fix(S3)!: S3Handle bug read/write return
mcakircali Feb 19, 2024
c07f52c
feat(S3): added S3Handle tests, S3Bucket/Object accessors and change …
mcakircali Feb 19, 2024
9cb443b
fix(S3): S3Client and IMDS bugs
mcakircali Feb 19, 2024
b4c8f7f
feat(S3): tests 1000 object writes
mcakircali Feb 19, 2024
b9e30d2
feat(S3): write performance tests
mcakircali Feb 20, 2024
a9f0cd1
feat(S3): improved perf 20% .. reuse S3Client
mcakircali Feb 20, 2024
17fd771
fix(S3): removed S3Context
mcakircali Feb 20, 2024
b481597
feat(S3): move uri to base
mcakircali Feb 21, 2024
f46d47c
fix(S3): remove debug log
mcakircali Feb 22, 2024
2e53988
chore(S3): removed unused macros file
mcakircali Feb 26, 2024
7d76c42
feat(S3): read credentials from user/default file
mcakircali Feb 27, 2024
650594d
fix(S3): missing headers
mcakircali Feb 28, 2024
a682dfa
fix(S3): more missing headers
mcakircali Feb 28, 2024
27e8f8a
fix(S3): turned off verifySSL
mcakircali Mar 4, 2024
c1a586c
Merge branch 'develop' into s3
mcakircali Dec 17, 2024
3647ce8
feat(S3): add config and credentials from file
mcakircali Dec 17, 2024
3adb373
fix(S3): uri bug
mcakircali Dec 18, 2024
14f3579
feat(S3): added S3ObjectPath
mcakircali Dec 19, 2024
701fe9e
fix(S3): bucket name parse
mcakircali Dec 20, 2024
3c29ecf
feat(S3): added make to S3Config
mcakircali Dec 20, 2024
042667c
fix(StringContent): compareString problem
mcakircali Dec 20, 2024
9771afa
feat(S3): made strongly typed
mcakircali Dec 24, 2024
bf630c7
Merge branch 'develop' into feature/backend-s3
mcakircali Dec 24, 2024
2cc56af
test(S3): improved
mcakircali Dec 25, 2024
fcc5f37
test(S3): improved common config
mcakircali Dec 25, 2024
07b936b
fix(S3): asString output and test config
mcakircali Dec 25, 2024
25d98c7
test(S3): removed generated header [skip ci]
mcakircali Dec 25, 2024
50e3a25
feat(S3): added compare operator
mcakircali Dec 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chore(S3): remove advanced attribute from AWS_S3 cmake option
@mcakircali
mcakircali committed Feb 8, 2024
commit 028a4eeafce3e68fd3e72f0fe57649ba2e352dca
1 change: 0 additions & 1 deletion CMakeLists.txt
View file
Original file line number Diff line number Diff line change
@@ -107,7 +107,6 @@ ecbuild_add_option( FEATURE RADOS

ecbuild_add_option( FEATURE AWS_S3
DEFAULT OFF
ADVANCED
REQUIRED_PACKAGES "AWSSDK REQUIRED COMPONENTS s3"
DESCRIPTION "Enables AWS S3 support" )