Add other backends (HTTP/WebDAV, Globus, S3/blob) to internal Pelican by bbockelm · Pull Request #3234 · PelicanPlatform/pelican

bbockelm · 2026-03-13T16:30:14Z

This is an initial draft of moving other backends to be embedded inside the Pelican process. Quite a bit left to go for code review and testing - but I thought it'd be good to open for other eyeballs ASAP.

@whwjiang - thought you'd have fun playing with this one.

turetske

Please deal with the merge conflicts as well.

turetske · 2026-05-13T18:48:57Z

+	// If per-export S3 credentials were provided, set them in the environment
+	// so the gocloud AWS credential chain picks them up.
+	//
+	// FIXME(deploy): this mutates global process environment, which has two


Is there an existing issue for this FIXME? Otherwise please create one as I worry this will just get lost.

turetske · 2026-05-14T20:23:00Z


-	if server_structs.OriginStorageType(param.Origin_StorageType.GetString()) != server_structs.OriginStorageGlobus {
-		return errors.Errorf("failed to initialize Globus backend: Origin.StorageType is not Globus: %s",
+	ost := server_structs.OriginStorageType(param.Origin_StorageType.GetString())


Small nitpick, but xrootdGid doesn't apply if using OriginStorageGlobusv2 and should be behind a conditional, I think.

turetske · 2026-05-14T20:31:18Z

+	oldKey := blobKey(oldName)
+	newKey := blobKey(newName)
+
+	if err := fs.bucket.Copy(ctx, newKey, oldKey, nil); err != nil {


This isn't a recursive copy. I believe this will rename oldKey to newKey, but any child objects in the collection will be under oldKey. Which is obviously bad. If this will ever be used on a collection rather than an individual "leaf" object, it needs to iterate recursively through and do a copy/delete.

turetske · 2026-05-14T20:33:26Z

 	// Handle POSIXv2 and SSH-specific initialization now that the web server is running
 	storageType := param.Origin_StorageType.GetString()
-	useXRootD := storageType != string(server_structs.OriginStoragePosixv2) && storageType != string(server_structs.OriginStorageSSH)
+	useXRootD := storageType != string(server_structs.OriginStoragePosixv2) &&


This is an identical repeated code segment to lines 59-63. Should useXRootD become part of a helper function to avoid repeated code?

turetske · 2026-05-14T20:35:16Z


+// globusIssuerURL returns the configurable Globus OIDC issuer URL.
+func globusIssuerURL() string {
+	if v := param.Origin_GlobusIssuerURL.GetString(); v != "" {


This is set in defaults.yaml, is it possible for it to be an empty string? Unless someone sets it explicitly, which seems odd.

turetske · 2026-05-14T20:35:32Z

+
+// globusTransferAPIBaseURL returns the configurable Globus Transfer API base URL.
+func globusTransferAPIBaseURL() string {
+	if v := param.Origin_GlobusTransferAPIBaseUrl.GetString(); v != "" {


Same comment as above.

turetske · 2026-05-14T20:40:12Z

 	github.com/youmark/pkcs8 v0.0.0-20201027041543-1326539a0a0a
 	github.com/zsais/go-gin-prometheus v0.1.0
 	go.uber.org/atomic v1.11.0
+	gocloud.dev v0.45.0


So, I think this is pretty large. Since we are splitting the client/server binaries (and that's merged into main already, but may not be part of this PR) and this should only be in the server binaries, make sure this dependency is only pulled in there as the whole point of the split is to reduce the client binary size.

turetske · 2026-05-14T20:46:36Z

+	if _, peekErr := peekIter.Next(ctx); peekErr == nil {
+		// It is a directory — return a lazy dir handle (a fresh iterator
+		// will be created when Readdir is called).
+		return &blobDirFile{name: name, bucket: fs.bucket, prefix: dirPrefix}, nil


Should this store the context for the later Readdir call?

turetske · 2026-05-14T20:48:19Z

+
+func (fi *blobFileInfo) Name() string      { return fi.name }
+func (fi *blobFileInfo) Size() int64       { return fi.size }
+func (fi *blobFileInfo) Mode() os.FileMode { return 0444 }


Are we always read only? If so, this is fine, but I want to check.

…thz code

- Do not chown Globus files if we aren't accessing via XRootD - Use simpler detection of whether XRootD is in use. - Switch secret keys from being injected via environment to explicitly (and redacting, as needed) on open.

github-advanced-security AI found potential problems Mar 13, 2026

View reviewed changes

bbockelm linked an issue Mar 14, 2026 that may be closed by this pull request

Convert other backends to be embedded inside the Pelican process #3245

Open

bbockelm added enhancement New feature or request origin Issue relating to the origin component labels Mar 14, 2026

github-advanced-security AI found potential problems Mar 14, 2026

View reviewed changes

Comment thread origin_serve/backend_blob.go Fixed

bbockelm requested a review from whwjiang March 14, 2026 21:52

bbockelm force-pushed the native-s3-https-globus-backends branch 2 times, most recently from fd6656b to 188f86d Compare March 21, 2026 13:35

github-advanced-security AI found potential problems Mar 21, 2026

View reviewed changes

Comment thread origin_serve/handlers.go Fixed

Comment thread origin_serve/handlers.go Fixed

Comment thread origin_serve/handlers.go Fixed

Comment thread origin_serve/handlers.go Fixed

bbockelm force-pushed the native-s3-https-globus-backends branch from a05ebe9 to efff8da Compare March 21, 2026 16:45

bbockelm force-pushed the native-s3-https-globus-backends branch from efff8da to bd2536c Compare April 6, 2026 00:42

bbockelm requested review from turetske and removed request for whwjiang April 28, 2026 11:39

bbockelm force-pushed the native-s3-https-globus-backends branch 2 times, most recently from b4baff2 to dc87f72 Compare May 2, 2026 14:03

turetske requested changes May 14, 2026

View reviewed changes

bbockelm force-pushed the native-s3-https-globus-backends branch from dc87f72 to 330d1b0 Compare June 17, 2026 00:02

bbockelm added 10 commits June 18, 2026 07:52

Add other backends (HTTP/WebDAV, Globus, S3/blob) to internal Pelican

9e8d2e8

Round of review fixes

89795ab

Add a regression test ensuring path.Clean is always present in the au…

7647598

…thz code

Add extensive Globus and WebDAV E2E tests

e6f514e

Fix range read; minio on mac

8e3b057

Unify minio startup scripts

91452dc

Allow retry of concurrent MKCOL

32c5d13

Checkpoint of some self-code review

b2c5594

Bugfix handler to not buffer HTTP PUTs in-memory

6768e11

Update to reflect review comments:

2222182

- Do not chown Globus files if we aren't accessing via XRootD - Use simpler detection of whether XRootD is in use. - Switch secret keys from being injected via environment to explicitly (and redacting, as needed) on open.

bbockelm force-pushed the native-s3-https-globus-backends branch from 1532734 to 2222182 Compare June 18, 2026 13:02

Conversation

bbockelm commented Mar 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

turetske left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants