publish switch to S3 uploading unchanged files starting in version 1.6

[msqr]

I am using aptly on macOS, installed via Homebrew. I have several S3 hosted repositories where I publish updates to like this:

aptly publish switch ... bookworm s3:${S3_BUCKET}: ${APTLY_SNAPSHOT_NAME}

Up through version 1.5.0 this worked as expected, and only new packages added to the repo were actually uploaded to S3, so the process was very quick.

Starting in version 1.6.1 (I didn't try in version 1.6.0 sorry) when I run the same aptly publish switch command, all files in the repository are uploaded to S3, even if they are not changed. I have let the command complete, and tried again several times thinking maybe the next times around it would not re-upload all files, but it still did.

I then reverted back to version 1.5.0, and aptly publish switch went back to working as expected: only newly added packages are uploaded to S3.

I noticed the 1.6 release notes mentioned changes to the S3 component of aptly so wondered if something broke with the switch command with those changes?

[toofewtooextra]

same bug present when publishing to local filesystem with symlinks.
500: Unable to update: unable to process packages: symlink ... file exists

[neolynx]

Thanks for reporting !

the publish switch has been updated, and there are new commands for managing publish endpoints and components. Maybe @cfiehe can help here ?

[cfiehe]

I was not able to find a direct relationship between the component management, added some time ago, and the publishing issue described above.

@msqr Could you please try and test if you get the same behaviour, when you use the newly introduced CLI commands for component management. I think so, because both commands make use of the same lower-level function:

aptly publish source update -component=<component> <published-repo> <snapshot/repo>

Could you please give us the full commands, starting from repo creation over snapshot switching that result in the issue?

Unfortunately, 1.5.0 is pretty old, and a lot of changes were done between 1.5.0 and 1.6.0. What comes to my mind is that the package publication may has some trouble with the formerly introduced multi-distribution feature. The multi-distribution feature adds an additional folder to the pool hierarchy. My guess is that the source of the problem may be located in this function, but it is just a guess:

aptly/deb/publish.go

Line 815 in ab18da3

func (p *PublishedRepo) Publish(packagePool aptly.PackagePool, publishedStorageProvider aptly.PublishedStorageProvider,

[cfiehe]

I am not familiar with the S3 implementation, but a package upload should be done here, if I got that right:

aptly/s3/public.go

Line 333 in ab18da3

func (storage *PublishedStorage) LinkFromPool(publishedPrefix, publishedRelPath, fileName string, sourcePool aptly.PackagePool,

This should happen, when the file does not exist or when they have different checksums. Right?

[cfiehe]

@neolynx
I cannot test the S3 implementation, but it seems to me that this line does not look right, and it was changed between 1.5.0 and 1.6.0:

aptly/s3/public.go

Line 341 in ab18da3

paths, md5s, err := storage.internalFilelist(filepath.Join(storage.prefix, publishedPrefix, "pool"), true)

The storage.prefix is put in front of the string and gets put in front again in the method internalFilelist:

aptly/s3/public.go

Line 407 in ab18da3

prefix = filepath.Join(storage.prefix, prefix)

When storage.prefix is not empty, the result would be that the lookup string contains storage.prefix twice. If that is the case, that would be an explanation why the code cannot detect that a package already exists in the S3 storage and gets uploaded again. Maybe the issue is gone, when the argument given to internalFilelist is not prefixed with storage.prefix.

What do you think?

[cfiehe]

I setup an S3 and I was able to reproduce the issue. I found a problem in path comparision. I have created a pull request that should fix the issue.

[msqr]

@cfiehe Thank you for digging in and finding the issue! I really appreciate it. 🙏

[neolynx]

@msqr Could you verify if the problem is solved with the test artifacts from https://github.com/aptly-dev/aptly/actions/runs/14571537492 ?

• aptly_1.6.1+20250421100150.276225a3_darwin_arm64 (Apple Silicon)
• aptly_1.6.1+20250421100150.276225a3_darwin_amd64 (Intel)

You can just use the binary inside the zip file.

Thanks for testing and letting us know !

[neolynx]

@toofewtooextra there should be also a test build for your environment in https://github.com/aptly-dev/aptly/actions/runs/14571537492

Thanks for testing and letting us know if that change fixes the problem !

[msqr]

@msqr Could you verify if the problem is solved with the test artifacts from https://github.com/aptly-dev/aptly/actions/runs/14571537492 ?

I tested out the build and can confirm it is working as expected for me: only changed packages are published once again when I run switch.

🎉

Thank you for fixing the issue!