Azure Storage blob versioning

[MariuszBartnik]

I have a Azure Storage account with blob versioning functionality enabled. I want to upsert file, by providing the same path for blob as the one, that already exists on the storage and create a new version of this file in this way. I was doing this without any problems on my previous setup, which was an Express app with multer streaming files to Azure, but now I want to migrate to using tusd instead.

I have noticed an issue with this approach though. When I try to download the updated file I am still getting the original one, instead of a new one, even if the upload was a success. I've notice that the issue is connected to this piece of code in the getOffset function in azureservice.go file.

getBlock, err := blockBlob.Blob.GetBlockList(ctx, azblob.BlockListAll, azblob.LeaseAccessConditions{})

I might be missing something, so I wanted to ask if we really need to get the block list for azblob.BlockListAll instead of azblob.BlockListUncommitted? With this setup function is returning blocks for both files - updated and original.

[Acconut]

The code of the azurestore has slightly changed since we migrated to a newer Azure SDK in #1205, but it seems you are referring to these lines:

tusd/pkg/azurestore/azureservice.go

Lines 196 to 210 in a50bc42

	resp, err := blockBlob.BlobClient.GetBlockList(ctx, blockblob.BlockListTypeAll, nil)
	if err != nil {
	return 0, checkForNotFoundError(err)
	}
	// Need committed blocks to be added to offset to know how big the file really is
	for _, block := range resp.CommittedBlocks {
	offset += *block.Size
	indexes = append(indexes, blockIDBase64ToInt(block.Name))
	}
	// Need to get the uncommitted blocks so that we can commit them
	for _, block := range resp.UncommittedBlocks {
	offset += *block.Size
	indexes = append(indexes, blockIDBase64ToInt(block.Name))
	}

The code after GetBlockList does take committed block explicitly into consideration, which seems to be the reason why BlockListAll is used here. However, my knowledge of Azure Blob Storage is unfortunately not deep enough to tell whether we can get rid this somehow. The best way to know is to just try it out. Feel free to give it a shot and let us know if it works. If it does, I am open to changing the logic to only consider uncommitted blocks.

Maybe @omBratteng can shed some light on this as well. As far as I know, he doesn't work with tusd anymore but he wrote the initial version of azurestore and might remember some details why BlockListAll is used.

[omBratteng]

I don't exactly remember.

@MariuszBartnik how are you downloading the blob?

[MariuszBartnik]

@omBratteng Downloading the blobs from Azure storage is currently implement without the use of tusd package. I am doing this in the NestJs server using Azure SDK for Typescript.

@Acconut I am currently working on the implementation with BlockListAll changed to BlockListUncommitted. So far it looks good, but I will spend some more time on testing

[Acconut]

@MariuszBartnik Just wanted to check in and ask how things have turned out? Did you manage to get tusd running in a way to replace the previous file?

[MariuszBartnik]

@Acconut Yes, I've been able to use tusd for uploading files in my project with the blob versioning enabled. As mentioned previously I had to change BlockListTypeAll to BlockListTypeUncommitted when getting a list of uploaded blocks. I have not noticed any issues with this approach. Everything seems to be working as expected

[Acconut]

That's great to hear! Do you it would be helpful to include this change in tusd as well so other users can benefit as well? Would you be willing to open a PR for that?