Public presigned Genome URL yields "Error loading genome: Access forbidden"

[brainstorm]

Tested on IGV 2.19.2. Thanks @minw2828 for reporting.

To reproduce the error:

1. Create a presigned URL from a .fasta, hosted on an AWS S3 bucket.
2. DO NOT login via the Amazon menu in IGV.
3. Since it's a publicly accessible resource, the genome URL should download a .fasta on a regular (incognito mode) browser or via cURL.
4. Using the Genomes->Load Genome from URL, paste the presigned URL in the form dialog.
5. "Error loading genome: Access forbidden".

Expected outcome: The (custom) genome should be downloaded and loaded into IGV successfully. There's no redirects, just a straight 200 OK from a cURL request to the presigned S3 url.

I suspect that the presigned URL goes through some logic that requires AWS SDK to load the genome (and therefore authentication-checking is executed)... it shouldn't require auth since it should be treated as any other publicly accessible HTTP resource?

I'll investigate this one, @jrobinso, unless you are already familiar with it and you think it's an easy/quick fix you can address directly?

[jrobinso]

@brainstorm You are probably correct in your diagnosis. I will look at it when I get time. Why would a presigned URL be necessary for a public resource?

[brainstorm]

Why would a presigned URL be necessary for a public resource?

Good question! Not strictly necessary, but say you want to quickly share a custom genome on AWS and share it with peers?

Sure, fronting it with CloudFront and with a stable domain name would be "better" next step but also more complex (unnecessary?) setup.

[jrobinso]

You don't need cloudfront, all of our hosted data is on S3, we just use the provided https urls, and they are stable. Egress charges can become an issue, but I don't think a signed url helps with that.

It's still a bug, which I will fix, I was just curious about the use case.

[jrobinso]

The issue here is actually caused by inferring the name of the fasta index. We don't have a dialog that allows you to enter the presigned url to the index. The index URL is assumed to be == the fasta url + ".fai". The added ".fai" corrupts the signature.

You can work around this by creating a json genome definition

The forbidden error is coming from the presumed index, which is fasta url + ".fai". In this case adding the ".fai" invalidates the signature, thus the cause of the error. This has nothing to do with amazon per se

I can work around this by creating a json genome configuration and loading that instead of the fasta directly. The json looks like this

{
"id": "test",
"name": "test",
"fastaURL": "https://igv-genepattern-org.s3.us-east-1.amazonaws.com/test/fasta/chr22.fa?response-content-disposition=inline&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Security-...",
"indexURL": "https://igv-genepattern-org.s3.us-east-1.amazonaws.com/test/fasta/chr22.fa.fai?response-content-disposition=inline&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Security-..."
}

However, as noted previously if this is a public resource the signing is not needed, another workaround is to just use the url to the fasta, i.e. in this case

https://igv-genepattern-org.s3.us-east-1.amazonaws.com/test/fasta/chr22.fa

I'm going to close this as there are 2 workarounds for the issue as reported (Public presigned...), (1) create a json genome config, and (2) don't presign. There is a more general issue of not being able to use a fasta directly if the index url does not follow the pattern of "fasta url + .fai", but the first workaround listed above covers that case as well.