"key does not exist" after direct S3 upload

[kevinansfield]

I'm running into issues with direct S3 uploads where the application is trying to save the model after the upload is complete but before the file upload has fully propagated across S3.

Is there any way to specify the number of retries or a timeout period when trying to copy from cache to store here https://github.com/refile/refile-s3/blob/master/lib/refile/s3.rb#L56?

[pvande]

+1

We've been (sporadically) seeing this issue in our error tracker for months. Would love to have an official response.

[pvande]

@ivobenedito While your error certainly seems easier to replicate, it's worth noting that this is also occurring without presigning (e.g. by copying to store before the file has fully persisted to cache).

[ivobenedito]

@pvande You're only getting this errors when using multiple file uploads, correct?

I've replaced the data-url on the multiple file upload HTML form to submit directly to the rails app instead of going throw the asset host CDN and it all worked fine.

[pvande]

@ivobenedito Incorrect; I'm actually seeing this error manifest on trivial codepaths.

class Image < ActiveRecord::Base
  attachment :key
end

Image.create(remote_key_url: image_url)  # => Occasionally raises this error.

[ivobenedito]

@pvande Sorry, it's a different issue then.
Mine is related to refile/refile#292
I'm removing my previous posts to leave this discussion "cleaner".

[jnicklas]

This is a tricky issue. The Amazon docs indicate that reads after writes should be consistent immediately:

Amazon S3 provides read-after-write consistency for PUTS of new objects in your S3 bucket

See: http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html

But it's unclear if this is also true for copy and move. I haven't been able to find any documentation to either effect. I guess we could always read the object instead of copying, but that seems also bad from a performance perspective.

[pvande]

I feel like I can say with some confidence that their documentation is (at least in part) incorrect on that subject. Logging HTTP requests has yielded some data on the subject.

# Normal Workflow
# Note that the first PUT to `cache` failed, but was automatically retried by Refile
2015-09-19T20:09:54Z - GET https://i.ytimg.com/vi/BjbilpBaA-U/hqdefault.jpg status="200"
2015-09-19T20:10:35Z - PUT https://bucketname.s3.amazonaws.com/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709 status="400"
2015-09-19T20:10:35Z - PUT https://bucketname.s3.amazonaws.com/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709 status="200"
2015-09-19T20:10:35Z - GET https://bucketname.s3.amazonaws.com/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709 status="200"
2015-09-19T20:10:36Z - PUT https://bucketname.s3.amazonaws.com/store/2dc27f78ac20c4a617c9f31f532e54c960341fba status="200"
2015-09-19T20:10:36Z - DELETE https://bucketname.s3.amazonaws.com/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709 status="204"

# Error Case
2015-09-19T20:14:37Z - GET https://i.ytimg.com/vi/JADbdP5x4Zk/hqdefault.jpg status="200"
2015-09-19T20:14:39Z - PUT https://bucketname.s3.amazonaws.com/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709 status="200"
2015-09-19T20:14:39Z - GET https://bucketname.s3.amazonaws.com/cache/da39a3ee5e6b4b0d3255bfef95601890afd80709 status="404"
# OpenURI::HTTPError is raised at this point.

If there's any further data that it would be useful to capture, I'm happy to do so.

EDIT: This specific exception may have originated inside my own code. Having said that, it is the same class of problem that we've been witnessing.

[Menghongli]

Hey Guys, I'm also running into this issue, is there any solutions to this issue now?

[brianlittmann]

Can confirm this crops up occasionally on direct uploads for single files as well, most recently with a 20 KB image.

[JoeWoodward]

Any updates on this issue?

[dcpomfret]

Also, requiring updates on this issue

[hale]

This issue might crop up elsewhere:

I'm seeing occasional NoMethodError: undefined method '>' for nil:NilClass errors thrown in BackendMacros#verify_uploadable from Attacher#get (given a cache_id).

Given that Refile::S3#get "will always return a {Refile::File} object, even if a file with the given id does not exist in this backend" [ref], this error makes sense if the object has not yet propagated.

That being said, AWS assert read-after-write consistency for all regions and all endpoints now. Most recently: https://forums.aws.amazon.com/thread.jspa?messageID=690666&#690666.

I'll continuing to look in the app code to see if there's something wrong there (e.g. maybe the file was somehow deleted from the cache) - in the meantime flagging in in case it helps anyone else with the same error.

[achung-miovision]

Any updates on this issue? I can consistently reproduce this error, but it is not related to S3 consistency.

In my model, if I have the following:

has_many :uploads, inverse_of: :upload_set accepts_attachments_for :uploads, append: true

In my simple form, I have the following:
= f.input :uploads do =f.attachment_field :uploads_files, multiple: true, direct: true

I get "the specified key does not exist" exception. The file is clearly sitting there in the S3 store, but Refile is still trying to copy an invalid key that doesn't exist. If I remove the append: true from my form, everything works as expected. This is ONLY for NEW upload_set records. If I'm editing an existing upload_set, the append: true works as expected.

[kevinansfield]

Closing to clean up my own issues. Can be re-opened if this is still an active problem.