Problems uploading large files > 75GB

[jeffreykoch]

I'm trying to save 75GB and 120GB encrypted files to Glacier with the cmd-interface. The 75GB uploads work 90% of the time the 120GB uploads never work. We also have seem to sometimes have problems with smaller 6GB files.

Here's the output when one of the uploads fails. Perhaps someone can help explain what's going on. We have temporarily fixed the issue by splitting files into segments no larger than 50GB.

It would appear the issue is with AWS and Glacier providing an 'UnexpectedHTTPResponseError' and the cmd-interface cannot recover. Are there any work arounds? We're trying to automate a backup procedure. Thanks

Traceback (most recent call last):
File "/usr/local/bin/glacier-cmd", line 9, in
load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')()
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 929, in main
args.func(args)
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 156, in wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 309, in upload
args.name, args.partsize, args.uploadid, args.resume)
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 65, in wrapper
ret = fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 232, in glacier_connect_wrap
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 65, in wrapper
ret = fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 253, in sdb_connect_wrap
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 65, in wrapper
ret = fn(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 1157, in upload
writer.write(part)
File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glaciercorecalls.py", line 129, in write
data)
File "/usr/lib/python2.7/dist-packages/boto/glacier/layer1.py", line 1279, in upload_part
response_headers=response_headers)
File "/usr/lib/python2.7/dist-packages/boto/glacier/layer1.py", line 119, in make_request
raise UnexpectedHTTPResponseError(ok_responses, response)
boto.glacier.exceptions.UnexpectedHTTPResponseError: Expected 204, got (408, code=RequestTimeoutException, message=Request timed out.)

[gburca]

You could try my version (d04ab69) which has handling for these kinds of errors. I've successfully uploaded files over 200Gb and files that took days to upload. Timeout (408) errors are graciously handled and retried.

[jeffreykoch]

Gabriel - thanks - I'll take a look - Jeff

…

On 1/30/2017 12:26 PM, Gabriel Burca wrote: You could try my version (d04ab69 <d04ab69>) which has handling for these kinds of errors. I've successfully uploaded files over 200Gb and files that took days to upload. Timeout (408) errors are graciously handled and retried. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYPqFEo93GL3uOxMrcFZyRVdQT32hNCTks5rXh1HgaJpZM4Lw72_>.

[jeffreykoch]

HI Gabriel: I'm new at GITHUB - how do I download/clone just your version? and for the install should I do the same: cd amazon-glacier-cmd-interface/; python setup.py install If I understand things correctly your patch is not in the main code - any reason why not? Regards, Jeff Koch

…

On 1/30/2017 12:26 PM, Gabriel Burca wrote: You could try my version (d04ab69 <d04ab69>) which has handling for these kinds of errors. I've successfully uploaded files over 200Gb and files that took days to upload. Timeout (408) errors are graciously handled and retried. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYPqFEo93GL3uOxMrcFZyRVdQT32hNCTks5rXh1HgaJpZM4Lw72_>.

[gburca]

You can clone from https://github.com/gburca/amazon-glacier-cmd-interface.git. Other than that there should be no difference.

I don't know why most of the patches are not being merged into this project. @uskudnik can speak to that.

[jeffreykoch]

Hi Gabriel: How can I confirm that I'm using your version of the glacier-cmd-interface versus the uskudnik version? I had the uskudnik version previously installed on my server. Regards, Jeff

…

On 5/24/2017 8:09 PM, Gabriel Burca wrote: You can clone from https://github.com/gburca/amazon-glacier-cmd-interface.git. Other than that there should be no difference. I don't know why most of the patches are not being merged into this project. @uskudnik <https://github.com/uskudnik> can speak to that. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYPqFDIO1kn7Wvo47mSBVLqSGwKG_KGaks5r9MangaJpZM4Lw72_>.

[gburca]

I would expect the latest one to be installed is the one you're using. You can also specify the installation directory if you want. To see the options available: python setup.py install -h

[jeffreykoch]

Hi Gabriel: After a successful ten days of Glacier uploads using your branch we've started getting numerous failures. Here's the traceback. Please let me know if you have any suggestions on what the problem is and how to troubleshoot. Thanks, Jeff This uploaded started at Sat Jun 17 10:35:27 EDT 2017 and failed at Sat Jun 17 10:48:23 EDT 2017 - 13 minutes later. The file size is 32GB Traceback (most recent call last): File "/usr/local/bin/glacier-cmd", line 9, in <module> load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')() File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 935, in main args.func(args) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 157, in wrapper return fn(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 310, in upload args.name, args.partsize, args.uploadid, args.resume) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 66, in wrapper ret = fn(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 237, in glacier_connect_wrap return func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 66, in wrapper ret = fn(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 258, in sdb_connect_wrap return func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 66, in wrapper ret = fn(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 1177, in upload writer.write(part) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glaciercorecalls.py", line 143, in write if '408' in e.message or e.code == "ServiceUnavailableException" or e.type == "Server": AttributeError: 'UnexpectedHTTPResponseError' object has no attribute 'type'

…

On 6/3/2017 11:45 AM, Gabriel Burca wrote: I would expect the latest one to be installed is the one you're using. You can also specify the installation directory if you want. To see the options available: |python setup.py install -h| — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYPqFPN9Lv_EbjVY3dTESOjoIKPG8vbxks5sAX-pgaJpZM4Lw72_>.

[jeffreykoch]

Hi Gabriel: Not sure if you've seen my emails about the problems we're now having with Glacier uploads using your version of the glacier-cmd-interface? Uploads worked great for about ten days and are now crashing regularly with the following traceback conclusion: File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/GlacierWrapper.py", line 1177, in upload writer.write(part) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glaciercorecalls.py", line 143, in write if '408' in e.message or e.code == "ServiceUnavailableException" or e.type == "Server": AttributeError: 'UnexpectedHTTPResponseError' object has no attribute 'type' I really need to get this going. Is there anything I can do to troubleshoot? Thanks in advance for your help Regards, Jeff Koch

…

On 6/3/2017 11:45 AM, Gabriel Burca wrote: I would expect the latest one to be installed is the one you're using. You can also specify the installation directory if you want. To see the options available: |python setup.py install -h| — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYPqFPN9Lv_EbjVY3dTESOjoIKPG8vbxks5sAX-pgaJpZM4Lw72_>.

[gburca]

I've been out of town and haven't had a chance to look at this. I'll take a look in the next few days.

[gburca]

Try the master branch from https://github.com/gburca/amazon-glacier-cmd-interface

[jeffreykoch]

Hi Gabriel: Tried the first upload with your new branch. I believe it worked but it generated the following error message. Please let me know what to change to fix this. - Thanks, Jeff Traceback (most recent call last): File "/usr/local/bin/glacier-cmd", line 9, in <module> load_entry_point('glacier==0.2dev', 'console_scripts', 'glacier-cmd')() File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 1007, in main args.func(args) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 174, in wrapper return fn(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 364, in upload else output_headers(results[0], args.output) File "/usr/local/lib/python2.7/site-packages/glacier-0.2dev-py2.7.egg/glacier/glacier.py", line 45, in output_headers output)) ValueError: Output format must be ['csv', 'json'], got:print

…

On 6/20/2017 1:00 AM, Gabriel Burca wrote: Try the master branch from https://github.com/gburca/amazon-glacier-cmd-interface — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#188 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYPqFJGobtLHrm2ksCeHJbR8ShpIXRaHks5sF1IDgaJpZM4Lw72_>.

[gburca]

Sorry. Didn't get to test the changes. Try the master branch again. We should move this discussion to my fork of the project to reduce confusion since at this point the discussion is strictly about https://github.com/gburca/amazon-glacier-cmd-interface