S3ResponseError: 400 Bad Request

[djandruczyk]

amiimporter.py --vboxfile /tmp/CentOS7-20161123_0.box --s3bucket my-amis --region=us-east-2 --verbose
INFO:root:Uploading ./tmpdir/box-disk1.vmdk to s3
Traceback (most recent call last):
File "/usr/local/bin/amiimporter.py", line 214, in
main(make_opt_parser().parse_args(sys.argv[1:]))
File "/usr/local/bin/amiimporter.py", line 208, in main
upload_vmdk_to_s3(opts, vmdkfile)
File "/usr/local/bin/amiimporter.py", line 96, in upload_vmdk_to_s3
bucket_location = bucket.get_location()
File "/usr/lib/python2.7/dist-packages/boto/s3/bucket.py", line 1144, in get_location
response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request

InvalidRequestThe authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.0B2D268921129D32uIt4GoSguVmj+Lh9VPtuhwb73affiu/2OOoqW7+l06m3cZIWYABs7zjTd+n0KJa5CvuaNco0Fk4=

I have verified my environment variabels are properly exported for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY_ID, and it still gives me the same results..

Ubuntu 16.04 64 bit using python2.7 and python-boto-2.38.0

[dliappis]

@djandruczyk TL;DR I couldn't reproduce this (Fedora 24)

This issue is related to newer regions like us-east-2, eu-central-1 etc. that force the AWS4-HMAC-SHA256 signature, but there's already a workaround in the code for that: https://github.com/dliappis/amiimporter/blob/master/amiimporter.py#L95 (see also the boto issue).

For reference, I tried creating a bucket in us-east-2 (Ohio) as you have and the process worked fine for me.

Your boto version is a bit old; I don't recommend relying on boto python libraries shipped with the OS.
Instead, please install the needed libs with pip either locally or in a virtualenv i.e.:

• pip install -U --user aws awscli boto boto3 botocore (-U will ensure it's updated if already installed)
  This command will install packages (on Linux) under ~/.local/lib

OR

• Use a python virtual environment

  virtualenv awsve
  source awsve/bin/activate
  pip install --user aws awscli boto boto3 botocore

Make sure that you don't mix both approaches shown above. If in doubt just pip show boto to see the current path of the boto lib.

Here are the versions of my AWS related pip libs:

aws==0.2.5
awscli==1.11.15
boto==2.43.0
boto3==1.3.1
botocore==1.4.72

It would be nice to switch to boto3 in amiimporter.py as boto3 receives better support going forward but I don't currently have the bandwidth.

Should also mention that if you can afford to create your bucket in a different older region like us-east-1, us-west-2, eu-west-1 it should work without issues too, even with older boto libs.

[dliappis]

@djandruczyk I was able to reproduce your problem. It seems boto2 has fallen behind with the new signature methods required by S3 in newer regions.

I took the plunge and switched much of the code to use boto3; you can try the version I pushed in this branch https://github.com/dliappis/amiimporter/tree/switch-to-boto3 and see if it helps. Keep in mind that as it is now, it will copy the produced artifact in 4 regions, which you can adjust here: https://github.com/dliappis/amiimporter/blob/switch-to-boto3/amiimporter.py#L200

[djandruczyk]

Using the pip method it still fails...
I uninstall the OS provided boto libs (apt-get purge boto -y) and installed them via pip with
pip install -U --user aws awscli boto boto3 botocore

./amiimporter.py --vboxfile /tmp/CentOS7-20161123_0.box --s3bucket s3://amis-ohio --region=us-east-2 --verbose
INFO:root:Uploading ./tmpdir/box-disk1.vmdk to s3
Traceback (most recent call last):
File "./amiimporter.py", line 214, in
main(make_opt_parser().parse_args(sys.argv[1:]))
File "./amiimporter.py", line 208, in main
upload_vmdk_to_s3(opts, vmdkfile)
File "./amiimporter.py", line 93, in upload_vmdk_to_s3
AWS_SECRET_ACCESS_KEY_ID)
File "/home/dave/.local/lib/python2.7/site-packages/boto/init.py", line 141, in connect_s3
return S3Connection(aws_access_key_id, aws_secret_access_key, **kwargs)
File "/home/dave/.local/lib/python2.7/site-packages/boto/s3/connection.py", line 197, in init
"When using SigV4, you must specify a 'host' parameter."
boto.s3.connection.HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter.

I tried both to a bucket in virginia (US east-1) and ohio (us-east-2)

boto 2.43.0
aws 0.2.5
awscli 1.11.21
boto3 1.4.1
botocore 1.4.78

python --version
Python 2.7.12

Ubuntu 16.04.1 LTS 64 bit

[dliappis]

@djandruczyk Can you try the boto3 version: https://github.com/dliappis/amiimporter/tree/switch-to-boto3 ?

[oxr463]

Any updates on this?

[dliappis]

Any updates on this?

Haven't touched this for several years. Did you try the boto3 branch: https://github.com/dliappis/amiimporter/tree/switch-to-boto3 I mentioned earlier? (4yrs old ... not sure if it works!)