"AWS was not able to validate the provided access credentials" when they work fine (0.3.7)

[ecweaver]

I upgraded to 0.3.7 in order to put instances in eu-central-1 and now I'm getting this. The credentials are working fine in direct runs of aws ec2. Ran with TF_LOG=1 and just get the same error message prefaced with "Error walking (resource_id)".

This on MacOS 10.9.5 Mavericks.

[catsby]

Hey @ecweaver ; how are you providing credentials? Environment variables, hard-coded into your *.tf file(s), etc ?

[ecweaver]

I was originally having them in the usual environment variables. As a test, I passed them in as --var args and explicitly passed them in the provider block, same result.

[catsby]

I downloaded 0.3.7 and was not able to reproduce this.
Here's the .tf file I used

provider "aws" {
  region="us-west-2"
}
resource "aws_instance" "web" {
    ami = "ami-21f78e11"
    instance_type = "t1.micro"
    tags {
        Name = "HelloWorld"
    }
}

and the invocation:

$ terraform version
Terraform v0.3.7

$ AWS_ACCESS_KEY_ID=****** AWS_SECRET_ACCESS_KEY=****** terraform plan -out=create.tfplan
Refreshing Terraform state prior to plan...


The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed.

Your plan was also saved to the path below. Call the "apply" subcommand
with this plan file and Terraform will exactly execute this execution
plan.

Path: create.tfplan

+ aws_instance.web
    ami:                 "" => "ami-21f78e11"
    availability_zone:   "" => "<computed>"
    block_device.#:      "" => "<computed>"
    instance_type:       "" => "t1.micro"
    key_name:            "" => "<computed>"
    private_dns:         "" => "<computed>"
    private_ip:          "" => "<computed>"
    public_dns:          "" => "<computed>"
    public_ip:           "" => "<computed>"
    root_block_device.#: "" => "<computed>"
    security_groups.#:   "" => "<computed>"
    subnet_id:           "" => "<computed>"
    tags.#:              "" => "1"
    tags.Name:           "" => "HelloWorld"
    tenancy:             "" => "<computed>"


$ AWS_ACCESS_KEY_ID=****** AWS_SECRET_ACCESS_KEY=****** terraform apply create.tfplan
aws_instance.web: Creating...
  ami:                 "" => "ami-21f78e11"
  availability_zone:   "" => "<computed>"
  block_device.#:      "" => "<computed>"
  instance_type:       "" => "t1.micro"
  key_name:            "" => "<computed>"
  private_dns:         "" => "<computed>"
  private_ip:          "" => "<computed>"
  public_dns:          "" => "<computed>"
  public_ip:           "" => "<computed>"
  root_block_device.#: "" => "<computed>"
  security_groups.#:   "" => "<computed>"
  subnet_id:           "" => "<computed>"
  tags.#:              "" => "1"
  tags.Name:           "" => "HelloWorld"
  tenancy:             "" => "<computed>"
aws_instance.web: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate
$ 

Can you reproduce your issue with those?

[ecweaver]

No, works with those.

Investigating one other issue...

[ecweaver]

Working to pare this down to a minimal reproducible example for you...

[Zeroto]

I'm getting the same problem. also on the eu-central-1 region. Maybe related to that?

[Zeroto]

I just checked with a different region and for eu-west-1 it works fine with the exact same plan. So the problem is most definitely the eu-central-1 region.

[catsby]

So the problem is most definitely the eu-central-1 region

This I have confirmed.. strange.

[ecweaver]

I just used this pared-down example and it fails in the mentioned fashion not only in eu-central-1 but also in us-west-2. Here is the file, which is called test.tf and lives in terraform_templates/test/ . The invoking command follows.

#  test deploy template
#
variable "ami" {} # prompted or supply on cmd line, e.g.:  --var ami=ami_01234567
variable "a_zone" {}
variable "b_zone" {}
variable "region" {}
variable "user" {}
variable "aaki" {}
variable "asak" {}

provider "aws" {
  region = "${var.region}"
  secret_key = "${var.aaki}"  # provide in AWS_SECRET_ACCESS_KEY env var and ignore prompt.
  access_key = "${var.asak}"  # provide in AWS_ACCESS_KEY_ID env var and ignore prompt.
}

resource "aws_instance" "test-a-zone" {
  count = 3
  ami = "${var.ami}"
  availability_zone = "${var.region}${var.a_zone}"
  instance_type = "t2.micro"
  tags = {
    Name = "test${count.index}.elk.${var.region}${var.a_zone}.aws.foobar.com"
    ShortName = "consul${count.index}"
    Owner = "joeblow"
    Hostgroup = "test"
    Environment = "Production"
    Creator = "${var.user}"
  }
}

resource "aws_instance" "test-b-zone" {
  count = 2
  ami = "${var.ami}"
  availability_zone = "${var.region}${var.b_zone}"
  instance_type = "t2.micro"
  tags = {
    Name = "consul${count.index}.elk.${var.region}${var.b_zone}.aws.foobar.com"
    ShortName = "consul{count.index}"
    Hostgroup = "consul"
    Owner = "joeblow"
    Hostgroup = "test"
    Environment = "Production"
    Creator = "${var.user}"
  }
}

(removed invocation by @catsby)

[catsby]

@ecweaver if those are valid AWS key/secrets I strongly encourage you to rotate them immediately.

[pmoust]

@ecweaver make sure you deactivate the keys you 've just shared.

[catsby]

For what it's worth, the example tf file I shared above is enough to reproduce this with eu-central-1

[pmoust]

@catsby signature version for said region is V4 as opposed to the 'old' regions.