retry non-duplicate ec2 register_image errors

[asher]

No description provided.

[asher]

Aminator currently only retries register_image exceptions caused by a duplicate image name. We see regular throttling due to 429 rate limiting from amazon, so hoping allowing these to be retried during registration, as well as being more aggressive at retrying tag failures will cut down on full bake retries.

[marchello2000]

should this be

Suggested change

	if (tries > 0):
	if (_tries > 0):

instead?

[asher]

yep, thanks!

[bmoyles]

Still bummed that Amazon doesn't support tag creation at image registration time like it does for other objects (volumes, snapshots, etc) as it would be nice to be able to get rid of the tagging step altogether and just do it in-line with registration.

@asher one thing I might offer up -- we could make failing on tag failure optional... We'll still have some identifying metadata embedded in the AMI name and the AMI description fields which gives the app, build number, the commit, the base AMI name, ID, and version. We would potentially lose one (or all) of:

• base_ami_version (redundant, in ancestor_version in description)
• creator (which is not always accurate, anyways, depending on where, when, and how the bake was triggered)
• creation_time (which, given creationDate exists on the AMI, is redundant -- our creation_time tag reflects the time at which tags were added, creationDate reflects registration time if I'm not mistaken, and looking at some recent bakes, they differ by seconds at most)
• appversion -- this might be more annoying to miss as I know there are some internal tools that use it. All of the information in appversion is also available from description EXCEPT the build job name

Given we have enough metadata in the name and description, seems like we could make tag failure optionally fatal and worst case, run something out-of-process later on that adds tags back to the image if they're missing...

[bmoyles]

Are the current crop of retry-able errors identifiable by either a specific exception or message in the exception?

What this effectively does is add an alternative path for ClientError exceptions wherein exception['Error']['Code'] does not match InvalidAMIName.Duplicate that catches all other forms, so if you're just looking to retry all exceptions from ClientError I might consider getting rid of the specific foo around checking for InvalidAMIName.Duplicate in the error code and just always retry ClientError exceptions (which is basically what your else: block does).

[bmoyles]

The bigger deal is if the exceptions thrown during registration are of some other type than ClientError...