Getting autoScaling information fails fast after rate limit exception

[claudiumacovei]

Actual behavior

The shredder start up service fails fast, when unable to fetch information about the instance auto scaling group, due to Rate Limit exception.

Expected Behavior

The shredder should be able to retry this requests, for a finite amount, in order to avoid leaving instances hanging in auto scaling group

Steps to reproduce

Ramp up a decent number of instances at the same time, chances to hit a Rate Limit exception goes higher directly with the number of instances ramped up. Usually when trying to ramp up 40 instances, we get around 3-4 instances getting stuck.

StackTrace:

2023-07-27 08:01:23.032 [WrapperSimpleAppMain] ERROR c.a.a.s.c.a.s.AwsAutoScaleGroupHelper - Unable to fetch current AutoScaleGroup for instance: i-070c5880e7ccd1fc4 {}
com.amazonaws.services.autoscaling.model.AmazonAutoScalingException: Rate exceeded (Service: AmazonAutoScaling; Status Code: 400; Error Code: Throttling; Request ID: 0ed76c14-1f08-45a6-a699-fb587016e020)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
	at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.doInvoke(AmazonAutoScalingClient.java:3806)
	at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:3782)
	at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.executeDescribeAutoScalingInstances(AmazonAutoScalingClient.java:1545)
	at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.describeAutoScalingInstances(AmazonAutoScalingClient.java:1518)
	at com.adobe.aam.shredder.core.aws.servergroup.AwsAutoScaleGroupHelper.getCurrentAutoScalingGroup(AwsAutoScaleGroupHelper.java:40)
	at com.adobe.aam.shredder.ec2.aws.InstanceDetailsRetriever.getInstanceDetails(InstanceDetailsRetriever.java:41)
	at com.adobe.aam.shredder.ec2.notifier.Notifier.notifyAutoScaleGroup(Notifier.java:36)
	at com.adobe.aam.shredder.ec2.service.StartupService.runStartup(StartupService.java:66)
	at com.adobe.aam.shredder.ec2.service.StartupService.getStartupResult(StartupService.java:49)
	at com.adobe.aam.shredder.ec2.Main.main(Main.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:325)
	at java.lang.Thread.run(Thread.java:748)
2023-07-27 08:01:23.032 [WrapperSimpleAppMain] WARN  c.a.a.s.e.a.InstanceDetailsRetriever - Unable to retrieve the Auto Scale Group for the current instance i-070c5880e7ccd1fc4.

Checklist

• Add a descriptive title
• Include explicit steps on how to reproduce the issue
• Link to a repository that reproduces the issue (optional, but greatly appreciated)

Tips

• If possible, create a small repo that reproduces the issue and provide a link
• Don't include full logging output -- link to a gist of the full logging output instead
• Don't paste screenshots of error messages -- copy and paste instead