AWS Network ELB take 4 minutes to recognise target as healthy

Question

Using AWS Network ELBs: takes at least four minutes for a registered instance to become 'healthy'. The instances and services have been running for days, I am simply de-registering and then registering on the same target group, as part of a deployment. If I use a script or use the AWS UI, CLI it makes no difference.

Healthcheck Settings are:

• Port: tried various, all have listening services tested via curl. 80,22,9001
• Healthy threshold: 2
• Unhealthy threshold: 2
• Timeout: 10
• Interval: 30

I can see the connection requests coming in on whatever port has been specified, the service responds appropriately and the connection is then closed. As far as I am aware this should be sufficient for the ELB to determine that the instance is healthy (once the threshold has been passed). Which should mean that my instances are up and running no more than 90 seconds past registration time. I have no idea why this should be happening, should be straight forward.

I cannot determine what would be causing such a long delay given that I have fulfilled the known criteria for my instances being healthy. They sit at the Elb.InitialHealthChecking reason for about 4 minutes. Any ideas on further tests to determine the cause of the delay?

Answer 1

We hit this issue with NLBs and raised it with AWS support on March 20, 2018. Their response:

This is a known issue, where a newly registered instance remains in initial state for longer period of time and our internal team is already working on the fix for this issue. Unfortunately, at this point in time we do not have an ETA for the fix.

They confirmed that, under normal circumstances, targets should remain in the initial state until HealthyThreshold health checks pass.