How can I perform automatic scaling based on metrics like CPU usage and Network usage?
For example create a new instance if my CPU usage is above 70% on all the nodes, and destroy an instance if the CPU usages falls below 30%. These would have to be sustained usages of course, like all nodes running at more than 70% CPU Usage for more than 5 minutes.
I did some research into this but its not very clear. For example Digital Ocean has a monitoring service which can send you an email if the CPU Usage is abnormally high (or Networking Load) based on a threshold you set. The problem is they don't have WebHook support yet so I can't send these signals to my application.
One idea I had was to set a special email for these notifications and check the email. Based on the email response I then create a modified terraform file and 'terraform apply' it to modify the infrastructure?
I'm not sure if this is good form either since the point of terraform is code as infrastructure, but if the infrastructure is designed to create new instances and destroy instances all the time based on CPU Usage or Network Usage I'm not sure how you convey that concept correctly in terraform?
Can anyone help me with this?