DODWELL.US DevOPS - For fun and profit

how to use asg

8 min read aws configuration management

The following explains how I utilize Auto Scaling Group’s (ASG’s) and puppet to automatically expand clusters during high load.

asg

Auto Scaling allows you to scale your Amazon EC2 capacity up or down automatically according to conditions you define. With Auto Scaling, you can ensure that the number of Amazon EC2 instances you’re using increases seamlessly during demand spikes to maintain performance, and decreases automatically during demand lulls to minimize costs. Auto Scaling is particularly well suited for applications that experience hourly, daily, or weekly variability in usage. Auto Scaling is enabled by Amazon CloudWatch and available at no additional charge beyond Amazon CloudWatch fees.

creation of asg

The following script was used to create the RabbitMQ ASG:

#!/bin/sh
ASG_GROUP="web-asg"
LAUNCH_GROUP="web-l"
EC2_AMI="ami-09f26b33"
EC2_USERDATA="user-data-file.txt"
EC2_SECURITYGROUPS="sg-ffffffff,sg-fffffffe"
EC2_LB="web"
EC2_MIN="2"
EC2_MAX="10"
EC2_TYPE="t1.micro"
echo -n "Checking if group already exists.."
if [ "`as-describe-auto-scaling-groups | grep ${ASG_GROUP}`" ]; then
  echo "exists"
  as-delete-auto-scaling-group --force-delete --auto-scaling-group ${ASG_GROUP}
  echo ""
  printf "Waiting until Amazon actually removes group |"
  rotate='|/-\'
  while [ "`as-describe-auto-scaling-groups | grep ${ASG_GROUP}`" ]; do
    rotate="${rotate#?}${rotate%???}"
    printf '\b%.1s' "$rotate"
  done
  echo
else
  echo "none"
fi
echo -n "Checking if launch config already exists.."
if [ "`as-describe-launch-configs | grep ${LAUNCH_GROUP}`" ]; then
  echo "yes"
  as-delete-launch-config --force --launch-config ${LAUNCH_GROUP}
  echo ""
  printf "Waiting until Amazon actually removes config |"
  rotate='|/-\'
  while [ "`as-describe-launch-configs | grep ${LAUNCH_GROUP}`" ]; do
    rotate="${rotate#?}${rotate%???}"
    printf '\b%.1s' "$rotate"
  done
  echo
else
  echo "nope"
fi
as-create-launch-config ${LAUNCH_GROUP} --image-id ${EC2_AMI} --instance-type ${EC2_TYPE} --monitoring-enabled --key root --group ${EC2_SECURITYGROUPS} --user-data-file ${EC2_USERDATA}
as-create-auto-scaling-group ${ASG_GROUP} --launch-configuration ${LAUNCH_GROUP} --min-size ${EC2_MIN} --max-size ${EC2_MAX} --load-balancers ${EC2_LB} --desired-capacity ${EC2_MIN}
mon-put-metric-alarm ${ASG_GROUP}-add \
  --alarm-actions "`as-put-scaling-policy ${ASG_GROUP}-add --type ChangeInCapacity --adjustment 1 --auto-scaling-group ${ASG_GROUP}`" \
  --comparison-operator GreaterThanThreshold \
  --dimensions "AutoScalingGroupName=${ASG_GROUP}" \
  --evaluation-periods 3 \
  --metric-name CPUUtilization \
  --namespace "AWS/EC2" \
  --period 60 \
  --statistic Average \
  --threshold 90 \
  --unit Percent
mon-put-metric-alarm ${ASG_GROUP}-del \
  --alarm-actions "`as-put-scaling-policy ${ASG_GROUP}-del --type ChangeInCapacity --adjustment='-1' --auto-scaling-group ${ASG_GROUP}`" \
  --comparison-operator LessThanThreshold \
  --dimensions "AutoScalingGroupName=${ASG_GROUP}" \
  --evaluation-periods 10 \
  --metric-name CPUUtilization \
  --namespace "AWS/EC2" \
  --period 60 \
  --statistic Average \
  --threshold 30 \
  --unit Percent \

A copy of the user-data-file.txt file is as follows:

#!/bin/bash
INSTANCE_TYPE=something
DOMAIN=dodwell.us
INSTANCE_ID=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id | cut -d- -f2`
HOSTNAME=$INSTANCE_TYPE-$INSTANCE_ID
#set also the hostname to the running instance
hostname $HOSTNAME.ec2-int.$DOMAIN
export DEBIAN_FRONTEND=noninteractive
# Set APT to fetch packages from a apt-cacher-np install
echo 'Acquire::http::Proxy "http://puppet.ec2-int.dodwell.us:3142";' > /etc/apt/apt.conf
aptitude -y install puppet
echo "
[agent]
server = puppet.dodwell.us
certname = puppet.ec2-int.dodwell.us
" >> /etc/puppet/puppet.conf
/usr/bin/puppet agent -t --pluginsync true
sleep 10
/usr/bin/puppet agent -t --pluginsync true
sleep 10
/usr/bin/puppet agent -t --pluginsync true
sleep 10
reboot

overview

In the first part of the script we define bash variables that we can use later. This is done so we can easily use this script for the Web and Elastic Search ASG’s. After we’ve defined the variables we check if the ASG’s have already been created and delete them. This will cause instances to also be Terminated. Once the groups have been Terminated the script will then create a Launch configuration. Once the launch configuration has been created we create the auto scaling group. Then we need to create scaling policies and then link CloudWatch events to these polices to create and destroy instances.

launch configs

The first command to setting up autoscale is as-create-launch-config. Using this command, you tell AWS:

  • a unique name for the configuration
  • which AMI ID you want to use as your template for creating more EC2 instances
  • the EC2 instance type (the size and power of the server) to launch using your AMI
  • your access key
  • a security group to deploy the instances into
$ as-create-launch-config ${LAUNCH_GROUP} --image-id ${EC2_AMI} --instance-type ${EC2_TYPE} --monitoring-enabled --key root --group ${EC2_SECURITYGROUPS} --user-data-file ${EC2_USERDATA}
OK-Created launch config.

The API replies with: “OK-Created launch config.” Please refer to the below ‘Puppet’ section for more detail on the user-data-file.

auto scaling group

Use the as-create-auto-scaling-group command to define the properties for your group of servers. Auto scaling groups are the core component of an auto-scaling configuration. This command takes the launch_config_name you defined from the step before as a parameter, the name of the ELB you want to use, and most importantly, lets you define the minimum and maximum number of servers you want to have in your cluster. In the example below, we define a group with a minimum of 2 servers and a maximum of 10.

$ as-create-auto-scaling-group ${ASG_GROUP} --availability-zones ${REGION}b,${REGION}a --launch-configuration ${LAUNCH_GROUP} \
--min-size ${EC2_MIN} --max-size ${EC2_MAX} --load-balancers ${EC2_LB} --desired-capacity ${EC2_MIN}
OK-Created ASG.

auto scaling policies

Once we have our EC2 AMI, an AS launch config, and an AS group defined to deploy our instances into, we’re ready to define the auto scaling policies that will actually cause more (or fewer) EC2 instances to be launched and attached behind the ELB.

The command used to change the number of servers in the group is the as-put-scaling-policy command. With auto scaling, you use EC2 monitoring within CloudWatch to trigger a certain policy, but before we can do that, we need to define the actual policies that will be triggered. You can use this command to manually trigger scaling events as well, for testing before your traffic burst arrives, and in doing so, you can not only see the effect of scaling up and down, but you can watch AWS work its magic by refreshing your Instances view—new server instances appear in the AWS Management Console as your traffic increases beyond the thresholds you set.

The as-put-scaling-policy command takes the auto scaling group name we defined in step 1, a name for the policy, such a “scale-up” or “scale-down,” the type of scaling change the policy defines, and a cool down period. Again, the cool down period is used to prevent AWS from executing multiple policies within a very short time.

$ as-put-scaling-policy ${ASG_GROUP}-add --type ChangeInCapacity --adjustment 1 --auto-scaling-group ${ASG_GROUP}
arn:aws:autoscaling:ap-southeast-2:000000000048:scalingPolicy:ffffffff-ffff-ffff-ffff-9030fffffffff:autoScalingGroupName/web-asg:policyName/rabbitmq-asg-add

Above you can see the basic upscale policy we defined, named “rabbitmq-asg-add,” a ChangeInCapacity policy to add 1 server and wait 3 minutes before another policy can be triggered. Below is the reverse operation, or a “rabbitmq-asg-del” policy to remove 1 server from our group.

$ as-put-scaling-policy ${ASG_GROUP}-del --type ChangeInCapacity --adjustment='-1' --auto-scaling-group ${ASG_GROUP}
arn:aws:autoscaling:ap-southeast-2:000000000048:scalingPolicy:ffffffff-ffff-ffff-ffff-9030fffffffff:autoScalingGroupName/web-asg:policyName/rabbitmq-asg-del

In both cases, AWS replies with a return message including the unique auto-generated name of our two new auto scaling policies. We’ll use those unique policy identifiers to connect to our CloudWatch events in the final step.

At the moment we have everything we need for an intelligent auto-scaling configuration except one thing—the intelligence! The smarts come from choosing a CloudWatch event, such as 80% CPU utilization of an EC2 instance in our group, and wiring up that condition to automatically trigger the scale-up policy we defined. We’re also going to want to do the same in reverse for scaling back down at 20% CPU utilization.

The command to do this comes from the CloudWatch command line tools, and is called mon-put-metric-alarm. This command takes several parameters:

  • a name for the alarm that you choose
  • a description for what the alarm is monitoring,
  • the namespace for the alarm (in this case, AWS/EC2)
  • the name of the [namespace] metric that you want to monitor
  • the statistic type of the monitoring metric, such as Average or Percent,
  • a period or time interval,
  • a threshold for the statistic you choose,
  • a comparison operator, such as greater than or lesser than
  • a dimension, which is the ID of an EC2 instance to monitor
  • and the number of evaluation periods during which the metric you choose has to consistently return over or under the average or percent unit you define

As you can see, there’s a lot to this command, but once we look at every parameter, you can see that without each of them, you wouldn’t have the ability to control auto scaling changes with enough granularity. The name and description are shown back to you later when using the mon-describe-alarmscommand. The statistics you’re watching, and the thresholds and time intervals, are important to test for your particular application. For example, we chose to monitor average CPU utilization for a period of 60 seconds, and an evaluation period of 3 intervals (or 3 minutes), for an event of 80% or greater level. Here’s the command to achieve this.

$ mon-put-metric-alarm ${ASG_GROUP}-add \
  --alarm-actions "arn:aws:autoscaling:ap-southeast-2:000000000048:scalingPolicy:ffffffff-ffff-ffff-ffff-9030fffffffff:autoScalingGroupName/web-asg:policyName/rabbitmq-asg-add" \
  --comparison-operator GreaterThanThreshold \
  --dimensions "AutoScalingGroupName=${ASG_GROUP}" \
  --evaluation-periods 3 \
  --metric-name CPUUtilization \
  --namespace "AWS/EC2" \
  --period 60 \
  --statistic Average \
  --threshold 90 \
  --unit Percent \
OK-Created Alarm

In English, the above command says, “If the average CPU utilization of instances in the ASG group ‘rabbitmq-asg’ is measured at 80% or greater 3 times over 3 minutes, then trigger our scale-up policy.

Here is the reverse mon-put-metric-alarm command we used to terminate one of the servers if the CPU utilization drops below an average of 20% over 3 minutes.

$ mon-put-metric-alarm ${ASG_GROUP}-del \
  --alarm-actions "arn:aws:autoscaling:ap-southeast-2:947682202348:scalingPolicy:1a31d3d1-b180-4fcb-b288-bb43e651056d:autoScalingGroupName/rabbitmq-asg:policyName/rabbitmq-asg-del" \
  --comparison-operator LessThanThreshold \
  --dimensions "AutoScalingGroupName=${ASG_GROUP}" \
  --evaluation-periods 3 \
  --metric-name CPUUtilization \
  --namespace "AWS/EC2" \
  --period 60 \
  --statistic Average \
  --threshold 30 \
  --unit Percent
OK-Created Alarm

In the above script example we nest the ‘mon-put-metric-alarm’ and ‘as-put-scaling-policy’ commands to simplify running the scripts. However this is pulled apart in this documentation to explain better.

For more information and examples, refer to the Auto Scaling section on the Amazon developer documentation.

instance creation via cloud-init

cloud-init is the Ubuntu package that handles early initialization of a cloud instance. It is installed in the Ubuntu Cloud Images and also in the official Ubuntu images available on EC2.

cloud-init’s behavior can be configured via user-data. User-data can be given by the user at instance launch time.

Using the user-data script functionality we can bootstrap puppet into action.