packer.io
In a previous post I wrote about how we can use Auto-Scaling Groups (ASG’s) to quickly adapt to user load. In this post I intend to explain a method to create custom Amazon Machine Image’s (AMI’s) using a project called packer.io
Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration. Packer is lightweight, runs on every major operating system and is able to create machine images for multiple platforms in parallel. Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software.
A machine image is a single static unit that contains a pre-configured operating system and installed software which is used to quickly create new running machines. Machine image formats change for each platform. Some examples include AMIs for EC2, VMDK/VMX files for VMware, OVF exports for VirtualBox, etc.
Packer is distributed as a binary package for different OS’s. Our OS is Linux based with a 64-bit CPU.
mkdir ~/bin/packer
wget https://dl.bintray.com/mitchellh/packer/0.6.0_linux_amd64.zip
unzip -d ~/bin/packer 0.6.0_linux_amd64.zip
export PATH=$PATH:~/bin/packerNow that packer is installed lets create a json file to build an Amazon EC2 AMI.
example.json:
{
"variables": {
"aws_access_key": "",
"aws_secret_key": ""
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "ap-southeast-2",
"source_ami": "ami-09f26b33",
"instance_type": "t1.micro",
"ssh_username": "ubuntu",
"ami_name": "example-ami {{timestamp}}"
}],
"provisioners": [{
"type": "shell",
"script": "build_example.sh",
"execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo {{ .Path }}"
}]
}And let’s make a shell script that will be executed to customize the AMI:
build_example.sh:
#!/bin/bash
export DEBIAN_FRONTEND=noninteractive
echo 'Package: *
Pin: origin "packages.dodwell.us"
Pin-Priority: 1001' > /etc/apt/preferences.d/dodwell-precise-pin-1001
echo 'deb http://apt.puppetlabs.com precise main' > /etc/apt/sources.list.d/puppetlabs.list
echo 'deb http://apt.puppetlabs.com precise dependencies' > /etc/apt/sources.list.d/puppetlabs-dependencies.list
echo 'deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main' > /etc/apt/sources.list.d/pgdg.list
echo 'deb http://apt.dodwell.us precise main' > /etc/apt/sources.list.d/dodwell.list
# puppetlabs
apt-key adv --keyserver keyserver.ubuntu.com --recv 4BD6EC30
# pgdg
curl https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
# dodwell repo
curl https://packages.dodwell.us/keys/keyfile.asc | apt-key add -
# Lets not allow services to be started on install
mkdir /tmp/fake
for COMMANDS in initctl invoke-rc.d restart start stop start-stop-daemon service
do
ln -s /bin/true /tmp/fake/${COMMANDS}
done
PATH=/tmp/fake:$PATH aptitude -y update
PATH=/tmp/fake:$PATH aptitude -y upgrade
PATH=/tmp/fake:$PATH aptitude -y install puppet lsb-release ruby ruby1.9.1 ruby1.9.1-dev ri1.9.1 build-essential libssl-dev zlib1g-dev rubygems fail2ban openjdk-7-jre-headless postgresql-client-9.3 proftpd-basic unzip nginx php5-fpm php5-suhosin
echo "
[agent]
server=puppet.ec2-int.dodwell.us
" >> /etc/puppet/puppet.conf
echo "/usr/bin/puppet agent -o" >> /etc/rc.local
/usr/sbin/update-alternatives --install /usr/bin/ruby ruby /usr/bin/ruby1.9.1 400 \
--slave /usr/share/man/man1/ruby.1.gz ruby.1.gz /usr/share/man/man1/ruby1.9.1.1.gz \
--slave /usr/bin/ri ri /usr/bin/ri1.9.1 \
--slave /usr/bin/irb irb /usr/bin/irb1.9.1 \
--slave /usr/bin/rdoc rdoc /usr/bin/rdoc1.9.1
/usr/sbin/update-alternatives --set gem /usr/bin/gem1.9.1Now that we have these 2 files in our current directory lets build the AMI:
root@earth:~# packer build \
-var 'aws_access_key=YOUR ACCESS KEY' \
-var 'aws_secret_key=YOUR SECRET KEY' \
web.json
amazon-ebs output will be in this color.
==> amazon-ebs: Creating temporary keypair: packer 536acd17-6080-23b8-5a0e-884d4a05035d
==> amazon-ebs: Creating temporary security group for this instance...
==> amazon-ebs: Authorizing SSH access on the temporary security group...
==> amazon-ebs: Launching a source AWS instance...
amazon-ebs: Instance ID: i-ffffffff
==> amazon-ebs: Waiting for instance (i-ffffffff) to become ready...
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Provisioning with shell script: build_web.sh
amazon-ebs: Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /tmp/tmp.4SGucrYWX8 --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv 4BD6EC30
...
...
...
amazon-ebs: update-alternatives: using /usr/bin/gem1.9.1 to provide /usr/bin/gem (gem) in manual mode.
==> amazon-ebs: Stopping the source instance...
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Creating the AMI: web-ami 1399508247
amazon-ebs: AMI: ami-XXXXXXXX
==> amazon-ebs: Waiting for AMI to become ready...
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' finished.
==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs: AMIs were created:
ap-southeast-2: ami-XXXXXXXX
root@earth:~#Packer gives us a method to consistently create base AMI’s that can be further configured with tools like Puppet. Packer ensures that these AMI’s are documented with code so that at later dates we can upgrade instances and know exactly how they were created. By using custom-built AMI’s you’re able to speed up the time it takes ASG to boot new instances.