This is the second part of a two part presentation related to infrastructure automation. You may like to take a look at the first part that provides an overview of the tools and benefits of virtualization and automation and can be found here
As virtual machines are becoming increasingly important by the day both for organizations and individuals, we will attempt here to provide a brief introduction of the tools used in NCR Edinburgh for the automation and provisioning of our infrastructure.
These tools can be used either in a corporate environment in order to provide infrastructure standardization and automation both for development and production systems. All of the tools mentioned here are open source projects and can be downloaded at no cost.
This is the second part of a two part presentation related to infrastructure automation. You may like to take a look at the first part that provides an overview of the tools and benefits of virtualization and automation and can be found here
The goal is to create a fully configured vm artifact ready to be imported in the virtualization software of choice. We will use VirtualBox for the development system.Our technology stack is the following:
Packer is an open source tool for creating identical machine images for multiple platforms from a single source configuration.
Packer does not replace configuration management like Chef or Puppet. In fact, when building images, Packer is able to use tools like Chef or Puppet to install software onto the image.
A machine image is a single static unit that contains a pre-configured operating system and installed software which is used to quickly create new running machines. Machine image formats change for each platform. Some examples include AMIs for EC2, VMDK/VMX files for VMware, OVF exports for VirtualBox, etc.
In order to setup and configure a new system, Packer uses a number of files, called kickstarter files, that guide the whole setup procedure step by step, by providing input to all the various menus that come through the usual installation of a system (both windows and or linux).
Although nowadays Packer supports Chef itself, we are basically using its Vagrant post-processor to create custom Vagrant boxes from the net installation iso of Centos 6.6. Originally Packer did not support configuration management systems, so we used Vagrant instead. Nowadays its feature list is quite close to the Vagrant.
The following example retrieves the ubuntu server iso and prepares a system using shell as a provisioner and executing the script setup_things.sh
{
"builders": [
{
"type": "virtualbox-iso",
"guest_os_type": "Ubuntu_64",
"iso_url": "http://releases.ubuntu.com/12.04/ubuntu-12.04.5-server-amd64.iso",
"iso_checksum": "769474248a3897f4865817446f9a4a53",
"iso_checksum_type": "md5",
"ssh_username": "packer",
"ssh_password": "packer",
"ssh_wait_timeout": "30s",
"shutdown_command": "echo 'packer' | sudo -S shutdown -P now"
}
],
"provisioners": [
{
"type": "shell",
"script": "setup_things.sh"
}
]
}
berks
will download the cookbooks and
berks vendor
will upload them to the new vm. You dont need to do this manually, there is a Vagrant plugin that takes care of everything
The Berksfile defines the behaviour of the berkshelf tool. There is a community cookbook repository called chef supermarket and Berkshelf can access them by using the following syntax:
source "https://supermarket.chef.io"
cookbook 'cookbook_name', "~> 2.6"
Berkhelf can also use cookbooks from alternative locations (e.g git repos, github, gitlab etc) with a small change in the Berksfile syntax:
cookbook 'packages', git: '[email protected]:mattray/packages-cookbook.git'
Chef is one of the most prominent configuration management and vm provisioning opensource systems. It is mainly consisted by a number of execution blocks called resources, bundled together in files called recipes. A collection of recipes that can be combined to perform a specific task are often combined in larger packages called cookbooks. For more details about the internal structure of chef, please refer to the first part of the vm automation.
For the purposes of this blog post we are going to use a very simple cookbook, that takes a list of packages and installs it on the new vm. It is developed by a Chef employee Matt Ray and is distributed under the Apache License. It does not matter which distribution we choose to use, it will work on all of them, Chef takes care of the differences in package managers. By using a tool like Chef, we can ignore the underlying differences in package managers, repositories and initialization scripts and focus at the job in hand. Chef allow us to describe very complex structures in a declarative way.
One important feature of attributes is that they can be overridden during runtime, both from the calling scripts and the cookbooks themselves. Cookbook attributes are saved inside the attributes directory, by default there is one file there called default.rb. In our example cookbook it contains two attributes:
default['packages'] = []
default['packages_default_action'] = 'install'
These are the default values defined in the cookbook. An empty array list of packages and the default action of the cookbook.
A run-list is:
In the chef-solo context, a run_list is limited to a specific chef-solo run and isolated on the vm it is running, it does not require any external chef resources (e.g chef server)
Plainly put, a run_list is the list of the recipes we want to execute from one or more cookbooks. These cookbooks need to be available and accessible on a predefined location.
It looks like this:
{ "run_list": ["recipe[packages::default]", "recipe[another thing]"] }
Vagrant Berkshelf is a Vagrant plugin that adds Berkshelf integration to the Chef provisioners. Vagrant Berkshelf will automatically download and install cookbooks onto the Vagrant Virtual Machine.
If the Vagrant Berkshelf plugin is installed, it will detect when a Berksfile is present in the same working directory as the Vagrantfile.
Vagrant-berkshelf is installed using the following command:
vagrant plugin install vagrant-berkshelf
Vagrant is a tool for building complete development environments. With an easy-to-use workflow and focus on automation.
It uses pre packaged templates called boxes as a starting point and is using an open extendable interface in order to provide more features in the form of plugins. Vagrant-berkshelf is one of those plugins, and that makes work with chef a lot easier.
There are a lot of freely available community vagrant boxes for any platform imaginable. In NCR Edinburgh we build our own from the official CentOS sources using Packer
Vagrant supports out of the box a number of provisioners, among themselves chef and salt. In the following example we are going to use the chef provisioner.
It also supports a number of virtualisation providers, with VirtualBox as a default and VMWare as the suggested platform for production environments. All these providers are distributed as vagrant plugins
Vagrant is controlled by a single file called Vagrantfile
All the vagrant vms appear as VirtualBox vms, so you can use VirtualBox interface to export a vm to and ova
A Vagrant file looks like this:
# -*- mode: ruby -*-"
# vi: set ft=ruby :
box_type = "chef/centos-6.6"
Vagrant.configure("2") do |config|
config.vm.box = "#{box_type}"
config.berkshelf.enabled = true
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Add packages
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
config.vm.provision "chef_solo" do |chef|
chef.json = {
"packages" => ['git', 'screen', 'vim','openssl' ]
}
chef.add_recipe "packages::default"
end
end
We can see here, that the Vagrantfile syntax is actually a DSL (Domain Specific Language) it allows us to use a relatively simple, human readable, declarative syntax to define the various attributes we want our new vm to have.
In this file, we are defining the box type, we are using one of the community vagrant boxes provided by opscode the company that creates chef, we are enabling the vagrant berkshelf support since we want to use berkshelf to handle the cookbook management for us and use the chef provisioner in order to call the default recipe of the packages cookbook.
Vagrant will override the default attribute with name packages and provide to chef-solo a list of packages that need either to be installed or removed. The default action (also defined by an attribute) is install, so these files will be installed. As we can see we do not define anywhere how this is going to be done. The chef run is completely declarative. All the complexities related to package managers, packages and repos are handled under the hood by chef.
The cookbook recipe we are executing with this vagrant file goes through the following operations:
Chef::Log.info "packages:#{node['packages'].inspect}"
if node['packages'].is_a?(Array)
node['packages'].each do |pkg|
package pkg do
action node['packages_default_action'].to_sym
end
end
elsif node['packages'].is_a?(Hash)
node['packages'].each do |pkg, act|
package pkg.to_s do
action act.to_sym
end
end
else
Chef::Log.warn('`node["packages"]` must be an Array or Hash.')
end
Chef recipes are also a ruby DSL, and even though it is not pure ruby they are a bit more technical than Vagrant directives. Despite that, the recipes are also declarative, and as we can see in this example, we are using the resource package in order to define the concept of a package and its installation action and not the exact package name and the name of the package manager.
This recipe will check if the provided attribute is a hash or an array, and act accordingly. Ruby is a dynamically typed language so there is no way to infer the type of the data types at the beginning of the execution.
After the installation of the packages the chef run terminates.
A reasonable question is why do we use two tools that are doing a similar job. Packer and Vagrant are developed by the same company. Originally Packer did not support all the external provisioners it supports today, so its use was mostly limited in creating vagrant boxes. Today it supports most of the tools Vagrant supports and can be used instead of Vagrant.
However there are a few points that you may want to consider before using Packer for everything.
At this point we should have a live virtual machine created from scratch by our scripts. We can log in to this vm and use it with the command
vagrant login
or create a new vagrant box out of it that can be uploaded to the public vagrant repository of boxes. The vm can also be exported to an ova, if we use the VirtualBox tools and be distributed to other systems, or members of our organization.
This vm is completely disposable since it can be recreated at any moment through the use of the scripts either manually or through an automation system like Jenkins.
Nick Apostolakis
April 6th 2015