Infrastructure automation with python

In NCR Edinburgh, we are focused in infrastructure automation. We are using Chef and Vagrant to automate our infrastructure, but we are always interested in the development of other automation technologies.

Technology recap

Chef is only one part of a whole stack of technologies that is used to enhance the automation and the reproducibility of virtual environments. Other important technologies are:

Vagrant

Vagrant allows us to combine the configuration management systems with the virtualization technologies. It is:

• a tool for building complete virtualized environments.
• supports lots of different virtualization backends
  • Virtualbox (the default)
  • VmWare
• supports lots of different provisioning backends:
  • shell
  • chef
  • salt
  • ansible
  • powershell
• platform independent
  • windows
  • linux

Saltstack

Saltstack is the response of the python community in the automatic provisioning and configuration management needs of modern information technology. Salt is:

• systems management tool. can be used for:
  • configuration management
  • systems orchestration
• written in python
• supports all operation modes and more:
  • masterless mode
  • master/minions mode
  • one off custom command execution either to whole minions environment, or to isolated minions (ansible, capistrano)
• uses extensively YAML
• supports templating

Chef salt 1-1 comparison

In NCR Edinburgh we are using Chef, so it makes sense to do a 1-1 comparison or the best possible approximation between the two techologies features and components so:

• chef recipe -> salt state
• chef resource -> salt state module
• chef attributes/databags -> salt pillars
• chef ohai -> salt grains
• chef cookbook -> salt formula
• chef uses ruby DSL -> salt uses YAML and Jinja
• chef can be extended with ruby -> salt can be extended with python

Please keep in mind that this is an approximation. The components are not 100% equivalent. The above mapping means that you can more or less achieve a similar outcome by using the different components in this way.

Chef salt architectural differences

• Chef enforces a specific workflow. The cookbooks have a strictly defined structure, and the relationship of cookbooks is also strictly defined
• Salt does not enforce a specific workflow. Its formulas structure does not follow a specific pattern, but they can be structured in a loosely defined tree.

Vagrant – salt integration

A minimal Vagrant file that can be used to create a centos6 virtual machine and provisionit with Saltstack is the follwing:

box_type  = "chef/centos-6.6"
Vagrant.configure("2") do |config|
  config.vm.box = "#{box_type}"
  config.vm.provider "virtualbox" do |v|
    v.memory = 2048
    v.cpus = 1
  end
  # For masterless, mount your salt file root
  config.vm.synced_folder "salt/roots/", "/srv/salt/"
  ## Use all the defaults:
  config.vm.provision :salt do |salt|
    salt.minion_config = "salt/minion"
    salt.run_highstate = true
  end
end

As we can see the only thing that changes between Chef and Salt runs is really the provisioner. Vagrant supports both technologies and many more ( e.g puppet, powershell, bash etc). There is no reason, other than good design best practices, that prevents us from mixing different provisioners inside the same Vagrantfile.

This Vagrantfile, will visit the defined directory, find the minion configuration file, read it and since we set the highstate flag to true, execute it. It will converge the system to the state defined in the minion configuration.

Salt execution modes

Salt same as Chef, supports various execution modes. It can be used either in a master/minion configuration (a server/client architecture) or as in masterless mode, which is equivalent to chef-solo execution mode. Comparatively setting up a salt master and registering minions is a lot simpler than setting up a Chef server with its chef nodes. Saltstack server can distribute configuration files to its minions using gitfs as the distribution mechanism. This file distribution mechanism is not supported when salt is running in masterless mode.

Masterless mode is relatively simpler. Salt reads a number of configuration files already found on the system, compiles a state list, compares the system against the desired states and converges all the diverging resources to it.

Masterless mode

is defined by a:

• "salt/minion" file with contents
  • file_client: local
• it defines the filesystem backend used by salt.
• other backends are:
  • salt server backend
  • gitfs backend
• gitfs backend is the chosen method for community formula distribution
• is unsupported by masterless mode
• setting highstate to true, is the salt way to start converging the minion

Salt state files layout

• Salt excepts a salt directory and a minion file in that
• It also expects a directory named roots and a file named top.sls in it
• top.sls is the root of all the state files
  • it defines what kind of environment exists
  • which sls files are included in which environment
  • the order by which these sls files are goind to be included
• in roots directory we can find any number of sls files, or directories of sls files, the usage of which are defined in top.sls

top.sls content

base:
  '*':
    - common-tools
    - httpd.apache
    - dovecot.dovecot
    - bind.bind
    - db.mysql
    - db.postgres
    - supervisord.supervisord

This sls file, will go ahead and – install a number of common tools in the new vm, – install apache as a webserver, – dovecot as an imap server, – bind as a dns server, – mysql and postgres databases systems, and – supervisord as a service management system.

This is an automation of a pretty common technology stack used in traditional server setup.

Sls file format

Sls files need to conform to a number of rules:

• Sls files, can include other sls files forming thus a tree. The shape of the tree is fully defined by the developer
• Sls files contain mostly YAML, but can also contain Jinla expressions
• Sls file entries are separated in execution blocks that are consisted of
  • a unique block id
  • a number of salt modules statements
• The different execution blocks in an sls file are executed sequentially

typical sls file

In this example the pkg mysql and mysql-server are the identifiers (block ids) and pkg and service are the salt modules.

mysql:
  pkg:
    - installed
mysql-server:
  pkg:
    - installed
  service:
    - name: mysqld
    - running

sls example with custom commands and file transfer

Even though in masterless mode we do not have a master to access and retrieve files from, we can still access files in the local filesystem. This is important for automation purposes because, even in an isolated run we can still update a repo of configuration files using an external command over e.g ssh and propagate configuration changes over a number of masterless minions.

supervisor:
  cmd.run:
    - cwd: /
    - user: root
    - name: pip install supervisor
    - env:
      LC_ALL: C.UTF-8
supervisor-init-file:
  file.managed:
    - name: /etc/init.d/supervisor
    - source: salt://supervisord/supervisor
    - user: root
    - group: root
    - mode: 755

jinja attribute file example

As we mentioned earlier, salt uses Jinja for its configuration. In this example we can see a list of different postgres package versions available in the different distrubutions supported by this specific salt configuration.

{% set pg_version = salt['grains.filter_by']({
    'CentOs' : { 'id': '8.4' },
    'RedHat' : { 'id': '9.1' },
    'Arch'   : { 'id': '9.1' },
    'Debian' : { 'id': '9.3' },
}, merge=salt['grains.filter_by']({
    '14.04'  : { 'id': '9.3' },
    '14.10'  : { 'id': '9.4' },
}, grain='lsb_distrib_release', merge=salt['grains.filter_by']({
    'jessie' : { 'id': '9.4' },
    'wheezy' : { 'id': '9.1' },
}, grain='lsb_distrib_codename', merge=salt['pillar.get']('postgres:lookup')))) %}

Execution and pitfalls

• The usual command: vagrant up will start the vm build
• The vagrant-salt plugin provides the necessary vagrant/salt integration
• Salt always does a full upgrade of the vm before the provisioning. This may cause delays if the base image is old.
• After salt has been installed, the logs can be found in /var/log/salt
• a yaml parser and verifier can save you hours of debugging
• sometimes salt fails silently, the verifier may help with that
• at the time of writting, there seems to be a bug in the auto bootstrap script that affects centos 7 provisioning

Nick Apostolakis
Tags: infrastructure
June 8th 2015