This guide is an attempt to document the "best practices" which have emerged at Bloomberg during our journey of deploying Chef in a large enterprise environment. It is intended to serve as a reference to Bloomberg's infrastructure engineers, as well as the Chef community as a whole, on how to write high-quality, re-usable infrastructure code which is capable of being used infront or behind a corporate firewall.

During the initial phase of bringing Chef into a large enterprise, especially one with several thousand developers, we realized that there needed to be some tight constraints put on what (and how) code was written. Therefore after dragging our feet for a long time we decided to invest some time in writing this style guide. Since we contribute and consume the open source cookbooks from the Chef community it made sense to publish this document. We hope that it can be of some guidance on how to write confident Chef cookbooks.

The core Chef framework is written in the Ruby programming language. The same programming language is used to write Chef cookbooks. Therefore, it is natural for us to start with the best practices of the Ruby programming language. Luckily for us this has already been done by the amazing Ruby community.

It is very important to understand that this document models itself from the excellent Ruby Style Guide which is actively being maintined by the Ruby community. Most of the principles laid out in this guide are directly applicable to writing confident infrastructure code with Chef.

• How to Contribute
• Guiding Principles
  • Infrastructure as Code
  • Continuous Integration
  • Unit and Integration Testing
  • Service Discovery
  • Managing Secrets
  • Semantic Versioning
• Platform Considerations
  • Filesystem Paths
  • Service Management
• Cookbook Design
  • Attributes
  • Custom Resources
  • Recipes
  • Templates
• Policy Development
  • Chef Repository
  • Hoisting Attributes
• Cookbook Patterns
  • Application Cookbook
  • Library Cookbook
  • Wrapper Cookbook
• Cookbook Development
  • Chef Development Kit
  • Test Kitchen
  • Policyfiles

It is very easy, just follow the contribution guidelines.

The open source software development community has begun to gravitate towards a standard of versioning software to accurately express the impact of changes between releases. By adhering to a strict policy of semantic versioning we are able to indicate to contingent cookbooks the impact of an impending release.

Of course for cookbooks which are fast-moving this can quickly become a burden on developer productivity. We rely on our continuous integration pipeline to automatically bump the patch revision of the cookbook for each promoted release. This means that developers need only to consider the major and minor revision numbers to express changes in the target cookbook.

Our aim is to produce quality cookbooks which are capable of converging in a world of multi-platform operating environments. For most community cookbooks it is sufficient to support a few flavors of Linux - usually Ubuntu and CentOS - but in an enterprise environment we often have to install the same software across many different platforms.

This often goes beyond simply concatenating a different package CPU architecture or filename extension. The two examples that we run into the most often below are Filesystem Paths and Service Management.

While writing an application cookbook it is often the case where you may need to write out a configuration file to disk. In the general case one can simply use Ruby string interpolation to create the proper configuration file from a few variables. To illustrate this point let's use the common example of downloading a file from an HTTP server and then installing that package.

url = 'http://mirrors.rit.edu/epel/6/x86_64/collectd-4.10.9-1.el6.x86_64.rpm'
checksum = '549978cc77f9466925701668bfffa147bcb65ef5fa77dad4bee3d85231090010'
basename = File.basename(url)
remote_file "#{Chef::Config[:file_cache_path]/#{basename}" do
  source url
  checksum checksum
end

package 'collectd' do
  source "#{Chef::Config[:file_cache_path]/#{basename}"
  action :upgrade
end

The above snippet will work as intended on a POSIX operating system, but where this will not work is on Windows. A typical path on a Windows file system is addressed with backslashes instead of forward slashes. A better approach is to use Ruby's built-in API to create the filesystem path based on the operating system that it is running on. Let's visit that example again using the Ruby File.join API.

url = 'http://mirrors.rit.edu/epel/6/x86_64/collectd-4.10.9-1.el6.x86_64.rpm'
checksum = '549978cc77f9466925701668bfffa147bcb65ef5fa77dad4bee3d85231090010'
basename = File.basename(url)
remote_file File.join(Chef::Config[:file_cache_path], basename) do
  source url
  checksum checksum
end

package 'collectd' do
  source File.join(Chef::Config[:file_cache_path], basename)
  action :upgrade
end

This subtle change would now have a correct filesystem separator on the Windows platform and thus make our recipe a little less error prone for someone attempting to use it.

The whole purpose of a application cookbook is to provide Chef primitives to install and configure an application on a node. This more often than not requires enabling, starting and restarting system services. To illustrate our point let's take a look at the Cassandra Cluster cookbook.

The Cassandra Cluster cookbook is an application cookbook which installs and configures a node to be a member of a Cassandra database cluster. This cookbook has an extremely simple and straight forward default recipe. An abbreviated version of the Cassandra cookbook's default recipe is given as an example below:

cassandra_config service_name do |r|
  owner node['cassandra-cluster']['service_user']
  group node['cassandra-cluster']['service_group']

  node['cassandra-cluster']['config'].each_pair { |k, v| r.send(k, v) }
  notifies :restart, "cassandra_service[#{name}]", :delayed
end

cassandra_service service_name do |r|
  user node['cassandra-cluster']['service_user']
  group node['cassandra-cluster']['service_group']

  node['cassandra-cluster']['service'].each_pair { |k, v| r.send(k, v) }
end

Let's focus on the second resource in this snippet of the recipe. This is a custom resource which uses the Poise Service library cookbook for service management. The cassandra_service resource has attributes which control how the Cassandra software is installed and where the configuration file is located. In the default recipe these are driven through the node['cassandra-cluster']['service'] attribute Hash.

The Poise Service library cookbook provides a reusable pattern for creating custom resources that manage services. In practical terms this means that the same code referenced above will work with the native system management routines for any of the supported platforms. Take a look at the table below to further illustrate how the Poise Service library cookbook would configure the cassandra_service resource.

Furthermore, the Poise Service library allows for the "service provider" to be specified as a custom attribute or the default set through a node attribute. So, for instance, if we would like to use the Runit service management framework instead of the native provider that can simply be done by including a new cookbook and setting a node attribute in a wrapper cookbook.

node.default['poise-service']['provider'] = 'runit'

Why is this important? By using the Poise Service library cookbook we can abstract away the concerns of service provider management and build a clear and concise cookbook. It reads a lot easier, and it is a whole lot more flexible out of the box. It also means that the same cookbook can be used on all of the above platforms without the need for the management of service provider templates. This is an extremely valuable pattern when you are managing dozens of application cookbooks. We now have a single library cookbook instead of a dozen application cookbooks with different service management templates.

Since we have a diverse set of platform requirements for most of our cookbooks we have made several decisions regarding the usage of node attributes, custom resources and templates. Additionally, because our engineers do not have knife access this limits the custom values which could be set in node attributes. It also means that we must support looking up variable data such as data bags through an HTTP service.

In the general case we prefer writing custom plugins which populate Ohai attributes from either the operating system or external services. This is important as it allows us to describe the state of a system with the ohai command-line tool outside of a Chef convergence.

By limiting the default value of attributes it allows for deployment specific overrides to happen from outside of a cookbook. There should be very few cases where this is used instead of a service discovery mechanism, but in a pinch it may be necessary.

A library cookbook abstracts common patterns into resources and providers which can be used to build both wrapper cookbooks and application cookbooks.

A great example of a library cookbook is the libarchive cookbook which provides a Chef resource primitive to manage compressed archives of all different formats. This makes it extremely useful to write application cookbooks that are agnostic to the underlying compression format. The example below downloads a compressed archive of the GitHub Hub command for Linux x86_64 and extracts it to the /opt directory.

archive_url = 'https://github.com/github/hub/releases/download/v2.2.1/hub-linux-amd64-2.2.1.tar.gz'
remote_file File.join(Chef::Config[:file_cache_path], File.basename(archive_url)) do
  source archive_url
end

libarchive_file File.join(Chef::Config[:file_cache_path], File.basename(archive_url)) do
  extract_to '/opt/hub/2.2.1'
end

A more complex example of using the libarchive_file resource as a primitive can be found in the libartifact cookbook. This cookbook goes a step further and manages application artifacts on disk using symbolic links. It utilizies the libarchive_file resource for managing the extraction of compressed artifacts.

The application cookbook is the most common cookbook pattern. Its purpose is to install, configure and manage the lifecycle of an application on a node. Most cookbooks which are publically available on the Chef Supermarket are of this type.

The complexity of application cookbooks can vary very widely. A cookbook such as our own Collectd cookbook installs and configures the collectd monitoring daemon. Because there are several tuning knobs on the daemon itself we take the approach of breaking out additional Chef resource primitives to manage the configuration and service separately. There is an additional resource which manages the configuration of collectd plugins.

This process of modeling an application cookbook with several primitives allows for the maximum flexibility for testing and deployment. Because we are testing the validity of the input properties of a resource we're able to fail the Chef convergence prior to configuration being modified on the target.

In an enterprise environment if the decision is made to include a public cookbook available on the Chef Supermarket it almost always should be included via a wrapper cookbook. If the underlying application cookbooks are flexible enough - that is, the Chef resource primitives appropriately model the configuration and application - writing a wrapper cookbook is very straightforward and simple.

There are a few types of wrapper cookbooks which are generally accepted as best practices within the Chef community. These patterns represent a purposeful adaptation of the concept of a wrapper cookbook.

A base cookbook is a type of wrapper cookbook which is in the expanded run-list of each and every node within an organization. The base cookbook itself generally wraps core cookbooks which harden and configure the operating system itself. It is also the place where mirrors and the chef-client are configured.

A cluster cookbook is a type of wrapper cookbook which targets a specific configuration of a cluster of nodes. The cluster cookbook may set purposeful node attributes to fine tune the underlying application running on the cluster. A recipe within a cluster cookbook is generally one of the only recipes directly applied to a node's run-list.

chef generate cookbook clojure-service
cd clojure-service
bundle install
chef install

After generating a cookbook there will be several files in the newly minted directory. There is one file in particular which drives the resolution of cookbook dependencies. The Policyfile.rb is what is read when the chef install command executes. A fairly plain example file can be seen below.

name clojure-service'
run_list clojure-service::default'
default_source :community
cookbook 'clojure-service', path: '.'

Some cookbooks which are less recent may continue to use [Berkshelf][16] for managing cookbook dependencies. This tool still ships with the Chef Development Kit. The workflow for developing on these cookbooks is very similar to using the chef command.

git clone https://github.com/bloomberg/zookeeper-cookbook
cd zookeeper-cookbook
bundle install
berks install