Designing systems for Immutable Infrastructure requires a deep understanding of how compute is deployed and configured in virtual and cloud environments. In order to help practitioners better understand this architectural paradigm I have created a 5 Layer Model for Compute in Immutable Infrastructure. The following is a high level overview of this model, what the layers are, and how they are deployed. It is meant to be used as an outline to frame deeper conversations in the topic of Immutable Infrastructure, and help bring understanding to people trying new to the topic.
Layer 1: The Node
The node, a single bare metal server, is the basic atomic unit of compute in the datacenter. It is the lowest layer of the infrastructure stack and it is where automated provisioning begins.
Automated Bare Metal provisioning follows this pattern:
1. The Node boots and connects to a PXE server.
2. The Provisioning server determines which image to assign the Node based on a rules engine.
3. The Node downloads a microkernal image via TFTP from the PXE server.
4. The image installs and configures the OS.
5. The image runs post-processing logic.
6. The Node reboots with the locally installed and configured OS.
At this point the Node is now a Host.
Layer 2: The Host
The primary job of the Host is one of the following:
1. Run virtual machine Instances
2. Run Containers
3. Schedule tasks and jobs
It is practically not possible to deploy the Host as immutable. Partial immutability (some, but not all Hosts) is possible, but the trade off is consistency of process. Ideally all hosts are automated in the same fashion. The best method for accomplishing this is a combination of configuration management and service registration/discovery, including a shared key/value store for configuration data.
Layer 3: The Instance
The Instance is first layer above abstraction for Infrastructure-as-a-Service. It is an OS that has been installed in a virtual machine on a Host. The virtual machine is a file or collection of files. The files can be packaged into Images for portability. Examples of images are AMIs, OVFs, containers, and Glance Images. Immutablilty also becomes possible at this layer.
The process for deploying Instances of immutable images looks like this:
1. Configuration Management recipes are developed for all Processes running in the Instance
2. Packer (an image build tool) takes a declarative input file specifying the image base (ISO, AMI, etc), the configuration management recipes, and the output format and creates a binary artifact.
3. The binary artifact is stored in a binary repository.
4. Terraform (an infrastructure deployment tool) deploys an Instance of the binary artifact Image to a Host (or abstracted cloud endpoint in the case of IAAS).
For this process to work in an automated way it important that Service Discovery and Registration be utilized in the target production environment.
Layer 4: The Long Running Process
The Long Running Process (LRP) is the first layer above abstraction for Platform-as-a-Service. A LRP is a service or application that when started will run for an indeterminate amount of time (examples: a web server, applications, etc). Most modern application stacks are comprised of several LRPs that may or may not run on distributed Instances.
In Immutable Infrastructure the binary to execute an LRP will be installed on the Instance OS at the time the binary artifact Image is created. The configuration of the process can also be added to the image at this time if it will not change. If the configuration will change during runtime it should be stored in an external key/value store. Configuration Management can be used at runtime to ensure the process is running in the desired state.
In the case of a PaaS the binaries and configurations for executing the LRPs will be bundled in container Images or buildpacks. To preserve immutability these images or buildpacks are treated the same way as the Layer 3 Image binary artifacts and should be stored in a binary repository after creation and deployed from the same repo.
Layer 5: Stateful Data
Stateful Data is information that is required to persist regardless of the lifecycle of a process. Examples include Binary Large OBject (BLOB) Stores (file servers), Databases, and Key/Value Stores. Persisting data on a Instance would by definition violate immutability, however data must exist. This is accomplished by attaching external volumes to the Instances or connecting the LRPs to an external Object Store via APIs.
There are other methods, however they concern proprietary hardware or services that are outside the scope of this definition. As a rule it is almost always easier to account for data being persisted outside of the Instance at the time of the development of the LRP than to solve the problem with infrastructure.