Infrastructure as Code (IaC) makes the life of a cloud engineer easier in many ways. But it is still complex to manage the cloud and its operations in large enterprises. When we think of a cloud-agnostic solution for IaC in a multi-cloud environment, we can think of HashiCorp’s Terraform.
Infrastructure as Code (IaC) automates, templatizes and reuses infrastructure configuration and setup through code in a cloud platform. It is useful for setting up different environments (dev, test, UAT, PROD) quickly in a cloud environment and when you want to replicate a setup for disaster recovery or multi-region setup of application instances. With its template based scripting for infrastructure provisioning, service configuration and logic development, IaC makes the life of a cloud engineer easier.
It is easy to develop and maintain IaC on the cloud in the case of greenfield software development. But when you already have cloud architecture ready and deployed, IaC can be developed manually. However, this could be tedious if you are using more cloud resources or many architectures for the various applications developed.
Creating IaC from infrastructure services and cloud resources can be done using a reverse engineering tool like TerraCognita, which was developed as an open source facility by Cycloid. TerraCognita can be used to develop IaC code from live cloud accounts in AWS, Azure or Google and from private cloud setups in OpenStack and VMWare. Cycloid has multiple open source projects including TerraCognita, InfraMap (for cloud diagrams) and TerraCost (for predictive cloud cost estimation). TerraCognita gets IaC scripts as Terraform templates from infrastructure services and uses InfraImport to get this infrastructure service converted as code.
Why we need Infrastructure as Code
During cloud adoption, we have repeated activity for infrastructure provisioning, which could be simple or complex when it comes to multiple environment provisioning. Hence, automating and reducing the manual effort in such activities (including compute, storage, network and security configurations) helps to reduce overall DevOps operational activities and improve efficiency in engineering activities (Figure 1).
This kind of IaC can be available as a native facility in Azure as Azure Resource Manager (ARM) templates, in AWS as CloudFormation templates and in GCP as Google Deployment Manager (GDM). All these are in JSON format to prepare scripts for infrastructure provisioning and configuration.
For instance, an AWS feature like CloudFormation accomplishes this by defining provisioning code in a script using Typescript or Python for resource development and deployment. With a cloud development kit (CDK), this CloudFormation is taken to the next level of IaC. By using CDK, we can develop AWS constructs embedding CloudFormation templates, and configure services and business logic in native code using Java, Python, JavaScript or Typescript.
When we have a large number of microservices in terms of containers and serverless applications, it would be a Herculean task to handle code build, deployment, infrastructure provisioning, and platform management of cloud services across different stages, monitoring and management activities, and platform updates.
Platform teams automate many of these activities through Infrastructure as Code (IaC), but it is still a complex activity to manage the cloud and its operations in large enterprises. In the case of Microsoft Azure, using a templatized solution with Azure Resource Manager (ARM) helps to replicate configuration and application infrastructure provisioning. Here, pre-defined code scripts are used for building a descriptive model that defines VMs, networks, security and connectors easily.
Azure DevOps can be integrated with the ARM group for secure deployment of applications to different stages by selective users, and for tag deployment and code promotion with different parallel releases. This is so that we can promote (when regression checks are successful) or demote (when there is a reason to rollback to the previous release) the code build, as needed, quickly. It can also be used to store reusable artifacts like library or container images, so that it can be reused for deployment hooks without being built again.
Foundational services of Terraform
When we think of an agnostic solution for IaC in a multi-cloud environment, we can think of HashiCorp’s Terraform template. This can be a JSON (to export/reuse from native scripts) or YAML script based simplified solution that can be integrated with any cloud service provider’s native deployment pipeline or third-party pipeline like Ansible to enable infrastructure provisioning activities on-the-go with flexibility.
It is easy to do large scale infrastructure creation, spin-up development, test instances during the implementation cycle or create pre-production or production instances during horizontal scaling, with Terraform scripts. These can be run with resource schedulers, and can create network configuration or secured environment setups for software defined networks (SDN) using Terraform templates.
Terraform scripts can be used to spin-up instances or to destroy environments for disposable temporary test environments in a cloud platform. Terraform templates can be validated before actual provisioning. Terraform workflow has three steps — write (create Terraform templates for IaC), plan (preview the changes before applying them in the cloud environment to validate) and apply (to provision infrastructure quickly).
For a cloud-agnostic solution like multi-cloud architecture, it is easy to adapt Terraform templates for IaC to quickly create a uniform setup and configuration for multiple environments, reduce complexity in creating multiple parallel scripts, and reuse these scripts as much as possible for each cloud service provider in the multi-cloud setup.
Some claims on the Internet, though, state that Terraform is not cloud-agnostic but a wrapper running on top of the native cloud services of any cloud service provider.
Cloud-agnostic provisioning using Terraform
Infrastructure as Code (IaC) helps to look at infrastructure as a software module and programmatically handle it through scripts. This can be quite useful for cloud platforms to quickly provision virtual infrastructure components like compute, storage, network, security and databases.
Infrastructure as Code (IaC) templates can be classified as cloud native templates (like AWS CloudFormation, Azure Resource Manager and Google Deployment Manager templates), and cloud-agnostic templates (like Terraform templates).
Cloud native templates are flexible and customisable to the native platform, and can be integrated with any native service easily. Cloud-agnostic templates are suitable for multi-cloud application services in order to have uniform infrastructure scripts across cloud service providers to reduce rework in IaC script activities. For example, if you want to deploy a Web application and API services with disaster recovery services (active-passive deployment) in the AWS platform, it would be better to choose cloud-agnostic templates.
Google Cloud Platform (GCP) has a native template service called Cloud Foundation Toolkit (CFT), which contains a set of reference templates that can be used as is or customised to your IaC needs. The reference templates available in CFT are deployment manager templates (GCP native) or Terraform templates (cloud-agnostic). They are simple Python scripts that can be customised easily.
Other popular cloud native IaC facilities
Infrastructure as Code as well as cloud automation for provisioning and virtualisation requirements are very helpful to accelerate build, deployment, templatizing and cloud management activities. They are even more helpful when it comes to multi-cloud management, as preparing a provisioning script for each cloud is inconsistent and inefficient. Infrastructure as Code helps to automate this across different cloud environments with ease.
Though cloud native facilities like CloudFormation in AWS, ARM templates in Azure and CDM templates in GCP help to achieve this Infrastructure as Code, they are mostly native to the cloud platform and don’t have the agility and flexibility for multi-cloud management. On the other hand, Google Anthos, HashiCorp Terraform, and Red Hat Ansible templates can help in multi-cloud management, enabling the development of zero-touch deployment scripts and provisioning templates for cloud tower setup.
There are various VM image building tools like Packer and Terraform, but Azure Image Builder is a comprehensive tool that can be used in combination with Azure Resource Manager (ARM) templates or with Terraform templates and create VHD (virtual hard disk) files of a virtual machine (VM). This can be imported to any cloud platform like AWS EC2 using the VM import tool in AWS or the GCP VM import tool in Google Cloud.
Azure Image Builder has two major functions — to create images from the template builder, and to run the build image with an integrated CI pipeline to deploy it to the target VM in Azure. The image build from this Image Builder service can be a Windows or Linux image, and hence suitable for importing by other cloud service providers as well as for using the VM import tool.
Integrating a CI pipeline with Azure Image Builder is very simple. Integrate it with Azure DevOps or any other customised DevOps pipeline with Ansible, Bamboo or Jenkins. Azure Image Builder also provides role based access control (RBAC) to access the image repository and template repository in a secure way.
AWS CDK has many pre-configured templates available, which can be reused to build a provisioning script. It usually works hand-in-hand with the AWS CloudFormation service. One has to develop an IaC script using AWS CDK and provisions using CloudFormation.
If you are familiar with Terraform, you can use the aws-tf construct library, and if you are familiar with Kubernetes and want to develop a container deployment script, you can use the aws-cdk8s construct library and deploy it in the AWS Elastic Kubernetes Engine (EKS).
Nowadays, Low-code/No-code and Infrastructure as Code (IaC) are getting popular as they are less complex and more efficient for operations. The Systems Manager Service of AWS can be used to group cloud resources, get operational insights and automate operational tasks. It helps to reduce the time and effort used in troubleshooting operational problems.
This service can handle orphaned cloud resources and group cloud resources, get monitoring and management activities onto a single UI dashboard quickly, and simplify the operation and management of cloud infrastructure at scale. When you are managing large applications and infrastructure estate in a cloud platform, it is tedious to manage the server provisioning and deployment across different stages. Infrastructure as Code (IaC) simplifies this with template based scripts.
Google Anthos, HashiCorp Terraform and Red Hat CloudForms are leading this space equally for creating, changing and provisioning infrastructure for any cloud platform including service management, security, and compliance and optimisation. When you work with more applications and services during cloud adoption, you may end up with repeated work in infrastructure provisioning, platform setup, and application development and deployment. If you can automate many of these jobs, it will ease the software development journey particularly when you need to repeat activities at different stages like development, testing, pre-production and production.
