The need to develop software and applications more efficiently, to save redundancy, shorten time to market, and improve reliability and speed, has spawned the DevOps project. Traditionally developed apps are not adaptable to the rapid and sweeping changes engulfing the IT arena. DevOps provides a way to overcome these obstacles. And DevOps powered by open source is a formidable combination.
DevOps comprises a set of methods, cultural changes, practices, organisational structures and tools that increase the ability to deliver applications at high velocity to clients. Organisations that adopt DevOps are more easily able to release and monitor next-generation applications. It helps in responding to new requirements quickly and to the problems that occur in production. This speed enables organisations to better serve their customers and compete more effectively in the market.
DevOps aims at unifying the software development and software operation processes. The emphasis is on people and culture — to improve collaboration between the development and operations groups. Having said that, let us look at what DevOps is not:
- A role, person or organisation
- Something only systems administrators can do
- Something only developers can do
- Writing Chef and Puppet scripts
- Tools
Organisations today are looking more and more towards automation to achieve faster delivery, quick feedback from customers, providing quality software, lower recovery cycle times from any crashes, minimal defects, etc. Thus, one needs to ensure that all the tools used result in better integration between the development and operations teams, helping them collaborate or communicate better.
Industry adoption of DevOps tools
A new global study indicates that 69 per cent of enterprises in the Asia Pacific and Japan region have implemented DevOps (https://www.ciena.com/insights/articles/Five-DevOps-habits-for-highly-successful-enterprises.html).
Fifty per cent of organisations are now implementing DevOps. The questions and discussions with clients have shifted from, “What is DevOps?” to “How do I implement it at scale?” (https://go.forrester.com/blogs/2018-the-year-of-enterprise-devops/).
Facebook, Google, Twitter and Amazon are all working the DevOps way. Everything falls in place when these companies develop, test and successfully deploy code in a matter of 1 to 23,000 minutes per day (https://devops.com/most-popular-open-source-devops-tools/).
Also, organisations spend around 73 per cent of their DevOps budget in tool acquisition (https://devops.com/devops-adoption-the-driving-force-of-the-industry/).
The year 2018 has been a special journey for the IT industry, primarily because of the increasing DevOps implementation and the growing inclination towards it (https://devops.com/integrated-tool-ecosystem-helps-implement-devops/).
This article covers the characteristics of open source DevOps tools, and how the open source based DevOps life cycle phases are mapped to tooling when an enterprise adopts this model. It also presents the appropriate mapping of open source options with DevOps tool providers.
Drivers for open source DevOps tools
There is no single product in the market that can fulfil the DevOps requirements across the organisation. Thus, there is a demand for a collection of tools, potentially from a variety of vendors, used in one or more stages of the life cycle.
The key reasons for DevOps tool adoption are:
- Increasing cost of ownership in the tooling landscape
- Increased complexity when integrating and managing the tools acquired to achieve DevOps
- Burgeoning legacy tools and home-grown point solutions, leading to support issues
- Tools with duplicate functionality and those that are under-utilised
- The need for a wide range of skills to support the landscape
- Process variation due to tool limitations or capability variations
- Lack of tool catalogues and usage documentation
Tool selection for DevOps implementation depends on the following design principles:
- Maximising tool usage and coverage across all services and focusing on value
- Utilising fit-for-purpose contractor tools or leveraging customer tools where required
- Integration with various sources for analytics
- Focusing on automation and productivity tools for operations
- Application performance and currency through a combination of dynamic and static analysis tools
Characteristics of open source DevOps tools
Tools for release management, provisioning, configuration management, systems integration, monitoring, control and orchestration are important for DevOps adoption. In addition, the principles of open data, open standards and open APIs should be ingrained in the DevOps tool consolidation decision-making process. The open source DevOps tools should have the following characteristics:
- There should be no vendor lock-in and the seamless integration of enterprise wide tools, applications, products and systems developed/deployed by different organisations and vendors should be ensured.
- The DevOps tool chain should increase productivity, speed up time to market, reduce risk and increase quality.
- Vendor monopoly should be avoided, allowing for the use of free and open source DevOps tools. With data transferability across open data formats, there are greater opportunities to share data across interoperable platforms.
Adoption of open source DevOps tools enhances the interoperability with other enterprise tools because of the option of reusing recommended software stacks and libraries/components.
Figure 1 depicts the high-level characteristics of open source DevOps tools.
DevOps life cycle phases and the mapping of open source tools
A DevOps toolchain is a set or combination of tools that helps in the delivery, development and management of applications throughout the system’s development life cycle. At the enterprise level, the software teams need to automate the entire cycle of building, provisioning and deploying test environments, including the tools, scripts and test data to ensure rapid delivery. These teams need to collaborate via the application’s architecture and monitor event-based mechanisms for seamless data flow across the tool chains.
Listed below are the different stages of software/app development in the DevOps life cycle:
- Continuous planning
- Continuous development
- Version control
- Continuous testing
- Continuous integration
- Continuous deployment
- Configuration management
- Containerisation tools
- Repositories
- Release automation
- Continuous monitoring
The following sections briefly describe the phases of the DevOps life cycle and the appropriate open source products.
Continuous planning: DevOps readiness assessment across an enterprise is done during this phase. Requirements for DevOps implementation, the development approach and how this transits into operations is also done at this stage. Plans on how to deploy DevOps that involve people, processes and tools are made. The definitions of the target stage, transformation plan and execution plan are set during this phase. Companies need to continuously plan, measure and introduce business strategy and customer feedback into the development life cycle.
Continuous development: DevOps establishes the interdependence of software development and IT operations. It helps an organisation produce software and IT services more rapidly, with frequent iterations. Code development is done in any language, but maintained by using version control tools. The most popular tools used are Git, SVN, SonarQube, Maven and Ant.
Version control: Versions are maintained in a central repository that acts like a single source of truth. It helps the developers to collaborate on the ‘latest committed’ code and the operations team can access the same code when it plans to make a release. Whenever there is a fault during the release, the ops team can quickly roll back the deployed code and revert to the previous stable state. Git is the leading version control system. It allows developers to collaborate with each other on a distributed version control system.
Continuous testing: This promotes organisation-wide cultural changes that promote capabilities like testing early, testing faster and automated testing. Continuous testing synchronises testing and QA with development and operation, and is optimised to achieve business and development goals. Tools like Tosca, Selenium, TestNG and JUnit are used to automate the execution of test cases.
Continuous integration (CI): This helps developers to integrate code into a shared repository several times a day. It allows teams to detect problems early and verifies each check-in. By integrating regularly, errors are detected and located easily. The most popular CI tool in the market is Jenkins. Other popular CI tools are Bamboo and Hudson.
Continuous deployment: This happens when every change goes through the pipeline and is automatically put into production, resulting in many production deployments every day with greater delivery speed and frequency for complex applications. Ansible, Kamatera and Vagrant are the most useful continuous deployment tools.
Configuration management: This helps in establishing and maintaining consistency in an application’s functional requirements and performance. Configuration management tools work based on the master-slave architecture. The popular choices are Puppet, Chef, Ansible and SaltStack.
Containerisation tools: Containerisation tools help in maintaining consistency across the environments in which the application is developed, tested and deployed. It eliminates failures in the production environment by packaging and replicating the same dependencies and packages used in development, testing and the staging environment. Docker is the most used containerisation tool.
Repositories: The artefact repository is a collection of binary software artefacts and metadata stored in a defined directory structure. A repository stores two types of artefacts — releases and snapshots. Release repositories are for stable, static release artefacts, and snapshot repositories are frequently updated repositories that store binary software artefacts from projects under constant development.
GitHub is the central repository in which the code is maintained. Bitbucket and Nexus are the other repository tools.
Release automation: Deployment automation solves the problem of deploying an application by using an automated and repeatable process. DeployHub and Serena are widely used release automation tools.
Continuous monitoring: Continuous monitoring across all phases of the application development, testing and deployment processes is crucial for a successful DevOps implementation. The capability to improve service quality by monitoring the application’s performance and log management solves the problem of aggregating, storing, and analysing all logs in one place. Splunk, ELK Stack, Nagios, Sensu and NewRelic are some of the popular tools for continuous monitoring.
Key open source tools for DevOps
Open source DevOps tools are designed and developed using open source technologies in order to provide the DevOps toolchain capabilities that are listed below:
- Planning tools that provide transparency to stakeholders and participants
- Collaboration tools that help teams to work together, anywhere and anytime
- Source control tools are the single source of truth
- Issue tracking tools increase responsiveness and visibility
- Configuration management tools enforce the desired state
- Open source continuous integration tools work together
- Binary repositories manage builds, releases and dependencies
- Monitoring tools ensure service uptime and optimal performance
- Open source automated test tools provide higher quality
- Open source deployment tools improve time to market
Some of the key open source DevOps tools are briefly described below.
Ansible: Red Hat owns Ansible. It automates various common tasks related to IT operations such as application deployment, configuration management and cloud provisioning. It integrates with numerous DevOps tools including Jenkins, JIRA, Git and many others. The free open source version of Ansible is available on GitHub.
Chef: Chef is an open source automation platform that transforms infrastructure into code. It operates in the cloud, on-premise or in a hybrid environment. The Chef development kit provides the tools to develop and test infrastructure automation code from a local workstation prior to deploying changes into production.
Docker: Docker is software used for OS level virtualisation. It is used to create, deploy and run application packages called containers, which allow the developer to package an application with all the parts it needs, such as libraries and other dependencies, before shipping it as one package.
Docker has two parts. Docker Engine is a tool responsible for creating and running Docker Containers. Docker Hub is a service application based on the cloud, covering the concept of application sharing and workflow automation.
GitHub: This is a collaborative code review tool supporting around 200 software languages. It also supports all the version control features of check-in, commits, branches, merging, labels, task management, wikis, push and pull to/from GitHub, etc. Git fits in very well as a popular and distributed version control system for teams located at different geographical locations.
Many DevOps teams use it for managing the source code of their applications. It consists of plugins that can link with Jenkins to facilitate deployment and integration.
Hudson: This is a continuous integration tool developed in Java. This tool is used for managing, monitoring, continuous testing and integration. It supports various systems for the management of source code, application servers, code analysis tools, testing frameworks, build tools, real-time notifications of test failures, the changing of set support, and offers an easy process for installation and configuration.
Jenkins: Jenkins is a continuous integration tool that helps to automate the activities of build, code analysis and storing the artefacts. These activities are triggered once a developer or the team commits the code to the version control repository.
Jenkins has many plugins and works as a CI tool for various different languages like C/C++, Java/J2EE, .NET, Angular JS, etc. It also provides plugins to integrate with SonarQube for code review, JFrog Artifactory for storing binary artefacts, testing tools like Selenium, etc, as a part of the automation process.
Jenkins helps to automate deployments to app servers like Tomcat, JBoss and Weblogic through plugins and also to container platforms like Docker.
Kubernetes: Open source Kubernetes is free and downloadable from its repository on GitHub. Administrators must build and deploy the Kubernetes release to a local system or cluster; or to a system or cluster in a public cloud, such as AWS, Google Cloud Platform (GCP) or Microsoft Azure.
Puppet: This is a tool for operating and delivering software. It automates deployment to boost auditability, reliability and agility. It provides continuous automation and delivery across the complete software delivery life cycle. The latest version of Puppet features Node Manager and Puppet Apps, which help handle a large number of dynamic, variable systems.
Selenium: This is an automated, functional testing tool for Web applications. Installed as a Firefox browser plugin, it helps to record and playback test scenarios. In a DevOps scenario, once the application is deployed in a test environment, Selenium automated testing is invoked.
Supergiant: This is an open source platform for container management. It is used for Kubernetes deployment on multiple clouds in a matter of minutes. The Supergiant API is used for streamlining production deployments.
Benefits of DevOps tools
The high-level benefits of DevOps tools are:
- Reduce deployment times by 80 to 90 per cent, e.g., from 12 hours to 2 hours
- Reduce cost and time to delivery – they can be deployed often, faster and with repeatable, reliable processes
- Improve development productivity and overall quality by 20 per cent with automated and early detection of defects in the cycle
- Improved code coverage, from 30 per cent to 95 per cent
- Reduce regression cycle time from months to days
- Improved quality – automated testing reduces cost and time to test
- Increased service quality, reliability and the use of virtualised environments
- Reduced defective cycle time
DevOps tools shorten the time-to-market and improve quality. The DevOps toolchain improves productivity, time to market and reduces risks, facilitating collaboration between the product team, the IT development resources and operations staff through all the stages of the software development life cycle for seamless communication.
Just deploying the right set of tools to the right users will not serve the purpose, however. Proper connectivity between and across the DevOps toolchain is necessary to create a collaborative work environment for the development and operations teams. An integrated tools system saves considerable time and effort compared to manual processes.
If enterprises are looking to build enterprise DevOps toolsets, they should consider open source DevOps tools in addition to commercial tools.
