10 Useful Tools for DevOps Engineers

0
464
DevOps Engineers
DevOps Engineers

Here’s an overview of ten tools that can be used by DevOps practitioners.

DevOps is a culture and its implementation helps us to manage the application life cycle effectively by using automation. DevOps has three pillars:

  • People
  • Processes
  • Tools

Tools play a critical part in DevOps or culture transformation initiatives. However, people and processes help to maintain uniformity and sustainability across business units or projects in an organisation. DevOps enables faster time to market with high quality products.

I have used a few tools and believe they will be very important in the implementation of DevOps and DevSecOps practices in 2023 and beyond. Let’s take a brief look at what they offer.

1. GitLab

GitLab is the DevOps platform that provides features for different phases of software development, from planning to production. It helps to plan, develop, secure, and operate software in a single application. Its features include cache and artifact management.

GitLab is one of the most futuristic and forward-looking tools for the implementation of DevOps practices.

Initial release 2014
Stable release 15.7.1
Written in Ruby, Go and JavaScript
Licence Community Edition – MIT License
Website about.gitlab.com
GitHub repository https://gitlab.com/gitlab-org/gitlab
https://gitlab.com/gitlab-org/gitlab-foss/
Features
  • Highly scalable
  • Supports collaboration
  • Supports on-premise and cloud installation
How is it useful in DevOps practices implementation? DevOps platform

  • Version control
  • CI/CD
  • Package management
  • Resource management
  • Security
  • Distributed architecture
  • Cloud support
Can we integrate it with Pipeline as Code? Yes;
Gitlab-ci.yaml
Is a commercial flavour available? Yes
Pricing DevSecOps platform: https://about.gitlab.com/pricing/
Free, Premium, Ultimate
Community Number of wider community contributors: 778
Use cases
  • Digital transformation
  • Security and governance
  • Value stream management
  • Agile development
  • Source code management
  • Continuous integration (CI/CD)
  • Out-of-the-box pipelines
  • Security (DevSecOps)
  • GitOps
  • The DevOps platform
Important Git commands
Description Command
Set a user name git config –global user.the name “Mitesh”
Set email address git config –global user.email “xxxxxxxxx@xxxxx.com”
Initialise a Git repository git init
Clone a repository git clone https://github.com/xxxxxxxx/Angular9786-DevOps
Verify the status of files in the repository git status
Add files in the current directory for commit operation git add
Commit files git commit -m “First commit in the repository”
List your branches git branch
Switch branch git checkout
Create a new branch git branch new-branchname
Merge current branch with the target branch git merge target-branch
Show all commits git log
Save modified and staged changes git stash
Fetch and merge any commits from the remote branch git pull

 

 

2. Jenkins

Jenkins is an open source tool that provides integration with the existing tools used in application life cycle management to automate all activities — build, test, and deploy. It is almost a synonym for DevOps, and is one of the easiest to learn. It has a huge community for support. We can easily install Jenkins in the Kubernetes cluster and utilise multiple cloud services to make it highly available.

Jenkins provides a simple way to create a pipeline that includes continuous integration and continuous delivery. It can be used creatively for managing entire life cycle management activities easily. Plugin based architecture helps us to use the existing tools in the pipeline and hence too many changes in culture can be avoided in one go.

Initial release 2011
Stable release 2.375.1 LTS. More information at https://www.jenkins.io/changelog-stable/
Written in Java
Licence MIT
Website www.jenkins.io
GitHub repository https://github.com/jenkinsci/jenkins
Features
  • Jenkins installation is supported for Docker, FreeBSD, Gentoo, MacOS X, OpenBSD, OpenSUSE, Red Hat/Fedora/CentOS, Ubuntu/Debian, Windows, and Generic Java package (.war)
  • Multiple views are supported along with folders to manage jobs in an easier manner
  • New UI for Jenkins dashboard with ease of use
  • New user interface to create Pipeline as a Code in an easier way
  • Pipeline as a Code/multi-branch pipeline to manage pipelines easily in repositories
  • Distributed architecture to manage Jenkins effectively
  • Authorisation: Role-based access to users based on build jobs/folders/pipelines/configurations/actions
  • Authentication: Integration with active directory, LDAP and with internal database
How is it useful in DevOps practices implementation? Automation server that helps to automate activities involved in application life cycle management
Can we integrate it with Pipeline as Code? Pipeline as Code using Jenkinsfile

  • Declarative pipeline
  • Scripted pipeline
  • Pipeline as YAML
  • Blue Ocean
Is a commercial flavour available? CloudBees Jenkins
Awards
  • DevOps Dozen 2017: Best DevOps Open Source Project
  • DevOps Dozen 2016: Best DevOps Open Source Project
  • Bossie Awards 2014: The best open source application development tools
  • Geek Choice Award 2014
  • SDTimes 100, 2014: DevOps & SCM
  • SDTimes 100, 2013: Development Tools & CI
  • SDTimes 100, 2012: Development Tools & CI
  • InfoWorld Bossies Award, 2011
  • O’Reilly Open-Source Award, 2011
  • SDTimes 100, 2011: ALM&SCM
  • SDTimes 100, 2010: ALM Platforms
  • GlassFish Community Innovation Award 2008
  • Duke’s Choice Award
Third party integrations
  • More than 1500 plugins to integrate Jenkins with many open source and commercial tools (https://plugins.jenkins.io/)
  • Platforms (iOS development, .NET, Android development, Ruby development)
  • User interface (List view column plugins)
  • Administration (Agent controllers, Page decorators, Users and security, Cluster management, CLI extensions)
  • Source code management (SCM connections, SCM related)
  • Build management (Build triggers, Build wrappers, Build notifiers, Deployment plugins, Build parameters, Clean-up actions, Build tools, Build reports, Artifact uploaders)

3. SonarQube

SonarQube helps to clear bugs, vulnerabilities and code smell in the code. SonarQube 9.8 brings new rules across JavaScript, Kotlin, C++, and Python. It supports more than 20 programming languages for static code analysis (SCA).

It is much easier to integrate SCA using SonarQube in Pipeline as Code. It helps to transform the culture of an organisation from manual to automated code reviews with quality gates and quality profiles.

Initial release 2006-07
Stable release 9.8, https://www.sonarsource.com/products/sonarqube/whats-new/sonarqube-9-8/
Written in Java
Licence Lesser GNU General Public License
Website https://www.sonarqube.org/
GitHub repository https://github.com/SonarSource/sonarqube
Features
  • Support for the programming languages Java (including Android), C#, C, C++, JavaScript, TypeScript, Python, Go, Swift, COBOL, Apex, PHP, Kotlin, Ruby, Scala, HTML, CSS, ABAP, Flex, Objective-C, PL/I, PL/SQL, RPG, T-SQL, VB.NET, VB6, and XML
  • Quality gates
  • Quality profiles
How is it useful in DevOps practices implementation? Static code analysis
Can we integrate it with Pipeline as Code? Yes
Is a commercial flavour available? Yes

 

4. Docker

A Docker container image is a standalone package that contains all the dependencies required for running an application. This open source lightweight tool helps to create, deploy, and manage containers on a different host operating system using resource isolation features, such as cgroups and Linux kernels. Docker Desktop is a very helpful application for beginners and advanced users of MacOS, Linux, and Windows machines for building and sharing containerised applications and microservices. It can be used for free as part of a Docker Personal subscription.

A container can package application code, libraries, and configurations. The container engine is installed on the host OS.

Docker and Kubernetes have changed the game as Infrastructure as Code and Pipeline as Code have become a norm in recent times.

Initial release 2013
Stable release 20.10.22
Written in Go
Licence Apache License 2.0
Docker Community Edition
Website https://hub.docker.com/ https://www.docker.com/
GitHub repository https://github.com/docker
Features Easy management of applications
Docker Hub – Public Registry
Uniform packaging
Integrates well with Kubernetes
How is it useful in DevOps practices implementation? OS-level virtualisation, Containers
Can we integrate it with Pipeline as Code? Yes
Is a commercial flavour available? Docker Enterprise Edition
Docker commands
Description Command
Log in to the registry docker login
To build an image from the Docker file docker build -t sampleapp :1.0
List all the images docker images
Delete an image docker rmi imagename
Create a container docker run –name containerName dockerImage
Pull an image from the registry docker pull imageName
Push an image docker push imagename: tag
Clean and unused image docker image prune
Switch branch git checkout
Create a new branch git branch new-branchname
Merge current branch with the target branch git merge target-branch
Show all commits git log
Save modified and staged changes git stash
Fetch and merge any commits from the remote branch git pull

5. Trivy

Trivy is an open source yet simple and comprehensive vulnerability/misconfiguration scanner for containers and other artifacts. It can scan targets such as container image, file system, Git repository (remote), virtual machine image, Kubernetes, and AWS.
In my experience, it is easy to integrate Trivy scanning in Pipeline as Code and make sure that images are secure.

Stable release V0.36.0
Written in Go
Licence Apache-2.0 License
Website https://aquasecurity.github.io/trivy/v0.36/
GitHub repository https://github.com/aquasecurity/trivy
Features
  • Simple, fast and easy to install
  • Vulnerabilities detection for Docker images, Red Hat Enterprise Linux, CentOS, Oracle Linux, Debian, Ubuntu, Amazon Linux and other OS packages
  • Vulnerabilities detection for language-specific packages such as npm and maven
  • Secret and misconfiguration detection
How is it useful in DevOps practices implementation? To find vulnerabilities of Docker images;
DevSecOps
Can we integrate it with Pipeline as Code? Yes.
The pipeline stage can be used to analyse Docker images and publish the report
Example trivy image python:3.4-alpine
Docker image docker pull aquasec/trivy
Installation
  • apt-get install trivy
  • yum install trivy
  • brew install aquasecurity/trivy/trivy
  • docker run aquasec/trivy

6. Terraform

Terraform is an open source ‘Infrastructure as Code’ tool, created by HashiCorp, which helps us to create immutable infrastructure to avoid the problem of configuration drift. This platform-agnostic declarative coding tool helps developers use a high-level configuration language to manage infrastructure.

Terraform supports different cloud providers and different services provided by cloud providers, as listed below:

AWS Azure Google Cloud Platform Oracle Cloud Infrastructure
Alibaba Cloud VMware vSphere GitHub Kubernetes
Apigee GitLab Artifactory Octopus Deploy

 

Terraform Cloud is HashiCorp’s managed service offering that helps you with infrastructure automation.

In my experience, this is yet another tool that has changed the game.

Initial release 2014
Stable release 1.3.6
Written in Go
Licence Mozilla Public License v2.0
Website https://www.terraform.io/
GitHub repository https://github.com/hashicorp/terraform
How is it useful in DevOps practices implementation? Infrastructure as Code
Can we integrate it with Pipeline as Code? Yes.
It can be part of Pipeline as Code and different strategies can be used to create resources and deploy applications in different cloud platforms.
Important Terraform commands
Description Command
Verify Terraform version terraform –version
Install Terraform modules terraform init
Check configuration and create a plan terraform init -input=false
Execute the plan terraform plan
Delete all resources terraform apply
Validate/check the syntax of the Terraform files terraform apply -auto-approve
Reformat Terraform configuration in the standard style terraform destroy
Update the state to match remote systems terraform plan –destroy
Show the current state or a saved plan terraform validate
Show the current Terraform version terraform fmt
Workspace management terraform refresh
Show all commits terraform show
Save modified and staged changes terraform version
Fetch and merge any commits from the remote branch terraform workspace
Tutorials https://learn.hashicorp.com/terraform

 

7. Kubernetes

Kubernetes or k8s is an open source, extensible, and portable container orchestration platform founded by Google. It provides declarative configuration and automation for deployment automation. It has become one of the most popular tools in DevOps practices implementation with its features related to deployment, management, and scaling applications in Kubernetes clusters. It works as a booster in managing resources effectively.

We can create a Kubernetes cluster in an on-premise environment as well as different cloud deployment models such as private and public clouds. Public cloud provides managed services for Kubernetes. Google Cloud Platform has Google Kubernetes Engine, AWS has Amazon Elastic Kubernetes Services (EKS), while Azure has Azure Kubernetes Service (AKS) for managing Kubernetes cluster in a cloud environment.

Some major removals, deprecations and changes in version 1.26 are:

  • Based on inclusive naming initiative, the –prune-whitelist flag has been replaced with –prune-allowlist
  • Dynamic kubelet configuration has been removed from the API server
  • Deprecations for kube-apiserver command line argument –master-service-namespace
  • Deprecations for kubectl run command line arguments such as –cascade, –filename, –force, –grace-period, and so on
  • The v1beta1 flow control API group and the v2beta2 HorizontalPodAutoscaler API have been removed
Initial release 2014
Stable release 1.26
Written in Go
Go Apache License 2.0
Website https://kubernetes.io/
GitHub repository https://github.com/kubernetes/Kubernetes
Features
  • Distributed architecture
  • Easy integration with Docker
  • Supports hybrid, on-premise, or cloud
  • environments
How is it useful in DevOps practices implementation? Container orchestration and management.
Combined with Infrastructure as Code practice, automation can be achieved in an end-to-end manner for resource provisioning based on the support provided by IaC tool.
Can we integrate it with Pipeline as Code? Yes
Is a commercial flavour available? Cloud flavours are available such as EKS, AKS, and GKE
Important Kubernetes commands
Description Command
Get Kubectl Version kubectl version
Get details on Kubernetes cluster kubectl cluster-info
Get details about a node kubectl describe nodeName
Get details about a pod kubectl describe podName
List of all pods kubectl get pods
List of all services kubectl get services
List of all deployments kubectl get deployments
List one or more nodes kubectl get node
List all resources in the cluster kubectl get all –all-namespaces

8. Lens

Lens Desktop is an exciting platform for working with Kubernetes. It is a distribution of the OpenLens repository with specific customisations, and is free of charge. For more details, you can visit https://k8slens.dev/pricing.html.

Lens is extremely useful for all those who are managing infrastructure, as it gives a 360 degree view of all the resources available in a cluster along with description and logs. You can also view YAML files for deployments, services, pods, secrets, and so on.

Initial release 2019 (https://github.com/lensapp/lens/tags?after=v3.2.0-rc.1)
Stable release 6.3.0
Written in TypeScript, JavaScript, Makefile
Licence MIT License
Website https://k8slens.dev/
GitHub repository https://github.com/lensapp/lens
How is it useful in DevOps practices implementation?
  • Simplicity and increased productivity – no need to learn kubectl commands
  • Visibility in existing Kubernetes resources
  • Built on open source
  • Observability — live statistics, events, and log streams in real-time
  • Errors and warnings are directly available on the Lens dashboard
  • Supports EKS, AKS, GKE, Minikube, Rancher, k0s, k3s, OpenShift
  • Huge community support — 450,000 users and 17,000 stars on GitHub
Can we integrate it with Pipeline as Code? N/A.
Helps to view all the resources available in the Kubernetes Cluster.
Is a commercial flavour available? Lens Pro is a complete solution for cloud native professionals.
Monthly and annual billing included in Lens Pro:

  • Lens Desktop
  • Lens Teamwork
  • Lens Security
  • Lens Desktop Kubernetes
  • Community Forums and Slack
  • 24 x 5 commercial support

9. OWASP ZAP

In software security testing, we assess and verify a system against security risks and vulnerabilities. System security testing can be categorised as per the Open Web Application Security Project (OWASP) in the following way:

  • Vulnerability assessment, where scanning and analysing of security risks are performed
  • Penetration testing, where simulated malicious attackers attack the system and analyse it
  • Code review, where review and analysis of the system takes place to find vulnerabilities.

OWASP’s Zed Attack Proxy (ZAP) is a flexible, extensible and open source penetration testing tool, also known as a ‘man-in-the-middle proxy’. ZAP can intercept and inspect messages sent between browser and web applications, and perform other operations as well.

Automated pen testing is an important part of the CI/CD pipeline. Pen tests have stages such as explore, attack and report. OWASP ZAP fills the important gap of verifying applications and protecting them from multiple security threats.

Initial release 2014
Stable release 2.12.0
Written in Java
Licence Apache License 2.0
Website https://owasp.org/www-project-zap/
GitHub repository https://github.com/zaproxy/zaproxy
Features
  • Intercepting proxy
  • Active and passive scanners
  • Traditional and Ajax spiders
  • Brute force scanner
  • Port scanner
  • Web sockets
Risk categories
  • High
  • Medium
  • Low
  • Informational
  • False positive
How is it useful in DevOps practices implementation? Penetration testing
Can we integrate it with Pipeline as Code? Yes
Is a commercial flavour available? N/A
ZAP Docker image scan Docker stable image can be obtained using docker pull owasp/zap2docker-stable:

  1. ZAP – baseline scan: It executes the ZAP spider against the specified target for 1 minute and then completes the passive scanning.
    zap-baseline.py -t <target> [options]
  2.  ZAP – full scan: It executes ZAP spider against the specified target, an optional Ajax spider scan and then a full active scan.
    zap-full-scan.py -t <target> [options]
  3. ZAP – API scan: This is suitable to perform scans against APIs defined by OpenAPI, SOAP, or GraphQL
    zap-api-scan.py -t <target> -f <format> [options]

10. Grafana

Grafana is available as a fully managed tool with Grafana Cloud, or you can run the enterprise version on your infrastructure with self-managed options. You can visit publicly available dashboards from https://grafana.com/grafana/dashboards/ and import them in your Grafana instance to visualise data quickly.

In my experience, Grafana is one of the most useful tools while using Kubernetes on-premise or cloud providers such as AWS, Microsoft Azure or GCP. It supports Windows, Linux, and MacOS. It is easy to install, configure, and manage Grafana and monitoring metrics keeping Prometheus as a data source.

Initial release 2014
Stable release 9.3.1
Written in Go, TypeScript
Licence GNU Affero General Public License, version 3.0
Website https://grafana.com/
GitHub repository https://github.com/grafana/grafana
Supported data sources
  • Alertmanager
  • AWS CloudWatch
  • Azure Monitor
  • ElasticSearch
  • Google cloud monitoring
  • Graphite
  • InfluxDB
  • Jaeger
  • Loki
  • Microsoft SQL Server (MSSQL)
  • MySQL
  • OpenTSDB
  • PostgreSQL
  • Prometheus
  • Tempo
  • Testdata
  • Zipkin
Features Panels – heatmaps to histograms
Plugins – https://grafana.com/grafana/plugins/
Alerts
Annotations
How is it useful in DevOps practices implementation? Visualisation of monitoring metrics. Another important thing to note is that you can convert the logs of third party tools into normal logging such as Google Cloud Logging -> Create Log Router Sink to route logs to supported Google services.
Is a commercial flavour available? https://grafana.com/pricing/

LEAVE A REPLY

Please enter your comment!
Please enter your name here