1. Overview
In this article, we’ll understand the basics of DevOps principles and practices. We’ll see why this is relevant and helpful in software development. We’ll also understand how we can adopt DevOps meaningfully and what tools are there to help us along this journey.
2. Historical Context
We won’t be able to appreciate DevOps as it stands today without looking back into history a little bit. The early days of software development were mostly characterized by what we call waterfall methodology. What this effectively means is that software was conceptualized, designed, developed, tested, and distributed in succession.
Every step was as detailed as possible, as going back was very costly. What this effectively meant was a much higher waiting period between thought and action. However, this was not such a problem as the technology landscape was much less volatile and disruptions far too spread.
Interestingly, this model didn’t last long. As the pace of technology changed and disruptions started to happen often, businesses started to feel the heat. They needed new ideas to be tested faster. This meant faster changes in all aspects of the business, including software.
This gave birth to a whole new world of software development methodologies that are loosely seen under the umbrella of Agile. The agile manifesto sets forth a set of principles to be followed for software delivery in small increments with a faster feedback loop. There are several agile frameworks like Scrum and Kanban in practice.
3. What Is DevOps?
We’ve seen that incremental development with faster feedback has become the cornerstone of software delivery today. But how do we achieve that? While traditional agile methodologies take us to a reasonable point, is it still not ideal.
Agile methodologies keep refining themselves as they continuously strive to break silos.
Traditionally, we always had different teams that were responsible for developing and delivering software. These teams often operated in their silos. This effectively translated into a much longer feedback cycle, which is not something we desire with agile methodologies.
So, it doesn’t require a lot of reasoning to understand that well-integrated, cross-functional agile teams are much better suited to deliver their objectives. DevOps is the practice that encourages communication, collaboration, integration, and automation between software development and operations teams. This better enables us to realize incremental development with faster feedback.
The following diagram explains a possible workflow for practicing DevOps:
While we will go through the details of these steps later in the tutorial, let’s understand some of the key principles of DevOps:
- Value-centric approach (as realized by end-user)
- Collaborative culture (with effective communication, processes, and tools)
- Automation of processes (to enhance efficiency and reduce errors)
- Measurable outcomes (to measure against the goals)
- Continuous feedback (with a tendency to improve quicky)
4. How to Start the Journey?
While the theory is straightforward and appealing, the real challenges are in practicing DevOps meaningfully. As we’ve gathered so far, DevOps is mostly about people, rather than teams.
Common objectives, effective communication, and cross-functional skills are the hallmarks of such teams. Since a large part of this change is cultural, it’s often slow and not without friction.
4.1. Motivation
Just because there’s a popular practice out there does not necessarily make it suitable for us. We need to understand our motivation for any shift — more so if we’re making a change towards agile. It’s useful to set out by defining the goals we want to achieve.
The goals of DevOps in any organization are dependent on that organization’s ambition, culture, and maturity. Here are some of the more common DevOps goals:
- Better experience to end-users
- Faster time to market
- Improved mean time to recovery
4.2. Adoption
Remember that DevOps is not an end state but a continuous process of improvement to achieve the goals. Hence, everyone on the team must strive to identify impediments and remove them swiftly. Here are a few activities that can help us get started:
- Clearly understand the current state of ideation to the production cycle
- Gather some of the obvious bottlenecks and use metrics to make factual decisions
- Prioritize the bottlenecks that will add the most value when removed
- Define an iterative plan to deliver value incrementally for prioritized items
- Follow the short cycles of Develop-Deploy-Measure to achieve the goals
5. DevOps Practices
There are several practices to follow, but the idea should not use any as a gold standard. We should carefully examine every practice in the background of our state and objectives and then make informed decisions. However, almost all the practices tend to focus on automating processes as much as possible.
5.1. Agile Planning
Agile planning is the practice of defining the work in short increments. While the end objective should be clear, it’s not necessary to define and detail the entire application upfront. The key here is to prioritize work based on the value it can deliver.
Then, it should be broken in an iteration of short but functioning increments.
5.2. Infrastructure-as-Code (IaC)
This is the practice of managing and provisioning infrastructure through machine-readable configuration files. We also manage these configurations in a version control system like we manage our codebase. There are many domain-specific languages available to create these configuration files declaratively.
5.3. Test Automation
Software testing has been traditionally a manual effort often conducted in silos. This does not marry well with agile principles. Hence, it’s imperative that we try to automate software testing at all levels, such as unit testing, functional testing, security testing, and performance testing.
5.4. Continuous Integration (CI)
Continuous integration is the practice of merging working code more often in small increments to a shared repository. Usually, there are automated builds and checks frequently running on this shared repository to alert us of any code breaks as soon as possible.
5.5. Continuous Delivery/Deployment (CD)
Continuous delivery is the practice of releasing software in small increments as soon as it passes all checks. This is often practiced together with Continuous Integration and can benefit from an automated release mechanism (referred to as Continuous Deployment).
5.6. Continuous Monitoring
Monitoring – perhaps the center of DevOps – enables faster feedback loops. Identifying the right metrics to monitor all aspects of the software, including infrastructure, is crucial. Having the right metrics, coupled with real-time and effective analytics, can help identify and resolve problems faster. Moreover, it feeds directly into the agile planning.
This list is far from complete and is ever-evolving. Teams practicing DevOps are continuously figuring out better ways to achieve their goals. Some of the other practices worth mentioning are Containerization, Cloud-Native Development, and Microservices, to name a few.
6. Tools of the Trade
No discussion on DevOps can be complete without talking about the tools. This is one area where there has been an explosion in the last few years. There may be a new tool out there by the time we finish reading this tutorial! While this is tempting and overwhelming at the same time, it’s necessary to exercise caution.
We mustn’t start our DevOps journey with tools as the first thing in our minds. We must explore and establish our goals, people (culture), and practices before finding the right tools. Being clear on that, let’s see what are some of the time-tested tools available to us.
6.1. Planning
As we’ve seen, a mature DevOps always starts with agile planning. While clear on the objectives, it’s only necessary to prioritize and define work for few short iterations. The feedback from these early iterations is invaluable in shaping the future ones and the entire software eventually. An effective tool here would help us exercise this process with ease.
Jira is a top-rated issue tracking product developed by Atlassian. It has a lot of built-in agile planning and monitoring tools. Largely, it’s a commercial product that we can either run on-premise or use as a hosted application.
6.2. Development
The idea behind agile is to prototype faster and seek feedback on the actual software. Developers must make changes and merge faster into a shared version of the software. It’s even more important for communication between team members to be fluid and fast.
Let’s look at some of the ubiquitous tools in this domain.
Git is a distributed version control system. It’s fairly popular, and there are numerous hosted services providing git repositories and value-added functions. Originally developed by Linus Torvalds, it makes collaboration between software developers quite convenient.
Confluence is a collaboration tool developed by Atlassian. Collaboration is the key to success for any agile team. The actual semantics of collaboration is pretty contextual, but a tool that makes an effort seamless is nevertheless invaluable. Confluence fits this spot accurately. Moreover, it integrates well with Jira!
Slack is an instant messaging platform developed by Slack Technologies. As we discussed, agile teams should be able to collaborate and communicate, preferably in real-time. Apart from instant messaging, Slack offers many ways to communicate with a single user or a group of users — and it integrates well with other tools like Jira and GitHub!
6.3. Integration
Changes merged by developers should be continuously inspected for compliance. What constitutes compliance is specific to team and application. However, it’s common to see static and dynamic code analysis, as well as functional and non-functional metric measurements, as components of compliance.
Let’s look briefly at a couple of popular integration tools.
Jenkins is a compelling, open-source, and free automation server. It has been in the industry for years and has matured enough to service a large spectrum of automation use cases. It offers a declarative way to define an automation routine and a variety of ways to trigger it automatically or manually. Moreover, it has a rich set of plugins that serve several additional features to create powerful automation pipelines.
SonarQube is an open-source continuous inspection platform developed by SonarSource. SonarQube has a rich set of static analysis rules for many programming languages. This helps detect code smells as early as possible. Moreover, SonarQube offers a dashboard that can integrate other metrics like code coverage, code complexity, and many more. And, it works well with Jenkins Server.
6.4. Delivery
To deliver changes and new features to software quickly is important. As soon as we’ve established that the changes merged in the repository comply with our standards and policies, we should be able to deliver it to the end-users fastly. This helps us gather feedback and shape the software better.
There are several tools here that can help us to automate some aspects of delivery to the point where we achieve continuous deployment.
Docker is a prevalent tool for containerizing any type of application quickly. It leverages OS-level virtualization to isolate software in packages called containers. Containerization has an immediate benefit in terms of more reliable software delivery. Docker Containers talk to each other through well-defined channels. Moreover, this is pretty lightweight compared to other ways of isolation like Virtual Machines.
Chef/Puppet/Ansible are configuration management tools. As we know, an actual running instance of a software application is a combination of the codebase build and its configurations. And while the codebase build is often immutable across environments, configurations are not. This is where we need a configuration management tool to deploy our application with ease and speed. There are several popular tools in this space, each having their quirks, but Chef, Puppet, and Ansible pretty much cover the bases.
HashiCorp Terraform can help us with infrastructure provisioning, which has been a tedious and time-consuming task since the days of private data centers. But with more and more adoption of cloud, infrastructure is often seen as a disposable and repeatable construct. However, this can only be achieved if we’ve got a tool with which we can define simple to complex infrastructure declaratively and create it at the click of a button. It may sound like a dream sequence, but Terraform is actively trying to bridge that gap!
6.5. Monitoring
Finally, to be able to observe the deployment and measure it against the targets is essential. There can be a host of metrics we can collect from systems and applications. These include some of the business metrics that are specific to our application.
The idea here is to be able to collect, curate, store, and analyze these metrics in almost real-time. There are several new products, both open source, and commercial, which are available in this space.
Elastic-Logstash-Kibana (ELK) is a stack of three open-source projects — Elasticsearch, Logstash, and Kibana. Elasticsearch is a highly-scalable search and analytics engine. Logstash provides us a server-side data processing pipeline capable of consuming data from a wide variety of sources. Finally, Kibana helps us visualize this data. Together, this stack can be used to aggregate data like logs from all the applications and analyze them in real-time.
Prometheus is an open-source system monitoring and alerting tool originally developed by SoundCloud. It comes with a multi-dimensional data model, a flexible query language, and can pull time-series data over HTTP. Grafana is another open-source analytics and monitoring solution that works with several databases. Together, Prometheus and Grafana can give us a real-time handle on pretty much any metric that our systems are capable of producing.
7. DevOps Extensions (Or are they really!)
We have seen that DevOp, fundamentally, is a continuous effort to remove impediments towards faster and iterative value-based delivery of software. Now, one of the immediate conclusions is that there can not be an end-state here.
What people realized as friction between development and operations teams is not the only friction. Breaking silos within an organization to increase collaboration is the central idea. Now, people soon started to realize that similar frictions exist between development and testing teams, and between development and security teams. Many traditional setups have dedicated security and performance teams.
The full potential of DevOps can never be realized until we can break almost all boundaries between teams and help them collaborate much more efficiently. This inherently means bringing teams like testing, security, and performance into the fold.
The confusion is largely in its nomenclature. DevOps makes us understand that it’s mostly about development and operations teams. Hence, over time, new terms have emerged, encompassing other teams. But largely, it’s just DevOps being realized more effectively!
7.1. DevTestOps
The cornerstone of DevOps is delivering high-quality software in small increments and more often. There are many aspects to the emphasis on quality here. In a sense, we often assume that the DevOps practices we adopt will help us achieve this. And, it’s also true that many of the practices we discussed earlier focus on ensuring high quality at all times.
But functional testing of software has a much wider scope. Quite often, we tend to keep the higher-order testing like end-to-end testing towards the end of software delivery. More importantly, this is often the responsibility of a separate team that engages late in the process. This is where things start to deviate from the DevOps principles.
What we should rather do is to integrate software testing, at all levels, from the very beginning. Right from the planning stage, software testing should be considered an integral aspect of delivery. Moreover, the same team should be responsible for developing and testing the software. This is what the practice of DevTestOps is widely known as. This is often also referred to as Continuous Testing and Shifting Left.
7.2. DevSecOps
Security is an integral part of any software development and has its share of complexity. This often means that we have a separate team of security specialists who we engage with right when we’re ready to ship the product. The vulnerabilities they identify at this stage can be costly to fix. This again does not resonate well with the principles of DevOps.
By this time, we should already have the solution we need to apply, and that is we should bring on the security concerns and personnel early in the game. We should motivate teams to think about security at every stage. Security is no doubt a very specialized domain, and hence we may need to bring in a specialist within the team. But the idea here is to consider some of the best practices right from the beginning.
As we move along, there are several tools available that can automate scanning for a bulk of vulnerabilities. We can also plug this into our Continuous Integration cycles to obtain fast feedback! Now, we can’t integrate everything into the Continuous Integration as we must keep it light, but there can always be other periodic scans running separately.
8. Conclusion
In this article, we went through the basics of DevOps principles, practices, and tools available for use. We understood the context where DevOps is relevant and the reasons it can be of use to us. We also discussed briefly where to begin in the journey of adopting DevOps.
Further, we touched upon some of the popular practices and tools available for us to use in this journey. We also understood some of the other popular terms around DevOps like DevTestOps and DevSecOps.
Finally, we must understand that DevOps is not an end-state, but rather, a journey that may never finish! But the fun part here is the journey itself. All the while, we must never lose sight of our goals, and we should focus on the key principles. It’s quite easy to fall for the shine of a popular tool or term in the industry. But we must always remember that anything is only useful if it helps us deliver value to our audience more efficiently.