Nowadays you will hear that if you want your SDLC (Software Development Lifecycle) to be efficient, you need to have a proper, well-established CI/CD setup. In most cases, it’s true. In this blog post I am going to focus on how to go about creating such a setup when creating a new CI/CD pipeline and optimizing an existing one. I will also look at best practices to follow and outline errors to avoid. Finally, I will analyze the possible reasons why your CI/CD setup may not work as expected and propose solutions to turn things around.
What is a CI/CD pipeline—should I call a plumber?
My colleague Maciej does a good job succinctly explaining what CI/CD is, while Wikipedia delves further into the topic. With such good sources at the ready, there is no need to go into further detail about it here. But what the heck is this pipeline thingy? The easiest explanation would be to call it a group of events or jobs that are connected together in a sequence. However, it's a bit more complicated than that as each stage or group of events/jobs is dependent upon the output of the previous stage. In other words, all stages are grouped into a sequence, where the output of a stage triggers the next stage to start. Of course, events or jobs in such stages of groups may also be acyclic (see Figure 1).
Figure 1. CI/CD pipeline—a diagram
If Stage 2 has failed, then Stage 3 will also fail. The entire pipeline is all about creating such an automated setup in which every step will trigger its successor. From building the code, through automated tests to automated deployment—it’s a complex process, yet one that is both fast and reliable.
Can I have a CI/CD, please?
Everyone has their own CI/CD, so naturally, you want one, too, right? But are you sure you need one? This can be a complicated question to answer. First, you should outline your workflow and decide about your development environments. Then, think carefully and answer the following questions:
- How often are you going to release your apps?
- Where and how often do you want to deploy your applications?
- Do you want or need to create end-to-end automation?
- Do you want or need to use orchestration tools?
- Do you want to use Chat/Ops or deploy from your slackbot?
If your answer to these questions is mostly yes, you can start planning the first step. If speeding up your development life cycle and delivering more consistent product is your goal, the Continuous Integration/Continuous Delivery approach will definitely be the right scenario for your organisation. Just remember the correct direction: CI feeds CD. This process is complex and involves quality assurance engineers, devops team, ops engineers, scrum masters and project managers, among others. Smooth collaboration is a key to success and cannot be automated.
-> Want to learn more about CI/CD? Check out our other articles:
- What is CI/CD - all you need to know
- Best CI/CD pipeline tools you should know
- Continuous monitoring and observability in CI/CD
- CI/CD process—how to handle it
- Business benefits of CI/CD
- Sharing configuration between your CI, build and development environments
- How to build a test automation framework in the cloud
CI/CD pipeline best practices
Choosing the right tools strongly depends on your application specifics, but before you have the tools in place, think about the people and process. It is definitely a good idea to have an evangelist in your organization, a person to actively explain and promote this new approach among the teams. The evangelist will ensure that everybody understands the reasons behind it and is fully committed to the process.
When it comes to choosing tools used in your first CI/CD pipeline, some best practices will cover the vast majority of use cases.
- You need a version control system (while that may well seem obvious, this advice isn’t always followed): once your code is pushed to the repository, the system will automatically trigger the entire pipeline. Thus pushing the code to the repo will be the pipeline trigger point.
- Conduct a careful analysis of the available CI/CD tools (one of many handy comparisons can be found here) and choose the one that suits your needs.
- Make sure that you have at least two separate configurations, one for development and one for production. Keep your environments defined as code.
- In your testing suite, run the quickest tests first in order to shorten the feedback loop. Then move to the more time-consuming tests (this approach is called “fail fast”). An example of a proper flow could be as follows: code quality → unit tests → build → staging deployment → e2e test.
- Focus on implementing CI first. After the CI is proved to be reliable, move on to set up the CD part.
- Once you implement monitoring, compare the velocity of the team before and after setting up the CI/CD pipeline. This will tell you if the new approach has improved efficiency.
Once you have done the above, follow the guidelines for specific environments, like for example Kubernetes (k8s). Also, don’t hesitate to look at CI/CD from different perspectives, especially security, but also the tool has to be carefully chosen from a usability point of view (you’ll be using it day-to-day) and again the choice would strongly depend on your individual requirements.
CI/CD pipeline—things to avoid
Of course, there are also things that should be avoided when creating your first pipeline. Remember that over-automation is a myth, yet you need to think carefully if you really need everything to be automated. Before moving forward with any automation, ask yourself the following questions:
- How often is the scenario repeated?
- How long is the process?
- What people and resource dependencies are involved in the process? Are they causing delays in CI/CD?
- Is this process error-prone and automation would help?
- How urgently does this process need to be automated?
Answering these questions will allow you to understand what process should be automated first.
It is easy to confuse continuous deployment with continuous delivery. Here’s a quick look at what each of these terms mean:
- Continuous delivery - each code-base change goes through the pipeline up to the point where it is deployed to the non-production environments.
- Continuous deployment - each code-base change will be deployed pretty much immediately into the production environment upon the successful result of the whole pipeline.
Both approaches have their own distinct characteristics. While they can be combined, they mustn’t be confused.
Needless to say, a failure to properly monitor the crucial metrics could well cause you to detect issues only after it’s too late to fix them. Moreover, a dashboard in the form of a single pane of glass would allow all stakeholders to gain good insight into what is actually happening in the entire CI/CD pipeline (e.g. compliance code stats, release velocity, etc.).
Last but not least, CI feeds CD. There are many stakeholders involved in the CI/CD process: quality assurance, devops team, ops engineers, scrum masters, project managers, etc. You should do your best to foster smooth communication and collaboration between these people, as it cannot be automated.
Why does my CI/CD pipeline not work like I expect it to?
So, you have your stack working, but you can’t trumpet success just yet. It takes hours to create builds and they often fail. Worse still, you have no clue where the problem may lie. For most users, the time it takes to create a build tops the list of frustrating issues. To ensure your pipeline is efficient enough, you need to be able to plan and manage your software delivery lifecycle. Once you have one in place, you can create, verify and package your software that needs to be secured and released and should thereafter be monitored and defended against security threats.
Below you can find some hints and best practices on how to optimize your CI/CD pipeline setup.
- Build only once - the artifacts (the binary result of a build) should be built once and only once. That is to ensure that you have a clean environment in CI, and can trust the build. If you try to rebuild binaries, it’s difficult to make sure that the one you are building and the one passing CI into production are one and the same.
- Reduce branching through small and incremental commits - all changes should be merged to the main repository and branching should be minimized. Encourage developers to commit atomically and implement features incrementally. Educate your people how to make pull requests that are easy to run through a code review, and enforce them to merge as soon as possible.
- Plan test priority carefully - to be able to fail fast, prioritize which test should go first. This will help you test CI itself. For example, you can arrange some smaller and quicker tests (e.g. unit test) at an early stage.
- Use version control for everything (including documentation, deployment and configurations) - this is one of the must-have elements in every CI/CD. Your developers should separate build artifacts by versioning them. This will enable you differentiate each of the versions as well as to audit the changes.
- Isolate CI/CD pipeline (for security reasons) - this one should be obvious: you don’t want your codebase and internal data being exposed to the public, do you? We strongly recommended isolating the CI/CD pipeline as it will be from there that you have full access to sensitive data such as your codebase and credentials.
- Deploy via CI/CD only - if you keep using other channels for deployment and bypassing CI/CD, you'll be exposed to human error (deploying by mistake other versions that have not been tested). A CI/CD pipeline is the best way to ensure integrity while testing and deploying software.
- Consider using container technology for testing - it will help you set up a testing environment swiftly and at the same time ensure the cleanliness of your environments.
You can make things even better, if you do the following. First, automate QA. In many organisations quality assurance is a real bottleneck. This is not the fault of the people, but the environment-provisioning process in which they are supposed to carry out their job. Some remediation here is to fully automate the QA process, from QA-ready code coming in the door to one-click promotion on the other side. In between the two, QA should have a constant feedback loop with development where communication about and tracking of bugs is clear and easy to follow, allowing development to easily repair issues and recommit the code for further testing. Automation and communication are the overriding imperatives here. They will be exhibited by reports to developers that are communicated immediately and code commits that are fail-rejected automatically.
Use identical environments. Development and testing environments should mirror production as closely as possible. It should be quick and easy to set up new environments, to the point where developers can, with the push of a button, deploy new environments for their development and testing needs. That will result in increasing efficiency and effectiveness and will allow developers to fail fast, breaking and re-creating environments without worrying that they’re wasting time.
The next phase of automation and step towards streamlining the entire process is to ensure one-click promotion to the production environment. This allows changes to be seamlessly moved to the last stage (overall production environment) with as little friction as possible between the development and operations teams. To that end, this reduction in friction should occur across all the steps of the process, with efficient code delivery pipelines that are designed to balance security and agility from the outset. Cloud infrastructure coupled with container technology and automation can easily optimize the process across teams and throughout the DevOps journey.
Finally, fully automate deployments. Full automation is the ultimate attribute of an efficient and mature CI/CD pipeline. All configuration changes made by development and accepted by QA should be applied automatically. In this way, continuous integration helps to avoid system downtime and the number of human errors decreases considerably. Needless to say, deployments should be continuously monitored to detect potential problems before your clients inform you about them. This would also allow for faster rollback and code returned to development so anything not functioning properly can be addressed. The use of canary releases is considered a best practice. Depending on your application, such releases may give you many precious insights and keep your reputation intact in the clients’ eyes. The final phase is to reach feature toggles and dark launches.
Still not there?! Where’s the glitch?
There will be times when you believe you’ve done everything correctly, but something is still not working right. At such times, your pipeline and feedback loop are most probably still slow. To solve this problem, review the design of your pipeline setup. While this is a subject for another article, here are some hints on how to ensure that the important areas are working efficiently, look for pain points and speed up your feedback loop with developers.
- Fail fast - feedback has no preventative value when it comes late. Look carefully at each stage of your pipeline and make sure none of the tests are overdone (especially limit e2e tests as they are often a main source of slowness). It is better to optimize for failure, expect it and embrace it.
- Monitoring - your CI/CD pipeline needs to be properly monitored. Look for the right toolstack (ELK, Splunk, Prometheus + Grafana, etc.) that will suit your needs and give you actionable insights into your CI/CD setup. Once that is in place, focus on fine tuning it for tracking trends that will allow you to act proactively.
- Post mortems - make sure to revise every issue retrospectively and collect all necessary data. This data needs to be stored and analyzed carefully and the conclusions drawn should be turned into meaningful action items so that nothing is missed.
CI/CD pipeline—an ideal flow
Throughout the whole planning phase, you will definitely have to reply to questions about tools to use (open-source or proprietary) as well as whether to use cloud service or self-hosted solution. The right answer will definitely help you get the most out of your CI/CD pipeline. You should remember that small but more frequent releases are much easier to conduct and considerably less risky than big-bang integrations done ‘by hand’. Bear this fundamental principle in mind when designing your pipeline. Of course, there is no universal model that would work in every scenario. Nonetheless, I will briefly outline the most common models.
With on-prem solutions, small-scale implementations in particular benefit from setup simplification such as one gets with vendor solutions offered by the likes of GitLab or Atlassian. The most popular cloud-based CI/CD solutions are GitLab, Bitbucket Pipelines, Azure PipeLines and AWS CodePipeline. Different scenarios may require a variety of tool coupled together. One of possible examples is Zuul CI, which was used for Tungsten Fabric. Bear in mind that sometimes it is impossible to limit vendors, for example when you need a fully fledged artifact repository (i.e Artifactory) in your CI/CD pipeline or a dedicated tools for code analysis like SonarQube (Code Quality and Security) and/or BlackDuck (Open Source Security & License Compliance). As every scenario is different, before making your final decision about tools, do the research to gain a better overview of what is available on the market. Still, regardless of your choice, when designing or optimizing your CI/CD pipeline, keep in mind these three DevOps-endorsed principles:
- Work always flows in one direction—downstream
- Create, shorten and amplify feedback loops (if necessary, do some stages in parallel)
- Do the experiments to learn from mistakes and achieve mastery
CI/CD is an iterative process that may seem overly complex and difficult to manage when you are looking at it from a wider perspective. It is therefore wise to start with an element that is small and simple, yet the most problematic in a given process. Then you can add new elements from there.
