youri babakhanians - Fotolia
Changes in the CI/CD pipeline
Continuous integration/continuous delivery is responsible for pushing out a set of “isolated changes” to an existing application. We look at how big these changes can be
A core advantage gained through continuous integration/continuous delivery (CI/CD) environments is the ability to garner immediate feedback from the user base. This, in theory, enables software development teams to develop an application’s functionality “on the fly”, with each new feature put into what is known as a “branch” of the “master” source code tree.
In Forrester’s Tech tide for continuous software delivery Q4 2019 report, the analyst firm says the adoption of such technologies is a crucial step to increase development team efficiency, agility, and business alignment.
“Shifting away from protracted upfront planning and manual repetitive development steps saves teams time and money,” wrote Forrester analyst Diego Lo Giudice. “It empowers them to iterate quickly over smaller changes and adopt new practices like creating minimal viable products.”
Lo Giudice says application development and deployment teams can quickly gain customer feedback to assess whether their work is delivering the business value they expect. “If it doesn’t, continuous software delivery (CSD) permits teams to nimbly redirect their efforts and deliver business value sooner,” he wrote.
The Computer Weekly Developer Network recently ran a number of guest blog posts from industry experts looking into contemporary CI/CD practices and methodologies. The experts agree that there is a strong link between CI/CD and DevOps.
Ido Benmoshe, vice-president of product and solution consulting at Perforce Software, says CI/CD is aligned with agile iterative development. However, it is a build, test and release automation process that complements an agile development process. “Smaller increments in development make CI/CD more efficient and allows faster feedback,” says Benmoshe.
According to Paul Farrington, Europe, Middle East and Africa (EMEA) CTO at Veracode, in live production environments, different CI/CD release patterns can be extremely powerful for DevOps teams. For instance, he says, isolated code can be controlled by feature flags and routing rules that limit how many users experience new code. The “canary release pattern” is a popular example technique, as it allows deployments to take place at any time of day, but the user experience is carefully controlled so that, initially, only a trickle of users are exposed to new functionality. The gradual increase of flow of users can be controlled by scripts that look for errors appearing in logs.
Read more about CI/CD
To realise the benefits of Agile for infrastructure, network teams must think differently about how to deliver services -- embracing radical change and a trial-by-error approach.
Before embracing a NoOps model, be sure IT team members have a clear understanding of how the methodology changes the technologies they'll work with, along with their day-to-day roles.
“Based on certain tests, the roll-out could be automatically paused or rolled back, without engineers needing to be paged in the middle of the night,” says Farrington. “These tests might also include automated security scans, for example with dynamic analysis, which could conceivably ‘vote’ for a roll-back if say, an SQL injection flaw is found on the main login page.
“Once the isolated code has passed a series of tests during the release pipeline and once in production, it can then be allowed to survive if negative test results are below a designated threshold.”
CI/CD relies on new functionality being pushed through an automated workflow, known as a pipeline, which runs software tests and quality assurance, before the code is deployed. At any stage in this pipeline, the work can be rolled back or, to use industry parlance, “shifted left” – back to the individual team responsible for the previous step in the workflow.
Farrington says developers typically write unit tests to help validate the correctness in a software function. These can range in complexity, but in essence, they should be deterministic. Such tests could be used to validate that a function correctly sums inputs properly, or perhaps that it sorts a range of data points in the expected way. “Because of their nature, unit tests will usually run quickly and are a normal part of the build process, fitting well within a CI/CD pipeline,” he adds.
In Farrington’s experience, developers are increasingly using tools to perform security unit tests on code changes. He says functional tests can also be undertaken within a CI/CD pipeline, but will require more time to complete. “Depending on the nature of the project, the development team may elect for some functional tests to be performed following deployment, while retaining the ability to roll back the release if need be,” he says.
The experts Computer Weekly spoke to recommend that application teams launch a CI pipeline using the smallest code increments possible. Jason Lenny, director of product management, CI/CD at GitLab, says: “Smaller changes mean code is easier to review, easier to test and easier to roll back when there is a problem. This can result in fewer bugs and reduced defects in production. But if you aren’t reviewing, aren’t testing adequately, and aren’t monitoring to know when to roll back, then you won’t realise these benefits.”
Keeping tight control of changes
In practice, a CI/CD pipeline is fed each time new code is checked into a source code repository or at a high frequency that suits the specific application, such as hourly or daily. Benmoshe says: “However, keeping tight control over the number of changes made to each iteration requires developers to be disciplined – and even minor changes may still have an impact on critical application functionality. Reducing the chance of integration issues requires rigorous testing at multiple tiers – and it requires a high percentage coverage of unit, static code and functional testing, as well as load and endurance testing.”
GitLab’s Lenny says: “A key limiting factor is how long it takes your pipeline to complete. If you are deploying more frequently than your total pipeline time, you run the risk of breaking the master (source branch). The potential for conflict arises when additional changes get merged while a pipeline is still running on a previous change. High velocity means everything can pass in staging, but still fail in master. Very sophisticated engineering organisations like Google, Facebook and Uber mitigate this by building complex custom functionality to queue and sequence changes, ensuring that once a pipeline starts, other code can’t get merged ahead of that change.
As Marco Rottigni, chief technical security officer EMEA at Qualys, points out, CI/CD pipelines rely on automation in order to achieve results and deliver releases out quickly. He says: “Security teams can support this process through providing data into that pipeline to guide developer teams when it comes to improvements that are needed and where potential issues have been discovered. Automating the discovery process around new vulnerabilities is one option, but it can quickly lead to security alert overload.”
With so many changes coming through, it is no longer sufficient for IT security to act as a gatekeeper at the end of the process. Instead, says Rottigni, security should be embedded into the whole approach to software development.
“Embedding security into development involves making the tools used for tracking security vulnerabilities easy to consume within the software development pipeline,” he says. “Rather than providing these tools separately, they should instead provide any necessary updates or recommendations directly into the developer environment. This makes it easier for the developers to consume this information in the same way as other bugs or feature requests, then go about fixing them.”
Similarly, log management has a role to play in the continuous improvement of application functionality in a CI/CD pipeline. Mark Pidgeon, vice-president of technical services at Sumo Logic, says: “Each stage of the development process will create data, which can be tapped into to support a more accurate feedback loop for development, for deployment and for business teams.”
According to Pidgeon, linking data sets from application components ahead of release into production provides a strong “before and after” set of data. This, he says, can be useful for looking at the impact of changes as part of any move to production. “The data is created continuously during any development, and can be used over time for making better decisions across the software process,” he says.
Not every company is a software company, but CI/CD promises to help organisations build and release code far more quickly than they are used to. IT decision-makers need to ensure CI/CD is run using small incremental code changes, where feedback from customers can quickly be incorporated into the next code release. The CI/CD pipeline requires strong links between the tools for coding, all aspects of testing, deployment and post-deployment analysis to ensure apps built using CI/CD are continuously improved.
