For years, the DevOps industry has tried to simplify how developers create automation scripts or workflows to automatically test a code change and to perform a task with the resulting artifact or binary. Today, we are introducing CI/CD Steps, a programming language for DevSecOps automation in experiment phase, as a solution to this challenge. With CI/CD Steps, software development teams can easily create complex automation workflows within GitLab.
The path to CI/CD Steps
Early in the company's history, GitLab founders and engineers decided that there must be a tight integration between source code management, the place you store your code, and continuous integration, the automation workflows that test your code changes. And we've continued to evolve that integration, focusing on workflow automation tasks and differentiating from the approaches of CI engines across the industry, including Jenkins CI's domain-specific language, GitHub Actions, and many more.
And, yes, I did mean to use the term workflow automation tasks rather than CI and continuous deployment (CD). This is simply a result of the code that I have seen our customers develop. In a lot of cases, the platform engineering teams that support development teams using GitLab are writing complex automation scripts (workflows). So we need to embrace a more expansive construct beyond simply CI and CD. In fact, I have seen some developers rave about the flexibility of new CI/CD solutions that allow for modularity and conditionals in writing automation workflows.
At GitLab, our initial approach for CI authoring was based on YAML. We can endlessly debate the pros and cons of such a choice, but for me, as a DevOps practitioner coming from a large Fortune 50 company with a moshpit of Jenkins Groovy code and hundreds of permutations of scripts basically performing the same job, the GitLab CI authoring and execution approach was a breath of fresh air.
The first time I read a GitLab CI file – this was back in mid-2019 – my first thought was, "No, it could not be that simple." A non-developer can easily grasp the intent of a basic GitLab CI pipeline without prior knowledge of all of the intricacies of the syntax of the execution model. In fact, I had just spent a year working on a team that spent several hours each day helping other development teams debug Jenkins pipelines written in Groovy and trying to figure out how to test, and in some cases build, large Java monoliths; in other cases, tons of microservices.
While there are benefits to a GitLab CI YAML-based authoring and a bash script execution type approach, there are also limitations. Limitations that developers or platform engineers bump into as they integrate more complex workflows into their CI pipelines. These issues seem to be amplified at enterprise scale as platform teams are trying to simplify or standardize workflows across multiple development teams. In fact, one of the quotes from a recent customer survey states: “GitLab needs to embrace a post-YAML world for CI.”
So, over the past two years, our pipeline authoring team, led by Product Manager Dov Hershkovitch, has been working extensively on improving the pipeline authoring experience. They've also been improving the management experience of the building blocks for workflow automation – especially at scale. In fact, a part of this work, the GitLab CI/CD Catalog, recently became generally available.
The logical next step was to build a new language for workflow automation.
Understanding CI/CD Steps
GitLab CI/CD Steps is a concept incubated by our top-notch engineers. In our documentation, we describe CI/CD Steps as reusable and composable pieces of a CI job that can be referenced in a GitLab CI pipeline configuration. But what does that really mean and what is the long-term value proposition?
As I was giving this some thought, a comment from one of our customers (paraphrased here) came to mind:
“CI Steps enables you to compose inputs and outputs for a CI/CD job. With CI/CD Steps, developers can define inputs and outputs and, therefore, use CI/CD Steps as a function as we do in any modern programming language. A key differentiator to a normal CI/CD component is that CI/CD Steps allows the use of the outputs of other steps without GitLab having to know certain values before running the pipeline. With CI/CD Steps, you could more easily auto-cancel redundant jobs when all jobs are running as part of the parent pipeline versus having to use child pipelines.”
Having CI/CD Steps alongside the current GitLab CI/CD execution mechanism and the CI/CD component catalog unlocks so many possibilities for creating and maintaining the most complex CI/CD workflows.
A key feature is reusability. Now, I am not suggesting that once we release CI/CD Steps as generally available, you would immediately start refactoring your currently working CI/CD jobs to CI/CD Steps. Instead, you likely will find opportunities to introduce CI/CD Steps to optimize complex pipeline workflows, and, in doing so, you will begin to reuse a CI/CD Step that you author in multiple pipelines.
CI/CD Steps is a marathon, not a sprint. When we release this in beta (currently targeted for late 2024) and start getting feedback from you, we will learn new information that will guide the evolution of this new CI programming language as well as the new Step Runner, which is designed specifically to run CI/CD Steps alongside the current CI/CD jobs.
I'm sure there will be questions about our strategy: Why did we make certain syntax choices? Why didn't we use Starlark as the basis for this new approach? Why did we create something new that we all have to learn? My boilerplate response is: At GitLab we develop our software in the open. More importantly, as a customer, user, and community member, if you have an idea of how to make it better, we invite you to create a merge request so we can improve this feature together.
We are the only enterprise software platform where, as users and customers, you have a direct say in how the platform evolves and you can see the changes happening transparently and in real time. That’s the power of GitLab – we iterate and we collaborate. You have invested in a platform and community that is able to evolve with the ever-changing software industry.
Create your own CI/CD step
To get a deeper understanding of CI Steps and our direction, take a look at the detailed refactoring proof-of-concept writeup in this issue. Principal engineer Joe Burnett walks through in great detail the thought process for refactoring a CI/CD job used as part of our GitLab Runner automated test framework. There are also recommendations noted at the end that will inform the evolution of the CI Steps syntax.
Then check out the CI/CD Steps tutorial and try creating your own CI/CD step. We recently released the run keyword, so testing out a CI/CD step will be simpler than previous examples that required using environment variables. This feature set is experimental so please share your experiences on the feedback issue. There also is a separate feedback issue if you are testing the Run GitHub Actions with CI/CD Steps experimental feature.
We look forward to working with you on this journey to continuously improve the GitLab CI/CD authoring experience.
