I've always found it fascinating that my father, a heavy-duty mechanic by trade, would make his own tools for challenging jobs for which his industry had not yet built a fit-to-purpose tool. Little did I realize I'd become a tool builder in IT, which has been one of my loves for many years now.
I have been building GitLab CI/CD includable, shared templates since starting with GitLab over four years ago. They were designed in a specific way for others to depend directly on them – similar to the dependency managers you see in application languages like Node.js NPM, Python Pypi, and .NET NuGet.
GitLab itself has had long experience in building these shared CI dependencies through Auto DevOps and all of our security scanning suite of tools.
With the introduction of GitLab CI/CD Catalog, this long-running approach is formalized into a way for everyone to publish GitLab CI/CD components for use by anyone in the world.
Some of they key upgrades compared to the shared templates approach include:
-
Independent component versions are a new versioning mechanism that no longer relies on inheriting containers versions. GitLab CI/CD component versions bundle together the CI code and any number of containers (or no containers) behind a single CI/CD component version. The concepts of stable, production-grade DevSecOps require the ability to peg dependency versions in automation – for exactly the same reasons and benefits that this is done in production-grade application code.
-
Global visibility (with control) is available through the catalog at GitLab.com (or global to your company on a self-managed instance). Individual component visibility is also subject to the security settings of it's source project - so you can publish components to secure groups.
-
Catalog metadata, like most code-sharing mechanisms, is needed data to make decisions about which components to use.
Let's show some code
I much prefer to show than tell, so let's look at a few component examples - all of which also publish their sources publicly (click on the title to access the component).
1. Hello World
I noticed that there was not yet a Hello World component that could show the minimum viable component, both the results and the source. This particular example shows how to "componentize" just CI code.
2. Hello World Container
Frequently a CI/CD component will require a container to be fully functional. This example includes a container that is published in the same project as the component itself.
3. GitVersion Ultimate Auto Semversioning
This component automates the venerable "GitVersion" utility, which completely automates selecting the next semversion for your software without having to store the last version – even for busy repositories where many production-possible candidates are being worked on at once. One of the building principles this component follows is the principle of "least configuration" or "default to doing the most useful thing with zero configuration." In this case, if your project does not contain a GitVersion.yml, the component creates the one that an individual unfamiliar with GitVersion might find to be the most useful starting point.
4. Amazon CodeGuru Secure SAST Scanner
This component is a security scanner and, as such, follows some security scanning best practices I have implemented during recent years. For instance, if it detects that you are licensed for GitLab Ultimate, it has the scanner output GitLab's SAST JSON format, which integrates the findings just like native GitLab scanner findings. The findings appear in MRs and dashboards and can be the target of security policy merge approvals. If, however, you are not licensed for GitLab Ultimate, the scanner outputs JUNIT XML so that you have some basic, non-diffed findings visualization in the pipeline "Test Results" tab. It also only activates if there are file types it can scan and disables if the GitLab SAST_DISABLED property is turned on.
5. Checkov IaC SAST
Checkov IaC SAST is another security scanner component that also follows the above security scanner principles, but specifically for the file types it is capable of scanning. A critical best practice of many of these components is pegging container tags for stability - but doing so through a "component input" with a default value. This allows component users to test with and peg to a newer or older version than you last tested with. So your shared dependency then offers stability, but with flexibility.
6. Super-Linter
Super-Linter is a community-driven conglomeration of many linters for many languages. It originally started life as a GitHub Action, so this particular example demonstrates some of the ease of porting open source GitHub Actions to GitLab CI/CD components. A best practice aspect to many of my components is to always link to working example code with the component in action. This also allows you to do easy testing when performing updates.
7. Kaniko
Kaniko is a container that can build containers without Docker-in-Docker (DinD) privileged mode requirement. This component supports many OpenContainers labels and multi-arch builds.
8. CI Component Publishing Utilities
As I built more components, I noticed that my "component publishing CI code" was being duplicated many times - and that makes it a candidate for becoming a component itself. All the other components here leverage this component. It also uses components itself, so it uses GitVersion Ultimate Auto Semversioning to get the next version.
And if you're wondering, yes, CI Component Publishing Utilities publishes itself. In many of my components I have expanded the standard "Inputs" README section to "Inputs and Configuration" and I have added a column to show whether configurations are happening via inputs or variables. While you generally want to favor inputs, there are times when variables give more flexibility or you just want to document that the user can get perform key configurations of the underlying utilities via environment variables that the utility already supports. CI Component Publishing Utilities also uses the Kaniko CI component to build a container with the same version if it finds a Dockerfile at the root of your project (or you tell it where one is with a variable). This synchronizes the version of components and containers that support them. It also handles multi-arch container builds - see the documentation linked above to learn more!
Getting started with component templates
The Hello World components function as my own personal templates for starting a new component. They incorporate the CI Component Publishing Utilities and a reasonably good README.
For components that contain only CI code, I start by copying the source of Hello World and for ones that require a container, I start with Hello World Container. I generally copy just the source into a new project so that I have a clean commit history.
When I feel the component is stable and well developed I do a manual pipeline run and force the version to 1.1.0 or greater. The CI Component Publishing Utilities will then auto-increment the version from there.
CI component Builders Guides and practices
Darwins CI Component Builders Guide - I was also interested in publishing my approach to building components and what better way to get visibility than as a CI/CD component? BTW, the GitLab Pipeline Authoring team that created the CI/CD component architecture and CI/CD Catalog has some great best practices published at CI components best practices. The practices I publish reference these ones, but I also have quite a few I follow that are specific to my own lessons learned.
Finding the CI/CD components and their sources
The GitLab CI/CD Catalog is still undergoing innovation in searchability. However, the description from the source project is free-form searchable, so by including standard text in the descriptions of all my component source projects, I have created the ability for users to find all of the ones I've created in the catalog.
To make my component source findable regardless of its location on GitLab.com:
- I add a repository topic to all the projects called DarwinJS Component Builder Library.
- I tag with the organic tag I found called
GitLab CICD Components.
Both of the above techniques can help you provide an index to your components and their source if you are inclined to do so.
I hope that my CI/CD component building journey will be helpful to you now and in the future.