At GitLab, our Red Team conducts security exercises that emulate real-world threats. When the team was first formed, these exercises were opportunistic and done in plain sight. As the GitLab Security organization matured, so did our Red Team.
We now perform a majority of our operations in stealth, meaning that only a small group of team members are aware of the details.
This blog dives into the steps we took as we matured and lessons we learned along the way. We also share highlights of a recent stealth operation and the value it provided our organization.
If you're building an offensive security practice, or looking to mature an existing one, you may find some inspiration below.
Where we started
Our Red Team was formed in July 2019 - about four years ago. We started off as three engineers and one manager spread across the U.S., Australia, and Europe.
Back then, GitLab's security maturity was at an earlier stage. Some of the more advanced capabilities we have in place today were still being planned or improved.
As newly hired hackers, it was tempting to jump right into emulating advanced threat actors in top-secret operations. But we weren't just hackers - we were a Red Team with a mission to help make our organization more secure. It wasn't just about attacking all the things, it was about identifying and addressing realistic threats.
Getting to know GitLab
Before we started hacking, we did the following:
- Wrote down what we were doing, why were doing it, and what rules we would stick to. This was critical to our success, especially as a team that worked asynchronously across time zones.
- Met with our counterparts in Security Incident Response (SIRT) to understand how they could benefit from an offensive security practice.
- Met with our counterparts in Engineering and IT to build relationships and help them understand our overall goals and approach.
- Read. A lot. Documentation, runbooks, architecture diagrams. Whatever we could find to understand GitLab's environment and attack surface.
Getting to work
Finally, it was time to hack.
We started out doing what we called "open-scope" work, which was similar to a penetration test but without the bureaucracy and boundaries of a typical time-based engagement. We wrote enumeration scripts, scanned publicly exposed cloud resources, and hunted for leaked secrets.
When we found something that could be hacked, we hacked it and reported it in an issue to prevent it from happening again.
As we noticed patterns emerging, we developed automation to more efficiently find and report them.
This was great - it reduced risk at GitLab and gave our team a chance to better understand our environment and its risks.
But it wasn't quite Red Teaming.
We were finding, exploiting, and reporting vulnerabilities, but we weren't providing GitLab with an opportunity to practice detecting and responding to real-life attackers.
How we planned to mature
Over time, we found systemic solutions to more and more of the opportunistic findings. A new Vulnerability Management group was formed, taking ownership of our custom scanners and implementing more robust and permanent solutions. Visibility and control over endpoints increased as did the ability to monitor and alert across our entire organization.
As GitLab's defensive capabilities matured, it became important for the Red Team to do the same. We needed to emulate more advanced attackers and provide more realistic opportunities to detect and respond to these attacks.
We needed a plan.
We created a maturity model with unique stages showing where we started, where we were, and where we were headed. Each stage had a list of behaviors the team strived to demonstrate, or states we hoped to achieve.
This gave us a broad roadmap that we could work towards for the next two-to-three years. Looking back, it was worth the effort. We use the roadmap extensively, leveraging it to guide tricky decisions and to plan quarterly goals that moved us further on our journey.
The inspiration for our model came from many places, including:
- The general-purpose Capabilities Maturity Model
- The Red Team maturity model from Jordan Potti, Noah Potti, and Trevin Edgeworth
- The Red Team maturity model from Brent Harrell and Garet Stroup
We used a GitLab issue board to build the model. You can read about the logistics and benefits of using an issue board in our handbook.
This is what our model looks like:
Key milestones along the way
When we first wrote our maturity model, we were sitting somewhere in the second column. Moving beyond that would require a big shift - from opportunistically finding and exploiting vulnerabilities to emulating adversaries and providing opportunities for detection and response.
For us, that path started with Purple Teaming and then moved on to stealth operations.
We used GitLab epics to make high-level plans for each of these stages. Epics allow you to group individual issues, breaking down long-term projects into actionable tasks.
Implementing Purple Teaming
Purple Teaming was a pathway to stealth operations. It would give us an opportunity to build and practice our processes transparently and in collaboration with our Blue Team.
We made a plan to develop these processes and to test them out by conducting a small-scale Purple Team operation. This was done in the context of an OKR (Objectives and Key Results), and took us about three months to complete.
Here is the description from the epic we opened to get started:
OKR: Purple Team Foundations & Initial Run
Our SIRT team continues to grow and implement more robust detection and response capabilities. Recently, they have begun to adopt the MITRE ATT&CK framework for classifying attack techniques.
These strategies are highly aligned with our own, and build an excellent framework for a more collaborative approach in planning, designing, and executing attack emulations. When both teams are involved in all stages of a campaign, we are more likely to produce an outcome that is actionable and beneficial to the organization.
This OKR will allow us to focus on ensuring all of the foundational/logistical pieces are there, and then to execute a smaller controlled operation to make sure we got it right.
At a high-level, the OKR contained the following tasks:
- Meet with various teams at GitLab to discuss what we were trying to accomplish, how we would work together across timezones, what rules we should put in place, etc.
- Plan for specific changes/additions to our handbook to capture the results of those discussions.
- Collaborate across teams to plan and execute a small operation using these new processes.
When the quarter was complete, we had the following to show for it:
- Purple Teaming at GitLab: A handbook page describing our methodology
- Red Team issue templates: Public, re-usable templates for planning, executing, and reporting on operations
- "How we run Red Team operations remotely": A blog talking about the how and why we do this work asynchronously across time zones
We then used those processes and issue templates to plan and execute a small Purple Team operation. The brainstorming stage allowed us to work with our friends in SIRT, identifying recurring security themes and selecting attack techniques that would allow them to improve their detection and response capabilities.
We replicated a token leak where an attacker leveraged legitimate credentials to establish persistence and move laterally within the GitLab.com environment. This provided an opportunity to test existing security information and event management (SIEM) alerts, validate the ability to locate all malicious activity in log files, and to implement earlier detection and prevention capabilities.
We made changes to our Purple Teaming processes based on lessons learned. In following quarters, we moved on to full-scale emulation of relevant adversaries using a Purple Team process that was developed and tested in collaboration with groups across our organization.
Implementing stealth operations
Shifting to stealth was a natural evolution from Purple Teaming. We continued to work from our maturity model, operating from the plan that was already established and communicated across the organization.
Just as we did with Purple Teaming, we created an epic to shift to stealth operations by default and aligned it with our quarterly OKR.
This epic was opened with the following description:
OKR: Improve the maturity of the Red Team by shifting to stealth operations by default
As part of our general team roadmap, we are focusing on maturing the Red Team's processes and procedures this year. This quarter, we will complete various tasks allowing us to shift to a "stealth by default" way of performing operations.
This will provide the organization a better opportunity to practice detecting and responding to the most relevant and realistic threats.
We will do this by:
- Refreshing the Red Team Rules of Engagement by collaborating with SIRT and agreeing on processes and procedures.
- Researching, documenting, and automating architecture requirements for stealth operations.
We ended up breaking those two bullet points into separate child epics, as there was a lot of work to do in each.
The first child epic, around processes, resulted in output that is mostly public. Some examples are:
- A short summary of Stealth Operations in our general handbook page
- A new Stealth Operations section in our rules of engagement
- Example Stealth Operation Techniques section, also in the rules of engagement
- Iterations to our Red Team issue templates
We were very transparent with all of these changes. Each change was a merge request, which was visible to everyone at GitLab. We opened a dedicated issue to discuss any concerns and used an all-company Slack channel to invite everyone to provide feedback.
As an internal Red Team, building relationships across the organization is key to our success. We find that transparency about how we operate helps us maintain this trust.
The second child epic, around technical research, resulted in outputs that are mostly not public and involved things like:
- Using "Attacker VMs" with Parallels on our corporate laptops. This provides us a space without security monitoring, where we can use commercial VPNs to appear as separate entities when emulating remote attackers.
- Working with our IT department to acquire our own AWS accounts with exceptions to standard security monitoring. This gives us a space to install our C2 infrastructure, phishing sites, etc.
- Testing various command and control (C2) frameworks, agents, and redirectors. Designing automation to deploy these environments from scratch with each new operation.
- Establishing private communication channels and a wiki for Red Team engineers and trusted participants.
- Testing encrypted secret management tools for temporary storage during operations.
Results from a recent stealth operation
With our new tools and processes in place, stealth operations became our default.
One recent operation began with selecting an attack group that had been in the news for targeting organizations similar to ours. This operation spanned three months - the majority of which was spent on researching the adversary and developing capabilities to emulate them.
We started with a volunteer from a non-security team at GitLab. They were one of our "trusted participants", meaning they were briefed on the operation. We had them visit a website we created which mimicked the download page of a popular open-source desktop utility. They downloaded the utility and followed the on-screen instructions to install and authorize it.
The application was a modified fork of the legitimate tool, created just for this operation. It contained an embedded script which downloaded our command and control (C2) agent and provided the Red Team access to the laptop. This scenario mirrored the adversary we were emulating, who would deploy malware to engineers' laptops.
Using an insider to launch the initial payload is a common Red Team technique called an "assumed breach." This allows the Red Team to focus their efforts on emulating post-exploitation activities, where there is more value in practicing detection and response.
With remote access achieved, the Red Team conducted various attack techniques locally on the laptop to steal web browser cookies and impersonate their active sessions. From there, we pursued further objectives similar to those of our emulated adversary.
These techniques triggered an alert from our SIEM system. This created an incident with our SIRT team, who immediately took action to contain and investigate the incident.
Select members of security leadership were included as trusted participants in the operation. We were all closely monitoring the investigation from a Slack room set up for this purpose. This allowed the SIRT engineers to experience responding to a very realistic attack while preventing the incident from escalating too far.
At some point during the investigation, it was revealed that the attacker was in fact the Red Team. SIRT had performed a thorough investigation, collaborating across the team to trace the attack back to our initial access vector.
This operation helped us validate some existing detection capabilities, recommend improvements for more, and give the team a chance to work together to solve an interesting challenge in a safe and controlled environment. This type of experience only comes from conducting attack operations in stealth, which is exactly why we have an internal Red Team at GitLab.
What we learned
At GitLab, we believe that performing Red Team operations in stealth provides the most realistic opportunity to practice detecting and responding to real-life attacks.
We also realize that every organization is different, and your security evolution may follow a different path.
We learned that having a plan defined early on and shared transparently across the organization was key to success. Here are the things that helped us the most:
- Defining a maturity model and using it as a roadmap.
- Committing to broad goals defined in GitLab epics, and breaking them down into manageable tasks inside GitLab issues.
- Thoroughly documenting processes in our handbook and in GitLab issue templates.
We would love to hear your thoughts on Red Teaming and how you've managed your own security evolution. If there are any specific topics you'd like our team to write about in the future, please let us know. Feel free to comment below or to open issues or merge requests in any of our public projects.
