GitLab Critical Patch Release: 16.10.10, 16.9.11, 16.8.10, 16.7.10, 16.6.10, 16.5.10, 16.4.7, 16.3.9, 16.2.11, 16.1.8, 16.0.10

Today we are releasing versions 16.10.10, 16.9.11, 16.8.10, 16.7.10, 16.6.10, 16.5.10, 16.4.7, 16.3.9, 16.2.11, 16.1.8, and 16.0.10 for GitLab Community Edition (CE) and Enterprise Edition (EE). This extends the security fixes previously added to 17.3.3, 17.2.7, 17.1.8, 17.0.8, 16.11.10.

These versions contain backports of an important security fix which was previously released for GitLab versions 17.3.3, 17.2.7, 17.1.8, 17.0.8, and 16.11.10. We strongly recommend that all affected self-managed GitLab installations be upgraded to one of these versions immediately. GitLab.com is already running the patched version.

GitLab releases fixes for vulnerabilities in patch releases. There are two types of patch releases: scheduled releases, and ad-hoc critical patches for high-severity vulnerabilities. Scheduled releases are released twice a month on the second and fourth Wednesdays. For more information, you can visit our releases handbook and security FAQ. You can see all of GitLab release blog posts here.

For security fixes, the issues detailing each vulnerability are made public on our issue tracker 30 days after the release in which they were patched.

We are committed to ensuring all aspects of GitLab that are exposed to customers or that host customer data are held to the highest security standards. As part of maintaining good security hygiene, it is highly recommended that all customers upgrade to the latest patch release for their supported version. You can read more best practices in securing your GitLab instance in our blog post.

Special thanks goes to Roger Meier (@bufferoverflow) who originally created the merge request in Canonical.

Recommended Action

We strongly recommend that all installations running a version affected by the issue described below be upgraded to the latest version as soon as possible.

When no specific deployment type (omnibus, source code, helm chart, etc.) of a product is mentioned, this means all types are affected.

Security fixes

Table of security fixes

SAML authentication bypass

Updates dependencies omniauth-saml to version 2.2.1 and ruby-saml to 1.17.0 to mitigate CVE-2024-45409. This security vulnerability applies only to instances which have configured SAML based authentication.

Self Managed GitLab: Known Mitigations

The following mitigation for self-managed GitLab installations prevents successful exploitation of CVE-2024-45409:

1. Enable GitLab two-factor authentication for all user accounts on the GitLab self-managed instance (NOTE: Enabling identity provider multi-factor authentication does not mitigate this vulnerability) and
2. Do not allow the SAML two-factor bypass option in GitLab.

Self Managed GitLab: Identifying & Detecting Exploitation Attempts

Evidence of attempted or successful exploitation of Ruby-SAML (CVE-2024-45409) will be present in the GitLab application_json and auth_json log files.

Unsuccessful Exploit Attempt - Hunting

Unsuccessful exploitation attempts may generate a ValidationError from the RubySaml library. This could be for a variety of reasons related to the complexity of crafting a working exploit.

Two examples are shown below, but the error may manifest with other descriptions. The common string to search for is RubySaml::ValidationError inside the application_json log.

1. Invalid ticket due to incorrect callback URL
   1. Example log event:
   2. {"severity":"ERROR","time":"2024-xx-xx","correlation_id":"xx","message":"(saml) Authentication failure! invalid_ticket: OneLogin::RubySaml::ValidationError, The response was received at https://domain.com/users/auth/saml/incorrect_callback instead of https://domain.com/users/auth/saml/callback"}
2. Invalid ticket due to a certificate signing issue
   1. Example log event:
   2. "message":"(saml) Authentication failure! invalid_ticket: OneLogin::RubySaml::ValidationError, Fingerprint mismatch"

Successful Exploitation - Hunting

Successful exploitation attempts will trigger SAML related log events. However, there may be differences that make an exploit attempt unique from legitimate SAML authentication events.

A successful exploitation attempt will log whatever extern_id value is set by the attacker attempting exploitation. Therefore, identifying a unique extern_uid that is not common in your organization could be an indicator of potential exploitation.

1. Example exploit authentication event in the application_json log file, with a extern_id set in exploit PoC code:
   1. Log event:
   2. {"severity":"INFO","time":"2024-xx-xx","correlation_id":"xx","meta.caller_id":"OmniauthCallbacksController#saml","meta.remote_ip":"0.0.0.0","meta.feature_category":"system_access","meta.client_id":"ip/0.0.0.0","message":"(SAML) saving user [email protected] from login with admin =\\u003e false, extern_uid =\\u003e exploit-test-user"}

When crafting an exploit, there are many SAML assertions an attacker would need to craft to perfectly replicate a legitimate login. These include both the key and value fields that you specify at your IdP, and may be unknown to unauthorized individuals - especially if you have customized these attributes.

You can review your auth_json log file to look for SAML responses with incorrect or missing information in the attributes section.

1. Example of a SAML authentication event in the auth_json log file.
   1. "severity":"INFO","time":"2024-xx-xx","correlation_id":"xx","meta.caller_id":"OmniauthCallbacksController#saml","meta.remote_ip":"0.0.0.0","meta.feature_category":"system_access","meta.client_id":"ip/0.0.0.0","payload_type":"saml_response": {"issuer": ["xxx"],"name_id": "xxx","name_id_format": "xxx","name_id_spnamequalifier": null,"name_id_namequalifier": null,"destination": "xxx","audiences": ["xxx"],"attributes": {"first_name": ["xxx"],"last_name": ["yyy"], "email": ["zzz"]}}

Detecting Exploitation Attempts

For self managed customers forwarding GitLab application_json logs to a SIEM, creating detections to detect Ruby-SAML (CVE-2024-45409) exploitation attempts is possible. Our team is sharing two threat detections rules, written in Sigma format, to detect potential exploitation.

Note: These detections may need to be tuned and modified to customer environments in order to deliver effective results, and due to varying configurations of different customer environments, customers should validate the legitimacy and accuracy of any events identified by these detections.

Users with more than 1 unique extern_uid over time

This detection is designed to identify an authenticated SAML user with more than one extern_uid values linked to authentication events, a potential indication of malicious authentications with an attacker set extern_uid field.

title: Multiple extern_ids
description: Detects when their are multiple extern_id's associated with a user.
author: Gitlab Security Engineering
date: 09/15/2024
schedule: "*/10 * * * *"
pseudocode: |
  select log source application.log
  where 7d < event_time < now()
  where severity="INFO" and meta_caller_id="Groups::OmniauthCallbacksController#group_saml"
  regex(message, "saving user (?<user_email>\S+) .*extern_uid \S+ (?<extern_id>[\S]+)")
  count extern_id by user_email as total_extern_ids
  where total_extern_ids > 1
verify: Review Gitlab application logs for the source IP of the SAML authentications. If there is a singular IP for all extern_ids this could point to a false positive. Cross reference the SAML authentication source IP/s with the known user's IP from sso authentication logs.
tuning: N/A

GitLab SAML authentication from a different IP address than other iDP events for the same user over time

This detection is designed to correlate authentication events, grouped by user, against both GitLab SAML authentication events as well as other iDP authentication events in an effort to identify any change in user IP address, which could be an indication of attacker authentication sessions.

title: Gitlab SAML IP differs from SSO IP
description: Detects when the source IP for the SAML authentication to Gitlab from application.log differs from the users known IP from SSO MFA logs.
author: Gitlab Security Engineering
date: 09/15/2024
schedule: "*/10 * * * *"
pseudocode: |
  select log source application.log
  where severity="INFO" and meta_caller_id="Groups::OmniauthCallbacksController#group_saml"
  regex(message, "saving user (?<user_email>\S+) ")
  #Create sub-query to bring in table from SSO authentication data
  select meta_remote_ip, user_email
  where user_email in
    (
    select log source authentication
    where 1d < event_time < now()
    where event_type="user.authentication.auth_via_mfa"
    group by user_email, sso_source_ip
    )
  where sso_source_ip!=meta_remote_ip
verify: False positives can arise when the user is traveling. Review SSO authentication logs to see if the geo-location is similar to the SAML authentication to Gitlab. If any discrepancies are found, reach out to the user for verification. If the user is not traveling, temporarily lock the user's Gitlab account and review their activity through Gitlab's application logs.
tuning: If the query is producing high false positives, consider using geolocation functions on IPs to compare the cities and countries that are generating the authentications.

Updating

To update GitLab, see the Update page. To update Gitlab Runner, see the Updating the Runner page.

Receive Patch Notifications

To receive patch blog notifications delivered to your inbox, visit our contact us page. To receive release notifications via RSS, subscribe to our patch release RSS feed or our RSS feed for all releases.

We’re combining patch and security releases

This improvement in our release process matches the industry standard and will help GitLab users get information about security and bug fixes sooner, read the blog post here.