What Is Insufficient Credential Hygiene?

Insufficient credential hygiene, listed as CICD-SEC-6 on the OWASP Top 10 CI/CD Security Risks, involves inadequate management and protection of credentials used in the CI/CD pipeline. Poor practices such as using hard-coded credentials, shared keys, and weak passwords give attackers easy access to CI/CD systems.

Factors that contribute to insufficient credential hygiene in CI/CD include the lack of a formal process for managing credentials, which often leads to uncontrolled proliferation of access keys and passwords. Breaches resulting from this CI/CD security risk typically cause significant damage to organizations, particularly those in highly regulated industries.

CICD-SEC-6: Insufficient Credential Hygiene Explained

Insufficient credential hygiene refers to the risks associated with an attacker's ability to obtain and use secrets and tokens spread throughout the CI/CD pipeline due to oversights in access controls, insecure secret management, and overly permissive credentials.

CI/CD environments consist of multiple systems that communicate and authenticate with each other. These interactions pose significant challenges in protecting credentials due to the diverse contexts in which credentials can exist:

• Application credentials are used during runtime.
• Pipelines use credentials to deploy infrastructure, artifacts, and apps to production.
• Engineers use credentials in their testing environments and within their code and artifacts.

The variety of contexts, combined with numerous methods and techniques for storing and using credentials, leads to a high likelihood of insecure credential usage practices.

Oversights and Risk Factors

Code Containing Credentials Pushed to an SCM Repository Branch

Hard-coded credentials pushed to an SCM repository branch can occur by mistake or by intention, without understanding the risk. But once in the repository branch, the credentials are exposed to anyone with read access to it. Even if deleted from the branch, they continue to appear in the commit history, viewable by anyone with repository access.

Insecure Use of Credentials in Build and Deployment Processes

Credentials in build and deployment processes access code repositories, read from and write to artifact repositories, and deploy resources and artifacts to production environments. Given the number of pipelines and target systems they need access to, DevOps teams need to know:

• The context and use method of each set of credentials in use
• Whether each pipeline has access only to the credentials needed to fulfill its purpose
• Whether unreviewed code flowing through the pipeline has access to credentials
• How credentials are called and injected to the build
• Whether credentials are accessible only in runtime and only from the contexts that requires them

Credentials in Container Image Layers

Credentials required only for building the image can remain in one of the image layers, available to anyone who can download the image.

Credentials Printed to Console Output

Pipelines often print credentials to the console output, either deliberately or inadvertently. Such actions can expose credentials in cleartext within logs, making them accessible to anyone with access to the build results. These logs can potentially flow into log management systems, broadening their exposure.

Unrotated Credentials

Credentials, spread throughout the engineering ecosystem, are exposed to numerous employees and contractors. Failure to rotate credentials leads to an ever-increasing population of people and artifacts possessing valid credentials. Pipeline credentials, such as deploy keys, exemplify this issue. Often managed under the "if it isn’t broken, don’t fix it" principle, these credentials can remain unrotated for years.

Credential Hygiene Defined

Playing a critical role in CI/CD security, credential hygiene refers to the practices associated with managing and protecting authentication mechanisms, such as passwords, tokens, keys, and certificates used to control access to systems and resources. It involves secure credential creation, storage, usage, rotation, and retirement.

In the context of CI/CD, maintaining superior credential hygiene helps prevent unauthorized access to development, testing, and production environments, ensuring the integrity and confidentiality of applications throughout their lifecycle.

Effective credential hygiene consists of several components:

• Secure credential storage: Credentials should be stored using encryption and secure secret management solutions.
• Credential rotation: Regularly changing credentials minimizes the window of opportunity for an attacker to use compromised ones.
• Least-privileged access: Granting minimal necessary privileges for each user or process can limit potential damage if credentials are compromised.
• Audit logging: Regular audits of credential use can help detect suspicious activities and ensure compliance with best practices.

How Exploitation of CICD-SEC-6 Happens

Attackers exploit insufficient credential hygiene by identifying exposed credentials and using them to infiltrate systems. For example, they might discover a container image with an exposed credential, either through manual inspection or automated scanning. Once found, they extract the credential, often using specialized tools that can retrieve information from container images.

After extraction, the attacker validates the credential to ensure it’s active, often from a compromised or disposable system to avoid detection. With a valid credential, the attacker gains access to the system or service. Depending on the permissions associated with the credential, they might access sensitive data, execute commands, or perform other malicious actions.

If the credential provides limited access, the attacker might use it as a steppingstone to escalate privileges or move laterally within the network. This could involve exploiting system vulnerabilities, launching phishing attacks against other users, or finding additional credentials within the accessed system.

Once the attacker has gained sufficient access, they begin exfiltrating customer data, intellectual property, or other sensitive information. They either store the data on a remote server controlled by the attacker or relocate it within the victim’s network for later retrieval. To maintain access to the system, the attacker might create new accounts, install backdoors, or periodically refresh the exploited credential to prevent it from expiring.

Importance of Credential Hygiene in CI/CD

Attackers covet credentials. Promising access to valuable resources or the ability to deploy malicious code and artifacts, credentials quite literally give bad actors the keys to the kingdom. And all too often, engineering environments unwittingly provide attackers with ample opportunities to acquire credentials.

Risks Associated with Insufficient Credential Hygiene

Insufficient credential hygiene exposes an enterprise to significant risks involving sensitive systems, data breaches, and disruption of development and deployment process. Poor credential hygiene can also result in reputational damage and regulatory fines.

Case Study 1: Exposed Link to Reset Credentials and a Misconfig with Role Assignments Gives Hacker Admin Privileges

In May 2019, Stack Overflow fell victim to a security breach. An unidentified user exploited a loophole to gain unauthorized moderator and developer access across all Stack Exchange Network sites. Using a sophisticated approach — initially probing the infrastructure and gradually escalating their privilege level — the attacker accessed Stack Overflow's source code. Credentials had been obtained through site settings, which led to the breach that exposed the personal data of 184 users. Stack Overflow's response team quickly intervened, revoking privileges, suspending the account, and initiating an audit. This incident led to a significant overhaul of Stack Overflow's credential management and security practices.

Case Study 2: Samsung's Misconfigured GitLab Instance Leads to Unintentional Source Code Exposure

In 2019, Samsung's development lab inadvertently leaked sensitive source code, credentials, and secret keys for several internal projects, including its SmartThings platform. The leak occurred due to the misconfiguration of a GitLab instance hosted on a Samsung-owned domain, Vandev Lab. The projects were set to "public" without proper password protection, allowing anyone to access and download the source code. The exposed data included credentials for an entire AWS account, private GitLab tokens, and private certificates for Samsung's SmartThings' iOS and Android apps.

Case Study 3: A Tale of Stolen Credentials and Concealed Cyberattacks

In 2017, Uber disclosed a significant data breach involving its Amazon Web Services account. Two hackers obtained login credentials to access data stored on Uber's Amazon Web Services, leading to the exposure of 57 million customers' and drivers' personal data. Uber's security team discovered that the attackers used stolen GitHub credentials to access Uber's private code repository. The company faced criticism for concealing the breach, which led to significant reputational damage.

Preventing Insufficiency in Credential Hygiene

A substantial potential for human error, paired with knowledge gaps pertaining to secure credentials management and the concern of breaking processes due to credential rotation, expose high-value resources to compromise. But DevSecOps teams can reverse risk profiles by replacing weak habits with strategic recommendations:

• Establish procedures to continuously map credentials found across the different systems in the engineering ecosystem — from code to deployment. Ensure each set of credentials follows the principle of least privilege and has been granted the exact set of permissions needed by the service using it.
• Avoid sharing the same set of credentials across multiple contexts. This increases the complexity of achieving the principle of least privilege as well as having a negative effect on accountability.
• Prefer using temporary credentials over static credentials. In case static credentials need to be in use — establish a procedure to periodically rotate all static credentials and detect stale credentials.
• Configure usage of credentials to predetermined conditions only (like scoping to a specific source IP or identity) to ensure that, even in case of compromise, exfiltrated credentials can’t be used outside your environment.
• Detect secrets pushed to and stored on code repositories. Use controls such as an IDE plugin to identify secrets used in the local changes. Implement automatic scanning with each code push, along with periodic scans on the repository and its past commits.
• Ensure secrets used in CI/CD systems are scoped in a manner that allows each pipeline and step access only to the secrets it requires.
• Use built-in vendor options or third-party tools to prevent secrets from being printed to console outputs of future builds. Ensure all existing outputs do not contain secrets.
• Verify that secrets are removed from any type of artifact — layers of container images, binaries, Helm charts.

Industry Standards for Credential Hygiene in CI/CD

Industry standards guide organizations in maintaining robust credential hygiene. The National Institute of Standards and Technology (NIST) recommends practices like frequent credential rotation, utilizing strong, complex passwords, and adopting multifactor authentication. ISO 27001 also provides guidance on access control, including the secure management of passwords.

Regularly Updating and Rotating Credentials

In CI/CD environments, it’s essential to regularly update and rotate credentials. Automated systems can help rotate credentials on a set schedule, reducing the risk of human error and ensuring consistent updates.

Implementing Multifactor Authentication (MFA)

MFA provides an additional layer of security by requiring more than one method of authentication to verify a user's identity. In a CI/CD pipeline, MFA can be implemented at various stages, such as during login or when making significant changes. Even if an attacker manages to compromise a password, without the second factor — like a fingerprint, a hardware token, or a time-based one-time password — they can’t gain access.

Auditing and Monitoring Access

Continuous auditing and monitoring of access logs can help detect unusual activity, potentially identifying a breach in progress. Automated systems can alert to suspicious behavior, such as multiple failed login attempts, logins from unusual locations, or unusual times of activity. These systems can also help ensure compliance with access policies and identify when accounts might have been compromised.

Avoiding Hardcoding of Credentials in Code

Hardcoding credentials in code is a common mistake that can lead to unintentional credential leaks. Instead, use secure methods of storing credentials, such as environment variables or secure vaults. Automated tools can scan code for hard-coded credentials, helping to prevent them from making it into production. This practice not only improves security but also makes it easier to rotate credentials, as code won’t need to be updated in the process.

Tools and Techniques to Strengthen Credential Hygiene

Modern tools can significantly enhance credential hygiene. Secrets management tools like HashiCorp Vault, CyberArk, and AWS Secrets Manager ensure secure storage and management of credentials. Additionally, credential management platforms provide features for automatic password rotation, auditing, and anomaly detection. It's also crucial to integrate these tools with the CI/CD pipeline to ensure secure deployment processes.

Establishing Credential Hygiene Policies

Organizations need to establish clear policies around credential hygiene, including the requirement for unique, complex passwords, regular password changes, and the avoidance of credential hardcoding. These policies should be regularly reviewed and updated in response to evolving threat landscapes.

Role of Automated Testing in Ensuring Sufficient Credential Hygiene

Automated testing is critical to maintaining adequate credential hygiene. Using static and dynamic analysis tools, organizations can automate the process of detecting weak or hard-coded credentials in the codebase.

Implementation of Secure Coding Practices

Developers need to adhere to secure coding practices to prevent credential-related security issues. This includes avoiding the hardcoding of credentials, utilizing environment variables or secure vaults for storing sensitive data, and regularly updating and rotating API keys and other credentials. Secure coding guidelines, paired with regular training, can help enforce these practices.

Insufficient Credential Hygiene FAQs