What Is Insufficient Logging and Visibility?

Insufficient logging and visibility refers to a CI/CD security risk arising from inadequate data capture, storage, and analysis within continuous integration and continuous deployment processes. Listed as an OWASP Top 10 CI/CD Security Risk, this insufficiency creates blind spots that hinder the detection of anomalies and malicious activities within CI/CD systems. While inadequate logging prevents traceability of actions from code commits to deployment, limited visibility obstructs access to and interpretation of logged data.

CICD-SEC-10: Insufficient Logging and Visibility Explained

Insufficient logging and visibility involves a lack of comprehensive logging and monitoring in the CI/CD pipeline that allows adversaries to execute malicious activities undetected throughout the attack kill chain, a term coined by Lockheed Martin. In addition to potentially masking malicious activities and delaying response times, this security risk impedes the identification of a threat actor’s techniques, tactics, and procedures (TTPs) during post-incident investigations.

With myriad attack vectors targeting engineering ecosystems, CI/CD environments require in-built capabilities to promptly detect attacks. Addressing this challenge involves providing complete visibility on human and programmatic access.

The sophisticated nature of CI/CD attack vectors also demands system audit logs to help identify unauthorized access, privilege escalations, and policy violations. Equally important, it requires applicative logs to aid in the detection of malicious code injection, unauthorized changes, and vulnerabilities introduced during development, testing, or deployment phases.

Without these capacities, CICD-SEC-10, as it’s referred to by OWASP, leaves the door open to all possible fallout of unauthorized access to the CI/CD pipeline.

Logging and Visibility Defined

Visibility encompasses the ability to observe and understand the flow of code, artifacts, configurations, and associated metadata throughout the software delivery process. Logging, on the other hand, refers to the process of capturing and storing relevant events, activities, and data points within the CI/CD pipeline.

Several types of logs play a role in CI/CD security, each contributing to identifying potential threats, vulnerabilities, and issues in the pipeline.

Access Logs

Access logs record information about who accessed the CI/CD environment, when, and from where to help identify unauthorized access attempts and trace the origin of security breaches.

Authentication and Authorization Logs

Authentication and authorization logs track user authentication and authorization activities, such as successful and failed login attempts, password changes, and access privilege modifications. These logs help monitor and detect suspicious activities and potential insider threats.

Build Logs

Build logs capture information about the build process, such as which code changes were included in a build, who initiated the build, and any errors or warnings that occurred during the build.

Deployment Logs

Deployment logs record information about the deployment process, including which versions of the application were deployed, when, and by whom. AppSec teams use deployment logs to track changes to the production environment and ensure that only authorized and tested code is deployed.

Code Repository Logs

Code repository logs help identify unauthorized or malicious changes to the source code by tracking changes to the codebase — including who made the changes, when, and what the changes were.

Security Scanning Logs

Security scanning logs capture the results of security scans and vulnerability assessments performed on the code, infrastructure, and deployed applications.

Infrastructure Logs

Infrastructure logs record information about the underlying infrastructure used to support the CI/CD pipeline, such as server logs, network logs, and firewall logs.

Audit Logs

Audit logs track changes to the CI/CD pipeline configuration to confirm whether the pipeline aligns with organizational policies and best practices. Important changes recorded in this log include modifications to build or deployment scripts, changes to security settings, and updates to third-party tools and integrations.

Effective CI/CD security, in addition to the logs listed above, draws on the applicative logs generated by individual applications. These can involve performance logs, transaction logs, debug logs, event logs — all of which provide AppSec teams with insights into a given application's behavior, performance, and potential issues.

Components of Effective Logging and Visibility

Effective logging and visibility require careful consideration of several components. These include:

• Selection of relevant log data
• Implementation of proper log formats
• Use of standardized event levels and severity levels
• Integration of logging mechanisms into the pipeline workflow
• Establishment of secure log storage and retention practices

Also multifaceted, comprehensive visibility entails tracking the movement and transformation of artifacts, managing dependencies, and capturing metadata associated with each stage of the CI/CD process.

Logging and Visibility in the CI/CD Context

Capturing all relevant pipeline events and benefitting from real-time awareness of the pipeline's security posture relies on the integration of logging and visibility tools and practices within the CI/CD workflow. Logging and visibility should be integrated throughout — from code commits, build processes, artifact storage, and testing to deployment and runtime. Each stage of the CI/CD pipeline presents valuable data and insights that contribute to overall security monitoring and incident detection.

By focusing on both audit and applicative logs, AppSec practitioners can correlate data across CI/CD stages, monitor the full pipeline, identify patterns and anomalies, and improve incident investigations.

How CICD-SEC-10 Happens

While maintaining a clear line of sight into system operations is paramount, full pipeline integration of logging and visibility doesn’t always happen. All too often, organizations grapple with the pitfalls of insufficient logging and monitoring, leaving them vulnerable to stealthy cyberthreats.

At its core, logging captures a digital trail of system activities, while monitoring interprets this data to detect anomalies. Blind spots result when these processes fall short, and attackers pounce, knowing that their malicious activities may go unnoticed.

Imagine a scenario where a threat actor infiltrates a CI/CD pipeline. Without detailed logs, tracing the origin of the breach becomes a daunting task. What’s more, without real-time monitoring, the breach could remain undetected for an extended period, giving the attacker an opportunity to compromise the deployment process.

The lack of comprehensive logging strategies and inadequate log capture mechanisms put organizations in harm’s way. Other inroads to exploitation of this risk involve incomplete or inconsistent log formats, insufficient event correlation, limited monitoring capabilities, and the absence of real-time visibility into pipeline activities.

Importance of Sufficient Logging and Visibility in CI/CD

With adversaries increasingly targeting engineering environments, timely and comprehensive data spells the difference between successful countermeasures and catastrophic outcomes.

Risks Sources and Outcomes

Insufficient logging and visibility pose significant risks to the security and integrity of the CI/CD pipeline. These risks include:

Inability to Detect and Respond to Security Incidents

Without sufficient logging and visibility, security incidents such as unauthorized access, code injection, or data breaches may go undetected. The lack of actionable log data hinders incident response teams' ability to promptly investigate and mitigate incidents. Persisting in the environment, attackers move laterally, further compromising systems and causing widespread damage.

Limited Forensic Analysis

A CI/CD breach without logs for forensic analysis compromises incident response and investigation. It becomes challenging to identify the breach, assess its scope, and perform root cause analysis, potentially leaving the environment exposed to further attacks. What’s more, tracing the attacker and attributing the attack to a specific threat actor could prove elusive without an IP address, user agent, or knowing the tools or tactics they used. The absence of logs also makes it difficult to determine the impact of the breach and can result in noncompliance with security regulations.

Ineffective Threat Hunting

Identifying potential threats and vulnerabilities within the CI/CD environment relies on vigorous logging and visibility functions, but insufficient data and limited visibility making it difficult to identify and address emerging risks.

Operational Challenges

Insufficient logging and visibility impact operational efficiency. Troubleshooting issues, diagnosing errors, and identifying performance bottlenecks become arduous without comprehensive log data and visibility into system activities.

Preventing Insufficiency in Logging and Visibility

Prioritize logging and visibility for your CI/CD pipelines with recommendations proven to potentiate security and enable effective incident management.

Map the Environment

To achieve strong visibility, you must intimately understand all systems vulnerable to potential threats. Any system involved in the CI/CD process — from SCM, CI, artifact repositories, package management software, container registries, and orchestration engines — could be a breach point.

Catalog every system your organization uses, including every instance of these systems, which is especially important with self-managed systems like Jenkins.

Enable Appropriate Log Sources

After identifying all systems, ensure activation of all relevant logs. Many systems don't enable these logs by default. Prioritize visibility for both human and programmatic access, emphasizing the identification of audit and applicative log sources.

Centralize Logs

Send logs to a centralized location, such as a security information and event management (SIEM) platform, to facilitate log aggregation and correlation across systems, enhancing detection and investigative capabilities.

Create Alerts

Set alerts to flag anomalies and potential threats. Monitor each system individually and watch for irregularities in the code shipping process, which spans multiple systems and demands a deep understanding of internal build and deployment processes.

Industry Standards for Logging and Visibility in CI/CD

Industry standards and best practices tailored for CI/CD pipelines help organizations stay ahead of attackers while preserving the agility of their deployment processes. In addition to defining logging requirements, using standardized log formats, and aligning with regulatory compliance requirements, ensure your organization has built a solid foundation for optimal logging and visibility.

Tools and Techniques to Improve Logging and Visibility

Organizations that integrate logging and visibility data with a security information and event management (SIEM) system benefit from the comprehensive analysis and correlation of security events. By merging logs from all stages of the CI/CD pipeline, they gain a complete view of their security landscape, enabling them to swiftly detect patterns or anomalies.

Knowledgeable of key indicators, DevSecOps teams can flag areas for enhancement in their logging and visibility practices. Such insights guide teams to refine their strategies, whether that involves optimizing log coverage, refining granularity, or extending log retention periods. Armed with the right logging and visibility solutions, teams can fortify the security posture of the CI/CD pipeline.

Real-Time Monitoring and Alerting

By actively monitoring the CI/CD pipeline, security teams can identify unauthorized access and vulnerabilities, allowing for timely intervention and mitigation of threats. Alerts for suspicious or anomalous activities ensure teams remain knowledgeable of critical security events, enabling rapid resolution and reduced risk of breach. Used together, real-time monitoring and alerting underpin the resilient CI/CD environment, safeguarding the integrity of the software development lifecycle.

Log Access Controls

Securing access to logs is vital to prevent unauthorized access and potential data tampering. Implementing role-based access controls ensures that only authorized users can view, modify, or delete logs. Regularly reviewing and updating permissions ensures that access remains restricted to relevant stakeholders.

Audits, Reviews, and Feedback Loops

Don’t neglect regular reviews of logging and visibility practices. Internal or external audits can identify gaps and weaknesses in logging configurations, alert thresholds, and log retention policies. With insights gained, AppSec teams can identify bottlenecks and enhance system performance. By continually iterating on logging practices from regular feedback, organizations can ensure that their pipelines remain agile and secure. Equally important, organizations can prevent insufficiencies that deny them the intel they need to achieve business outcomes.

Insufficient Logging and Visibility FAQs