Insider Threat Detection In Cloud Using Access Logs — Complete Documentation & Simulation

Overview & System Context

Insider Threat Detection In Cloud Using Access Logs: Comprehensive Web-Sourced Technical Investigation Exhaustive Research Dossier · Pure Web-Scraped Content: ~9505 Words · Real Software Screenshots: 1 1. Executive Overview & Technical Background An intrusion detection system (IDS) is a device or software application that monitors a network or systems for malicious activity or policy violations. Any intrusion activity or violation is typically either reported to an administrator or collected centrally using a security information and event management (SIEM) system. A SIEM system combines outputs from multiple sources and uses alarm filtering techniques to distinguish malicious activity from false alarms. IDS types range in scope from single computers to large networks. The most common classifications are network intrusion detection systems (NIDS) and host-based intrusion detection systems (HIDS). A system that monitors important operating system files is an example of an HIDS, while a system that analyzes incoming network traffic is an example of an NIDS. It is also possible to classify IDS by detection approach. The most well-known variants are signature-based detection (recognizing bad patterns, such as exploitation attempts) and anomaly-based detection (detecting deviations from a model of "good" traffic, which often relies on machine learning). Another common variant is reputation-based detection (recognizing the potential threat according to the reputation scores). Some IDS products have the ability to respond to detected intrusions. Systems with response capabilities are typically referred to as an intrusion prevention system (IPS). Intrusion detection systems can also serve specific purposes by augmenting them with custom tools, such as using a honeypot to attract and characterize malicious traffic.

Host intrusion detection systems (HIDS) run on individual hosts or devices on the network. A HIDS monitors the inbound and outbound packets from the device only and will alert the user or administrator if suspicious activity is detected. It takes a snapshot of existing system files and matches it to the previous snapshot. If the critical system files were modified or deleted, an alert is sent to the administrator to investigate. An example of HIDS usage can be seen on mission critical machines, which are not expected to change their configurations.

Network-based intrusion prevention system (NIPS): monitors the entire network for suspicious traffic by analyzing protocol activity. Wireless intrusion prevention system (WIPS): monitor a wireless network for suspicious traffic by analyzing wireless networking protocols. Network behavior analysis (NBA): examines network traffic to identify threats that generate unusual traffic flows, such as distributed denial of service (DDoS) attacks, certain forms of malware and policy violations. Host-based intrusion prevention system (HIPS): an installed software package which monitors a single host for suspicious activity by analyzing events occurring within that host.

Signature-based detection: Signature-based IDS monitors packets in the Network and compares with pre-configured and pre-determined attack patterns known as signatures. While it is the simplest and most effective method, it fails to detect unknown attacks and variants of known attacks. Statistical anomaly-based detection: An IDS which is anomaly-based will monitor network traffic and compare it against an established baseline. The baseline will identify what is "normal" for that network – what sort of bandwidth is generally used and what protocols are used. It may however, raise a False Positive alarm for legitimate use of bandwidth if the baselines are not intelligently configured. Ensemble models that use Matthews correlation co-efficient to identify unauthorized network traffic have obtained 99.73% accuracy. Stateful protocol analysis detection: This method identifies deviations of protocol states by comparing observed events with "pre-determined profiles of generally accepted definitions of benign activity". While it is capable of knowing and tracing the protocol states, it requires significant resources.

Figure 1: Figure 1: Verified System Architecture & Pipeline Workflow There are a number of techniques which attackers are using, the following are considered 'simple' measures which can be taken to evade IDS: Fragmentation: by sending fragmented packets, the attacker will be under the radar and can easily bypass the detection system's ability to detect the attack signature. Avoiding defaults: The TCP port utilised by a protocol does not always provide an indication to the protocol which is being transported. For example, an IDS may expect to detect a trojan on port 12345. If an attacker had reconfigured it to use a different port, the IDS may not be able to detect the presence of the trojan.

Coordinated, low-bandwidth attacks: coordinating a scan among numerous attackers (or agents) and allocating different ports or hosts to different attackers makes it difficult for the IDS to correlate the captured packets and deduce that a network scan is in progress. Address spoofing/proxying: attackers can increase the difficulty of the Security Administrators ability to determine the source of the attack by using poorly secured or incorrectly configured proxy servers to bounce an attack. If the source is spoofed and bounced by a server, it makes it very difficult for IDS to detect the origin of the attack. Pattern change evasion: IDS generally rely on 'pattern matching' to detect an attack. By changing the data used in the attack slightly, it may be possible to evade detection. For example, an Internet Message Access Protocol (IMAP) server may be vulnerable to a buffer overflow, and an IDS is able to detect the attack signature of 10 common attack tools. By modifying the payload sent by the tool, so that it does not resemble the data that the IDS expects, it may be possible to evade detection.

This article incorporates public domain material from Karen Scarfone, Peter Mell. Guide to Intrusion Detection and Prevention Systems, SP800-94 (PDF). National Institute of Standards and Technology. Retrieved 9 July 2025. Cloud computing security or cloud security refers to a broad set of policies, technologies, applications, and controls used to protect data, applications, services, and the associated infrastructure of cloud computing. It is a sub-domain of computer security, network security and, more broadly, information security.

Deterrent controls Administrative mechanisms intended to reduce attacks by informing attackers of consequences. Preventive controls Controls designed to reduce vulnerabilities and prevent unauthorized access. Detective controls Controls that detect and respond to security events. Includes monitoring, SIEM, IDS/IPS, malware detection. Corrective controls Controls that reduce the impact of an incident and restore systems.

Business continuity and data recovery Cloud providers have business continuity and data recovery plans in place to ensure service continuity and data protection. Log and audit trail In addition to producing logs and audit trails, cloud providers work with their customers to secure these logs and ensure they're accessible for forensic investigation (e.g., eDiscovery). Unique compliance requirements In addition to the requirements on customers, data centers used by cloud providers may be subject to additional compliance obligations. Using a cloud service provider (CSP) can lead to extra security concerns around data jurisdiction since customer or tenant data may not remain in the same location or provider's cloud.

Aside from the security and compliance issues already discussed, cloud providers and their customers negotiate terms around liability (stipulating how incidents involving data loss or compromise will be resolved, for example), intellectual property, and end-of-service (when data and applications are ultimately returned to the customer). These issues are typically addressed in service-level agreements (SLAs).

Identity threat detection and response (ITDR) is a cybersecurity discipline that includes tools and best practices to protect identity management infrastructure from attacks. ITDR can block and detect threats, verify administrator credentials, respond to various attacks, and restore normal operations. Common identity threats include phishing, stolen credentials, insider threats, and ransomware. ITDR adds an extra layer of security to identity and access management (IAM) systems. It helps secure accounts, permissions, and the identity infrastructure itself from compromise. With attackers targeting identity tools directly, ITDR is becoming more important in 2023 : according to Gartner, established IAM hygiene practices like privileged access management and identity governance are no longer enough. ITDR can be part of a zero trust security model. ITDR is especially relevant for multicloud infrastructures, which have gaps between cloud providers' distinct IAM implementations.

Closing these gaps and orchestrating identity across clouds is an ITDR focus. Access to this page requires authorization. You can try signing in or changing directories . This article contains recommended guidance and is not intended to serve as a certification standard, compliance attestation, or statement of required controls. The content was last reviewed on December 1, 2025, and might not reflect the latest developments in logging, monitoring, and threat detection capabilities.

Logging and threat detection enable organizations to identify, investigate, and respond to security incidents before they escalate into full-scale breaches. Unlike traditional periodic log reviews, modern cloud environments require continuous monitoring, behavioral analytics, and centralized correlation across identity, network, data, and application layers to detect multi-stage attacks that exploit rapid provisioning, ephemeral resources, and distributed architectures. Organizations implementing comprehensive logging and threat detection capabilities achieve rapid incident response and forensic readiness, while those neglecting these controls face prolonged adversary dwell time, undetected privilege escalation, and inability to reconstruct breach timelines.

Without comprehensive logging and threat detection capabilities, organizations face undetected threats operating for extended periods, incomplete forensic evidence preventing incident reconstruction, and regulatory compliance gaps. Enable detection capabilities: Before you can respond to threats, you must first deploy intelligent detection systems that continuously monitor for known attack patterns, anomalous behaviors, and indicators of compromise. Native threat detection services leverage behavioral analytics and threat intelligence to identify suspicious activities that traditional access controls cannot preventâIIfrom SQL injection attempts and malware uploads to credential abuse and data exfiltration. Implement detection across all critical resource types (compute, data, identity, network) to ensure no attack surface remains unmonitored.

Enable comprehensive logging: Implement systematic audit logging across all cloud tiersâIIresource logs (data plane operations), activity logs (management plane changes), identity logs (authentication and authorization events), and network logs (traffic flows and firewall decisions). Comprehensive logging provides the forensic evidence required to reconstruct attack timelines, scope incident blast radius, and support compliance requirements. Without complete audit trails spanning identity, network, and data access, incident responders lack the visibility to determine what was accessed, by whom, when, and from whereâIIprolonging breach dwell time and increasing regulatory exposure.

Centralize and analyze: Aggregate logs from all sources into a centralized Security Information and Event Management (SIEM) platform to enable correlation, advanced analytics, and automated response. Centralization transforms isolated log streams into actionable threat intelligence by correlating events across identity, network, and data planes to reveal multi-stage attack chains that single-source monitoring cannot detect. Establish disciplined retention policies aligned with compliance requirements and business needs, and ensure accurate time synchronization across all systems to maintain forensic integrity and enable precise incident reconstruction.

Enable threat detection capabilities across compute, storage, database, and identity services to identify known attack patterns, anomalous behaviors, and suspicious activities. Use behavioral analytics and threat intelligence to detect threats that traditional access controls cannot prevent. Adversaries exploit environments lacking native threat detection to conduct attacks that remain invisible to traditional security controls. Without behavioral analytics and threat intelligence-driven monitoring: Failure to deploy comprehensive threat detection increases mean time to detect (MTTD) from hours to weeks, amplifies breach costs, and enables adversaries to establish deep persistence across your cloud infrastructure.

Deploy native threat detection capabilities across all critical cloud services to identify malicious activities, anomalous behaviors, and known attack patterns through behavioral analytics and threat intelligence. This foundational layer provides the first line of defense against threats targeting compute, storage, database, and key management services. Implement the following steps to establish comprehensive threat detection coverage: Use cloud-native threat detection: Deploy the threat detection capabilities of your cloud security posture management platform for compute, storage, database, and identity services, providing behavioral analytics and threat intelligence-driven monitoring.

Enable comprehensive service coverage: Activate threat detection for all critical Azure services using Microsoft Defender for Cloud , covering virtual machines, storage accounts, databases, containers, and Key Vault instances. Review detection capabilities: Familiarize security teams with available detection capabilities using the Microsoft Defender for Cloud security alerts reference guide to understand alert types, severity levels, and response requirements.

Address detection gaps: For services without native threat detection capability, collect data plane logs and route to Microsoft Sentinel for custom analytics rules and behavioral detection. Optimize alert quality: Configure alert filtering and analytics rules to reduce false positives and extract high-quality alerts from log data, tuning detection sensitivity based on workload criticality and risk tolerance.

Enhance basic threat detection by centralizing security telemetry, deploying unified extended detection and response (XDR) platforms, and enriching alerts with threat intelligence. This advanced layer enables correlation of threats across multiple domains, custom detection for organization-specific attack patterns, and context-rich alerts that accelerate investigation and response. Build this advanced detection capability through the following steps: Centralize security telemetry: Aggregate alerts and log data from cloud security platforms, endpoint protection, identity systems, and application services into Azure Monitor or Microsoft Sentinel for unified analysis and correlation.

Deploy unified XDR platform: Enable Microsoft Defender XDR to correlate threat detection across Microsoft 365 Defender (endpoints, email, collaboration), Microsoft Defender for Cloud (Azure infrastructure), and Microsoft Entra ID (identity) with cross-domain incident grouping and automated investigation workflows. Build custom detection rules: Create custom analytics rules matching organization-specific threat patterns, attack indicators, and business logic violations that pre-built detections cannot address.

Enrich with threat intelligence: Integrate Microsoft Defender Threat Intelligence to enrich alerts with threat actor attribution, infrastructure reputation, vulnerability exploitation intelligence, and compromised indicator correlation. Leverage threat indicators: Implement threat intelligence indicators in Microsoft Sentinel for automated matching of observables (IP addresses, domains, file hashes, URLs) against known malicious infrastructure and threat actor campaigns.

A global manufacturing organization enabled comprehensive threat detection across 300+ Azure resources spanning storage accounts, SQL databases, Cosmos DB instances, and IoT production systems to reduce mean time to detect (MTTD) sophisticated attacks. Challenge: Traditional security monitoring relied on isolated service-specific alerts without correlation across storage, databases, identity, and email systems. Security team lacked unified visibility to detect multi-stage attacks spanning phishing campaigns, credential compromise, and data exfiltration. Mean time to detect sophisticated threats exceeded 30 days.

Outcome: Defender XDR detected and contained a phishing-to-exfiltration attack chain shortly after deployment, with Threat Intelligence identifying infrastructure matching known APT targeting manufacturing intellectual property. Automated cross-domain incident correlation enabled security teams to rapidly investigate and respond to multi-stage attacks that previously remained undetected for extended periods.

Monitor authentication and authorization events to detect compromised credentials, anomalous sign-in patterns, and account abuse. Identify behavioral anomalies including excessive authentication failures, impossible travel, deprecated account usage, and unauthorized privilege escalations. Identity-based attacks remain the primary initial access vector for cloud breaches, yet many organizations lack behavioral monitoring to detect credential abuse and anomalous access patterns. Without identity-focused threat detection: Failure to monitor identity and access anomalies enables attackers to operate using valid credentials, bypassing network and endpoint controls while maintaining low detection profiles.

Establish comprehensive visibility into identity and authentication activities by enabling audit logging and sign-in monitoring for all identity resources. This foundational telemetry enables detection of suspicious authentication patterns, unauthorized changes to identity configuration, and potential account compromise indicators. Configure the following monitoring capabilities: Enable comprehensive audit logging: Activate Microsoft Entra audit logs to provide complete traceability for all changes made to identity resources including user and group management, role assignments, application registrations, and policy modifications.

Monitor authentication activities: Track sign-in reports to capture all authentication events for managed applications and user accounts, establishing baselines for normal access patterns and identifying anomalies. Detect risky sign-in patterns: Enable monitoring of risky sign-ins to flag authentication attempts from suspicious sources, impossible travel scenarios, unfamiliar locations, or leaked credential usage indicating potential account compromise.

Identify compromised accounts: Monitor users flagged for risk to detect accounts showing multiple risk indicators requiring immediate investigation and remediation through automated or manual review workflows. Integrate with SIEM platforms: Route Microsoft Entra ID logs to Azure Monitor , Microsoft Sentinel , or third-party SIEM platforms for advanced correlation, long-term retention, and cross-domain threat detection analytics.

2. System Architecture & Core Working Principles Enhance identity monitoring with advanced risk detection, adaptive access controls, and AI-powered investigation capabilities. This advanced layer applies machine learning to detect sophisticated identity attacks, automates risk-based authentication enforcement, and extends protection to hybrid environments bridging cloud and on-premises infrastructure. Implement these advanced capabilities: Deploy identity risk detection: Enable Microsoft Entra ID Protection to detect and remediate identity risks including leaked credentials, sign-ins from anonymous IP addresses, malware-linked sources, and password spray attacks using machine learning and threat intelligence.

Implement risk-based access controls: Configure Identity Protection policies to enforce adaptive authentication requirements through Microsoft Entra Conditional Access, requiring MFA for medium-risk sign-ins and blocking high-risk authentication attempts until remediation. Monitor cloud infrastructure identity threats: Enable Microsoft Defender for Cloud workload protection to detect suspicious identity activities including deprecated account usage, excessive failed authentication attempts, and anomalous service principal behaviors.

Extend to hybrid environments: For organizations with on-premises Active Directory, deploy Microsoft Defender for Identity to monitor domain controllers, detect advanced threats, identify compromised identities, and investigate malicious insider actions spanning hybrid infrastructure. Accelerate investigations with AI: Leverage Microsoft Security Copilot to reduce investigation time through natural language queries, automated threat analysis, guided remediation workflows, and AI-powered correlation of identity signals across Microsoft Defender XDR, Sentinel, and Entra ID Protection.

A financial services organization with 8,000 employees implemented comprehensive identity threat detection to combat credential-based attacks targeting cloud banking applications and customer data repositories. Challenge: Traditional authentication monitoring lacked behavioral analytics to detect credential-based attacks including impossible travel, password spray campaigns, and dormant account exploitation. Security team relied on manual log reviews discovering compromises only after fraud incidents or customer complaints. Mean time to detect identity-based attacks exceeded 30 days.

Outcome: Identity Protection flagged compromised accounts shortly after deployment, Sentinel detected and contained credential stuffing attacks within minutes, and Security Copilot enabled security teams to rapidly investigate identity-based threats using natural language queries. The organization achieved substantially faster detection and response to identity-based attacks that previously remained undetected for extended periods.

Enable audit logging across data plane operations, control plane activities, and identity events to support incident investigation, forensic analysis, and compliance validation. Comprehensive logging provides the evidence trail required to reconstruct security incidents and determine breach scope. Comprehensive audit logging across all cloud tiers provides the forensic foundation for incident investigation, breach reconstruction, and compliance validation. Without systematic logging of resource, activity, and identity events: Inadequate logging transforms every security incident into a prolonged, high-cost investigation with incomplete forensic evidence and uncertain scope.

Establish the foundational audit trail by capturing all management plane operations and identity events across your cloud environment. This layer provides visibility into who is making changes to infrastructure, when those changes occur, and how identities are being usedâIIessential for detecting unauthorized modifications, privilege abuse, and compliance violations. Enable these core logging capabilities: Enable activity logs for management plane operations: Activate Azure Activity Logs for all subscriptions to capture management plane operations including resource creation, modification, deletion (PUT, POST, DELETE operations), role assignments, policy changes, and administrative actions.

Centralize activity log collection: Configure diagnostic settings at the management group or subscription level to route Activity Logs to centralized Log Analytics workspace for long-term retention, correlation analysis, and compliance reporting. Enforce consistent logging coverage: Deploy Azure Policy to enforce diagnostic settings across all subscriptions ensuring consistent Activity Log collection and preventing configuration drift as new subscriptions are created.

Capture authentication events: Enable Microsoft Entra sign-in logs to capture all user and service principal authentication events including interactive sign-ins, non-interactive sign-ins, service principal sign-ins, and managed identity authentications. Track identity changes: Enable Microsoft Entra audit logs to track all changes made to Microsoft Entra ID including user/group management, role assignments, application registrations, conditional access policy modifications, and administrative unit changes.

Extend identity log retention: Configure Microsoft Entra diagnostic settings to route sign-in and audit logs to Log Analytics workspace or Event Hub for extended retention beyond the default 30-day Microsoft Entra admin center retention period. Monitor hybrid identity infrastructure: For hybrid environments, integrate Microsoft Entra Connect Health logs to monitor synchronization events, authentication failures, and on-premises Active Directory integration health.

Correlate with network events: Enable network infrastructure logs as covered in LT-4 (NSG flow logs, Azure Firewall logs, VPN Gateway diagnostics, Application Gateway logs) to provide network context for security investigations correlating identity and control plane events with network traffic patterns. Extend audit coverage to data plane operations where sensitive business data resides and is accessed. Platform service logs capture the "what was accessed" and "by whom" details essential for investigating data breaches, insider threats, and compliance violationsâIIcovering storage, databases, secrets management, containers, and NoSQL platforms. Configure the following data plane logging: Enable resource-level data plane logs: Activate Azure resource logs for data plane operations performed within Azure services including read, write, and delete operations on data and configurations across all platform services.

Log storage operations: Enable Azure Storage diagnostic logs to capture all blob, file, queue, and table storage operations including StorageRead, StorageWrite, StorageDelete events with caller identity, source IP, and operation latency for forensic investigations. Audit database activities: Configure Azure SQL Database auditing to log all database queries, schema changes, permission grants, authentication attempts, and administrative operationsâIIroute audit logs to Log Analytics workspace or storage account for compliance and security monitoring.

Monitor secrets access: Enable Azure Key Vault diagnostic logs to capture all key, secret, and certificate access operations including retrieval, rotation, deletion, and permission changes with full audit context for sensitive asset tracking. Track NoSQL operations: Configure Azure Cosmos DB diagnostic logs to capture data plane operations, query performance, partition key access patterns, and throttling events for security and performance investigations.

Cover additional data platforms: Enable diagnostic logging for other data services including Azure Data Lake Storage, Azure Synapse Analytics, Azure Database for PostgreSQL/MySQL, and Azure Cache for Redis capturing data access and administrative operations. 0- Log Kubernetes control plane: Enable Azure Kubernetes Service (AKS) diagnostics to capture control plane logs including kube-apiserver (all API requests), kube-audit (security audit trail), kube-controller-manager, kube-scheduler, and cluster-autoscaler logs.

Monitor container runtime: Configure Container Insights to collect container-level metrics, logs, and performance data from AKS clusters, Azure Container Instances, and Azure Arc-enabled Kubernetes clusters including pod lifecycle events and resource utilization. Track container images: Enable Azure Container Registry diagnostics to log image push/pull operations, repository access, authentication events, webhook invocations, and vulnerability scan results.

Automate platform logging enablement: Use Microsoft Defender for Cloud to automatically enable and configure resource logs for supported Azure services across subscriptions reducing manual configuration overhead. Enforce consistent coverage: Deploy Azure Policy to enforce diagnostic settings for data services ensuring consistent log collection, preventing configuration drift, and remediating non-compliant resources automatically.

Complete your audit coverage by capturing application-layer activities, custom workload operations, and business logic execution. Application logs provide the deepest visibility into how users interact with your systems, what business data they access, and which application-layer attacks are attemptedâIIessential for detecting insider threats, application abuse, and sophisticated attacks that bypass infrastructure controls. Implement comprehensive application logging: Log web application activities: Enable Azure App Service diagnostics to capture application logs, web server logs (IIS/HTTP.sys logs), detailed error messages, failed request tracing, and deployment logs for web applications and APIs.

Monitor API gateway operations: Configure Azure API Management diagnostics to log API requests, responses, authentication failures, rate limit violations, policy execution details, backend service errors, and subscription management events. Track serverless function execution: Enable Azure Functions monitoring with Application Insights integration to capture function executions, dependencies, exceptions, performance metrics, and custom security events including authorization decisions and sensitive operation audits.

Log business process workflows: For Azure Logic Apps, enable diagnostics to log workflow runs, trigger events, action results, and integration failures supporting business process security investigations. Deploy VM monitoring agents: Deploy Azure Monitor Agent to Windows and Linux virtual machines to collect Security Event Logs, Syslog, performance counters, and custom log files.

Collect Windows security events: Configure Windows Event Log collection for security-relevant events including authentication attempts (Event ID 4624, 4625), privilege escalation (4672, 4673), account management (4720, 4726, 4738), and audit policy changes (4719). Gather Linux system logs: Configure Linux Syslog collection for authentication logs (/var/log/auth.log, /var/log/secure), system logs (/var/log/syslog, /var/log/messages), and application-specific security logs.

Monitor endpoint protection: Enable antimalware monitoring for Windows virtual machines to log malware detection events, scan results, signature updates, and policy violations. Implement structured logging: Implement structured application logging with security context including user identity, source IP address, request ID, operation type, data classification labels, authorization decisions, and business transaction identifiers to support correlation and forensic analysis.

Enable APM telemetry: Enable Application Insights or equivalent application performance monitoring (APM) solutions to collect telemetry, exceptions, custom security events, distributed tracing for microservices, and dependency tracking. Log application security events: Configure application-layer security event logging including authentication attempts, authorization failures, input validation failures, privilege escalations, sensitive data access, and business logic security violations.

Monitor web application attacks: For web applications and APIs, log HTTP security headers, content security policy violations, CORS policy enforcement, and session management events to detect application-layer attacks. 3. Software Implementation, Frameworks & Protocols Capture Layer 7 attack attempts: Enable logging for API gateways and web application firewalls to capture Layer 7 attacks including SQL injection attempts, cross-site scripting (XSS), remote code execution attempts, local file inclusion, XML external entity (XXE) attacks, and business logic abuse patterns.

Track API abuse: Configure logging for rate limiting enforcement, authentication failures, bot detection, and API abuse patterns supporting threat detection and incident response. A healthcare SaaS provider enabled comprehensive three-tier logging (infrastructure/identity, platform/data services, application/workload) to meet HIPAA audit requirements and support security investigations for electronic health record (EHR) systems serving 200+ hospital customers.

Challenge: Fragmented logging across isolated services prevented correlation of multi-stage attacks. Security team lacked complete audit trails for HIPAA compliance and could not reconstruct incident timelines spanning infrastructure changes, data access, and application abuse. Mean time to investigate incidents exceeded 2 weeks due to manual log aggregation across 120+ services.

Outcome: Cross-tier log correlation enabled rapid detection of sophisticated multi-stage attacks spanning role assignments, Key Vault access, and storage exfiltration. The organization achieved complete HIPAA audit trail compliance and substantially reduced incident investigation time through centralized log analysis across infrastructure, platform, and application layers. Enable network traffic logging including flow logs, firewall decision logs, web application firewall events, and DNS queries to support incident investigation and threat detection. Network logs provide forensic evidence for lateral movement, command-and-control communications, data exfiltration, and policy violations.

Network traffic logs provide critical forensic evidence for investigating lateral movement, data exfiltration, command-and-control communications, and application-layer attacks. Without comprehensive network logging: Capture comprehensive network traffic telemetry to detect lateral movement, data exfiltration, command-and-control communications, and application-layer attacks. Network logs provide the evidence trail for how attackers move between systems, what external destinations they contact, and which attack techniques they employâIIessential for investigating sophisticated multi-stage breaches. Enable the following network logging capabilities: Capture network flow logs: Enable network security group (NSG) flow logs to capture information about IP traffic flowing through NSGs including source/destination IPs, ports, protocols, and allow/deny decisions for lateral movement detection.

Monitor firewall activities: Enable Azure Firewall logs and metrics to monitor firewall activity, rules processing, threat intelligence hits, and DNS proxy logs for centralized egress monitoring and threat detection. Log application-layer attacks: Enable Web Application Firewall (WAF) logs to capture application-layer attack attempts including SQL injection, cross-site scripting, and OWASP Top 10 violations with request details and blocking decisions.

Collect DNS query data: Collect DNS query logs to assist in correlating network data and detecting DNS-based attacks including tunneling, DGA domains, and command-and-control communications. Deploy comprehensive monitoring: Use Azure networking monitoring solutions in Azure Monitor for comprehensive network visibility and centralized log correlation.

Enable traffic analytics: Send flow logs to Azure Monitor Log Analytics workspace and use Traffic Analytics to provide insights into network traffic patterns, security threats, bandwidth consumption, and policy violations. Challenge: A global e-commerce platform needed to detect lateral movement, data exfiltration, and application-layer attacks across multi-region infrastructure protecting payment processing and customer data systems.

Solution approach: Enabled comprehensive network logging by deploying NSG flow logs with Traffic Analytics across 200+ network security groups, configured Azure Firewall and WAF diagnostic logs at centralized egress points and application gateways, implemented DNS analytics for C2 detection, and integrated all network logs with SIEM for correlation with identity and resource activity signals.

Outcome: Traffic Analytics identified policy violations (exposed management ports), WAF logs detected and blocked SQL injection campaigns, and DNS analytics flagged DGA patterns enabling rapid VM isolation. Network logging enabled detection of lateral movement patterns and data exfiltration attempts that would have remained invisible without comprehensive flow and firewall log analysis.

Centralize security logs from all cloud services, identity systems, and network infrastructure into a unified platform for correlation and analysis. Centralized aggregation enables detection of multi-stage attacks spanning multiple services that isolated log sources cannot reveal. Distributed logs stored across disparate services and regions prevent correlation of multi-stage attacks and delay incident detection. Without centralized log aggregation and SIEM capabilities: Absent centralized SIEM/SOAR, organizations operate reactively with fragmented visibility, prolonged response times, and inability to detect coordinated attacks.

Transform fragmented logs scattered across services into unified visibility by routing all security telemetry to a central platform. Log aggregation provides the foundation for cross-service correlation, enabling detection of attack patterns that span infrastructure boundariesâIIfrom initial compromise through lateral movement to data exfiltration. Establish centralized log collection: Aggregate logs centrally: Integrate Azure activity logs into a centralized Log Analytics workspace along with resource diagnostic logs from all services to enable cross-resource correlation and unified investigation workflows.

Query aggregated logs: Use Azure Monitor with KQL queries to perform analytics on aggregated logs from Azure services, endpoint devices, network resources, and other security systems for pattern detection and investigation. Configure alerting: Create alert rules using the logs aggregated from multiple sources to detect security threats and operational issues through correlation logic spanning multiple log sources.

Establish data governance: Define data ownership, implement role-based access controls to logs, specify storage locations for compliance requirements, and set retention policies balancing investigation needs with cost and regulatory obligations. Elevate log aggregation into proactive security operations by deploying security information and event management (SIEM) with automated response capabilities. SIEM transforms raw logs into actionable intelligence through correlation rules, threat analytics, and automated incident workflowsâIIenabling security teams to detect and respond to threats at machine speed rather than manual investigation pace. Build your SIEM/SOAR platform: Deploy SIEM platform: Onboard Microsoft Sentinel to provide security information event management (SIEM) and security orchestration automated response (SOAR) capabilities for centralized security analysis and incident response.

Connect data sources: Connect data sources to Microsoft Sentinel including Azure services, Microsoft 365, third-party security solutions, and on-premises systems to establish comprehensive security visibility. Configure analytics rules: Create detection rules in Sentinel to identify threats and automatically create incidents based on correlated security events spanning multiple log sources and time periods.

Automate response actions: Implement automated response playbooks using Logic Apps to orchestrate incident response actions including containment, notification, and remediation workflows. Enable monitoring dashboards: Deploy Sentinel workbooks and dashboards for security monitoring, threat hunting, compliance reporting, and executive-level security posture visibility.

Integrate AI-powered analysis: Enable Microsoft Security Copilot integration with Sentinel to provide AI-powered incident investigation, threat hunting, and guided response recommendations using natural language queries across centralized security logs. Challenge: A multinational insurance company needed unified threat detection and investigation capabilities across 12 Azure subscriptions, 500+ resources, and 4 geographic regions processing policyholder personal information and claims data.

Solution approach: Deployed centralized Log Analytics workspace with management group-level diagnostic settings routing all Azure activity logs, identity telemetry, and critical resource logs (Storage, SQL, Key Vault, Firewall). Enabled Microsoft Sentinel SIEM with data connectors for Defender for Cloud, Entra ID Protection, Microsoft 365 Defender, and Defender Threat Intelligence. Configured analytics rules for insurance-specific threats (mass claims export, privileged access anomalies, multi-stage attacks), automated response playbooks for containment workflows, and integrated Security Copilot for natural language investigation capabilities. Established compliance workbooks mapping HIPAA, PCI-DSS, and SOC 2 requirements to Sentinel telemetry.

Outcome: Sentinel detected credential stuffing campaigns via cross-subscription correlation and identified lateral movement across geographic regions within minutes. Security Copilot enabled tier-1 analysts to conduct sophisticated investigations using natural language queries without requiring advanced query language expertise. Centralized SIEM substantially reduced mean time to detect and investigate security incidents across the global infrastructure.

Configure log retention periods aligned with regulatory requirements, compliance mandates, and investigation timelines. Balance forensic evidence preservation requirements against storage costs through tiered retention strategies. Insufficient or inconsistent log retention policies destroy forensic evidence before investigations complete and create compliance gaps. Without disciplined retention aligned with regulatory and operational requirements: Inadequate retention transforms long-running breaches into uninvestigatable incidents while creating regulatory and legal exposure.

Balance forensic preservation needs with storage costs by implementing tiered retention strategies aligned to regulatory mandates and investigation timelines. Different log types require different retention periodsâIIhot storage for active investigations, warm storage for recent history, and cold archival for long-term complianceâIIoptimizing both investigative capability and operational costs. Configure the following retention strategy: Route logs to appropriate storage: Create diagnostic settings to route Azure Activity Logs and other resource logs to appropriate storage locations based on retention requirements, compliance mandates, and investigation timelines balancing hot vs. cold storage costs.

Configure short-to-medium term retention: Use Azure Monitor Log Analytics workspace for log retention up to 1-2 years for active investigation, threat hunting, and operational analytics with KQL query capabilities. Implement long-term archival storage: Use Azure Storage, Data Explorer, or Data Lake for long-term and archival storage beyond 1-2 years to meet compliance requirements (PCI-DSS, SEC 17a-4, HIPAA) with significant cost reduction through cold/archive tiers.

Forward logs externally: Use Azure Event Hubs to forward logs to external SIEM, data lake, or third-party security systems outside Azure when required for multi-cloud visibility or legacy integrations. Configure storage account retention: Configure retention policies for Azure Storage account logs based on compliance requirements, implementing lifecycle management for automatic tier transitions and deletion.

Plan Sentinel log retention: Implement long-term storage strategy for Microsoft Sentinel logs since Sentinel uses Log Analytics workspace as its backend, requiring explicit archival configuration for extended retention beyond workspace limits. Archive security alerts: Configure continuous export for Microsoft Defender for Cloud alerts and recommendations to meet retention requirements since Defender for Cloud data has limited retention in the native portal.

4. Practical Deployment, Analysis & Case Studies Challenge: A regulated financial services organization needed to meet multiple retention requirements (PCI-DSS: 1 year, SEC 17a-4: 7 years) while managing 80 TB of annual security log data cost-effectively. Solution approach: Implemented tiered log retention strategy by configuring Log Analytics workspace with 1-year default retention and table-level overrides (identity logs: 2 years, network flows: 90 days), exporting all logs to Azure Storage accounts with lifecycle management policies (hotâIIcool at 90 days, coolâIIarchive at 1 year), and configuring immutable storage (WORM) on compliance-critical accounts. Deployed Azure Policy enforcing diagnostic settings and retention configuration consistency, created Log Analytics query packs for automated compliance reporting, and established legal hold process for forensic data preservation during active incidents.

Outcome: Achieved complete audit trail compliance for PCI-DSS and SEC 17a-4 requirements while substantially reducing log storage costs through tiered storage strategy. Successfully investigated historical security incidents using archived logs beyond previous retention capabilities, and streamlined quarterly compliance audits through automated evidence collection and retention verification queries.

Synchronize all systems to authoritative time sources to maintain accurate timestamps across security logs. Consistent time synchronization enables reliable log correlation, incident timeline reconstruction, and forensic analysis. Accurate and synchronized time across all systems is fundamental to log correlation, forensic analysis, and incident timeline reconstruction. Without consistent time synchronization: Time synchronization failures undermine the evidentiary integrity of all security logging and monitoring, rendering forensic analysis inconclusive and unreliable.

Ensure accurate and consistent timestamps across all systems to enable reliable log correlation and incident timeline reconstruction. Time synchronization is foundational to forensic integrityâIIeven small clock drift can corrupt investigation timelines, cause SIEM correlation failures, and create compliance audit findings. Configure all systems to use trusted time sources and monitor for drift continuously. Implement these time synchronization practices: Configure Windows time sync: Use Microsoft default NTP servers for time synchronization on Azure Windows compute resources leveraging Azure host time sources through virtualization integration services unless specific requirements dictate otherwise.

Configure Linux time sync: Configure time synchronization for Azure Linux compute resources using chrony or ntpd with Azure-provided NTP sources or appropriate external NTP servers. Secure custom NTP servers: If deploying custom network time protocol (NTP) servers, secure the UDP service port 123 and implement access controls restricting time service queries to authorized clients only.

Validate timestamp formats: Verify that all logs generated by Azure resources include timestamps with time zone information (preferably UTC) by default to enable unambiguous timeline reconstruction across global deployments. Monitor time drift: Implement continuous monitoring of time drift across systems and configure alerts for significant synchronization issues (>5 seconds) that could impact log correlation, forensic analysis, and time-based authentication mechanisms.

Challenge: A global retail organization needed forensic timeline integrity across hybrid infrastructure (on-premises point-of-sale systems, Azure cloud POS backend, payment processing) spanning 2,500 stores for PCI-DSS compliance and fraud investigation. Solution approach: Configured comprehensive time synchronization by verifying Azure PaaS services' automatic NTP sync, configuring Azure VMs (Windows/Linux) to use Azure host time sources, and implementing Azure Monitor alerts for time drift >5 seconds. Deployed Log Analytics queries detecting timestamp anomalies across correlated log sources and automated remediation runbooks forcing time resync on affected VMs. Established quarterly time synchronization audits validating NTP configuration and timestamp consistency across application, identity, and network logs.

Outcome: Achieved compliance with PCI-DSS time synchronization requirements and successfully correlated fraud investigations across hybrid log sources with consistent timestamp accuracy enabling precise incident reconstruction. Automated time drift monitoring and remediation eliminated false positive security alerts caused by out-of-order events and ensured forensic timeline integrity across globally distributed infrastructure.

Use Google Cloud Audit Logs with Chronicle SIEM to detect insider threat patterns including data exfiltration, privilege abuse, and unauthorized access. Insider threats are the hardest security problems to solve. Unlike external attackers who trip network-level detections, insiders already have legitimate access. The signs are subtle - a developer downloading an unusual amount of data, an admin granting themselves permissions they do not normally use, or someone accessing resources outside their normal work hours.

Google Cloud's audit logs capture the raw data you need, and Chronicle SIEM gives you the detection and analytics layer to find these patterns. In this post, I will walk through building insider threat detection rules that go beyond simple threshold alerts. First, get your GCP audit logs flowing into Chronicle. Google SecOps can ingest supported Google Cloud logs directly, and a Pub/Sub-based feed is also common when you need an explicit export pipeline: Chronicle uses YARA-L 2.0 for detection rules. Here are several rules targeting insider threat patterns.

The real power comes from combining multiple weak signals into a strong detection. This rule triggers when multiple insider threat indicators fire for the same user: Insider threat detection is never a single rule or a single tool. It is about building layers of behavioral signals, correlating them over time, and having a clear response process. Chronicle SIEM gives you the detection engine, GCP audit logs provide the raw data, and BigQuery lets you build the analytics layer on top. Start with the highest-confidence detections like self-privilege escalation and manual key creation, tune the thresholds to your environment, and gradually add more behavioral rules as you reduce false positives.

Every OneUptime blog post is open source. Found a typo, an inaccuracy, or have a clearer way to explain something? Anyone can contribute — your edits make this post better for everyone who reads it next. Your complete reliability stack unified: infrastructure monitoring, incident management, status pages, and APM. Open-source and self-hostable.

Microsoft Defender for Identity cloud service helps protect your enterprise hybrid environments from multiple types of advanced targeted cyber attacks and insider threats. This guide shows security practitioners how to onboard Google Cloud logs to be used in security analytics. By performing security analytics, you help your organization prevent, detect, and respond to threats like malware, phishing, ransomware, and poorly configured assets.

The information in this guide is part of Google Cloud Autonomic Security Operations , which includes engineering-led transformation of detection and response practices and security analytics to improve your threat detection capabilities. In this guide, logs provide the data source to be analyzed. However, you can apply the concepts from this guide to analysis of other complementary security-related data from Google Cloud, such as security findings from Security Command Center .

Provided in Security Command Center Premium is a list of regularly-updated managed detectors that are designed to identify threats, vulnerabilities, and misconfigurations within your systems in near real-time. By analyzing these signals from Security Command Center and correlating them with logs ingested in your security analytics tool as described in this guide, you can achieve a broader perspective of potential security threats.

The following diagram shows how security data sources, security analytics tools, and CSA queries work together. The diagram starts with the following security data sources: logs from Cloud Logging, asset changes from Cloud Asset Inventory, and security findings from Security Command Center. The diagram then shows these security data sources being routed into the security analytics tool of your choice: Observability Analytics in Cloud Logging, BigQuery, Google Security Operations, or a third-party SIEM. Finally, the diagram shows using CSA queries with your analytics tool to analyze the collated security data.

This section describe the steps to set up security log analytics in Google Cloud. The workflow consists of the three steps shown in the following diagram and described in the following paragraphs: Enable logs: There are many security logs available in Google Cloud. Each log has different information that can be useful in answering specific security questions. Some logs like Admin Activity audit logs are enabled by default; others need to be manually enabled because they incur additional ingestion costs in Cloud Logging. Therefore, the first step in the workflow is to prioritize the security logs that are most relevant for your security analysis needs and to individually enable those specific logs.

To help you evaluate logs in terms of the visibility and threat detection coverage they provide, this guide includes a log scoping tool . This tool maps each log to relevant threat tactics and techniques in the MITRE ATT&CK® Matrix for Enterprise . The tool also maps Event Threat Detection rules in Security Command Center to the logs on which they rely. You can use the log scoping tool to evaluate logs regardless of the analytics tool that you use.

Route logs: After identifying and enabling the logs to be analyzed, the next step is to route and aggregate the logs from your organization, including any contained folders, projects, and billing accounts. How you route logs depends on the analytics tool that you use. This guide describes common log routing destinations, and shows you how to use a Cloud Logging aggregated sink to route organization-wide logs into a Cloud Logging log bucket or a BigQuery dataset depending on whether you choose to use Observability Analytics or BigQuery for analytics.

Analyze logs: After you route the logs into an analytics tool, the next step is to perform an analysis of these logs to identify any potential security threats. How you analyze the logs depends on the analytics tool that you use. If you use Observability Analytics or BigQuery, you can analyze the logs by using SQL queries. If you use Google Security Operations, you analyze the logs by using YARA-L rules . If you are using a third-party SIEM tool, you use the query language specified by that tool.

In this guide, you'll find SQL queries that you can use to analyze the logs in either Observability Analytics or BigQuery. The SQL queries provided in this guide come from the Community Security Analytics (CSA) project. CSA is an open-source set of foundational security analytics designed to provide you with a baseline of pre-built queries and rules that you can reuse to start analyzing your Google Cloud logs.

The following sections provide detailed information on how to set up and apply each step in the security logs analytics workflow. To help you identify the logs that meet your security and compliance needs, you can use the log scoping tool shown in this section. This tool provides an interactive table that lists valuable security-relevant logs across Google Cloud including Cloud Audit Logs, Access Transparency logs, network logs, and several platform logs. This tool maps each log type to the following areas: The log scoping tool also generates a log filter which appears immediately after the table. As you identify the logs that you need, select those logs in the tool to automatically update that log filter.

The log filter that is automatically generated by the log scoping tool contains all of the logs that you have selected in the tool. You can use the filter as is or you can refine the log filter further depending on your requirements. For example, you can include (or exclude) resources only in one or more specific projects. After you have a log filter that meets your logging requirements, you need to save the filter for use when routing the logs. For instance, you can save the filter in a text editor or save it in an environment variable as follows: For each of the platform logs that you select in the log scoping tool, those logs must be enabled (typically at the resource level) on a service-by-service basis. For example, Cloud DNS logs are enabled at the VPC-network level. Likewise, VPC Flow Logs are enabled at the subnet level for all VMs in the subnet, and logs from Firewall Rules Logging are enabled at the individual firewall rule level.

Each platform log has its own instructions on how to enable logging. However, you can use the log scoping tool to quickly open the relevant instructions for each platform log. As you can see in the log scoping tool, the Data Access audit logs from Cloud Audit Logs provide broad threat detection coverage. However, their volume can be quite large. Enabling these Data Access audit logs might therefore result in additional charges related to ingesting, storing, exporting, and processing these logs. This section both explains how to enable these logs and presents some best practices to help you with making the tradeoff between value and cost.

Data Access audit logs—except for BigQuery—are disabled by default. To configure Data Access audit logs for Google Cloud services other than BigQuery, you must explicitly enable them either by using the Google Cloud console or by using the Google Cloud CLI to edit Identity and Access Management (IAM) policy objects. When you enable Data Access audit logs, you can also configure which types of operations are recorded. There are three Data Access audit log types: Note that you can't configure the recording of ADMIN_WRITE operations, which are operations that write metadata or configuration information. ADMIN_WRITE operations are included in Admin Activity audit logs from Cloud Audit Logs and therefore can't be disabled.

5. Performance Evaluation & Extended Technical Observations When enabling Data Access audit logs, the goal is to maximize their value in terms of security visibility while also limiting their cost and management overhead. To help you achieve that goal, we recommend that you do the following to filter out low-value, high-volume logs: Prioritize relevant projects such as projects that host production workloads as opposed to projects that host developer and staging environments. To filter out all logs from a particular project, add the following expression to your log filter for your sink.

Replace PROJECT_ID with the ID of the project from which you want to filter out all logs: Prioritize a subset of data access operations such as ADMIN_READ , DATA_READ , or DATA_WRITE for a minimal set of recorded operations. For example, some services like Cloud DNS write all three types of operations, but you can enable logging for only ADMIN_READ operations. After you have configured one of more of these three types of data access operations, you might want to exclude specific operations that are particularly high volume. You can exclude these high volume operations by modifying the sink's log filter. For example, you decide to enable full Data Access audit logging, including DATA_READ operations on some critical storage services. To exclude specific high-traffic data read operations in this situation, you can add the following recommended log filter expressions to your sink's log filter: Prioritize relevant resources such as resources that host your most sensitive workloads and data. You can classify your resources based on the value of the data that they process, and their security risk such as whether they are externally accessible or not. Although Data Access audit logs are enabled per service, you can filter out specific resources or resource types through the log filter.

Exclude specific principals from having their data accesses recorded. For example, you can exempt your internal testing accounts from having their operations recorded. To learn more, see Set exemptions in Data Access audit logs documentation. The following table provides a baseline Data Access audit log configuration that you can use for Google Cloud projects to limit log volumes while gaining valuable security visibility: 1 Enabling Data Access audit logs for IAP or Cloud Storage can generate large log volumes when there is high traffic to IAP-protected web resources or to Cloud Storage objects.

2 Enabling Data Access audit logs for Cloud Storage might break the use of authenticated browser downloads for non-public objects. For more details and suggested workarounds to this issue, see the Cloud Storage troubleshooting guide . In the example configuration, notice how services are grouped in tiers of sensitivity based on their underlying data, metadata, or configuration. These tiers demonstrate the following recommended granularity of Data Access audit logging: After the logs are identified and enabled, the next step is to route the logs to a single destination. The routing destination, path and complexity vary depending on the analytics tools that you use, as shown in the following diagram.

If you use Observability Analytics, you need an aggregated sink to aggregate the logs from across your Google Cloud organization into a single Cloud Logging bucket. If you use BigQuery, you need an aggregated sink to aggregate the logs from across your Google Cloud organization into a single BigQuery dataset.

If you use Google Security Operations and this predefined subset of logs meets your security analysis needs, you can automatically aggregate these logs into your Google Security Operations account using the built-in Google Security Operations ingest. You can also view this predefined set of logs by looking at the Exportable directly to Google Security Operations column of the log scoping tool. For more information about exporting these predefined logs, see Ingest Google Cloud logs to Google Security Operations .

If you use BigQuery or a third-party SIEM or want to export an expanded set of logs into Google Security Operations, the diagram shows that an additional step is needed between enabling the logs and analyzing them. This additional step consists of configuring an aggregated sink that routes the selected logs appropriately. If you're using BigQuery, this sink is all that you need to route the logs to BigQuery. If you're using a third-party SIEM, you need to have the sink aggregate the selected logs in Pub/Sub or Cloud Storage before the logs can be pulled into your analytics tool.

The routing options to Google Security Operations and a third-party SIEM aren't covered in this guide. However, the following sections provide the detailed steps to route logs to Observability Analytics or BigQuery: Open the Google Cloud console in the Google Cloud project that you want to aggregate logs into. BUCKET_LOCATION : the geographical location of the new Logging bucket. The supported locations are global , us , or eu .

To learn more about these storage regions, refer to Supported regions . If you don't specify a location, then the global region is used, which means that the logs could be physically located in any of the regions. (Optional) Set the retention period of the logs in the bucket. The following example extends the retention of logs stored in the bucket to 365 days: Open the Google Cloud console in the Google Cloud project that you want to aggregate logs into.

DATASET_LOCATION : the geographic location of the dataset. After a dataset is created, the location can't be changed. PARTITION_EXPIRATION : the default lifetime (in seconds) for the partitions in the partitioned tables that are created by the log sink. You configure the log sink in the next section. The log sink that you configure uses partitioned tables that are partitioned by day based on the log entry's timestamp. Partitions (including associated log entries) are deleted PARTITION_EXPIRATION seconds after the partition's date.

You route your organization logs into your destination by creating an aggregated sink at the organization level. To include all the logs you selected in the log scoping tool, you configure the sink with the log filter generated by the log scoping tool. In a Cloud Shell terminal, run the following gcloud command to create an aggregated sink at the organization level: The --include-children flag is important so that logs from all the Google Cloud projects within your organization are also included. For more information, see Collate and route organization-level logs to supported destinations .

Copy the entire string for writerIdentity starting with serviceAccount: . This identifier is the sink's service account. Until you grant this service account write access to the log bucket, log routing from this sink will fail. You grant write access to the sink's writer identity in the next section. In a Cloud Shell terminal, run the following gcloud command to create an aggregated sink at the organization level: The --include-children flag is important so that logs from all the Google Cloud projects within your organization are also included. For more information, see Collate and route organization-level logs to supported destinations .

The --use-partitioned-tables flag is important so that data is partitioned by day based on the log entry's timestamp field. This simplifies querying of the data and helps reduce query costs by reducing the amount of data scanned by queries. Another benefit of partitioned tables is that you can set a default partition expiration at the dataset level to meet your log retention requirements. You have already set a default partition expiration when you created the dataset destination in the previous section. You might also choose to set a partition expiration at the individual table level, providing you with fine-grained data retention controls based on log type.

Copy the entire string for writerIdentity starting with serviceAccount: . This identifier is the sink's service account. Until you grant this service account write access to the BigQuery dataset, log routing from this sink will fail. You grant write access to the sink's writer identity in the next section. After creating the log sink, you must grant your sink access to write to its destination, be it the Logging bucket or the BigQuery dataset.

Make sure that you've selected the destination Google Cloud project that contains the Logging bucket you created for central log storage. In the New principals field, enter the sink's service account without the serviceAccount: prefix. Recall that this identity comes from the writerIdentity field you retrieved in the previous section after you created the sink.

Click Add IAM condition to restrict the service account's access to only the log bucket you created. In the Dataset info tab, click the Sharing keyboard_arrow_down drop-down menu, and then click Permissions . In the New principals field, enter the sink's service account without the serviceAccount: prefix. Recall that this identity comes from the writerIdentity field you retrieved in the previous section after you created the sink.

After you grant access to the sink, log entries begin to populate the sink destination: the Logging bucket or the BigQuery dataset. Now that your log sink routes logs from your entire organization into one single destination, you can search across all of these logs. Use IAM permissions to manage permissions and grant access as needed.

To grant access to view and query the logs in your new log bucket, follow these steps. Make sure you've selected the Google Cloud project you're using to aggregate the logs. This role provides the newly added principal with read access to all views for any buckets in the Google Cloud project. To limit a user's access, add a condition that lets the user read only from your new bucket only.

In the Value field, enter the bucket's location and name, and the default log view _AllLogs as follows: To grant access to view and query the logs in your BigQuery dataset, follow the steps in the Granting access to a dataset section of the BigQuery documentation. When you route logs to a log bucket upgraded to Observability Analytics, you can view and query all log entries through a single log view with a unified schema for all log types. Follow these steps to verify the logs are correctly routed.

Make sure you've selected the Google Cloud project you're using to aggregate the logs. Expand the log views under the log bucket that you have created (that is BUCKET_NAME ) if it is not expanded already. Select the default log view _AllLogs . You can now inspect the entire log schema in the right panel, as shown in the following screenshot: Harvested Online Documentation & Reference Sources • https://en.wikipedia.org/wiki/Intrusion%20detection%20system • https://en.wikipedia.org/wiki/Cloud%20computing%20security • https://en.wikipedia.org/wiki/Identity%20threat%20detection%20and%20response • https://learn.microsoft.com/en-us/security/benchmark/azure/mcsb-v2-logging-threat-detection • https://oneuptime.com/blog/post/2026-02-17-how-to-detect-insider-threat-patterns-using-google-cloud-audit-logs-and-chroni cle-siem/view • https://learn.microsoft.com/en-us/defender-for-identity/ • https://docs.cloud.google.com/architecture/security-log-analytics