Automated Log Forensic Investigation Tool For Linux — Complete Documentation & Simulation

Overview & System Context

Automated Log Forensic Investigation Tool For Linux: Comprehensive Web-Sourced Technical Investigation Exhaustive Research Dossier · Pure Web-Scraped Content: ~9509 Words · Real Software Screenshots: 3 1. Executive Overview & Technical Background Computer Online Forensic Evidence Extractor (COFEE) is a tool kit, developed by Microsoft, to help computer forensic investigators extract evidence from a Windows computer. Installed on a USB flash drive or other external disk drive, it acts as an automated forensic tool during a live analysis. Microsoft provides COFEE devices and online technical support free to law enforcement agencies.

In mid to late 2009 a tool named Detect and Eliminate Computer Acquired Forensics (DECAF) was announced by an uninvolved group of programmers. The tool would reportedly protect computers against COFEE and render the tool ineffective. It alleged that it would provide real-time monitoring of COFEE signatures on USB devices and in running applications and that when a COFEE signature is detected, DECAF would perform numerous user-defined processes. These included COFEE log clearing, ejecting USB devices, and contamination or spoofing of MAC addresses. On December 18, 2009, the DECAF creators announced that the tool was a hoax and part of "a stunt to raise awareness for security and the need for better forensic tools".

Security information and event management (SIEM) is a field within computer security that combines security information management (SIM) and security event management (SEM) to enable real-time analysis of security alerts generated by applications and network hardware. SIEM systems are central to security operations centers (SOCs), where they are employed to detect, investigate, and respond to security incidents. SIEM technology collects and aggregates data from various systems, allowing organizations to meet compliance requirements while safeguarding against threats. NIST's definition for a SIEM tool is an application that provides the ability to gather security data from information system components and present that data as actionable information via a single interface. SIEM tools can be implemented as software, hardware, or managed services. SIEM systems log security events and generate reports to meet regulatory frameworks such as the Health Insurance Portability and Accountability Act (HIPAA) and the Payment Card Industry Data Security Standard (PCI DSS). The integration of SIM and SEM within SIEM provides organizations with a centralized approach for monitoring security events and responding to threats in real-time. First introduced by Gartner analysts Mark Nicolett and Amrit Williams in 2005, the term SIEM has evolved to incorporate advanced features such as threat intelligence and behavioral analytics, which allow SIEM solutions to manage complex cybersecurity threats, including zero-day vulnerabilities and polymorphic malware. In recent years, SIEM has become increasingly incorporated into national cybersecurity initiatives. For instance, Executive Order 14028 signed in 2021 by U.S. President Joseph Biden mandates the use of SIEM technologies to improve incident detection and reporting in federal systems. Compliance with these mandates is further reinforced by frameworks such as NIST SP 800-92, which outlines best practices for managing computer security logs. Modern SIEM platforms are aggregating and normalizing data not only from various Information Technology (IT) sources, but from production and manufacturing Operational Technology (OT) environments as well.

Log management: Focus on simple collection and storage of log messages and audit trails. Security information management (SIM): Long-term storage as well as analysis and reporting of log data. Security event manager (SEM): Real-time monitoring, correlation of events, notifications and console views. Security information and event management (SIEM): Combines SIM and SEM and provides real-time analysis of security alerts generated by network hardware and applications. Managed Security Service: (MSS) or Managed Security Service Provider: (MSSP): The most common managed services appear to evolve around connectivity and bandwidth, network monitoring, security, virtualization, and disaster recovery. Security as a service (SECaaS): These security services often include authentication, anti-virus, anti-malware/spyware, intrusion detection, penetration testing and security event management, among others. In practice many products in this area will have a mix of these functions, so there will often be some overlap – and many commercial vendors also promote their own terminology.

Oftentimes commercial vendors provide different combinations of these functionalities which tend to improve SIEM overall. Log management alone doesn't provide real-time insights on network security, SEM on its own won't provide complete data for deep threat analysis. When SEM and log management are combined, more information is available for SIEM to monitor. A key focus is to monitor and help manage user and service privileges, directory services and other system-configuration changes; as well as providing log auditing and review and incident response.

Figure 1: Zimmerman Forensic Tools _ Eric Zimmerman Log Viewer – YJCY SIEM architectures may vary by vendor; however, generally, essential components comprise the SIEM engine. The essential components of a SIEM are as follows: A data collector forwards selected audit logs from a host (agent based or host based log streaming into index and aggregation point) An ingest and indexing point aggregation point for parsing, correlation, and data normalization A search node that is used for visualization, queries, reports, and alerts (analysis take place on a search node) A basic SIEM infrastructure is depicted in the image to the right.

SIEM visibility and anomaly detection could help detect zero-days or polymorphic code. Primarily due to low rates of anti-virus detection against this type of rapidly changing malware. Parsing, log normalization and categorization can occur automatically, regardless of the type of computer or network device, as long as it can send a log. Visualization with a SIEM using security events and log failures can aid in pattern detection. Protocol anomalies that can indicate a misconfiguration or a security issue can be identified with a SIEM using pattern detection, alerting, baseline and dashboards. SIEMS can detect covert, malicious communications and encrypted channels. Cyberwarfare can be detected by SIEMs with accuracy, discovering both attackers and victims. Modern SIEM platforms support not only detection, but response too. The response can be manual or automated including AI based response.

SIEM systems can have hundreds and thousands of correlation rules. Some of these are simple, and some are more complex. Once a correlation rule is triggered the system can take appropriate steps to mitigate a cyber attack. Usually, this includes sending a notification to a user and then possibly limiting or even shutting down the system. Figure 2: MemoryGuard: Automated Forensic Investigation | Devpost In computing, data recovery is a process of retrieving deleted, inaccessible, lost, corrupted, damaged, or overwritten data from secondary storage, removable media or files, when the data stored in them cannot be accessed in a usual way. The data is most often salvaged from storage media such as internal or external hard disk drives (HDDs), solid-state drives (SSDs), USB flash drives, magnetic tapes, CDs, DVDs, RAID subsystems, and other electronic devices. Recovery may be required due to physical damage to the storage devices or logical damage to the file system that prevents it from being mounted by the host operating system (OS). Logical failures occur when the hard drive devices are functional but the user or automated-OS cannot retrieve or access data stored on them. Logical failures can occur due to corruption of the engineering chip, lost partitions, firmware failure, or failures during formatting/re-installation. Data recovery can be a very simple or technical challenge.

Therefore, specialized software companies and professionals work in this field to help extract data from faulty computer systems. A common misconception is that a damaged printed circuit board (PCB) may be simply replaced during recovery procedures by an identical PCB from a healthy drive. While this may work in rare circumstances on hard disk drives manufactured before 2003, it will not work on newer drives. Electronics boards of modern drives usually contain drive-specific adaptation data (generally a map of bad sectors and tuning parameters) and other information required to properly access data on the drive.

Replacement boards often need this information to effectively recover all of the data. The replacement board may need to be reprogrammed. Some manufacturers (Seagate, for example) store this information on a serial EEPROM chip, which can be removed and transferred to the replacement board. Each hard disk drive has what is called a system area or service area; this portion of the drive, which is not directly accessible to the end user, usually contains drive's firmware and adaptive data that helps the drive operate within normal parameters. One function of the system area is to log defective sectors within the drive; essentially telling the drive where it can and cannot write data. The sector lists are also stored on various chips attached to the PCB, and they are unique to each hard disk drive. If the data on the PCB do not match what is stored on the platter, then the drive will not calibrate properly. In most cases the drive heads will click because they are unable to find the data matching what is stored on the PCB.

The term logical damage refers to situations in which the error is not a problem in the hardware and requires software-level solutions. After data has been physically overwritten on a hard disk drive, it is generally assumed that the previous data are no longer possible to recover. In 1996, Peter Gutmann, a computer scientist, presented a paper that suggested overwritten data could be recovered through the use of magnetic force microscopy. In 2001, he presented another paper on a similar topic. To guard against this type of data recovery, Gutmann and Colin Plumb designed a method of irreversibly scrubbing data, known as the Gutmann method and used by several disk-scrubbing software packages. Substantial criticism has followed, primarily dealing with the lack of any concrete examples of significant amounts of overwritten data being recovered. Gutmann's article contains a number of errors and inaccuracies, particularly regarding information about how data is encoded and processed on hard drives. Although Gutmann's theory may be correct, there is no practical evidence that overwritten data can be recovered, while research has shown to support that overwritten data cannot be recovered. Solid-state drives (SSD) overwrite data differently from hard disk drives (HDD) which makes at least some of their data easier to recover. Most SSDs use flash memory to store data in pages and blocks, referenced by logical block addresses (LBA) which are managed by the flash translation layer (FTL). When the FTL modifies a sector it writes the new data to another location and updates the map so the new data appear at the target LBA. This leaves the pre-modification data in place, with possibly many generations, and recoverable by data recovery software.

Figure 3: Exploring Forensic Investigation: A Beginners Guide to Linux Log ... Phase 1 Repair the hard disk drive The hard drive is repaired in order to get it running in some form, or at least in a state suitable for reading the data from it. For example, if heads are bad they need to be changed; if the PCB is faulty then it needs to be fixed or replaced; if the spindle motor is bad the platters and heads should be moved to a new drive. Phase 2 Image the drive to a new drive or a disk image file When a hard disk drive fails, the importance of getting the data off the drive is the top priority. The longer a faulty drive is used, the more likely further data loss is to occur. Creating an image of the drive will ensure that there is a secondary copy of the data on another device, on which it is safe to perform testing and recovery procedures without harming the source. Phase 3 Logical recovery of files, partition, MBR and filesystem structures After the drive has been cloned to a new drive, it is suitable to attempt the retrieval of lost data. If the drive has failed logically, there are a number of reasons for that. Using the clone it may be possible to repair the partition table or master boot record (MBR) in order to read the file system's data structure and retrieve stored data. Phase 4 Repair damaged files that were retrieved Data damage can be caused when, for example, a file is written to a sector on the drive that has been damaged. This is the most common cause in a failing drive, meaning that data needs to be reconstructed to become readable. Corrupted documents can be recovered by several software methods or by manually reconstructing the document using a hex editor.

Data recovery cannot always be done on a running system. As a result, a boot disk, live CD, live USB, or any other type of live distro contains a minimal operating system. BartPE: a lightweight variant of Microsoft Windows XP or Windows Server 2003 32-bit operating systems, similar to a Windows Preinstallation Environment, which can be run from a live CD or live USB drive. Discontinued. Finnix: a Debian-based Live CD with a focus on being small and fast, useful for computer and data rescue Disk Drill: capable of creating bootable macOS USB drives for data recovery Knoppix: contains utilities for data recovery under Linux SystemRescue: an Arch Linux-based live CD, useful for repairing unbootable computer systems and retrieving data after a system crash Grml: A Debian-based live-system for system administrators. Windows Preinstallation Environment (WinPE): A customizable Windows Boot DVD (made by Microsoft and distributed for free). Can be modified to boot to any of the programs listed.

Foremost: an open-source command-line file recovery program, originally developed by the Air Force Office of Special Investigations and NPS Center for Information Systems Security Studies and Research Forensic Toolkit: by AccessData, used by law enforcement Open Computer Forensics Architecture: An open-source program for Linux The Coroner's Toolkit: a suite of utilities for assisting in forensic analysis of a UNIX system after a break-in The Sleuth Kit: also known as TSK, a suite of forensic analysis tools developed by Brian Carrier for UNIX, Linux and Windows systems. TSK includes the Autopsy forensic browser.

Clonezilla: a free disk cloning, disk imaging, data recovery, and deployment boot disk dd: common byte-to-byte cloning tool found on Unix-like systems ddrescue: an open-source tool similar to dd but with the ability to skip over and subsequently retry bad blocks on failing storage devices Team Win Recovery Project: a free and open-source recovery system for Android devices When a Linux server is suspected to be compromised, every second matters. Attackers may be actively running processes, maintaining persistence, or exfiltrating data.

This playbook provides a clear and actionable approach to collecting forensic evidence from a live Linux system without disrupting critical artifacts. It focuses on prioritizing volatile data , understanding attacker behavior , and ensuring that evidence is preserved in a forensically sound manner . Rather than listing generic commands, this guide helps you think like an investigator —what to collect first, what matters most, and how attackers typically leave traces on Linux systems.

Linux DFIR Acquisition Playbook provides a structured and practical workflow for collecting forensic evidence from compromised Linux systems during incident response. It covers volatile data, logs, memory, disk acquisition, persistence artifacts, and network indicators to support thorough and repeatable investigations. Linux systems maintain extensive logs that serve as primary evidence sources in forensic investigations. Unlike Windows Event Logs, Linux logs are typically plain-text files stored in /var/log/ and binary journal files managed by systemd-journald.

Key forensic logs include auth.log (authentication events, sudo usage, SSH sessions), syslog (system-wide messages), kern.log (kernel events), and application-specific logs. The Linux Audit framework (auditd) provides detailed security event logging comparable to Windows Security Event Logs. Forensic analysis of these logs enables investigators to reconstruct user sessions, identify unauthorized access, detect privilege escalation, trace lateral movement, and establish comprehensive event timelines.

2. System Architecture & Core Working Principles Linux uses a hierarchical file system. The root directory (/) is the top-level directory, and all other directories and files are organized beneath it. Understanding the common directories such as /etc (for system configuration files), /var (for variable data like logs), and /home (for user home directories) is crucial for forensic investigations.

Linux systems maintain various log files that record system events. Logs can be found in the /var/log directory. For example, syslog contains general system messages, and auth.log records authentication-related events. Linux uses a user-group-based permission system. User accounts are defined in the /etc/passwd file, and group information is stored in the /etc/group file. Forensic analysts can review these files to identify any unauthorized user accounts or group membership changes.

One of the first steps in a forensic investigation is to acquire a copy of the evidence. This can be done using tools like dd to create a bit-by-bit copy of a disk. The enhanced tool dc3dd adds on-the-fly hashing and progress reporting, making it preferable for forensic use. Guymager is a fast, graphical alternative that supports multiple image formats (E01, AFF, raw) and provides built-in hash verification.

Tools like fsstat from the Sleuth Kit can be used to analyze the file system structure of an acquired image. As mentioned earlier, log files contain valuable information. Tools like grep can be used to search for specific keywords in log files. Timeline analysis is a critical technique in Linux forensics that reconstructs the sequence of events on a system. Tools like log2timeline/plaso can automatically parse hundreds of artifact typesâIIincluding file system metadata, logs, browser history, and shell artifactsâIIand merge them into a single "super timeline." This allows investigators to correlate events across different sources and establish a clear chronology of attacker activity.

Volatile memory (RAM) contains critical forensic artifacts that do not persist on disk, including running processes, open network connections, encryption keys, and evidence of in-memory malware. On Linux systems, memory acquisition is typically performed using LiME (Linux Memory Extractor) , a loadable kernel module that captures RAM to a file. Once captured, memory dumps can be analyzed with the Volatility Framework , the leading open-source tool for memory forensics. Volatility supports Linux, Windows, and macOS memory samples and can extract process lists, network connections, loaded kernel modules, and more.

The ps command can be used to list all running processes. Analysts can look for any abnormal or unauthorized processes. On a live system, top or htop can reveal processes consuming unusual resources, which may indicate cryptominers or other malicious activity. Tools like ss (the modern replacement for the deprecated netstat ) can be used to view active network connections. This can help in identifying any suspicious network activity.

The Sleuth Kit's fls and icat commands can be used to recover deleted files from an image. It is essential to maintain a proper chain of custody for all evidence. This includes documenting who had access to the evidence, when it was acquired, and any changes made to it. A simple chain-of-custody log can be maintained in a text file. Open-source tools like the Sleuth Kit, Autopsy (a graphical front-end for the Sleuth Kit), Volatility (for memory forensics), and Wireshark (for network forensics) are widely used in Linux forensics. Specialized forensic distributions such as CAINE and the SANS SIFT Workstation come pre-loaded with hundreds of forensic utilities and provide a ready-to-use analysis environment.

Tools like Guymager offer fast, reliable disk imaging with built-in hash verification, while dc3dd enhances the standard dd command with on-the-fly hashing and forensic-specific features. These tools are cost-effective, allow code auditing for court admissibility, and are supported by active communities. All steps of the investigation should be documented, including the commands used, the findings, and any conclusions. This documentation can be used in legal proceedings and for future reference.

Linux forensics is a complex but essential field for investigating security incidents on Linux-based systems. By understanding the fundamental concepts of the Linux file system, logging, and user management, analysts can navigate the evidence landscape effectively. Using the right toolsâIIfrom disk imaging with dd and Guymager to memory analysis with Volatility and timeline reconstruction with plasoâIIenables investigators to uncover the full scope of an incident. Following best practices for chain of custody, documentation, and the use of validated open-source tools ensures that findings are reliable and admissible. Whether it's a small-scale investigation or a large-scale corporate security breach, Linux forensics provides the necessary tools and techniques to uncover the truth.

How forensic specialists locate, acquire, parse, and interpret Linux log files to reconstruct authentication history, system events, attacker activity, and log tampering on compromised systems. Linux log files are the primary audit trail for the operating system and its services. On a compromised Linux host, the logs in /var/log/ and the systemd journal are often where the most direct evidence of attacker activity lives âII authentication attempts, privilege escalation, service manipulation, cron job additions, and command execution history. Understanding the structure of each log source, what it records, what it does not record, and how to extract meaningful evidence from it is a fundamental competency for any forensic specialist working in Linux environments.

This article covers the major Linux log sources in technical depth: the syslog facility and severity architecture, the specific log files that appear on Debian/Ubuntu and Red Hat/RHEL-family systems, the auth.log format and authentication event patterns, the systemd journal binary format and journalctl query capabilities, supplementary sources such as wtmp/btmp, lastlog, and the Linux Audit framework (auditd), and the indicators that reveal when log files have been tampered with. The analysis patterns and command-line workflows throughout apply to both live systems and forensic images.

Linux logging has evolved through several generations. Understanding which logging infrastructure is active on a target system determines both where to find log data and how to parse it correctly. The original syslog daemon, defined by RFC 3164 and later RFC 5424, receives log messages from applications and the kernel through a Unix socket (traditionally /dev/log ) and routes them to files or remote destinations based on facility and severity rules. The facility identifies the source category — kernel (0), user (1), mail (2), system daemons (3), security/authorization (4), syslogd internal (5), printer (6), network news (7), UUCP (8), cron (9), authpriv (10, used by SSH and sudo on many systems in place of facility 4), FTP (11), local0 through local7 (16–23), and others. The severity level ranges from 0 (Emergency) through 7 (Debug). The combination of facility and severity as a dotted pair — for example, auth.info or kern.err — drives routing rules.

Rsyslog replaced the original syslogd on the major modern distributions. It preserves full compatibility with the facility/severity routing model but adds high-performance buffering, template-driven output formatting, database output, encrypted transport to remote syslog servers, and the ability to parse structured message content. Distributions including Debian, Ubuntu, RHEL, CentOS, Fedora, and SUSE ship rsyslog as the default syslog implementation, and its configuration at /etc/rsyslog.conf and /etc/rsyslog.d/*.conf determines which facility/severity combinations go to which files. In environments where rsyslog feeds log data to a centralized platform, standardizing log ingestion for SIEM correlation becomes essential for ensuring that facility/severity classifications translate correctly across heterogeneous sources.

3. Software Implementation, Frameworks & Protocols Journald (systemd-journald) is the logging component of the systemd init system, present on all systemd-based distributions âII which encompasses essentially all major distributions since approximately 2014âII2015. Journald collects messages from kernel, from processes via /dev/log and native journal socket, and directly from all systemd services via their stdout/stderr. It stores them in a binary, indexed format rather than plain text. On systems running both rsyslog and journald, rsyslog typically reads from journald's socket and writes its output to traditional text log files in /var/log/ , creating a parallel: the binary journal in /var/log/journal/ and the traditional text files side by side. This means analysts often have two sources for the same events, and discrepancies between them may indicate selective tampering.

The specific filenames and locations of Linux log files vary by distribution family. A forensic analyst examining an unfamiliar Linux system should verify the distribution and version first, then consult the appropriate file mapping. Cloud-deployed Linux instances add another layer of consideration. AWS EC2, Google Cloud, and Azure Linux VMs often forward system logs to cloud-native logging services (CloudWatch Logs, Cloud Logging, Azure Monitor Logs) either instead of or in addition to writing local files. The reason this matters forensically is that an attacker who compromises the Linux host can manipulate local log files, but log data that was already forwarded to the cloud provider's logging infrastructure is typically outside the attacker's reach — making the cloud-side copy a higher-integrity evidence source. On containerized workloads (Docker, Kubernetes), container stdout/stderr may be the primary log capture mechanism, with local /var/log/ files either absent or containing only host-level messages rather than workload activity. Kubernetes clusters typically aggregate container logs through a node-level agent (such as Fluentd or Fluent Bit) that ships logs to a centralized backend, meaning the evidence for a compromised container may exist only in the aggregation platform, not on the node's local filesystem. The analyst examining a cloud or container Linux host must determine the logging architecture before concluding that any particular source is absent.

Traditional syslog messages written to text files follow a consistent format. The RFC 3164 (BSD syslog) format, which remains the default output on the major Linux distributions, is: The fields are the timestamp (month, day, time in local time — notably without a year and without timezone), the hostname (the system that generated the message), the process name (the program that logged the message, without path), the PID in brackets (optional but common), a colon, and then the free-form message text. As RFC 3164 specifies, the timestamp uses local time in a 24-hour format where the hour, minute, and second fields each range from 00 to 59 (or 00 to 23 for the hour).

The lack of a year in the timestamp is a well-known limitation acknowledged in the RFC itself — when reading rotated log files that span year boundaries, the analyst must infer the year from file metadata and content context. The lack of timezone information is equally important: all timestamps are in the system's local timezone at the time of logging, which may differ from UTC and may have been adjusted by DST transitions within the log's time window.

The rsyslog template can be configured to produce different formats. Many organizations enable the RFC 5424 (IETF syslog) format, which superseded the earlier BSD format. RFC 5424 adds a version number, ISO 8601 timestamp with timezone offset, structured data fields, and a message ID, producing lines that look like: The priority value in angle brackets is the numerical encoding of facility multiplied by 8 plus severity âII a priority of 38 decodes to facility 4 (security/authorization) and severity 6 (informational). Decoding the priority value confirms the facility classification of a message independent of the file it appeared in.

Enter a numeric syslog priority value (0–191) from an RFC 5424 message header to decode its facility and severity, and understand its forensic significance. When examining syslog for forensic evidence, targeted pattern searches are far more efficient than reading the entire file. The patterns with the highest forensic yield in a typical investigation are messages from cron (recording scheduled job execution and any cron configuration changes), messages from systemd (recording service state changes, unit file loading, and process start/stop), messages from kernel (recording module loads, device attachment, and network interface changes), and messages from any service that does not have a dedicated log file on that distribution. Attackers who install a malicious cron job, start a new service, or load a kernel module will generate syslog entries for those events even if they subsequently delete other evidence.

Because BSD syslog timestamps record local time without timezone information, determining the system's configured timezone is a prerequisite for accurate event correlation. On a forensic image, the timezone is stored in /etc/timezone (Debian/Ubuntu, as a plain text region string like America/New_York ) or as a symlink at /etc/localtime pointing to a timezone definition file under /usr/share/zoneinfo/ . Running readlink -f /mnt/evidence/etc/localtime resolves the symlink and reveals the timezone. If the system used timedatectl , the configuration may also be reflected in /etc/adjtime (which indicates whether the hardware clock is set to UTC or local time).

The timezone matters because correlating syslog timestamps against other evidence sources — network packet captures (typically in UTC), cloud provider logs (UTC), web server logs (configurable, often UTC), and journal entries (stored internally as microseconds since the Unix epoch, which is UTC) — requires converting all timestamps to a common reference. A one-hour timezone offset that is not accounted for will produce false correlations or missed connections between related events across sources. The problem compounds when a system's timezone was changed during the incident period (whether by an administrator or by an attacker as an anti-forensic measure), because syslog entries before the change use the old timezone offset and entries after the change use the new one, with no marker in the log indicating where the change occurred.

Searching for timedatectl or timezone in both the journal and syslog can reveal timezone change events if they were logged. Convert between BSD syslog format, ISO 8601, and Unix epoch timestamps. Specify the system's timezone and an assumed year (BSD syslog omits both). NTP synchronization status is equally important for establishing clock accuracy. If the system's clock was unsynchronized or drifted significantly, all local timestamps are offset from real-world time by an unknown amount. Checking /etc/chrony/chrony.conf or /etc/ntp.conf on the forensic image reveals whether NTP was configured, and searching syslog or the journal for messages from chronyd , ntpd , or systemd-timesyncd reveals whether synchronization was active and whether any large time adjustments occurred during the relevant period. A sudden time step recorded by the NTP daemon means all timestamps before that adjustment are offset by the step amount relative to timestamps after it — a fact that must be accounted for in the forensic timeline.

Auth.log (or /var/log/secure on Red Hat systems) is the single highest-value forensic log file on a typical Linux system. It collects all messages in the auth and authpriv facilities — authentication events from SSH, PAM (Pluggable Authentication Modules), sudo, su, and the login daemon. For an incident involving unauthorized access, remote exploitation, or privilege escalation, this file is the first place to examine.

SSH authentication generates four primary message patterns in auth.log. A successful password authentication appears as: The public key fingerprint in the accepted publickey message is a significant forensic detail âII it identifies which specific key was used for the authentication. If the key fingerprint matches a known authorized key in the account's ~/.ssh/authorized_keys file, the authentication was expected. If it does not match any current authorized key, either the key was removed after authentication or the log entry reflects an authentication against a key that was added and then removed âII a pattern consistent with an attacker installing and then removing a backdoor SSH key.

Starting with OpenSSH 9.8 (released July 2024), the SSH server splits into two processes as part of improved privilege separation: the listener process retains the name sshd , but the per-connection child process that handles authentication and session management now logs as sshd-session rather than sshd . This means that on systems running OpenSSH 9.8 or later (including Ubuntu 24.10+, Fedora 41+, and Arch Linux), authentication log entries appear as sshd-session[PID]: Accepted publickey for... instead of sshd[PID]: Accepted publickey for... . A forensic analyst whose grep patterns use sshd\[ to match authentication entries will miss every entry on a post-9.8 system. The portable fix is to grep for sshd without the bracket anchor (which matches both sshd[ and sshd-session[ ), or to use sshd.*\[ as the pattern. When examining an unfamiliar system, checking sshd -V output (or the SSH version string in the log's server startup entry) determines which process name convention is in use.

For a brute-force or password spray investigation, counting failed authentication attempts per source IP is the standard first step. The command pattern is: This pipeline extracts the source IP from each failed password line, sorts, deduplicates with count, and shows the top twenty sources by failure count. An IP with hundreds or thousands of failures followed by a successful authentication from the same IP is the classic brute-force success pattern. A caveat: the awk '{print $11}' field position assumes the standard "Failed password for [user] from [IP]" message format. When the username is invalid, sshd inserts the additional words "invalid user" before the username, shifting the IP to field $13 instead of $11 . A more robust approach uses grep -oP 'from \K[0-9.]+' to extract the IP by pattern rather than field position, which handles both valid and invalid username messages correctly.

A second caveat that trips even experienced analysts: rsyslog's RepeatedMsgReduction setting (enabled by default on many distributions, set as $RepeatedMsgReduction on in older-format config or module(load="builtin:omfile" RepeatedMsgContraction="on") in newer RainerScript config) collapses consecutive identical messages into a single "last message repeated N times" line. During a rapid brute-force attack where the same "Failed password" message repeats hundreds of times per second, rsyslog may reduce what were 500 individual failed attempts into a handful of entries followed by "last message repeated 498 times." This collapses the per-attempt timing data that reveals the tool's request rate and whether attempts were serialized or parallelized. The journal does not perform this collapsing — every event is stored individually — which makes it the preferred source for brute-force timing analysis on systems running both rsyslog and journald. On a forensic image, checking /etc/rsyslog.conf for the RepeatedMsgReduction directive reveals whether this compression was active.

The fields record the invoking user, the terminal, the working directory, the target user (root in this case), and the exact command executed. This is a direct audit trail of privilege escalation. An attacker who gains access to a sudo-enabled account and then runs sudo /bin/bash or sudo su - will generate this entry. Failed sudo attempts âII typically because the user is not in the sudoers file âII generate: The "user NOT in sudoers" message is a red flag that warrants investigation âII it means an account that does not have sudo privileges attempted to use them, which is consistent with an attacker testing the capabilities of a compromised low-privilege account.

4. Practical Deployment, Analysis & Case Studies After landing a shell as a low-privilege user like www-data , your first move is capability testing: sudo -l to see what (if anything) this account can run as root, id to check group memberships, and cat /etc/sudoers (which will fail but the attempt itself generates no log if you lack read access). The sudo -l command is the dangerous one — it generates an auth.log entry even though it is just listing allowed commands. If the account is not in sudoers at all, trying sudo on anything produces the "NOT in sudoers" entry that immediately alerts defenders. Experienced attackers check /etc/sudoers permissions first and enumerate capabilities passively (reading /etc/group , checking for SUID binaries, inspecting /etc/passwd for accounts with shells) before touching sudo at all.

If sudo works, the next concern is the audit trail. Running sudo /bin/bash logs the exact command. Running sudo su - logs COMMAND=/bin/su - . Both leave a clear escalation record. Attackers who have found credential reuse or can crack a password hash may instead su directly to root, which generates a different log pattern ( su: pam_unix(su-l:session): session opened ) that some monitoring rules don't watch as closely as sudo events.

The "NOT in sudoers" entry is one of the most reliable early compromise indicators. Legitimate users almost never trigger it — they know their own access level. When you see it, immediately answer three questions: (1) What account triggered it? Cross-reference that account against /etc/passwd to determine if it is a service account or user account, and check whether it has an interactive shell assigned. (2) What was the source of the session? Trace the PID back to an SSH session or local login via the PAM session-open entry that preceded it. (3) What happened immediately after? If the attacker pivoted to a different escalation method (SUID binary exploitation, kernel exploit, or switching to a different compromised account), the subsequent log entries within seconds of the failed sudo reveal the attack path.

Check for the absence of evidence too. If you see a "NOT in sudoers" entry from www-data at 15:04, but no SSH session for www-data , the attacker likely gained a shell through the web application (command injection, webshell, deserialization exploit). Correlate the timestamp against web server access logs for the same time window to identify the exploitation request. Starting with systemd 256 (June 2024), the systemd project introduced run0 as a replacement for sudo that operates through a fundamentally different mechanism. Rather than elevating privileges within the current session via the setuid bit, run0 uses the systemd service manager to launch a new service under the target user's identity via systemd-run . The forensic implication is significant: run0 does not generate the traditional sudo: entries in auth.log described above. Instead, privilege escalation through run0 generates systemd service manager journal entries for the transient service it creates, and authentication events through polkit (PolicyKit) rather than PAM's sudo module. On systems where run0 has replaced or supplements sudo , an analyst searching auth.log exclusively for sudo patterns will miss privilege escalation events entirely.

The journal becomes the primary evidence source for run0 activity, and queries should include systemd-run and polkit events alongside traditional sudo patterns. As of early 2026, no major distribution ships run0 as the default sudo replacement, but it is available and may be in use on systems where administrators have adopted it. PAM (Pluggable Authentication Modules) records session events that establish the lifecycle of authenticated sessions.

Session open and close events appear as: The time between session open and close gives the session duration. Sessions that close immediately after opening may indicate automated tool use rather than interactive work. Sessions that open but never close before the log ends indicate a session that was still active at the time of log acquisition âII or a session that was killed abnormally. The PAM events also record su and login sessions, providing coverage for local console authentication that SSH events do not capture.

Account creation and modification events from useradd , usermod , and passwd also appear in auth.log when those operations go through PAM. An attacker who creates a backdoor account will generate a useradd entry here, providing a timestamp for the account creation that can be cross-referenced against the account's entry in /etc/passwd and /etc/shadow . When a suspicious source IP appears in auth.log, search for that same IP across all available log sources âII web server access logs, application logs, firewall logs, and syslog. An IP that appears in auth.log with failed SSH attempts may also appear in Apache or nginx access logs attempting web application exploits, or in firewall logs scanning multiple ports. The broader cross-log pattern establishes whether the SSH activity was part of a broader automated scan or a targeted attack against a specific service.

The systemd journal stores log data in a binary format that is significantly more structured and queryable than plain text syslog files. Understanding the journal's structure and how to extract data from it is essential for any forensic analysis of a modern Linux system. Journal files are stored in /var/log/journal/[machine-id]/ when persistent storage is enabled, where the machine-id is the 32-character hex identifier from /etc/machine-id . If persistent storage is not configured, journals are stored in /run/log/journal/ and lost on reboot. Each journal directory contains multiple files: system.journal for system-level messages and user-[UID].journal for per-user session journals.

Rotated journals are named with timestamps: system@[boot-id]-[timestamp]-[sequence].journal .

Whether journald stores logs persistently by default is one of the more confusing topics in Linux administration because the answer depends on the distribution, the version, and the upstream systemd compile-time default — and online resources frequently contradict each other. Here is the current state as of early 2026: Upstream systemd changed its default in version 259 (December 2025). Prior to v259, the compile-time default for the Storage option in journald.conf was auto , which behaved like persistent only if the /var/log/journal/ directory already existed, and behaved like volatile if it did not. As of systemd v259, the upstream default is now persistent , meaning journald will create the /var/log/journal/ directory and write persistent logs regardless of whether the directory existed previously. This is a significant change for forensic analysts, because systems running v259 or later are far more likely to have journal history available across reboots. However, distributions can and do override compile-time defaults, and existing installations that upgraded to v259 may still carry their previous configuration.

The distribution-level picture remains uneven. Debian enabled persistent journal by default with the upload of systemd 244.1-2 in February 2020, which landed in the Bullseye (Debian 11) release in August 2021. As Debian developer Michael Biebl announced on the debian-devel mailing list, the package creates a /var/log/journal directory on upgrades and new installs, which enables persistent journal in "auto" mode. Ubuntu has shipped with persistent journal storage for several releases as well. However, as Red Hat's own documentation states, RHEL 7, 8, 9, and 10 do not configure systemd to maintain logs persistently by default — the /var/log/journal/ directory is not created, meaning their journals are effectively volatile unless the administrator explicitly enables persistence. Note that RHEL 10 moved the main journald configuration file from /etc/systemd/journald.conf to /usr/lib/systemd/journald.conf , with /etc/systemd/ reserved for override configurations.

The practical consequence for forensic analysts is that you cannot assume journal persistence on any system — always verify by checking whether /var/log/journal/ exists and contains journal files, and inspect the Storage= value in /etc/systemd/journald.conf (or, on RHEL 10+, /usr/lib/systemd/journald.conf and any drop-ins in /etc/systemd/journald.conf.d/ ). The systemd version on the target system determines whether the upstream default is auto (pre-259) or persistent (259+), but the distribution's packaging decisions may override either.

Journal files begin with a header identifying the file as a systemd journal (magic bytes LPKSHHRH as defined in the official journal file format specification ), the file sequence number, the state (online, offline, or archived), the machine ID, the boot ID of the last writer, timestamps, and hash tables for indexed field lookup. Internally, journal files contain seven distinct object types: DATA objects (individual field=value entries), FIELD objects (field names), ENTRY objects (binding multiple DATA objects into a single log record), hash tables for DATA and FIELD lookup, ENTRY_ARRAY objects (sorted offset arrays for binary search), and TAG objects (FSS sealing tags, when enabled). All structures are aligned to 64-bit boundaries, and the file uses Jenkins lookup3 as its hash function. Every journal entry has guaranteed metadata fields including __REALTIME_TIMESTAMP (microseconds since epoch), __MONOTONIC_TIMESTAMP (microseconds since boot), _BOOT_ID , _PID , _UID , _GID , _COMM (process name), _EXE (full path to executable), and _HOSTNAME .

The presence of both __REALTIME_TIMESTAMP and __MONOTONIC_TIMESTAMP in every journal entry provides a forensic capability that syslog text files cannot replicate. The realtime timestamp (wall clock time) tells you when an event occurred in human-interpretable terms, but it can be manipulated by an attacker who changes the system clock. The monotonic timestamp (microseconds since the current boot) is immune to wall-clock manipulation because it is driven by the kernel's monotonic clock, which only moves forward and cannot be set by userspace. If an attacker changes the system time during an intrusion (a technique cataloged under MITRE ATT&CK T1070.006, Timestomp), the realtime timestamps in the journal will reflect the manipulated time, but the monotonic timestamps will continue their unbroken sequence. By correlating __MONOTONIC_TIMESTAMP values against __REALTIME_TIMESTAMP values within the same boot session, a forensic analyst can detect time manipulation: a sudden offset between the two timestamp series that was not present earlier in the boot indicates the wall clock was changed. This cross-referencing is not possible with traditional syslog, which records only wall clock time.

There is also a third timestamp that most analysts overlook: _SOURCE_REALTIME_TIMESTAMP . Unlike __REALTIME_TIMESTAMP , which is assigned by the journal daemon when it receives and stores the message, _SOURCE_REALTIME_TIMESTAMP is set by the originating process at the moment it generates the log message. The difference between the two reveals queueing delay — how long the message sat in the journal's buffer before being committed to disk. Under normal conditions this delta is negligible (sub-millisecond), but during a log storm (such as a brute-force attack generating thousands of auth messages per second), the delta can grow to seconds or more. If _SOURCE_REALTIME_TIMESTAMP is present and significantly earlier than __REALTIME_TIMESTAMP for entries during an incident, the actual event times are the source timestamps, not the storage timestamps. Not all log sources set this field — it requires the originating daemon to use the native journal API with sd_journal_sendv() — but when present, it provides the most precise event timing available.

From a forensic image or mounted filesystem, journal files can be copied directly from their path. The binary format requires journalctl to parse. On the analyst's forensic workstation, journal files from a target system can be read by specifying the directory path: This instructs journalctl to read from the specified path rather than the local system's journal. The analyst's workstation must run a systemd version compatible with the journal format version from the target system. For cross-version compatibility, the --file flag can target a single journal file. The --output=json-pretty flag produces structured JSON output suitable for programmatic processing or ingestion into SIEM platforms.

The journalctl query system supports filtering by any field, by time range, by boot ID, and by unit name. The following patterns cover the highest-value forensic queries. This shows each boot with its index, boot ID, and the first and last timestamps of journal entries from that boot. Boot gaps âII time periods between a recorded shutdown and the next recorded boot that are longer than expected âII may indicate unaccounted-for system activity or evidence of journal manipulation.

5. Performance Evaluation & Extended Technical Observations This is particularly valuable when investigating a compromised system where a Python-based backdoor or C2 agent may have been running âII all messages, errors, and output from that specific executable are queryable directly. The -k flag restricts output to kernel messages. This is equivalent to reading /var/log/kern.log but with the additional context of the boot ID and monotonic timestamps that allow correlation with other events from the same session.

The _BOOT_ID field in each journal entry is a UUID assigned at system boot and consistent across all entries from that boot session. It serves as an unambiguous anchor for correlating events to a specific system uptime period. When examining a system that may have been rebooted during or after an incident, grouping journal entries by _BOOT_ID allows the analyst to determine which events occurred in which boot session. If the journal shows a boot session that is absent from system uptime records (such as wtmp or last), it may indicate the system was booted in a forensically significant context âII recovery mode, live boot, or a period the attacker attempted to obscure.

Beyond syslog and the journal, several supplementary log files and databases provide evidence that does not appear in text log files and is not captured by journald. The following matrix shows which evidence types each major log source captures. No single source covers everything — forensic completeness requires cross-referencing multiple sources, and knowing which source to check for a specific evidence type is the difference between finding it and missing it entirely.

/var/log/wtmp is a binary file in the utmp format that records every login and logout event, including the username, terminal or tty device, the source IP for remote logins, and login/logout timestamps. It is the persistent log read by the last command. To read it from a forensic image without mounting the filesystem live: The output is tab-separated with type code, PID, tty, user, hostname/IP, and timestamps for each record. The type codes distinguish login sessions (USER_PROCESS), logout events (DEAD_PROCESS), system boot records (BOOT_TIME), and runlevel changes (RUN_LVL). Each reboot generates a BOOT_TIME record, providing an independent record of system restarts that can be correlated against journal boot IDs.

The last command reads wtmp in reverse chronological order and is the standard tool for examining it on a live system. On a forensic image, specifying the file path works: The -F flag shows full timestamps including the year and time of day, which is critical for investigations spanning year boundaries. Without it, last omits the year for entries in the current year.

/var/log/btmp uses the same binary utmp format as wtmp but records failed authentication attempts. The lastb command reads it. For forensic acquisition from an image, the same utmpdump approach applies. btmp complements auth.log for failed authentication analysis âII it records failed login attempts at the console and graphical display manager levels that may not generate syslog entries on all configurations, and it survives as a separate file even if auth.log has been cleared. As Sandfly Security's research on Linux forensics notes, attackers frequently "erase the files and replace them with ones that are zero bytes long" âII a crude but common anti-forensics technique that is immediately detectable by checking the file size of wtmp and btmp. A zero-byte wtmp on a system that has been running for weeks is itself strong evidence of tampering.

/var/log/lastlog is a sparse binary file indexed by UID. Each UID has a fixed-size record (a struct lastlog entry at byte offset UID * sizeof(struct lastlog) ) storing the last login timestamp, source terminal, and source hostname/IP for that account. Its sparse structure means the on-disk size is typically much smaller than the apparent size reported by ls -l , because UIDs that have never logged in do not consume physical disk blocks.

Unlike the last command which accepts a -f flag to specify an alternate wtmp file, the lastlog command does not support reading from an arbitrary file path — it reads only from /var/log/lastlog on the running system (consulting /etc/passwd for user enumeration). To examine lastlog from a forensic image, the analyst can either use a chroot environment ( chroot /mnt/evidence lastlog ) or parse the binary file directly with a script that reads the fixed-size records. The Python struct module can unpack each record's ll_time (login timestamp), ll_line (terminal), and ll_host (remote hostname/IP) fields from the raw binary data. Alternatively, copying the target system's /var/log/lastlog to /var/log/lastlog on the forensic workstation (with matching /etc/passwd entries) and running lastlog locally can produce readable output.

The lastlog file is forensically useful because it provides a per-account record that persists independently of log rotation. An account that shows a recent lastlog timestamp despite having no corresponding auth.log entries (because the auth.log was cleared) indicates the clearing was incomplete — the lastlog record survived as corroborating evidence of the login. Since lastlog is a binary database rather than a text log, attackers who clear text log files with simple commands like echo > /var/log/auth.log or cat /dev/null > /var/log/syslog often overlook it entirely.

A rarely discussed but forensically significant issue affects the long-term reliability of wtmp, btmp, and lastlog. On 64-bit bi-arch systems (such as x86-64 running glibc), the utmp/wtmp/lastlog binary structures use a 32-bit time_t field for timestamps — even on a pure 64-bit userland. This means these files are subject to the Year 2038 overflow problem, where the 32-bit signed integer representing seconds since the Unix epoch wraps around on January 19, 2038. After that date, timestamps in these files will be invalid on affected systems.

The glibc maintainers have indicated that rather than fixing the on-disk format (which would require an ABI-breaking change), they intend to deprecate the utmp/utmpx/lastlog interfaces entirely. The planned migration path involves replacing utmp reads with queries to systemd-logind (which gained the necessary API functions in systemd v254), and replacing wtmp with structured login records written to the systemd journal. The util-linux project has begun accepting patches for Y2038-safe alternatives, and the Linux-PAM project is evaluating deprecation of pam_lastlog .

For forensic analysts, the practical implications are twofold. First, on current systems: wtmp, btmp, and lastlog remain reliable evidence sources and should always be examined. Second, on future systems (particularly those running glibc 2.40+ and systemd 254+): the traditional binary login databases may be absent or no longer written to, and the equivalent evidence will live in the systemd journal under login session records. Analysts working on systems from 2026 onward should verify which login recording mechanism is active before concluding that login history is unavailable.

Web server logs ( /var/log/apache2/ or /var/log/nginx/ ), database logs, and application-framework logs are significant sources for web-targeted attacks. The Apache and nginx access logs record every HTTP request with source IP, request line, response code, and bytes transferred. Error logs record application errors that may reveal exploitation attempts. Understanding what web server logs reveal about automated attack patterns is a prerequisite for this correlation work. In a web server compromise investigation, correlating auth.log SSH events against web server access log source IPs establishes whether an attacker who gained shell access was the same source that preceded the compromise with web-based reconnaissance or exploitation attempts.

Many production Linux systems are configured to forward log messages to a centralized syslog server (via rsyslog's @@ TCP or @ UDP forwarding directives, or through systemd's systemd-journal-upload service). When remote forwarding is in place, the centralized log server holds a copy of every message that was transmitted before the attacker gained access to the local host — and often a copy of messages generated during the initial stages of the attack itself, before the attacker had the opportunity to disable forwarding or clear local logs. This makes the remote copy a higher-integrity evidence source than local files for the same time period.

On a forensic image, determining whether remote forwarding was configured requires examining /etc/rsyslog.conf and /etc/rsyslog.d/*.conf for action directives containing @@ (TCP) or @ (UDP) followed by a remote hostname or IP, and checking /etc/systemd/journal-upload.conf for journal forwarding configuration. If forwarding was active, the forensic investigation should request the remote log server's records for the target hostname and correlate them against local files.

Discrepancies between the remote and local copies — events present on the remote server but missing locally — are direct evidence of local log manipulation. A subtlety worth noting: because UDP syslog forwarding is fire-and-forget with no delivery guarantee, some messages may be absent from the remote server due to network loss rather than tampering. TCP forwarding (and TLS-encrypted TCP, configured via rsyslog's omrelp or omfwd with StreamDriver directives) provides reliable delivery and should be preferred in environments where log integrity matters. The forwarding protocol in use on the target system determines how much weight can be placed on the remote copy's completeness.

The kernel ring buffer is a fixed-size circular buffer in kernel memory that records kernel messages from boot onward, including hardware detection, driver loading, module insertion, device attachment and detachment, filesystem operations, and kernel-level errors. On a live system, the dmesg command reads this buffer; on a forensic image, the buffer content is not directly available because it exists only in kernel memory and is lost on shutdown. However, the kernel ring buffer's contents are typically written to /var/log/kern.log (Debian/Ubuntu) or included in /var/log/messages (RHEL-family), and on systemd systems they are captured in the journal under kernel facility messages (queryable with journalctl -k ).

The forensic value of kernel messages is distinct from userspace log sources. Kernel messages record hardware-level events that no other log source captures: USB device insertions and removals (which identify when external storage was connected), network interface state changes (which may reveal when an attacker brought up a tunnel interface or changed a NIC to promiscuous mode), kernel module loading (which reveals rootkit installation via insmod / modprobe ), and filesystem mount and unmount operations. A kernel message indicating that an out-of-tree module was loaded and "taints kernel" is a significant forensic indicator — legitimate kernel modules are typically signed and in-tree, while attacker-loaded rootkit modules trigger this taint warning.

Harvested Online Documentation & Reference Sources • https://en.wikipedia.org/wiki/Computer%20Online%20Forensic%20Evidence%20Extractor • https://en.wikipedia.org/wiki/Security%20information%20and%20event%20management • https://en.wikipedia.org/wiki/Data%20recovery • https://github.com/ilyess-sellami/Linux-DFIR-Acquisition-Playbook • https://github.com/plurigrid/asi/blob/main/plugins/asi/skills/performing-linux-log-forensics-investigation/SKILL.md • https://linuxvox.com/blog/linux-forensics/ • https://kandibrian.com/articles/linux-log-forensics-syslog-authlog-journal.html