đź”— Link to the Room

🏷️ Table of Contents

  1. Introduction and Lab connection
  2. Live Forensics: An Overview
  3. Tool of the Trade: Osquery
    3.1 Osquery Table Schemas
    3.2 Users Account
    3.3 Process Information
  4. System Profiling
    4.1 Basic System Information
    4.2 Hostname
    4.3 uptime
    4.4 Hardware Information
    4.5 Disk Free
    4.6 List of Block Devices
    4.7 Free Storage
    4.8 Debian Package Manager Packages
    4.9 Network Profiling
  5. Hunting for Processes
    5.1 Hunting for Suspicious Process
    5.2 Processes Running From the tmp Directory
    5.3 Hunting for Fileless Malware (Process)
    5.4 Orphan Processes
    5.5 Finding Processes Launched from User Directories
  6. Investigating Network Connections
    6.1 Listing Network Communication
    6.2 Network Connections
    6.3 Remote Connection
    6.4 Examining DNS Queries
    6.5 Listing Down Network Interfaces
    6.6 List Network Connections
  7. TTP Footprints on Disk
    7.1 Open Files
    7.2 Files Being Accessed From the tmp Directory
    7.3 Hidden Files
    7.4 Recently Modified Files
    7.5 Recently Modified Binaries
    7.6 Finding Suspicious Packages
  8. Persistence: Establishing Foothold
    8.1 Investigating the Initialization Services
    8.2 Hunting for a Backdoor Account
    8.3 Examining Cron Jobs

đź“š Study Notes

 

Introduction and Lab connection

 

Linux powers many servers, cloud services, and supercomputers due to its reliability and performance, which also makes it a major target for cyberattacks.

As a SOC analyst, your task is to perform live forensics on a compromised Linux server to identify attack footprints and assess the extent of the damage.

 

Live Forensics: An Overview

 

From a forensic perspective, it is crucial to collect volatile data from memory and disk, with primary focus on capturing data from the running system.

Key data investigators collect from a running Linux system:

Data Type Why It Matters
Running Processes Shows active programs and possible malicious processes.
Open Files Reveals which files processes are accessing or modifying.
Memory Structures May contain passwords, keys, or exploit traces.
Network Connections Identifies suspicious external connections or data exfiltration.
Listening Services Helps detect unauthorized services or backdoors.
Logged-in Users Shows who is currently accessing the system.
User Activity Command history can reveal attacker actions.
In-Memory Logs Provides real-time system events before they are written to disk.
Network Interfaces Shows IP, MAC, and routing details that may indicate tampering.
Temporary Files /tmp and /var/tmp may contain scripts or attack artifacts.

 

Tool of the Trade: Osquery

 

osquery is an endpoint monitoring tool that lets you query system information using SQL-like commands to gather detailed data for investigations.

To open the interactive session run osquery in the terminal as root. To display the help options run .help command.

image

 

Osquery Table Schemas

Check osquery website to view table schemas:

image

 

Users Account

To retrieve info about the user accounts created on the host use Search Query: Select username, uid, description from users;

image

 

Process Information

To get the info about the running processes use Query: Select pid, name, parent,path from processes;

image

 

âť“What hostname is returned after running the following query? Query: select * from etc_hosts where address = `0.0.0.0`;attacker.thm
âť“On the official website, how many tables are listed for Linux OS?154

 

System Profiling

 

The first step in an investigation is collecting detailed system information to understand the infected machine’s state and environment.

System profiling is an important forensic step used to gather key information about a system, such as configuration details, logged-in users, installed applications, and hardware setup.

Make sure to switch to root by using sudo su command.

 

Basic System Information

To extract basic information about the system use uname -a command. This command output provides comprehensive details about the system, including its kernel version, architecture, hostname, and the date and time when the kernel was compiled.

image

This command output provides comprehensive details about the system, including its kernel version, architecture, hostname, and the date and time when the kernel was compiled. Let’s break down the information we retrieved from this command:

 

Hostname

To see details about the hostname and related settings run hostnamectl command.

image

 

uptime

To see a quick snapshot of your system’s current status, including the time, how long it’s been running, how many users are logged in, and how busy the system is run uptime command.

image

 

Hardware Information

To display detailed info about the CPU architecture run lscpu command.

image

 

Disk Free

To see the amount of disk space used and available on the system run df -h command.

image

The df command is crucial for monitoring disk usage. It helps you see how much space is used and how much is available, which is essential for managing storage resources.

 

List of Block Devices

To see information about block devices, such as disks and partitions, including their sizes, mount points, and other relevant details use lsblk command.

image

 

Free Storage

To display memory usage info in human-readable format use free -h command.

image

 

Debian Package Manager Packages

Debian-based systems use package managers like dpkg and apt to install, update, and remove software.

1- dpkg

To list down the installed packages use dpkg -l command.

image

2 - apt

To list down all the packages installed through apt use apt command.

image

 

Network Profiling

Commands like ifconfig or ip a show network interface details such as IP address, MAC address, network mask, and connection status.

image

To display routing table use command ip r or route

image

To show socket statistics and active connections use command ss or netstat

image

 

âť“What is the Machine ID of the machine we are investigating?dc7c8ac5c09a4bbfaf3d09d399f10d96
âť“What is the architecture of the host we are investigating?x86_64

 

Hunting for Processes

 

From a forensics point of view, finding out what process is running on the suspected host and narrowing it down to identify the odd-looking process is crucial. It helps understand what’s happening in the system.

Command What It Does
ps Shows a snapshot of currently running processes.
top Shows real-time process activity and system resource usage.
htop Like top, but easier to read and manage with colors and extra features.
pstree Shows processes in a tree format to see parent-child relationships.
pidof Finds the process ID (PID) of a program by its name.
pgrep Searches for processes by name or other attributes.
lsof Lists open files and the processes using them.
netstat Shows network connections and listening ports for processes.
strace Tracks system calls of a process to see exactly what it’s doing.
vmstat Displays memory and CPU stats to monitor overall system performance.

 

Hunting for Suspicious Process

To list down all the running processes use Search Query: SELECT pid, name, path, state FROM processes;

image

The command retrieves a list of all currently running processes on the system and provides the following details for each process:

 

Processes Running From the tmp Directory

To narrow down the result to only show the processes running from the /tmp/ or /var/tmp/ directory use Query: SELECT pid, name, path FROM processes WHERE path LIKE '/tmp/%' OR path LIKE '/var/tmp/%';

image

Two processes were initiated from the tmp directories, indicating the host is infected.

 

Hunting for Fileless Malware (Process)

To list the processes executing on the host but not on the disk run Search Query: SELECT pid, name, path, cmdline, start_time FROM processes WHERE on_disk = 0;D

image

[!NOTE] Not every process without a presence on disk can be indicated as suspicious. We have to investigate them further to determine. In the above query, the column on_disk can have two values: 1 indicates it is on the disk, and 0 indicates the process is not on the disk.

A fileless malware can have the following characteristics that we can observe:

 

Orphan Processes

Normally, every process in Linux has a parent process. This parent-child relationship forms a process tree that can be viewed by the tool pstree. Intruders can create a process that becomes an orphan.

To list the processes without parent processes use Search Query: SELECT pid, name, parent, path FROM processes WHERE parent NOT IN (SELECT pid from processes);

image

 

Finding Processes Launched from User Directories

In the context of a server, if the process is running from the user directory, that process could be marked for further investigation. One of the main reasons is that, typically, system processes run from the standard system directories.

To search in the list of running processes and see which method is running from the user directory use Search Query: SELECT pid, name, path, cmdline, start_time FROM processes WHERE path LIKE '/home/%' OR path LIKE '/Users/%';

image

The above search queries demonstrate a few angles of examining the running processes from a suspicious mindset. They could be good Indicators of Compromise (IOC) and can be used for further investigation and the creation of host-based detection rules.

 

âť“What is the name of the process running from the tmp directory? (Note: Not Hidden one)sshdd
âť“What is the name of the suspicious process running in the memory of the infected host?.systm_updater
âť“What is the name of the process runniing from the user directory?rdp_updater

 

Investigating Network Connections

 

Let’s look at the network communication or connection initiated on this host, which could be identified as suspicious.

 

Listing Network Communication

To examine the network connections on the Linux host, there are various built-in command-line tools that we can use:

Command What It Does
netstat Shows current network connections and open ports.
ss Same as netstat, but faster and more detailed.
tcpdump Captures network traffic to see what’s being sent/received.
iftop Shows which programs are using the most internet bandwidth.
lsof Lists open files and network connections for each program.
iptables Manages firewall rules and monitors network traffic.
nmap Scans your network to find connected devices and open ports.
ping Checks if another device on the network is reachable.
traceroute Shows the path your data takes to reach another device.
dig Checks website addresses and DNS info.
hostname Shows the computer’s name and IP address.
ifconfig Shows info about your network interfaces.
ip Modern replacement for ifconfig; manages network settings.
arp Maps IP addresses to physical hardware addresses.
route Shows how your computer sends data to other networks.
curl Downloads or sends data to a website or server.
wget Downloads files from the internet.
netcat Sends or receives data across the network; useful for testing.
whois Shows who owns a domain name.
nslookup Finds the IP address for a website or vice versa.

 

Network Connections

To retrieve information about network connections established by various processes on the system run Query: SELECT pid, family, remote_address, remote_port, local_address, local_port, state FROM process_open_sockets LIMIT 20; It selects entries from the process_open_sockets table.

image

 

Remote Connection

To list down all the network connections with a remote connection use Search Query: SELECT pid, fd, socket, local_address, remote_address, local_port, remote_port FROM process_open_sockets WHERE remote_address IS NOT NULL;

 

Examining DNS Queries

To retrieve info about the DNS queries on this host use Search Query: SELECT * FROM dns_resolvers;

image

 

Listing Down Network Interfaces

To retrieve the info about the network interface use Search Query: SELECT * FROM interface_addresses;

image

 

List Network Connections

To lust down the listening ports use Search query: SELECT * FROM listening_ports;

image

 

âť“What is the state of the local port that is listening on port 80?ESTABLISHED

 

TTP Footprints on Disk

 

After spotting suspicious processes and network activity, the next step is to check for malicious or altered files on the disk.

 

Open Files

List all the files opened. Search Query: SELECT pid, fd, path FROM process_open_files; This query will list all files that have been opened and are associated with some process. We can locate them through their respective pid.

image

 

Files Being Accessed From the tmp Directory

Filter the query to only show the files accessed from the /tmp/. Search Query: SELECT pid, fd, path FROM process_open_files where path LIKE '/tmp/%'; This query will search for the processes that have opened files on the system. For this query, we only look at the files accessed from the /tmp/ directory. In an actual investigation, we will have to look at various other locations.

image

Check ID 556.

Identify the process name using this pid with Search Query: select pid, name, path from processes where pid = '556';

image

 

Hidden Files

Use Search Query: SELECT filename, path, directory, size, type FROM file WHERE path LIKE '/.%'; to examine the root directory to track down hidden files or folders. In a real investigation, we will also need to examine other locations.

image

Similarly, we can update the query to look at other directories like /tmp/, /etc/, /usr/bin/, etc.

 

Recently Modified Files

Use Search Query: SELECT filename, path, directory, type, size FROM file WHERE path LIKE '/etc/%' AND (mtime > (strftime('%s', 'now') - 86400)); to see which file was recently modified.

image

This query will look at the modified time (mtime) and list down the recently modified files. During a live investigation, this could be very useful in tracking down the system files or binaries that were recently modified.

 

Recently Modified Binaries

Look at the modification time and see which binary was modified recently by running Search Query: SELECT filename, path, directory, mtime FROM file WHERE path LIKE '/opt/%' OR path LIKE '/bin/' AND (mtime > (strftime('%s', 'now') - 86400));

image

The above query only looks at the files and binaries updated in the last 24 hours in the /opt/ or /bin/ directories. We can update the time to get the updated results.

Note: There is expected to be no result when it is executed on the attached VM.

 

Finding Suspicious Packages

Search for the term install in the /var/log/dpkg.log file, which contains all the information about installed / uninstalled packages.

image

There is a suspicious package installed on the 26th of June, 2024.

Run dpkg -l | grep <REDACTED> image

 

âť“Investigate the opened files. What is the opened file associated with the suspicious process running on the system?keylogger.log
âť“What is the name of the process that is associated with the suspicious file found in the above question?sshdd
âť“What is the name of the hidden binary found in the root directory?.systmd
âť“What is the name of the suspicious package installed on the host?datacollector
âť“The suspicious package contains a secret code. What is the code hidden in the description?{***_**_******_*******}

 

Persistence: Establishing Foothold

 

After gaining a foothold, an intruder would first try to establish persistence to maintain hidden, ongoing access, and we can detect this on a Linux host by examining key system areas.

 

Investigating the Initialization Services

Adversaries often set up services on the infected host to maintain persistence and evade detection, which are stored in /etc/systemd/system, so checking this directory for unusual services can reveal hidden malicious activity.

image

Use cat command to read the content of the services.

image

It seems one of the services is using the Netcat utility to open a listening port that can be used as a backdoor entry.

 

Hunting for a Backdoor Account

Creating a backdoor user account is a common technique used by attackers to maintain access, so examining the system’s user list can help identify suspicious or unexpected accounts.

Search Query: select username, directory from users;

image

You can see the suspicious user account created on this host.

List down the names of the users from the /etc/passwd file by running cut -d : -f1 /etc/passwd

image

 

Examining Cron Jobs

Cron is a time-based job scheduler in Unix-like operating systems. Intruders can create cron jobs to execute malicious scripts regularly, ensuring their activity continues after rebooting.

Examine the cron tables by running crontab -l

image

It looks like a process from a hidden directory located in the backdoor account’s directory.

 

âť“Which suspicious service was observed to be installed on this infected machine using netcat?systm.service
âť“What is the full path of the process found in the cron table?/home/badactor/storage/.secret_docs/rdp_updater

Â