Skip to content

Blog

  • in Blog, Linux
  • 7 min read

Linux: Automation and Orchestration

Ansible IaC Core Concepts

Ansible let users automate system configuration and management using clear, repeatable commands. Ansible is agentless. I uses SSH on Linux and WinRM on Windows.

Installing Ansible

# on RHEL-based system
dnf install -y ansible-core

# on Debian-based system
apt install -y ansible

# to test install
ansible --version
ansible localhost -m ping

Inventory

Inventory is a list of all the servers or devices. Inventory can store as simple text file using INI format, as structured YAML file, or dynamically from cloud platforms or CMDBs.

[web] Group servers for easy management

ex inventory file

# ./hosts
[local]
localhost ansible_connection=local

# to install htop on RHEL-based system
ansible -i hosts local -m dnf -a "name=htop state=present update_cache=yes" --become

# to create a new user
ansible -i hosts local -m user -a "name=bob state=present" --become

# to copy a file from the control node to the managed nodes
ansible -i hosts local -m copy -a "src=my_config.conf dest=/apps/myapp.conf" --become

Ad Hoc Mode

Ad Hoc Mode is used to run one-time commands to test settings or apply changes across systems.

ex:

# to ping all hosts listed in the inventory
ansible all -m ping

# to restart the nginx service in the web group
ansible web -m service -a "name=nginx state=restarted"

Module

A module is a built-in tool that handles specific tasks like installing software, restarting services, or managing users.

ex of modules:

  • yum: used to install, update, remove packages on RHEL-based systems
  • apt: used to install, update, remove packages on Debian-based systems
  • user: used to manage user accounts on the system
  • service: used to start, stop, restart, or enable services
  • copy: used to transfer files from the control node to remote machines
  • file: used to create directories, change permissions, or delete files

Playbook

Playbook are complex, repeatable tasks, and structured automation. It is a structured YAML file that defines a set of tasks for Ansible to carry out on managed system.

Facts

Facts allows Ansible to automatically gather information about each machine and make decisions based on that data. Data collected can include IP addresses, operating system, available memory, and disk space. They are gathered at the beginning of playbook execution so it can decide what action to take based on the current setup of the machine. Ansible collects facts only when users run a task, using a direct connection like SSH.

Collections

Collections help manage and reuse tools, making it easier to scale and maintain automation environment over time.

Puppet Core Usage

Puppet helps automate system configuration by letting admins describe what the system should look like. Puppet is agent-based.

The Puppet Agent is responsible for communicating with the Puppet server and applying configurations. It is also responsible of collecting facts.

The Puppet server is called Puppet Master.

Facts

Facts are information Puppet collects on the managed devices such as operating system, hostname, IP addresses, memory, and more. Puppet Agent collects facts on a regular schedule during each check-in with the Puppet server. Puppet is well suited for large-scale enterprise environment because it enforces regular automated configuration.

Classes

Classes group related configuration tasks together into one logical unit. They help apply consistent settings to many systems with minimal duplication of effort.

Modules

A module is a package that includes everything needed to manage a specific task or part of a system. It can include one or many classes, files, templates, or custom facts.

Certificates

Certificates ensure that only authorized machines are allowed to talk to the server and receive configurations. The certificates must be approved and signed by the server before configurations are exchanged.

OpenTofu Core Usage

OpenTofu is an open-source designed to help manage and automate cloud infrastructure with code.

Provider

Provider connects configuration code to the actual cloud platform or service that the user is trying to manage. OpenTofu talks to services like AWS, Azure, and GCP using APIs.

Resources

Resources are the specific pieces of infrastructure user wants to create or manage, such as virtual machine, a firewall rule, or a storage bucket. OpenTofu resources focus on provisioning and configuring cloud services from the ground up.

State

The state is how OpenTofu keeps track of what is already been created in the environment.

Unattended Deployment

It is the automation of installation and initial configuration of systems to avoid manual step-by-step administrations.

Kickstart

Kickstart is commonly used in traditional data center environments with RHEL-based systems. You automate the RHEL-based installation by specifying things like language, disk setup, network settings, package selection in configuration file. The general syntax to start a kickstart install from a boot prompt is linux ks=<LOCATION OF KICKSTART FILE> inst.repo=<INSTALLATION SOURCE>

# to start a kickstart install
linux ks=http://192.168.10.10/kickstart/ks.cfg inst.repo=http://192.168.10.10/rhel8

Cloud-init

Cloud-init is the standard for automating deployments in cloud platforms like AWS, Azure, or OpenStack. It reads a YAML configuration file in order to apply the changes during the fire boot of a cloud instance.

ex:

# to create an install and configure using cloud-init
aws ec2 run-instances --image-id ami-0adfads185141422356 --instance-type t2.micro --user-data file://init-script.yaml

CI/CD Concepts

CI/CD is a system of tools and practices that brings order and automation to modern software development.

Version Control

Version control is a system that tracks changes to files over time, allowing developers to collaborate, review history, and roll back if something goes wrong. Git is the most common version control tool used today.

Pipelines

A pipeline is a sequence of automated steps that take code from commit to deployment. It might include testing, security scanning, building the software, deploying to production.

Modern CI/CD Approaches

Shift Left Testing

Shift left testing moves testing earlier in the development cycle, right alongside coding. The common tools used are Jenkins and GitLab CI.

DevSecOps

DevSecOps = Development, Security, and Operations. It is an approach that builds on CI/CD by embedding security practices throughout the software lifecycle.

GitOps

GitOps is a way of managing infrastructure and deployments using Git as the single source of truth. Common tools used are Argo CD and Flux.

Kubernetes Core Workloads for Deployment Orchestration

Kubernetes is an open-source platform that automates the deployment, scaling and management of containerized applications.

Pods

Pods are where the applications run. They allow users to tightly couple containers that need to work together. Containers that run in the same pods can talk to each other like they are running in the same machine.

Deployments

Deployments make sure the right number of Pods are up and are kept up to date. A deployment acts like a controller that keeps track of the application and ensures the right number of Pods are always running and up to date.

Services

Services ensure the application is reachable by other apps or users. They provide stable endpoint so other applications or users can reliably connect to the app regardless of which Pod is curring running.

Kubernetes Configuration

Variables

Variables are the simplest way to pass configuration settings into the containers.

ex:

# to tell the pod to use production settings
ENVIRONMENT=production
ConfigMaps

ConfigMaps store larger sets of configuration data in a Kubernetes object.

Secrets

Secrets work similarly to ConfigMaps, but are specifically designed to store sensitive data such as passwords, API tokens, SSH keys, or SSL/TLS certificates. The defaults, secrets are encoded in base64. Kubernetes uses RBAC to control access to these secrets.

Volumes

Volumes provide a way for containers to store and access data that needs to persist beyond the life of a single container.

Docker Swarm Core Workloads for Deployment Orchestration

Docker Swarm is a tool that helps orchestrate container deployments, making sure everything runs reliably.

Nodes

A node is a physical or virtual machine that is part of the swarm cluster. A node runs the Docker engine and is classified as Manager Node which make decisions and assign tasks, or Worker Node which carry out the tasks.

Tasks

A task is the actual instance of a container running on a node. Each task maps to exactly one container, and Swarm monitors them all continuously. Pods can host multiple containers, while a swarm task maps one-to-one. Tasks help ensure that the application stays running as expected.

Service

A service is a top-level object in Docker Swarm that defines how the application runs.

Docker Swarm Configuration

Networks

Networks defines how containers communicate within the swarm.

Overlay Networks

Overlay Networks are virtual networks that span across all nodes in the swarm. They enable secure, seamless communication between containers on different nodes.

ex docker-compose.yaml

# define an overlay network called frontend.

networks:

  frontend:

    driver: overlay

Scaling

Scaling refers to how many replicas of a service are running at any given time.

ex docker-compose.yaml:

services:

  web:

    image:nginx

    deploy:

      replicas: 3

Docker/Podman Compose for Deployment

Compose file

docker-compose.yaml

version: "3.8"

services:
  web:
    image:nginx
    ports:
      - "8080:80"

  app:
    image:my-web-app:latest
    environment:
      - ENV=production
    depends_on:
      - db

  db:
    image:postgres
    volumes:
      - db-data:/var/lib/postgresql/data

volumes:
  db-data:

Podman Compose is more a security-focused container engine.

up and down commands

# to start or bring down containers
docker-compose up
docker-compose down

# to start or bring down containers
podman-compose up
podman-compose down

Viewing Logs

Viewing logs is essential for understanding what's happening in the application.

# to view logs
docker-compose logs
podman-compose logs

# to tail logs
docker-compose logs --follow web
  • in Blog, Linux
  • 11 min read

Linux: OS Hardening

sudo Configuration

sudo = Super User Do

sudo allows regular users to execute commands with elevated root privileges.

visudo

visudo is the tool used to edit the sudo configuration. The main configuration is at /etc/sudoers. Additional configuration files are stored in /etc/sudoers.d. It is highly recommended to use visudo because it prevents saving invalid configuration.

  • -c checks syntax of sudoers file without editing

  • -f <CONFIG FILE PATH> specifies a different file to edit with visudo

  • -s runs the editor in strict mode

  • -x exports the sudoers file in JSON format for automation or auditing

wheel group

The wheel group is a special group commonly used in Linux systems to grant its members permission to run administrative command with sudo. ex sudo usermod -aG wheel john

/etc/sudoers

/etc/sudoers is the main config file accessed by visudo.

  • admin ALL=(ALL) ALL give users in the admin group full access to the system

  • john ALL=(ALL) /usr/bin/systemctl restart nginx gives user john permission to run only systemctl restart nginx

/etc/sudoers.d

/etc/sudoers.d allows admins to break up sudo configurations into multiple smaller files. That is helpful in enterprise environments with different levels of access needs, supporting automation tools like Ansible or Puppet.

sudoers Directives

  • NOPASSWD allows users to run sudo commands without being prompted for their password. It should be used with care in production environment to prevent accidental misuse.
# to not prompt tom for a sudo password
tom ALL=(ALL) NOPASSWD:/usr/bin/systemctl restart nginx
  • NOEXEC prevents the approved user sudo commands from launching additional programs or subshells from within a sudo-allowed command. NOEXEC restricts behavior by disabling the ability of the command to spawn other processes.
# to not prompt tom for a sudo password
kyle ALL=(ALL) NOEXEC:/usr/bin/less /var/log/syslog

sudo User Groups

  • sudo group: used in Debian-based systems like Ubuntu

  • wheel group: used in RHEL-based systems

sudo Group

# to add a user to the sudo group
sudo usermod -aG sudo amy

wheel Group

It serves the same purpose has the sudo group in RHEL-based systems.

# to add a user to the wheel group
sudo usermod -aG wheel amy

sudo -i

sudo -i opens a full root shell for users with sudo privileges from the sudo or wheel group. If the sudo config contains %sudo ALL=(ALL:ALL) ALL or %wheel ALL=(ALL:ALL) ALL any user in those group can elevate their permission to full root user with sudo -i.

Root Shell

sudo su - switch to the root user role entirely. su stands for substitute user

File Attributes

File attributes provide an extra layer of control that goes beyond the standard file permissions.

lsattr

lsattr allows viewing a file's current attributes. 

# to view a file attributes
lsattr my-important-file.txt

Useful options:

  • -R list attributes recursively in subdirectories

  • -a includes hidden files

  • -d shows directory attributes instead of their contents

  • -v shows the version number of the file if supported

# to view attributes of all files in a directory including hidden files
lsattr -a -d -R /etc/config

Output may look like:

----i--------e-- /etc/config/setting.conf
-----a--------e-- /etc/config/logs.log

The i indicate that the file is immutable.

The a indicate that the file is append only.

chattr

chattr allows changing a file's attributes.

Common options include:

  • -R to apply changes recursively to directories and contents

  • -v to work with file's versioning

  • +i to set a file as immutable

  • -i to remove a file from immutable mode

# to protect a script from being modified or deleted
chattr +i /usr/local/bin/m_script.sh

# to remove immutable protection
chattr -i /usr/local/bin/m_script.sh

# to protect a directory from being modified or deleted
chattr -R +i /usr/local/bin/scripts/

File Permissions

chown

chown is used to change the ownership of a file and can also change its associated group at the same time. The general syntax is chown [OPTIONS] <NEW OWNER>[:NEW GROUP] file. -R option applies the change recursively to all files and subdirectories.

# to change the owner and group of a folder
chown -R tyler:accounting /data/reports

chgrp

chgrp focusses specifically on changing the group ownership of a file without affecting the user ownership. The general syntax is chgrp [OPTIONS] <NEW GROUP> file. 

# to change the group ownership of a directory
chgrp -R admins /scripts

File Modifications

chmod (change mode)

chmod allows changing file permissions by specifying user class and permission ot add, remove, or set. The general is chmod [who][operator][permission] file.

who can be:

  • u: user/owner
  • g: group
  • o: others
  • a: all

operator can be:

  • +: to add
  • -: to remove
  • =: to set

permission can be:

  • r: read
  • w: write
  • x: execute
# to give execute permission to a file
chmod u+x script.sh

chmod using Octal Notation

  • Read: 4
  • Write: 2
  • Execute: 1

The general syntax is chmod [mode] <FILE>.

# to give rwx permission to user, rx to group, and r to others
chmod 754 config.conf

File Special Permissions

Special permissions temporarily grant users additional access on certain conditions. There are mainly 3 special permissions: setuid, setgid, and the sticky bit.

setuid (Set User ID)

setuid allows a program to run with the privilege of the file's owner. The general syntax is chmod u+s <FILE>.

chmod u+s /scripts/run.sh

setgid

setgid is similar to setuid but focuses on group ownership. Files and subdirectories created inherit the group ownership of the directory. The general syntax is chmod g+s <FILE OR DIRECTORY>.

# to set the setgid of a folder
chmod g+s /data/

# to set of files and directories in a direct to belong to the same group
chgrp admins /shared/scripts && chmod g+s /shared/scripts

sticky bit

The sticky bit is a special permission used on shared directories to prevent deletion or renaming of files not owned. The general syntax of the sticky bit is chmod +t <DIRECTORY>.

# to make users be able to delete their own files only
chmod +t /shared

Default File Creation Mask

umask (User File Creation Mask)

The umask defines which permission bits should be masked out or removed from the system's default permissions when a new file or directory is created.

New files begin with a default permissions or 666 for user, group, and others. New directories start with the default permissions of 777.

umask 022 removes the write bit (value 2) from the group and others permission sets:

666 - 022 = 644 => rw-r-r for new files

777 - 022 = 755 => rwxr-xr-x for new directories

# to see the umask value
umask

# to remove all access for others and write access for the group
# 640 for new files and 750 for directories
umask 027

To make the change permanent, the command can be added to user's shell config in .bashrc or .profile.

Access Control Lists - ACLs

ACLs give more flexibility that file permission controls. It provides detailed file permission management by specifying unique access rights for individual users and groups beyond owner, group, and others. Two main commands are used to manage ACLs:

  • getfacl: to display current ACLs
  • setfacl: to modify or add new ACL entries

getfacl

getfacl is used to view ACL entries on files or directories. The general syntax is getfacl [OPTIONS] <FILE OR DIRECTORY>. -R allows displaying ACLs recursively.

ex:

# to view ACLs of directory and its content
getfacl -R data/

setfacl

setfacl is used to create or modify ACL entries, allowing admins to fine-tune file access. The general syntax is setfacl [OPTIONS] <PERMISSION> [FILE OR DIRECTORY].

Common options:

  • -m to modify or add an entry
  • -x to remove an entry
  • -R to apply changes recursively

ex:

# to give a user permission to rw a file via ACL
setfacl -m u:tom:rw config.conf

# to remove the ACL entry
setfacl -x u:tom config.conf

# to reset a file to use standard permissions
setfacl -b config.conf

# to set default ACL on a directory for new files to inherit
setfacl -d -m u:tyler:rw /data/reports

SELinux States

SELinux = Security-Enhanced Linux.

SELinux can be in one of the three states:

  • Disabled: No policy enforcement or logging
  • Permissive: No enforcement, but logs policy violations
  • Enforcing: Enforces policies and blocks unauthorized actions

SELinux - Disabled State

This mode is typically used for troubleshooting in extreme cases or when SELinux is not needed in a particular environment. SELinux configuration is located at /etc/selinux/config. Set SELINUX=disabled to disable SELinux. A reboot to required for the change to take effect.

# to view SELinux state
getenforce

SELinux - Permissive State

In this state, SELinux will log action violations that would have been blocked. 

# to temporarily set SELinux to permissive without reboot
setenforce 0

The change can be made persistent after reboot by editing /etc/selinux/config with the value SELINUX=permissive.

SELinux - Enforcing State

This is the default and most secured state. It is ideal for production environments.

# to temporarily set SELinux to enforcing without reboot
setenforce 1

Update /etc/selinux/config and set SELINUX=enforcing to make the change persistent after reboot.

SELinux File Security Contexts

To work with SELinux File Security Contexts, Linux provides 3 commands:

ls -Z

ls -Z is used to display the current SELinux context for files and directories with and added column showing context label including SELinux user, role, type, and level. The User part represent the SELinux user identity, such as system_u for system processes or unconfined_u for users not strictly controlled by SELinux. The Role part defines permissions available to process or user in a context such as object_r for files and directories. The Type part is the most important part of the context describing object purpose and used by SELinux policies to grant or deny access.

restorecon

restorecon is used to restore the default context of a file or directory based on SELinux policy. The general context is restorecon [OPTIONS] <PATH>.

# to recursively restore files context
restorecon -Rv /var/www/html

chcon

chcon is used to allow manual changes to a file context when necessary. The general syntax is chcon [OPTIONS] <FILE>.

Common options include:

  • -u to set the user
  • -r to set the role
  • -t to set the type
# to manually label a file for webserver access
chcon -u system_u -r object_r -t httpd_sys_context_t index.html

# then check with 
ls -Z index.html

SELinux System-wide Configuration

getsebool

getsebool check the current status of SELinux booleans, which are special on/off switches that control how strict or flexible SELinux is in certain situations.

# to list all booleans
getsebool -a

# to view a selected boolean
getsebool antivirus_can_scan_system

# to see whether webserver is allowed to access user home directories
getsebool httpd_enable_homedirs

# to view whether FTP services are allowed to access users's home directories
getsebool ftp_home_dir

# to view whether Apache webserver can initiate outbound network connections
getsebool httpd_can_network_connect

# to see whether the Samba file sharing service can share users's home directories over the network
getsebool samba_enable_home_dirs

setsebool

setsebool is used to turn a specific boolean on or off, and optionally make that change permanent across reboot. The general syntax is setsebool [-P] <BOOLEAN NAME> on|off. -P makes the change persist across reboots.

# to allow webserver to serve content from user directories
setsebool -P httpd_enable_homedirs on

# to allow Apache to connect to network services
setsebool -P httpd_can_network_connect on

semanage

semanage is used for managing SELinux settings persistently, including booleans, port contexts, and file labels. The general syntax for working with booleans is semanage boolean -m --on|--off <BOOLEAN NAME>.

semanage boolean -m --on httpd_enable_homedirs

Port Contexts are used to allow a service such as web or mail server to operate on a non-default port.

File Labels are used to define how SELinux should treat specific files or directories on the system.

SELinux Logging and Troubleshooting

sealert

sealert reads SELinux audit logs and provides clear, human-readable summaries of what denied and why. The logs are usually stored in /var/log/audit/audit.log. The general syntax is sealert -a <LOG PATH>.

# to read review SELinux relater logs
sealert -a /var/log/audit/audit.log

audit2allow

audit2allow is a tool that helps generate new policy rules based on denials to resolve issues by safely expanding SELinux policy when appropriate. The basic syntax is audit2allow -a that analyzes all current logs in the system's audit log. Alternatively audit2allow -i <LOG PATH>.

SSHD Secure Authentication and Access Control

SSHD: Secure Shell Daemon

Key vs Password Authentication

Password authentication requires users to type their password every time they login.

Key based authentication uses a private/public key pair for secure login. Key-based can be enforced by setting PasswordAuthentication no in /etc/ssh/sshd_config, then restart the service for the change to take effect.

PermitRootLogin

This controls whether the root user can login via SSH. It is common to disable this feature to reduce potential attack vectors. To disable this feature, set PermitRootLogin no in sshd_config.

AllowUsers

AllowUsers restricts SSH login to specific users.

# to edit the config
nano /etc/ssh/sshd_config

# allow user to login via ssh and block everybody else
AllowUsers tom tyler jessica@ws1

AllowGroups

AllowGroups allows SSH login for members of a specific group.

# allow ssh login to members of selected groups only
AllowGroups sshusers

SSHD Secure Configuration and Usage

Disabling X11 Forwarding

X11 allows graphical applications to be run on a remote system but displayed on the local machine. To disable it set X11Forwarding no in /etc/ssh/sshd_config, then restart sshd service to apply the changes

SSH Tunneling

Routes traffic through an encrypted SSH connection for securely accessing internal web applications or databases not exposed to the internet.

Secure Remote Access

SSH Agent

The SSH agent is a daemon that stores decrypted private keys in memory to avoid retyping the passphrase for each server connection.

# to load keys into daemon
ssh-add ~/.ssh/id_rsa

SFTP with chroot

A key aspect of secure remote access focused on controlling user interaction with the system during file transfers. SFTP with chroot restricts filesystem access during encrypted file transfer using SFTP.

# to limit sftp users to their designated directories
Match Group sftpusers
  ChrootDirectory /home/sftp/%u
  ForceCommand internal-sftp
  X11Forwarding no
  AllowTcpForwarding no

fail2ban

fail2ban is a security tool used on Linux systems that automatically blocks IP addresses which show signs of suspicious activity.

Configuration

The main config is located in /etc/fail2ban/jail.conf. It is advised to no change that file but to make a copy then change the copy like /etc/fail2ban/jail.local.

Each section in the configuration is called a jail. Each jail correspond a specific service such as ssh.

ex:

# to monitor ssh fail login attempt and block if necessary
[sshd]
enabled = true
port    = ssh
filter  = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime  = 10800

Restart the service to apply the changes with systemctl restart fail2ban.

Monitoring

The main log is located in /var/log/fail2ban.log

Triggering

Simulate triggering by simulating fail login attempts.

# to detect which IP address have been banned
fail2ban-client status sshd

Unblocking

fail2ban-client set [JAIL] unbanip [IP ADDRESS]

# to unblock an ip address
fail2ban-client set sshd unbanip 203.204.205.206

Avoid Unsecure Services

Telnet

Telnet sends everything in plain text without encryption. it has been replaced by SSH.

# to disable telnet
systemctl disable --now telnet.socket

# remove telnet package
dnf remove telnet

FTP

FTP transmit credentials and files in plain text. SFTP (Secure FTP using SSH) and FTPS (FTP over TLS) are secure alternatives.

# to disable ftp 
systemctl disable --now vsftpd

# remove the package
dnf remove vsftpd

TFTP

TFTP has no authentication and no encryption. SCP and SFTP are secure alternatives.

# to disable tftp
systemctl disable --now tftpd

# remove the package
apt remove tftp-hpa
dnf remove tftp-server

Disable Unused File Systems

Linux often comes with rarely used filesystems.

Disabling filesystem by disabling kernel modules

  • cramfs
  • hfs
  • udf

To disable them, edit or create a config file in /etc/modprobe.d/ and add line install cramfs /bin/false.

# /etc/modprobe.d/cramfs.conf
install cramfs /bin/false

# /etc/modprobe.d/hfs.conf
install hfs /bin/false

# /etc/modprobe.d/udf.conf
install udf /bin/false

Disabling filesystem by editing fstab

fstab tells Linux which file systems to automatically mount at boot time. Outdated entries in fstab file can create security risks. Disable unnecessary lines placing a # at the beginning.

Unnecessary SUID Permissions

SUID bit

SUID (Set User ID) bit is a special file permission that can be applied to executable files. Setting the SUID bit in the wrong executable can be a security risk.

# to look for SUID bit file: -rwsr-xr-x. The 's' indicates that SUID bit is set
ls -l

SUID binaries

SUID binaries are the programs or executables that have the SUID bit set.

# to search entire root file system for all files with SUID bit set
find / -perm -4000 -type f 2>/dev/null

# remove the SUID bit
chmod u-s /usr/bin/rcp

Secure Boot

Secure Boot is a feature designed to prevent unauthorized or malicious code from running during the system's startup process.

UEFI

UEFI = Unified Extensible Firmware Interface. It is the modern replacement for the older BIOS system that was used for decades to initialize hardware and start the operating system.

Secure boot can be configured via the UEFI menu with F2, Del, or Esc.

  • in Blog, Linux
  • 5 min read

Linux: Compliance and Audit

Detection and Response

Anti-malware tools

  • CalmAV: an open-source anti-malware option for Linux systems. Run clamscan -r /usr to scan each file against recent virus database.

  • Linux Malware Detect (LMD): LMD is built on ClamAV to automatically scan uploads for PHP backdoors and known malware families

  • rkhunter: used for rootkit detection. Run it with rkhunter --check to check for rootkits.

  • chkrootkit: is used to inspect the system for hidden binaries, suspicious configurations, and tampered libraries

Indicators of Compromise (IoCs)

IoCs are things bad actors leave behind such as unexpected processes, odd network connections, unauthorized file changes, and more.

# to hunt for brute force login attempts
grep -i 'failed password' /var/log/auth.log

# to see open ports
ss -tulnp

We can use specialized tools like YARA, auditd, and Wazuh to hunt for IoCs.

Vulnerability Categorization

Vulnerability categorization is the practice of systematically identifying and describing software flaws.

CVE: Common Vulnerabilities and Exposures

CVE-YYYY-NNNNN:

  • YYYY: Year of the CVE

  • NNNNN: Sequence number of the year

CVSS Common Vulnerability Scoring System

CVSS Categorization:

  • 0.0: None

  • 0.1-3.9: Low

  • 4.0-6.9 Medium

  • 7.0-8.9 High

  • 9.0.10.0 Critical

Vulnerability Management

Service Misconfigurations

A service misconfiguration occurs when a Linux daemon is left with unsafe defaults or overly permissive settings. For example:

  • leaving SSH configured to allow password-based root user login

  • binding critical services to all network interfaces (0.0.0.0) instead of localhost

Backporting Patches

Backporting patches is the process of taking security fixes from a newer version of a package and applying them to the older version running on the system.

Vulnerability Detection

Port Scanners

Port scanners detect open network ports and services running on those ports.

  • Nmap: run nmap -sS -sV 10.0.0.0/24 to process a stealth scan

  • Zenmap: GUI version of Nmap

Protocol analyzers (or packet sniffers)

Protocol analyzers allow deeper inspection of the data moving through these ports by capturing and examining the network traffic.

  • Wireshark: a GUI tool that offers advanced packet analysis for many network protocols

  • Tshark: The CLI version of Wireshark

  • tcpdump: useful for remote management. tcpdump -i eth0 port 443 -w capture.pcap

Standards and Audit

Center for Internet Security Benchmarks

CIS provide detailed, expert-developed best practices for configuring systems securely. From password policies to disabling unnecessary services. It gives a standardized way to protect systems and prove compliance.

OpenSCAP - Open Security Content Automation Protocol

OpenSCAP allows admins to scan systems for compliance, identify security gaps, and even apply fixes, all based on recognized standards. It is a free and open-source tool that uses SCAP content to scan systems and tell how secure they are.

File Integrity Verification

Signed Package Verification

Signed Package Verification helps confirmed that the package being installed originates from a trusted source and has not been modified since its publication.

Installed File Verification

Installed File Verification allows periodic checks to ensure that none of the system files have been changed unexpectedly.

File Integrity Tools

Rootkit Humber krhumter

rkhunter is a lightweight tool that scans systems for signs of rootkits, backdoors, and known vulnerabilities by comparing system files and settings against a database of suspicious patterns. 

# to do an interactive check for issues
rkhunter --check

# to do a non-interactive check
rkhunter --check --skip-keypress

# to update the toolkit database up to date
rkhunter --update

It is common to schedule rkhunter as a cron job to scan system and alert admin if anything suspicious is found.

Advanced Intrusion Detection Environment (AIDE)

AIDE builds a baseline snapshot of selected files and directories, captures details, and then compares the system to the snapshot during regular scans.

# to init AIDE
aide --init

# to detect changes
aide --check

Data Destruction Overwriting Tools

Data destruction overwriting tools overwrite deleted data with random or specific patterns to prevent recovery, used in sensitive and enterprise environments.

shred

shred securely deletes individual files by overwriting them with random data multiple times. 

# to destroy a file
shred -u -v -n 5 old_secrets.txt

dd if=/dev/urandom

This command is used to overwrite entire disks or partitions with random bits, preventing recovery of any previous contents.

# to overwrite a disk or partition with random bits
dd if=/dev/urandom of=/dev/sdc1 bs=1M status=progress

badblocks -W

badblocks is used to check disk errors, but in write mode it can also destroy data by repeatedly overwriting the disk with test patterns.

# to erase a device
badblocks -wsv /dev/sdc2

Cryptographic Data Destruction

cryptsetup with LUKS

cryptsetup is used to encrypt an entire disk or partition. It can also be used to permanently destroy encrypted data by simply erasing the encryption header or one or more keyslots. The keyslots stores and encrypted copy of the master key.

# to erase cryptsetup header
cryptsetup luksErase /dev/sdb

zuluCrypt

zuluCrypt is a GUI and CLI tool for managing encrypted volumes. It supports LUKS volumes.

# to wipe en encrypted device 
zuluCrypt-cli -z -d /dev/sdb1

Software Supply Chain

GPG - GNU Privacy Guard

GPS ensures that software comes from a trusted source and hasn't been tampered with.

SBOM - Software Bill of Materials

SBOM is the detailed ingredient list for software, listing libraries, dependencies, and open-source components included in the application.

CI/CD

CI/CD is the process that automates how code is built, tested, and released, and it's the engine that keeps the modern software supply chain moving. Popular tools that is used in CI/CD are Jenknis GitLab CI, and GitHub Actions.

Security Banners

Tools used to show banners:

/etc/issue

/etc/issue show messages before login on local terminals

example of message: Authorized access only. This is Service 1 - Production

# to change the message
echo "Authorized access only!" | sudo tee /etc/issue

/etc/issue.net

/etc/issue.net show messages before login over remote access like SSH. It is often used for legal warnings o policy notices.

example of message: Warning: Unauthorized access to this system is prohibited and will be prosecuted.

/etc/motd`

/etc/motd (motd = message of the day) show messages after successful login. It is commonly used to communicate helpful information to users.

example of message: System maintenance scheduled for Saturday at 11 PM. Please save your work.

# to change the message
echo "System maintenance scheduled for Saturday at 11 PM. Please save your work." | sudo tee /etc/motd
  • in Blog, Linux
  • 3 min read

Linux: Cryptography

Data at Rest - File Encryption

GNU Private Guard - GPG

GPG is a tool used to encrypt, decrypt, and digitally sign files using an asymmetric key

# to generate keys
# the keys are stored in ~/.gnupg
gpg --full-generate-key

# to encrypt a file using a asymmetric key
gpg --encrypt --recipient 'user@demo.com` secrets.txt

# to encrypt a file using a symmetric key
gpg -c secrets.txt

# to decrypt a file with a asymmetric key
gpg --decrypt secrets.txt.gpg

# decrypt a file with a symmetric key
gpg secrets.txt.gpg

# to digitally sign a file
gpg --sign secrets.txt

# to verify a signature
gpg --verify secrets.txt.gpg

A digital signature helps verify the identity of the sender and the integrity of the file.

Data at Rest - Filesystem Encryption

Linux Unified Key Setup version 2 - LUKS2

LUKS2 is a standardized, on-disk encryption container that wraps a filesystem in an impenetrable shell. argon2 is a lock mechanism used by LUKS2 to slow down attackers by requiring significant time and memory to test each passphrase. Argon2 has 3 variants: argon2i, argon2d, and argon2id.

# to install required tools
dnf install cryptsetup

# to check if a device contains a LUKS header
cryptsetup isLuks /dev/sdc2

# to add an extra passphrase or keyfile
cryptsetup luksAddKey /dev/sdc2 ./key.bin

# remove an existing
cryptsetup luksRemoveKey /dev/sdc2

# to view luks header
cryptsetup luksdump /dev/sdc2

# to encrypt a disk/partition
cryptsetup luksFormat --type luks2 /dev/sdc2

# to decrypt/open a disk/partition
# This will creates a mapped device in /dev/mapper/encrypted_disk.
cryptsetup luksOpen /dev/sdc2 encrypted_disk

# close the encrypted device
cryptsetup luksClose encrypted_disk

Data in Transit - OpenSSL

OpenSSL allows the creation and management of digital certificate and keys used to authenticate user identities.

TLS Certificate

TLS certificate is a like a digital passport for servers. It contains important information about the servers and is signed by a trusted Certificate Authority (CA).

# to generate a self signed certificate
openssl genpkey -algorithm RSA -out server.key

# to create a certificate signing request - CSR
openssl req -new -key server.key -out server.csr

s_client

s_client is used to probe any TLS-enabled service from the command line.

# to retrieve the server's certificate and verify the issuer, expiry date, and intermediates certs
openssl s_client -connect mail.example.com:993 -showcerts

Protection Methods

TLS Protocol Versions

TLS 1.2 and above are considered safe

LibreSSL

LibreSSL is a fork of the original OpenSSL library designed to be easier to audit and maintain.

# to install LibreSSL
dnf install libressl
WireGuard

WireGuard is a next gen VPN solution that operates inside the Linux kernel to secure entire network tunnels with modern cryptography.

Hashing

A hash function is a cryptographic hash algorithm that converts any size input into a fixed bit digest, ensuring data integrity by making it nearly impossible for two different inputs to produce the same output.

SHA-256

SHA-256 uses 256 bit digest

# to calculate the checksum of a file
sha256sum myfile,txt

Hash-based Message Authentication Code - HMAC

HMAC combines a secret key with SHA-256 to generate a keyed digest, allowing recipients who share the secret to verify both the integrity of the data and the authenticity of its sender.

# to calculate the hmac using SHA 256
openssl dgst -sha256 -hmac "secretkey" myfile.txt

Removal of Weak Algorithms

  • Defense Information Systems Agency Security Technical Implementation Guides (DISA STIGs). DISA STIGs sets the baseline for hardening Linux servers, and that includes explicitly turning off week cryptographic algorithms (disable legacy ciphers such as RC4, 3DES, prohibit MD5-based hashing)

  • FIPS 140-2 defines approved cryptographic modules and algorithms for federal systems, and FIPS compliance on a Linux machine ensures only approved algorithms are offered.

  • Disable SSHv1 sudo sed -i 's/^#*Protocol.*/Protocol 2/' /etc/ssh/sshd_config && sudo systemctl restart sshd

  • Use sslsanc to probe TLS-enable services and flag anything outdated.

Certificate Management

No Cost Trusted Root Certificate

  • Let's Encrypt: Free to use

Commercial Root Certificate Authorities

They charge fees in exchange for extended validation procedures, longer certificate lifetimes, insurance warranties, and hands-on support.

  • DigiCert

  • GlobalSign

  • Sectigo

  • Entrust

  • in Blog, Linux
  • 2 min read

Linux: Hardening Techniques

Password Composition Controls

Password Complexity

Password complexity requires different characters types in a password, like uppercase, lowercase, digits, and symbols. it ensures that user passwords include a mix of character types, such as uppercase letters, lowercase letters, numbers, and special characters. Password complexity is managed through PAM. To the user password complexity, edit /etc/security/pwquality.conf

# password should include at least 4 characters of different categories
minclass = 4

Password Length

Password length sets the minimum number of characters required in a password. It is also configured through the pam_pwquality module.

# set minimum password length
minlen = 12

Password lifecycle Controls

Password lifecycle controls require users to change their passwords regularly.

Password Expiration

Password expiration forces users to change their passwords after a certain number of days. chage is used to control this setting per user account basis.

# To change user password max age
chage -M 90 samuel

Password History

Password History keeps track of old passwords to support Password reuse.

Password Reuse

Password reuse prevents users from reusing old passwords. pam_pwhistory tracks old passwords in order to block password reuse. To change the settings, edit /etc/pam.d/common-password.

# to prevent any user from reusing their last 5 passwords
password requisite pam_pwhistory.so remember=5

Checking existing breach lists

Have I Been Pwned - HIBP

Checks email addresses against known public breaches.

Have I Been Pwned haveibeenpwned.com

Via API:

# check for email in breach
https://haveibeenpwned.com/api/v3/breachedaccount/jdoe@email.com

DeHashed

DeHashed provides deeper insight with email, phone number, username, ip address, and document searches in breach data.

Via API:

# to search break data for a selected email address
https://api.dehashed.com/search?query=jdoe@email.com&size=20

Intelx.io

Intelx.io provides enterprise-grade OSINT solution, aggregating data from dark-web forums, paste sites, and public breache dumps with powerful query syntax and API access.

Restricted shell use

/sbin/nologin

/sbin/nologin prevents interactive login.

ex:

# to create a user withn o shell access. Go for automated services
useradd -s /sbin/nologin backupbot

/bin/rbash

/bin/rbash provides limited shell access to users. It restricts actions like changing directories, modifying environment variables, or executing programs rom unexpected locations.

ex:

# create a user with restricted bash shell
useradd -s /bin/rbash -m reports

pam_tally2

pam_tally2 helps monitor and respond to failed login attempts.

/etc/pam.d/common-auth

/etc/pam.d/login

# to lock account after 5 failed attempts
# and automatically unlock it after 10 minutes
auth required pam_tally2.so onerr=fail deny=5 unlock_time=600

pam_tally2 # to view a summary of all failed attempts

pam_tally2 --user john # to view a summary of all failed attempts for a selected user

Avoid Running as root user

sudoers

/etc/sudoers is edited using visudo to prevent errors.

# give the user access to restart Nginx and nothing more
john ALL=(ALL) /sbin/systemctl restart nginx

PolKit (PolicyKit)

pkexec runs a command as another user

pkaction lists available privileged operations on the system

pkcheck checks whether a user is authorized to perform a specific action

pkttyagent provides a prompt for authentication in a terminal session

Linux: Firewall

firewall Configuration and Management

Zones

A zone is a named profile that carries itw own rule set for which services and ports are allowed through the zone. To create a zone, run firewall-cmd --permanent --new-zone=<ZONE-NAME> && firewall-cmd --reload. Run firewall-cmd --get-zones to see all zones.

Runtime Settings

Runtime settings take effect immediately and stays active until the next reboot or manual reload.

Permanent Settings

Permanent settings persists across reboots but does not touch the running firewall until reload.

firewall-cmd

firewall-cmd is the command line tool used to manage firewalld configurations. The general syntax is firewall-cmd <OPTION> <OPTION VALUES>.

Useful options include:

  • --get-zones: display all zones
  • --get-active-zones: shows only zones that currently have bound interfaces
  • --list-all --zone=<ZONE>: displays every rule in a given zone
  • --add-port=<PORT>/<PROTOCOL>: opens individual ports
  • --remove-port=<PORT>/<PROTOCOL>: closes individual ports
  • --runtime-to-permanent: to copy current rule set to disk
  • --set-default-zone=<ZONE>: to change the default zone assigned to new interfaces
  • --zone=<ZONE> --change-interface=<INTERFACE>: to assign an interface to a zone

Rules and Access Control

Ports
# to add port
firewall-cmd --zone=internal --add-port=8080/tcp --permanent

# to remove a port
firewall-cmd --zone=internal--remove-port=8080/tcp --permanent
Services
# to add https service
firewall-cmd --zone=internal --add-service=https --permanent

# to remove https service
firewall-cmd --zone=internal --add-service=https --permanent
Rich Rules

Rich rules extend firewalld with "if-this-then-that" logic.

# to add rich rule to a zone
firewall-cmd --zone=public --permanent --add-rich-rule='rule family="ipv4" source address="192.168.1.10" service name="ssh" accept' --permanent

Uncomplicated Firewall - (UFW)

By default UFW blocks all incoming traffic and allows all outgoing traffic. It writes every change directly to its configuration file and loads it boot.

ufw enable # to enable UFW service

ufw disable # to disable UFW service

ufw allow 8080/tcp # to add a allow rule

ufw allow ssh # to add a allow rule

ufw deny 23 # to add a deny rule

ufw delete allow http # to delete a allow rule

ufw allow from 192.168.1.10 # to allow traffic from specific IP address

ufw deny from 192.168.1.10 # to deny traffic from specific IP address

ufw allow from 192.168.1.0/24 to any port 22 # to allow subnet to access specific port

ufw status numbered # to see numbered rule set

ufw delete 2 # to delete numbered rule set

ufw default deny incoming # to set default incoming (deny)

ufw default allow outgoing # to set default outgoing (allow)

iptables

iptables is a command line utility used for traffic filtering and alteration. It is build around tables. The main tables are:

  • filter
  • nat
  • mangle
  • raw
  • security

Each table contains Chains:

  • INPUT: inspects packets destined for local system
  • OUTPUT: filters packets originating from local system
  • FORWARD: filters packets moving through the system

The general syntax is iptables [-t <TABLES>] -A <CHAIN> -p <PROTOCOL> [MATCH OPTION] -j <TARGET>.

ex:

# to accept SSH traffic into a host
iptables -t filter -A INPUT -p tcp --dport 22 -j ACCEPT

ipset

ipset groups many ip addresses or subnets into sets to let servers match and process packets more efficiently than checking each one individually. The generic syntax is ipset [OPTIONS] <COMMAND> <SETNAME> [PARAMS]. Common commands include: create, add, del, list.

# to keep a dynamic deny list of know-bad ip addresses and tie it back into iptables

# create new set
ipset create bad_hosts_list hash:ip

# add offending ip address to set
ipset add bad_hosts_list 172.0.0.25

# view ipset list
ipset list bad_hosts_list

nftables

nftables is a single framework that merges tables and rules. It is a modern successor of iptables. The general syntax is nft [OPTIONS] add rule <FAMILY> <TABLE> <CHAIN> <EXPRESSION>

# to allow ssh on port 22
nft add rule inet filter input tcp dport 22 ct state new accept

Netfilter Module

The Netfilter module is a Linux kernel module that acts like the digital gatekeeper, examining every data packet entering or leaving the system. and deciding if a packet should be blocked or allowed according to the predefined rules. It is the backend for iptables, ip6tables, and nftables.

Stateful and Stateless Firewall

Stateless Firewall

A stateless firewall treats each incoming packet independently using pre-defined rules. 

# accept http traffic
iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# drop all other traffic
iptables -A INPUT -j DROP

Every packet will be checked to see if it is destined to port 80. If not the packet will be dropped.

Stateful Firewall

Stateful firewall remembers ongoing communications sessions between computers.

# Allow established and related packets
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

# Allow new ssh connections
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j ACCEPT

# Drop everything else
iptables -A INPUT -j DROP

IP Forwarding

IP forwarding allows system to pass network traffic from one interface to another, acting like a router. IP forwarding is disabled by default. Set ip forwarding permanently with net.ipv4.ip_forward = 1 in /etc/sysctl.conf. For temporary enabling ip forwarding, run sysctl -w net.ipv4.ip_forward=1.

Linux: Authorization, Authentication, and Accounting

Local Authentication

PAM (Pluggable Authentication Modules)

PAM handles the core authentication process: validating usernames, password, and enforcing policies. PAM relies on other modules to handle specific part of the authentication process. These modules are configured in files located in /etc/pam.d/ directory.

PAM module types

  • auth: verifies user identity
  • account: enforces access policies
  • password: handle password updates
  • session: manages tasks that happen at the start or end of a session

PAM uses controls flags to determine how each module's result should affect the overall outcome.

Module flags:

  • required: the module must pass, processing continues even if it fails
  • requisite: the module must pass, failure causes immediate termination
  • sufficient: success means authentication may succeed early if no required module failed
  • optional: only evaluated if it's the only module in the group

Polkit (PolicyKit)

Polkit manages authorization: deciding if regular users can perform administrative or system-level actions without switching to root. The rules are configured in files in /etc/polkit-1/rules.d/ or /etc/polkit-1/localauthority/ directories.

Directory-based Identity Management

Kerberos

Kerberos handles secure authentication using a ticket-based system to prove identity without repeatedly sending passwords. It is a secured network authentication protocol that allows users and services to prove their identity without sending passwords over the network.

LDAP (Lightweight Directory Access Protocol)

LDAP provides a structured directory for storing user accounts, groups memberships, and organizational information. It is a standardized protocol used to access and manage directory information. It is where the usernames, group definitions, and user attributes are stored.

SSSD (System Security Service Daemon) and Winbind

SSSD and Winbind act as intermediary on Linux for connection and using these centralized services seamlessly.

Network / Domain Integration

realm

realm is a tool that simplifies the process of joining systems to domains and sets up authentication with minimal manual configuration. realm enables identity and login integration with Windows domains, but it doe snot handle file or printer sharing.

ex:

realm discover my.domain.com # to discover domains

realm join --user=admin my.corporation.com # to join my.corporation.com domain using the admin credentials

realm list # verify the configurations

realm permit --all # to permit all users to login

realm permit admin@my.domain.com # to allow specific user to login

realm permit -g "Administrators" # to allow a group to login

realm leave my.domain.com # to leave a domain

Samba

Samba provides a deeper integration with Windows environments. It is focussed on file sharing, printer access, and Windows-compatible network services. The main configuration is located in /etc/samba/smb.conf

example file share:

[global]
  workgroup = WORKGROUP
  server string = Samba Server
  security = user

[Public]
  path = /srv/samba/public
  browsable = yes
  writable = yes
  guest ok = yes

udo systemctl start smb nmb # to start samba service

sudo systemctl enable smb nmb # tp enable samba a system start

Logging

/var/log

/var/log is the central directory on most Linux systems where log files are stored.

  • messages General system messages
  • /var/log/syslog System-wide log
  • /var/log/kern.log Kernel-specific messages
  • /var/log/auth.log / /var/log/secure Authentication and authorization events
  • /var/log/boot.log Boot process messages
  • /var/log/dmesg Kernel ring buffer messages
  • /var/log/cron Cron job execution logs
  • /var/log/maillog / /var/log/mail.log Mail server logs
  • /var/log/Xorg.0.log / /var/log/X server graphical session logs
  • /var/log/apt/ / /var/log/yum/ Package manager logs
  • /var/log/journal/ Systemd journal storage

rsyslog

rsyslog is a high-performance logging service that receives and stores log messages from the kernel, services, and applications. The configurations are stored in /etc/rsyslog.conf and /etc/rsyslog.d/*.conf

ex:

auth.* /var/log/auth.log # to store all authentication messages to a file

kern.warning /var/log/kern.log # to log only kernel warning messages and above

*.* @@log.server.com:514 # to send log messages to a remote server

Message severity levels

emerg # system unusable

alert # immediate action required

crit # critical conditions

err # errors

warning # warnings

notice # normal but significant

info # informational messages

debug # debug messages

journalctl

journalctl is a systemd tool used to view messages store by systemd journal.

journalctl -b # to view all logs for the current boot

journalctl -b -1 # to view all logs for the previous boot

journalctl -f # to tail log

journalctl -k # to view logs from the kernel

journalctl -u nginx.service # to view logs nginx

logrotate

logrotate is a tool for managing the size and rotation of log files, ensuring that logs do not fill up the disk over time. The main configuration is located in /etc/logrotate.conf and /etc/logrotate.d/.

ex:

logrotate -d /etc/logrotate.conf # to check configuration

logrotate -f /etc/logrotate.conf # to force log rotation

System Audit

auditd

auditd is a service that records audit events to disk, and administrators control i witht he systemd utility.

audit.rules

audit.rules is the configuration file that tells the audit subsystem precisely which activity to record. The configuration is located in /etc/audit/rules.d/audit.rules.

ex:

-w /etc/passwd -p wa -k passwd_changes # to tell the audit system to watch for password changes

-w /var/log/lastlog -p wa -k login_logs # to watch for user login

-w /var/run/faillock -p wa -k failed_logins # to watch for failed logins

ausearch -k passwd_changes # to search logs for keys

Linux: Backup and Recovery

Basics

. refers to the current directory

.. refers to the parent directory of the current directory

~ refers to the home directory

Archiving

Archiving combines multiple files into one package, making them easier to backup, transfer, or organize. tar and cpio are popular tools used for archiving.

tar

tar packages multiple files or directories into a single archive file. The syntax is tar [OPTIONS] <ARCHIVE NAME> [FILE1 FILE2, DIR1...]. Common options includes:

  • -c to create an archive
  • -x to extract files
  • -t to list the contents of an archive
  • -v for verbose output
  • -r to append files to an existing archive
  • -f to specify te archive file name
  • -z for gzip
  • -j for bzip2
  • -J for xz

ex:

tar -czvf backup.tar.gz data/ # to create an archive of the data/ directory using gzip

cpio (copy in/out)

cpio get the list of file to archive from another command like find or ls. The general syntax using find is find [FILES] | cpio -ov > [ARCHIVE NAME].cpio. The following are the main 3 modes:

  • -o to create an archive (copy-out)
  • -i to extract and archive (copy-in)
  • -p to copy files (copy-pass)

additional options:

  • -d to create directories as needed
  • -v for verbose output
  • -u to override existing files
  • -t to list archive content

ex:

find /configs -type f | cpio -o > config_bk.cpio # to create an archive

cpio -id < backup.cpio # to extract an   archive

cpio -it < backup.cpio # to list the content of the archive

find data/ -name "*.conf" | cpio -pvd /backups/configs # to copy files

Compression Tools

Compression tools helps shrink files size.

gzip

gzip is widely used for its speed and simplicity. It uses the .gz format. For backup it is recommended to use tar + gzip (-cvfz). Common options include

  • -d to decompress files
  • -f to override files without asking
  • -n to skip storing the original file name and timestamp
  • -N to save the original file name and timestamp
  • -q for quiet mode
  • -r to compress directories recursively
  • -l to show statistics
  • -t to test the integrity of the compressed file
  • -v for verbose mode
  • -1...-9 to specify compression level

ex:

gzip myfile.txt # to compress a file and delete the original

gzip -k myfile.txt # to compress a file and keep the original

gzip -k myfile1.txt myfile2.txt myfile3.txt # to compress a file and keep the original

gzip -vr /var/log/ # to compress the content of the folder with verbose output

gzip -9 image.iso # to compress with maximum level (levels range 1-9 default is 6)

zcat myfile.txt.gz # to view compress file content

gunzip myfile.txt.gz # to uncompress an archive

bzip2

bzip2 offers a better compression but slower to complete compare to gzip. The syntax is bzip2 [OPTIONS] <FILE NAME>

  • bzip2 is used for compressing files
  • bunzip2 to uncompress files
  • bzcat to view content of a compressed file without extracting it
  • bzip2recover to attempt to recover data from a damaged archive
  • bzless and bzmore to scroll through compressed text files one page at a time

ex:

bzip2 myfile.txt # to compress a file and delete the original file

bzip2 -k myfile.txt # to compress a file and keep the original file

bunzip2 myfile.txt.bz2 # to decompress a file

bzip2 -t myfile.txt.bz2 # to test the integrity of a compressed file

bcat myfile.txt.bz2 # to list the content of a compressed file

xz

xz is a newer compression tool that offers a higher compression but is even slower than gzip and bzip2. It is great for archiving files that do not change often. The syntax is xz [OPTIONS] <FILE NAME>. Command options include:

  • -d to decompress an compressed archive
  • -f to override files
  • -q for quite mode
  • -v for verbose mode
  • -t to test compressed file

ex:

xz myfile.txt # to compress a file and delete the original file

xz -k myfile.txt # to compress a file and keep the original file

xz -d myfile.txt.xz # to decompress a file

unxz myfile.txt.xz # to decompress a file

xz -t myfile.txt.xz # to test the integrity of a compressed file

xz -l myfile.txt.xz # to list the content of a compressed file

7-Zip

7-Zip is used where compatibility with Windows system is needed. It is more flexible because it handles multiple archive format like .7z, .zip, and .tar. It is usually available through the p7zip package. Common options include:

  • -a to add files to an archive
  • -x to extract files from an archive
  • -l to list archive content
  • -t to test an archive
  • -d to delete files from an archive

ex:

7z a backup.7z file1 file2 data/ # to create a compressed archive

7z x backup.7z # to extract a compressed file

7z l backup.7z # to list the content of a compressed file

7z t backup.7z # to test a compressed file

7z a -mx=9 backup.7z image.iso

Data Recovery

dd (data duplicator)

dd copy data at the block level and is useful for creating exact images of disks or partitions. It is commonly used for disk cloning, creating bootable USB drive, doing backup and restore, and wiping disks. The basic syntax is dd if=<INPUT FILE> of=<OUTPUT FILE> [OPTIONS]. Common options include:

  • if= input file/device
  • of= output file/device
  • bs= block size. The default is 512 bytes
  • count= number of blocks to copy
  • skip= number of input blocks to skip
  • seek= number of output blocks to skip before writing
  • status=progress to show progress
  • conv=noerror,sync to copy pass read error in bad blocks.

ex:

dd if=image.iso of=/dev/sdb1 bs=4M status=progress # to create a bootable USB drive

dd if=/dev/sda of=diskA.img bs=1M status=progress # to create a disk image

dd if=diskA.img of=/dev/sda bs=1M status=progress # restore data from an image

dd if=/dev/zero of=/dev/sdb bs=1M status=progress # to completely erase a disk

dd if=/dev/zero of=test_file bs=1G count=1 oflag=dsync # to test the write speed of a disk

ddrescue

ddrescue is used to recover data from damaged drives. The basic syntax is dd [OPTIONS] <INPUT FILE> <OUTPUT FILE> <LOG FILE>

ex:

ddrescue /dev/sdb damaged.img rescue.log # to attempt rescuing /dev/sdb

rsync

rsync is used to synchronize files and directories over the network. After the first copy, it copies only differential changes in subsequent copy. The basic syntax is rsync [OPTIONS] <SOURCE> <DESTINATION>. Important options are:

  • -r# to copy recursively
  • -a# to copy in archive mode preserving permissions, symblinks, and timestampts
  • -n# to see what would be copied (Dry run)
  • -z# to enable compression during transfer
  • -h# to see a human-readable output
  • -v# for verbose mode
  • --progress# to show progress
  • --delete# to remove files in destination that are not present in the source

ex:

rsync -avh /home/user/ /mnt/backup/user # to copy user directory with all attributes preserved

rsync -avh user@server:/data/ /home/user/data/ # to sync from remote server to local

rsync -avh --bwlimit=4000 /home/user/ user@server:/backup/ # with a bandwidth limit = 4000KB/s

Compressed File Operations

zcat

zcat displays the full content of a compressed file.

zcat myfile.txt.gz # to show the content of the compressed file

zless

zless allows scrolling through the content of a compressed file interactively

zless myfile.txt.gz # to show the content of the compressed file in a scrollable mode

zgrep

zgrep allows searching through compressed data. The syntax is zgrep [OPTIONS] <SEARCH PATTERN> <FILE NAME> Common options include:

  • -i # to make the search case-insensitive
  • -n # to show line numbers
  • -v to show lines that do not match the query

ex:

zgrep "ERROR" logs.gz # to search for lines containing text 'ERROR'

zgrep -i "failed password" /var/log/auth.log.1.gz # to find all login attempts

Linux: Network Services and Configurations

Basics

Linux uses a layered approach (local files and external resources) to figure out how to resolve internal and external system names.

  • /etc/hosts
  • /etc/resolv.conf
  • /etc/nsswitch.conf

/etc/hosts

/etc/nosts is a plain text file where we manually map hostname to IP addresses so the system can resolve names without relying on DNS. It is useful for environment where DNS is not available.

ex:

192.168.1.101 server1.local

192.168.1.102 server2.local

/etc/resolv.conf

/etc/resolv.conf tells Linux which DNS servers to use when resolving names that are not listed in /etc/hosts. It is useful for troubleshooting internet related issues or the system cannot resolve external domain names.

/etc/nsswitch.conf

/etc/nsswutch.conf controls the order in which the system tries different methods to resolve names and other data.

hosts: files dns # check local /etc/hosts first before querying external DNS

hosts: dns files # check external DNS first before checking local /etc/hosts

NetworkManager

nmcli

nmcli is a command line interface for interacting with NetworkManager, allowing admin to monitor and manage network connections on Linux.

nmcli device status # to show the status of all devices

nmcli general status # to check overall networking health

nmcli connection show # to lists configured connections

nmcli connection up <CONNECTION-NAME> # to activate a specific network connection

nmcli connection don <CONNECTION-NAME> # to deactivate a specific network connection

nmcli connection edit <CONNECTION-NAME> # to open an interactive editor for detailed changes

nmcli connection reload # to reload the settings after editing

nm-connection-editor is a GUI tool for editing NetworkManager connection profiles without needing to use the command line. Just type nm-connection-editor in the terminal to start the GUI.

.nmconnection is a configuration file used by NetworkManager to store settings for a specific network profile. They are located in /etc/NetworkManager/system-connections/

Netplan

Netplan is the default tool used to configure and manage network settings in Debian Linux. It uses YAML files to centralize network configurations.

Configuration Files

Network configuration files are written in YAML. They are stored in /etc/netplan/ directory. Configurations are not applied automatically. They must be activated before they take effect.

ex:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses: [192.168.1.50/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]

This tells netplan to configure a static IP address on eth0.

netplan try

netplan try is used to try a new network configuration temporarily, with a built-in safety mechanism. This commands applies changes for 120 seconds by default before reverting the configuration if not confirmed.

netplan apply

netplan apply is used to permanently apply changes made in the configuration files. This command reads the configurations from /etc/netplan/, applies the changes to the system, and activates the associated network interfaces.

netplan status

netplan status verifies which configuration is active, which interfaces are managed by which renderer, and their current settings.

IP Network Management Tool

ifconfig

ipconfig is a legacy tool used to configure ip network on Linux system. It is getting deprecated in for of ip command suite.

ex:

ifconfig # to view currently active network interface.

ip address

ip address is used to view and manage IP address configuration of system network interfaces.

ex:

ip address show # to view ip address configuration. or simply "ip a"

ip address show dev eth0 # to view ip address config of interface eth0

ip address add 192.168.1.100/24 dev eth0 # to add a ip address to eth0

ip address del 192.168.1.100/24 dev eth0 # to delete a ip address to eth0

ip address flush dev eth0 # to flush all ip addresses from eth0

ip link focuses on network interface link layer (Layer 2 of the OSI model)

ip link show # to view interfaces

ip link set eth0 up # to bring interface up

ip link set eth0 down # to bring interface down

ip link set eth0 mtu 9000 # to change the MTU

ip link set eth0 dev eth0 address 12:23:34:45:56:67 # to change the MAC address of the NIC

ip route

ip route is used to display and manage the kernel's routing table, which determines how network traffic is forwarded.

ip route add 10.0.0.0/24 via 192.168.1.99 # to add a network route

ip route add 172.16.0.0/16 via eth1 # to add a network route via an interface

ip route add default via 192.168.1.1 # to add a default route

ip route del 10.0.0.0/24 # to delete a route

Network Configuration Tools

hostname

hostname is used to view or set the system's network name. Use hostnamectl set-hostname server1 to permanently set the hostname of the system to server1.

arp

arp is used to show or manage the system's Address Resolution Protocol (ARP) table.

arp -n # to all arp cache

arp -a # to display all arp entries

arp -s 192.168.1.10 12:23:34:45:56:67 # to set a static entry

arp -d 192.168.1.10 # to delete an entry

ethtool

ethtool is used for querying and configuring Ethernet network interface configurations. It is used to query driver information, test link status, change interface speed, and change duplex settings.

ex:

ethtool eth0 # to view NIC details such as speed, duplex mode, link status, and firmware info

ethtool eth0 | grep "Link detected" # to check the link status

ethtool -s eth0 autoneg on # to enable speed auto negotiation

ethtool -s eth0 autoneg off # to disable speed auto negotiation

ethtool -s eth0 speed 100 duplex full # to set the speed and duplex mode

ethtool -t eth0 # to test the link

ethtool -S eth0 # to display statistics

Network Connectivity Tools

ping/ping6

ping is used to test basic reachability and round-trip response time to remote systems over IPv4. The syntax is ping <DESTINATION>

ex:

ping 192.168.1.1 # to send ICMP echo request to IP address

ping mysite.com # to send ICMP echo request to hostname

ping -c 3 mysite.com # to send 3 ping

ping -c 4 -i 2 mysite.com # to send 4 ping with a 2 second interval between each ping

ping -s 1400 mysite.com # to send pings with 1400-byte payloads

ping6 is used to test basic reachability and round-trip response time to remote systems over IPv6.

traceroute

traceroute shows the full path packets take to a destination. The syntax is traceroute <DESTINATION>

ex:

traceroute mysite.com # to see all hops along the path including response time

traceroute -I mysite.com # to use ICMP instead of UDP

traceroute -T -p 80 mysite.com # to use TCP port 80

traceroute -m 10 mysite.com # to limit number of hops to 10

traceroute -q 2 mysite.com # to change number of probe packet per hop. Default is 3

tracepath

Similar to traceroute but does not require root privilege. The syntax is tracepath <DESTINATION>.

ex:

tracepath -m 15 google.com # to set the maximum number of hops

mtr

mtr is a short for My Traceroute. it combines the functions of both ping and traceroute into a live, interactive view of each network hop. The syntax is mtr mysite.com.

ex:

mtr mysite.com

iperf3

iperf3 is an advanced tool used to test actual network throughput between systems and assess bandwidth performance under real conditions. The general syntax is iperf3 -c <DESTINATION>.

ex:

iperf3 -s # to start the server

iperf3 -c 192.168.1.50 # to test client

iperf3 -c 192.168.1.50 -t 60 # to test client for 60 seconds

iperf3 -c 192.168.1.50 -P 8 # to set 8 parallel streams

iperf3 -c 192.168.1.50 -J # to output JSON

Network Scanning and Traffic Analysis Tools

ss

ss is the quickest way to look at the sockets the system is using.

ss -t # to show all TCP connections

ss -u # to show all UDP connections

ss -l # to show listening sockets only

ss -a # to show all connections

ss -p # to show processes using sockets

ss -lnt src :22 # to show SSH connections

nc (netcat)

nc or netcat is a tool to talk to network services.

nc mysite.com 80 # to connect to the host on port 80

nc -l -p 2345 # to listen for connections in server mode

tcpdump

tcpdump is used for network traffic capture and analysis.

tcpdump -i eth0 # to capture packets from eth0

tcpdump -i eth0 -c 100 -w netlog.pcap # to capture first 100 packets from eth0 and write to a file

tcpdump -i eth0 # to capture packets from eth0

tcpdump -i eth0 tcp # to capture only TCP

tcpdump -i eth0 udp # to capture only UPD

nmap (Network Mapper)

nmap is a reconnaissance tool used to scan networks.

ex: 

nmap 192.168.0.50 # to scan a single host

nmap 192.168.0.50 192.168.0.51 # to scan multiple hosts

nmap -p 80,443 192.168.0.52 # to scan host or network for specific ports

nmap -p 1-3000 192.168.1.55 # to scan a host for a range of ports

nmap -sV 192.168.1.50 # to detect services running on the host

nmap -O 192.168.1.50 # to guess OS running on the host

nmap -sS -p 22,80,443 -T4 -Pn 192.168.10.0/24 # to scan the network for hosts with open common ports (22, 80,and 443)

nmap -sS 192.168.0.55 # to perform a half open SYN scan which is faster and stealthier

nmap -F 192.168.0.60 # scan top 100 ports

nmap -Pn 192.168.0.55 # scan without ping

nmap --script=vuln 192.168.0.55 # to run vulnerability scan

DNS Tools

nslookup

nslookup is used to look up domain names. nslookup <DOMAIN> [DNS SERVER].

ex:

nslookup # enter the interactive mode

nslookup mysite.com # to query a domain records

nslookup -type=A mysite.com # to query specific record type such as A,AAAA,MX,TXT

nslookup mysite.com 9.9.9.9 # to query using a specific DNS server

dig (Domain Information Groper)

dig is used to get the full DNS exchange when looking up for domain names. It is more powerful than nslookup. dig [@SERVER] <DOMAIN> [TYPE] [OPTIONS]

ex:

dig mysite.com # to lookup a simple domain name

dig mysite.com AAAA # to query a specific record type

dig mysite.com +short # to obtain a short output

dig -x 88.89.90.91 # to perform a reverse DNS lookup

dig @8.8.8.8 mysite.com # to query using a specific DNS server

resolvectl

resolvectl is a DNS resolver service in systemd. resolvectl <VERB> [ARGUMENTS]

ex:

resolvectl query mysite.com # to query DNS

Linux: Containers

A container is a lightweight, portable environment that that packages an application along with everything it needs to run, including applications. library, and configuration files.

The container runtime is the software responsible for running and managing those containers.

There are many runtimes to choose from:

  • runC
  • containerd
  • Docker
  • Podman

Basics

runC

runC is a lightweight command line tool that creates and runs containers directly from the command line.

containerd

It is runtime that handles the entire lifecycle of containers. Uses runC under the hood but provide high level APIs.

Docker

A popular runtime that includes everything needed to build, run, and manage containers. It uses runC and containerd under the hood.

Podman

Podman is like Docker but designed to run without a central daemon. It works the same way as Docker and most Docker commands work with Podman. Podman supports running containers as a regular user without needing root privileges.

Building an Image

FROM

FROM tells what base image to start with; which operating system and environment the container will be build on top of.

ex:

FROM python:3.11-slim

to start with a Debian Linux that comes with Python 3.11 installed.

USER

USER defines who inside the container will run the remaining commands and processes.

ex:

RUN useradd -m appuser # to create a user

USER appuser # process subsequent command under this user

ENTRYPOINT

ENTRYPOINT defines the main command that will always run when the container starts.

ex:

ENTRYPOINT ["python", "app.py"] # to run this command every time the container starts

CMD

CMD provides default arguments to the ENTRYPOINT, or acts as the command to run if no ENTRYPOINT is set.

ex:

CMD ["--debug"] # to include a default option. So the container will run "python app.py --debug" by default

Example of Dockerfile

Dockerfile

FROM python:3.11-slim

RUN  useradd -m appuser

USER appuser

COPY app.py /home/appuser/app.py

WORKDIR /home/appuser

ENTRYPOINT ["python", "app.py"]

CMD ["--debug"]

then:

  • docker build -t myapp . to build the image
  • docker run myapp to start the container and run the app
  • docker run myapp --test to override CMD line and start the container

Image Retrieval and Maintenance

An container image contains the application code and everything needed to run it.

Image Pulling

Image pulling is the process of downloading a container image from a remote registry to local machine so it can be used to run containers. The general syntax is docker pull <IMAGE-NAME>[:TAG]. The latest tag is pull if the TAG is omitted.

ex:

docker pull ubuntu:20.04 # to pull Ubuntu image with tag 20.04

Image Tags

Tags are labels attached to container images that help identify versions or variants.

ex:

docker pull nginx:latest # to pull the latest version of nginx

Image Layers

Layers are the building blocks of container images. When pulling an image, docker pulls only images that are not previously pulled to increase efficiency.

Image Pruning

Pruning is the process of cleaning up unused containers, images, networks, and volumes to free up space.

Run:

docker system prune # to prune the system

Container Lifecycle Management

Run

docker run is used to create a new container from an image and start it immediately.

ex:

docker run -it ubuntu:20.04 bash # to create and start a Ubuntu:20.04 container with interactive bash shell

Start and Stop

docker start <CONTAINER-NAME> to start a container if it is stopped.

docker stop <CONTAINER-NAME> to stop a container if it is running.

ex:

docker start web-app # to start the container named "web-app"

docker stop web-api # to stop the container named "web-api"

Delete

docker rm is used to delete stopped container that is no longer needed. docker rm <CONTAINER-NAME or ID>

ex:

docker rm webapp-test # to permanently delete a stopped container named "webapp-test"

Prune

docker system prune is used to remove unused resources and free up space. Use -f flag to skip user prompt.

Container Inspection and Interaction

Environment Variables

They are used to pass config variables into a containers at startup. The general syntax is docker un -e <KEY>=<VALUE> <IMAGE-NAME>

ex:

docker run -e NODE_ENV=production node:18

Read Container Logs

Use docker logs <CONTAINER-NAME or ID> to see container logs.

ex:

docker logs web-api # to see the container output and error stream

Inspect Containers

Inspecting a container gives a detailed view of a container's configuration, network settings, mounted volumes, environment variables, an more. Use docker inspect <CONTAINER-NAME or ID> to inspect a container.

docker inspect web-api

Exec

exec is a command that lets users run a command directly inside a running container. The general syntax is docker exec -it <CONTAINER-NAME or ID> <COMMAND>

ex:

docker exec -it db-app bash # open bash shell in interactive mode from the db-app container

Container Storage

Mapping Container Volumes allows users to link a folder from their host machine to a folder inside the container. The general syntax is docker run -v <HOST-PATH>:<CONTAINER-PATH> <IMAGE-NAME>.

ex:

docker run -v /home/user/data:/app/data webapp

Volume Management and Operations

Create Volume

docker volume create <VOLUME-NAME>

ex:

docker volume create apidata # to create a volume named "apidata"

Map Volume

Mapping the volume connects the volume created to a specific location inside a container. docker run -v <VOLUME-NAME>:<CONTAINER-PATH> <IMAGE-NAME>

ex:

docker run -v apidata:/app/data webapi

Prune Volume

Use docker volume prune to remove unused volumes.

Network Management Operations

Create Network

A virtual network gives containers a way to interact with each other or with the outside world securely and efficiently. The general syntax is docker network create [OPTIONS] <NETWORK-NAME>.

ex:

docker network create --driver bridge apps-net # to create a bridge type network named apps-net

Port Mapping

Port Mapping allows containers to communicate with the outside world. The general syntax to map port is docker run -p <HOST-PORT>:<CONTAINER-PORT> <IMAGE-PORT>.

ex:

docker run -p 8080:80 webapp

Local Networks

Bridge Network

The bridge network is the default local network mode for Docker containers on a single host. The general syntax to create a container with a bridge network is docker run --network bridge -p <HOST-PORT>:<CONTAINER-PORT> <IMAGE-NAME>.

ex:

docker run --network bridge -p 8080:80 webapi

Host Network

The host network mode allows the container to share the host system's network stack directly. The container uses the same ip address and port as the host machine. The general syntax is docker run --network host <IMAGE-NAME>

ex:

docker run --network host webapp

none Network

The none network mode disables networking entirely for the container. Containers with this network type cannot communicate with other containers or with the outside world. They are completely isolated. The general syntax is docker run --network none <IMAGE-NAME>

ex:

docker run --network none webapi

Advanced and Overlay Networks

IPvlan

IPvlan network driver allows containers to receive IP addresses from the same subnet as the host, while still maintaining logical isolation between containers.

Macvlan

Macvlan network driver gives each container itw own MAC address and full presence on the physical network, making containers behave like independent network nodes.

Overlay

Overlay network driver is used to link containers across multiple Docker hosts, allowing them to communicate securely and seamlessly.