Security Hardening

0%
25 minsecurityhardeningselinuxapparmor

Security Hardening

Learn security hardening concepts for DevOps interviews

12 — Security & Hardening

“How do you harden a Linux server?” is a near-guaranteed DevOps interview question. This file gives you a layered answer: SSH, firewall, least privilege, SELinux/AppArmor, fail2ban, patching, and auditing — with the reasoning behind each control.

1. The hardening mindset

Security is layers (defense in depth) + least privilege + reduce attack surface. For a server: expose less, patch fast, authenticate strongly, limit blast radius, and watch.

🎯 Interview signal: structure your answer as access → network → identity → OS/MAC → monitoring, not a random list of chmods.

2. SSH hardening (the #1 answer)

SSH is usually the only inbound door — lock it down. Edit /etc/ssh/sshd_config:

PermitRootLogin no            # never log in directly as root; use a user + sudo
PasswordAuthentication no     # KEY-ONLY auth — kills brute-force
PubkeyAuthentication yes
Port 22                       # (changing it reduces noise, not real security)
AllowUsers deploy admin       # allow-list who can SSH
MaxAuthTries 3
ClientAliveInterval 300       # drop idle sessions
Protocol 2
sudo systemctl reload sshd    # apply (reload, keeps your current session)

⚠️ Test in a second session before closing your current one — a bad sshd_config + reload can lock you out. Set up your key first:

ssh-copy-id deploy@server     # install your public key

🎯 Interview signal: “Key-only auth with PasswordAuthentication no and PermitRootLogin no eliminates password brute-force entirely — the single biggest SSH win.” Keys > passwords because a strong keypair isn’t guessable and isn’t reused.

3. Firewall — expose only what’s needed

Default-deny inbound, allow only required ports (file 06). ⚠️ Allow SSH first.

# Ubuntu (ufw)
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 443/tcp
sudo ufw enable
# RHEL (firewalld)
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --permanent --add-service=https
sudo firewall-cmd --reload

On cloud, the security group / NACL is a second, separate firewall layer — harden both.

4. Least privilege & account hygiene

  • No shared root: individuals use their own accounts + sudo (audited). Lock the root password (sudo passwd -l root) and rely on sudo.
  • Scoped sudo (file 02): grant specific commands per group, use visudo.
  • Service accounts run daemons as unprivileged users with nologin shells (file 04).
  • Audit setuid binaries (privilege-escalation surface): 
    find / -perm -4000 -type f 2>/dev/null   # list setuid; remove/limit unexpected ones
  • Password/key policies: expiry (chage), remove unused accounts, disable inactive users.

5. SELinux / AppArmor — Mandatory Access Control (MAC)

Standard permissions are discretionary (owners decide). MAC adds a kernel-enforced policy layer that confines even root/processes to what policy explicitly allows.

SELinux (RHEL family) AppArmor (Ubuntu family)
Model Labels on everything (type enforcement) Per-program path-based profiles
Modes enforcing, permissive, disabled enforce, complain
Tools getenforce, setenforce, semanage, ausearch aa-status, aa-complain, aa-enforce 
getenforce                    # SELinux mode
sudo setenforce 0             # temporarily permissive (troubleshooting only)
sudo ausearch -m avc -ts recent   # what SELinux denied (AVC denials)
aa-status                     # AppArmor profiles loaded

⚠️ Never permanently disable SELinux to “fix” a problem — that’s a common bad habit and a red flag in interviews. Instead, read the AVC denial, then adjust the policy (semanage for ports/file contexts, restorecon to relabel). “My app can’t bind port 8443” under SELinux is often a missing port label, not a reason to disable it.

🎯 Interview signal: “SELinux/AppArmor is mandatory access control — a kernel policy that confines processes beyond user/group permissions, so a compromised service can’t step outside its profile. I troubleshoot denials with ausearch/aa-status and fix the policy rather than disabling it.”

6. fail2ban — block brute-force

Watches logs (e.g., auth.log) and temporarily bans IPs that fail auth repeatedly by adding firewall rules.

sudo apt install fail2ban
sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd     # jailed IPs

Pairs with key-only SSH to cut log noise and stop credential-stuffing.

7. Patching & attack-surface reduction

  • Patch fast, especially security updates (file 05: unattended-upgrades/dnf-automatic scoped to security, version-lock the risky packages).
  • Remove/disable unused services (systemctl disable --now) and packages — less surface.
  • Mount options as hardening: noexec,nosuid,nodev on /tmp, /var, removable media.
  • Kernel tunables (sysctl): disable unused protocols, enable SYN cookies, restrict dmesg/ptrace, etc.

8. Auditing & monitoring

  • auditd: kernel-level audit trail (file access, syscalls, logins) — for compliance/forensics. 
    sudo ausearch -f /etc/passwd        # who touched a sensitive file
  • auth logs: /var/log/auth.log / secure for logins, sudo, SSH (file 11).
  • File integrity: AIDE/Tripwire detect unexpected changes to system files.
  • CIS Benchmarks: the industry checklist for OS hardening; tools like OpenSCAP/Lynis scan against them. 🎯 Naming CIS Benchmarks and Lynis/OpenSCAP signals you know the standards, not just ad-hoc tweaks.

9. The “harden a server” checklist (say this in an interview)

Access:   key-only SSH, no root login, allow-list users, fail2ban
Network:  default-deny firewall (host + cloud SG), expose only needed ports
Identity: no shared root, scoped sudo via visudo, service accounts w/ nologin, audit setuid
OS/MAC:   SELinux/AppArmor enforcing, hardened mount options, sysctl tuning
Patch:    automatic security updates, version-lock critical pkgs, remove unused services
Monitor:  auditd, centralized auth logs, file integrity (AIDE), CIS scan (Lynis/OpenSCAP)
Data:     encryption at rest (LUKS), TLS in transit, secrets in a vault not on disk

Interview Questions

Q1. How do you harden SSH?

Disable password auth (PasswordAuthentication no) in favor of SSH keys, disable direct root login (PermitRootLogin no), allow-list who can connect (AllowUsers), limit MaxAuthTries, drop idle sessions, and add fail2ban to ban brute-forcers. Key-only auth is the biggest win because it eliminates password guessing. I always test a new session before closing the current one and reload (not blindly edit) sshd to avoid locking myself out.

Q2. Walk me through hardening a fresh Linux server.

Layered: lock down access (key-only SSH, no root login, fail2ban), default-deny firewall on both host and cloud security group exposing only needed ports, least-privilege identity (no shared root, scoped sudo, service accounts with nologin, audit setuid binaries), enable SELinux/AppArmor enforcing plus hardened mount options and sysctls, patch automatically for security and version-lock critical packages, remove unused services, and add monitoring — auditd, centralized auth logs, file-integrity checks, and a CIS benchmark scan.

Q3. What is SELinux/AppArmor and how is it different from normal permissions?

They’re Mandatory Access Control — a kernel-enforced policy that confines processes to exactly what’s allowed, on top of the discretionary user/group/other permissions that owners control. Even root or a compromised service can’t act outside its policy/profile. SELinux uses labels/type enforcement (RHEL); AppArmor uses per-program path profiles (Ubuntu).

Q4. An app fails under SELinux. What do you do?

Not disable SELinux. I’d check the AVC denials (ausearch -m avc -ts recent or sealert) to see exactly what was blocked, then fix the policy — e.g., add a port label with semanage port, correct file contexts with semanage fcontext + restorecon, or set the right boolean. Temporarily setenforce 0 (permissive) can confirm SELinux is the cause, but the fix is a correct policy, not disabling enforcement.

Q5. Why prefer SSH keys over passwords?

A keypair uses strong asymmetric crypto — the private key never leaves the client and isn’t guessable, so password brute-force and credential stuffing become ineffective. Passwords are reused, phishable, and guessable. With PasswordAuthentication no, the entire class of password attacks against SSH disappears.

Q6. How do you find and reason about setuid binaries?

find / -perm -4000 -type f 2>/dev/null lists them. Each setuid-root binary runs with root privileges, so a bug in one is a privilege-escalation path. I review the list, remove the setuid bit from anything that doesn’t need it, and keep only trusted, patched binaries — it’s a standard attack-surface-reduction step.

Q7. What does fail2ban do?

It monitors logs (like auth.log) for repeated auth failures and temporarily bans the offending IPs by inserting firewall rules, then removes them after a timeout. It cuts down brute-force/credential-stuffing noise and complements key-only SSH.

Q8. (Senior) How do you approach compliance-grade hardening across a fleet?

Standardize on a benchmark — CIS — and scan with Lynis/OpenSCAP to measure against it. Bake the hardened baseline into a golden image or config management (Ansible) so every host is consistent and auditable, with drift detection. Enforce MAC (SELinux/AppArmor), auditd for a tamper-evident trail, centralized log shipping, automated security patching with canary rollout, secrets in a vault, and encryption at rest/in transit. Then continuously scan and report on compliance rather than hardening hosts by hand.

Next: 13 — Scheduling & Automation.