15 minfilesystemfhsdirectories

Filesystem Hierarchy Standard (FHS)

Learn filesystem hierarchy standard (fhs) concepts for DevOps interviews

01 — Filesystem & FHS

“Everything is a file” is the Linux mantra. This file makes you fluent in the directory layout (FHS), what an inode actually is, the real difference between hard and soft links, and how mounts stitch storage into one tree. Inodes and links are classic interview filters.

1. One unified tree (no drive letters)

Linux has a single rooted tree starting at /. There are no C:/D: drives — every storage device is mounted onto a directory in that one tree.

/                      root of everything
├── bin  → /usr/bin    essential user commands
├── sbin → /usr/sbin   system/admin commands
├── boot               kernel, initramfs, GRUB
├── dev                device files (sda, null, tty...) — "everything is a file"
├── etc                system-wide configuration (text files)
├── home               user home directories (/home/arpit)
├── lib  → /usr/lib    shared libraries
├── media / mnt        mount points for removable / temporary media
├── opt                optional third-party software
├── proc               VIRTUAL: kernel + process info as files
├── root               root user's home
├── run                VIRTUAL: runtime data (PIDs, sockets) since boot
├── srv                data served by services
├── sys                VIRTUAL: kernel/device tree (sysfs)
├── tmp                temporary files (often cleared on boot)
├── usr                userland programs, libraries, docs (read-mostly)
│   ├── bin  └ local   /usr/local for locally-installed software
└── var                VARIABLE data: logs, caches, spool, databases
    ├── log            system + app logs
    ├── lib            state (e.g., /var/lib/docker)
    └── spool          queues (cron, mail, print)

This is the Filesystem Hierarchy Standard (FHS).

🎯 Interview signal: know the purpose distinctions that get tested:

/etc = configuration, /var = variable data (logs/state), /usr = programs, /opt = third-party, /tmp = scratch.
/proc and /sys are virtual — not on disk; the kernel generates them live.
/bin, /sbin, /lib are now symlinks into /usr on modern distros (“usr merge”).

2. `/proc` and `/sys` — the kernel as files

These aren’t real files on disk; the kernel synthesizes them, which is how you read and tune the running system.

cat /proc/cpuinfo          # CPU details
cat /proc/meminfo          # memory stats
cat /proc/loadavg          # load average
cat /proc/<pid>/status     # a process's state, memory, uids
ls -l /proc/<pid>/fd       # a process's open file descriptors
cat /proc/mounts           # what's mounted right now
sysctl -a                  # kernel tunables exposed under /proc/sys
echo 1 > /proc/sys/net/ipv4/ip_forward   # enable IP forwarding (runtime)

🎯 Interview signal: “/proc/<pid>/fd lets me see exactly which files (and deleted files) a process holds open — key for the ‘disk full but du shows nothing’ incident.” (See §6.)

3. Inodes — what a file really is

A filename is just a label. The actual file — its metadata and pointers to data blocks — is the inode.

An inode stores: file type, permissions, owner UID/GID, size, timestamps (atime/mtime/ctime), link count, and pointers to data blocks. It does not store the filename — the name lives in the directory entry that maps a name → inode number.

ls -i file.txt        # show inode number
stat file.txt         # full inode metadata
df -i                 # INODE usage per filesystem (not blocks!)

⚠️ A filesystem has a finite number of inodes set at creation. You can run out of inodes (millions of tiny files) while df -h still shows free space — writes fail with “No space left on device.” Check with df -i. This is a top-tier interview gotcha.

4. Hard links vs soft (symbolic) links

Hard link:                          Soft (symbolic) link:
name_A ─┐                           symlink ──▶ "path/to/target"  (a tiny file
name_B ─┼─▶ [ inode 1234 ]                       holding a PATH string)
        │     (data blocks)         target  ───▶ [ inode 5678 ]
link count = 2                                    (data blocks)

	Hard link	Soft/symbolic link
Points to	The same inode (same data)	A path (a name)
Cross filesystems?	❌ No (inodes are per-fs)	✅ Yes
Link to a directory?	❌ No (normally)	✅ Yes
If original deleted?	Data survives (link count > 0)	Symlink breaks (dangling)
Shows as	Indistinguishable from the “original”	`l` type, `->` in `ls -l`

ln  target hardlink      # hard link (same inode; ls -i shows identical numbers)
ln -s target symlink     # symbolic link
ls -li                   # see inode numbers + link counts + -> targets

🎯 Interview signal: “A hard link is another directory entry pointing at the same inode; the data is deleted only when the link count hits zero and no process holds it open. A symlink is a separate file that just stores a path, so it can cross filesystems and dangle.”

⚠️ “How does rm work?” — rm unlinks (removes the directory entry and decrements the inode’s link count). The blocks are freed only when link count = 0 and no process has the file open (see §6).

5. Mounting — attaching storage to the tree

A device (partition/LVM volume/network share) is made usable by mounting it onto a directory (its mount point).

lsblk                        # tree of block devices + mount points
mount /dev/sdb1 /mnt/data    # mount a filesystem
umount /mnt/data             # unmount
findmnt                      # readable mount tree
cat /proc/mounts             # what the kernel has mounted

Persistent mounts live in /etc/fstab (covered in depth in file 07):

# device                     mountpoint  fstype  options        dump  pass
UUID=1234-abcd               /data       ext4    defaults        0     2

⚠️ A bad /etc/fstab entry can block boot (drops to emergency shell). Use UUID= not /dev/sdb1 (device names can reorder across reboots), and test with mount -a before rebooting.

6. The “deleted but open” file (must-know incident)

When you rm a file that a process still has open, the directory entry disappears but the inode and its blocks stay allocated until the process closes the file (or dies). So:

du / ls show the file is gone, but df still shows the space used → “disk full but I can’t find what’s using it.”

lsof +L1                       # list open files with link count 0 (deleted-but-open)
lsof /var/log                  # what has files open under a path
ls -l /proc/<pid>/fd | grep deleted

Fix: truncate via the fd, or restart the holding process (often a service still writing to a log you deleted). ⚠️ Deleting a huge log file doesn’t free space if the service still has it open — you must restart or signal the service to reopen its log.

🎯 Interview signal: naming this exact scenario and the lsof +L1 fix is a strong “I’ve done ops” signal.

7. File types (the first char of `ls -l`)

-  regular file      d  directory       l  symlink
c  char device       b  block device    s  socket        p  named pipe (FIFO)

ls -l /dev/sda   # b  → block device
ls -l /dev/null  # c  → char device
file /bin/ls     # identify a file's real type/content

Interview Questions

Q1. What is an inode and what does it store?

An inode is the on-disk structure representing a file’s metadata and data-block pointers: type, permissions, owner UID/GID, size, timestamps, link count, and pointers to the data. It does not store the filename — the name lives in the directory entry mapping name → inode number. stat shows inode contents; ls -i shows the inode number.

Q2. df -h shows free space but writes fail with “No space left on device.” Why?

Likely inode exhaustion. A filesystem has a fixed number of inodes set at creation; if you have millions of tiny files you can run out of inodes while data blocks remain free. df -i reveals it. Fix by deleting unneeded small files (often cache/session/mail spool) or recreating the filesystem with more inodes.

Q3. Difference between a hard link and a symbolic link?

A hard link is another directory entry pointing to the same inode, so both names are equal and the data lives until the link count hits zero. It can’t cross filesystems or link directories. A symbolic link is a separate file that stores a path to the target, so it can cross filesystems and link directories, but it breaks (dangles) if the target is removed.

Q4. What actually happens when you rm a file?

rm calls unlink: it removes the directory entry and decrements the inode’s link count. The data blocks are freed only when the link count reaches zero and no process still has the file open. That’s why a deleted file held open by a process keeps consuming disk space.

Q5. A service’s log was deleted to free space but df still shows it full. What’s going on and how do you fix it?

The service still has the log file open, so the inode/blocks stay allocated even though the name is gone (deleted-but-open). lsof +L1 or ls -l /proc/<pid>/fd | grep deleted shows it. Fix by making the service reopen its log — restart it or send the signal it uses to reopen logs (or truncate through the fd). Deleting the name alone doesn’t reclaim space.

Q6. Why are /proc and /sys special?

They’re virtual filesystems the kernel generates in memory, not stored on disk. /proc exposes process and kernel info (e.g., /proc/<pid>/status, /proc/meminfo, /proc/cmdline) and tunables under /proc/sys; /sys (sysfs) exposes the device/kernel object tree. They let you read and tune the running kernel through the file interface.

Q7. Explain the purpose of /etc, /var, /usr, /tmp, and /opt.

/etc holds system-wide configuration (text). /var holds variable data — logs, caches, spool, and service state like /var/lib/docker. /usr holds installed programs, libraries, and docs (read-mostly). /tmp is scratch space, often cleared on boot. /opt is for self-contained third-party software.

Q8. Why use UUID= instead of /dev/sdb1 in /etc/fstab?

Kernel device names (/dev/sdb) can change across reboots or when disks are added/removed, so a hardcoded name may point at the wrong disk. UUIDs (or labels) are stable identifiers tied to the filesystem, so the right filesystem always mounts at the right place. A wrong device in fstab can fail boot into an emergency shell.

Q9. (Senior) You mounted a new disk at /data but the old files that were in /data seem to have vanished. Explain.

They’re not gone — mounting a filesystem over a non-empty directory shadows the existing contents. The original files still occupy the underlying directory on the parent filesystem but are hidden while the mount is active. Unmount /data and they reappear. Best practice is to mount onto empty directories and migrate data intentionally.

Next: 02 — Users, Groups & Permissions — where most “permission denied” incidents actually live.