Linux performance tuning is essential for running efficient production workloads. Understanding how the kernel manages CPU, memory, disk, and network resources allows you to identify bottlenecks and optimize accordingly.

Linux Performance Tuning

The USE Method

Brendan Gregg's USE (Utilization, Saturation, Errors) method provides a systematic approach to performance analysis:

  • Utilization : What percentage of the resource is busy?

  • Saturation : How much extra work is queued?

  • Errors : How many error events are there?

Apply this to CPU, memory, storage, and network resources to quickly identify the bottleneck.

CPU Performance Tuning

Monitoring Tools

Real-time CPU monitoring

htop

Per-process CPU usage

top -o %CPU

CPU statistics and context switches

vmstat 1 5

Detailed per-CPU utilization

mpstat -P ALL 1

High context switch rates (above 50,000 per second per core) may indicate inefficient application architecture. Use pidstat -w to identify processes causing excessive context switches.

Kernel Parameters

cat /etc/sysctl.d/99-performance.conf

kernel.sched_min_granularity_ns = 3000000

kernel.sched_wakeup_granularity_ns = 4000000

kernel.sched_migration_cost_ns = 500000

kernel.sched_nr_migrate = 32

These scheduler parameters reduce latency for interactive workloads. Adjust carefully -- aggressive settings can hurt throughput.

Memory Tuning

Monitoring Memory

Memory usage overview

free -h

Detailed memory breakdown

cat /proc/meminfo

Page fault statistics

sar -B 1

Top memory consumers

ps aux --sort=-%mem | head

Check sar -B for page fault rates. High pgmajfault values indicate the system is swapping -- add more RAM or reduce memory pressure.

Swappiness

Reduce swapping tendency (default is 60)

vm.swappiness = 10

Set temporarily

sysctl vm.swappiness=10

For database servers, set swappiness to 1 to avoid swapping. For desktops and general-purpose servers, 10-20 balances responsiveness with memory efficiency.

Transparent Huge Pages

Disable THP for database workloads:

echo never > /sys/kernel/mm/transparent_hugepage/enabled

echo never > /sys/kernel/mm/transparent_hugepage/defrag

THP can cause latency spikes in database systems due to memory defragmentation pauses.

Disk I/O Tuning

I/O Scheduler

Choose the right I/O scheduler for your workload:

Check current scheduler

cat /sys/block/nvme0n1/queue/scheduler

Set to none for NVMe, mq-deadline for spinning disks

echo none > /sys/block/nvme0n1/queue/scheduler

Modern NVMe drives perform best with the none (or nvme) scheduler. Spinning disks benefit from mq-deadline which minimizes seek times.

Monitoring Disk Performance

I/O statistics per device

iostat -x 1

Process-level I/O

iotop

File system latency

bcc-tools/biolatency

High await times (above 20ms) indicate disk saturation. Check iowait in top and svctm in iostat for confirmation.

Network Tuning

Kernel Network Parameters

/etc/sysctl.d/99-network.conf

net.core.somaxconn = 65535

net.core.netdev_max_backlog = 50000

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_fin_timeout = 15

net.ipv4.tcp_keepalive_time = 300

net.ipv4.tcp_keepalive_intvl = 60

net.ipv4.tcp_keepalive_probes = 5

net.core.rmem_max = 134217728

net.core.wmem_max = 134217728

net.ipv4.tcp_rmem = 4096 87380 134217728

net.ipv4.tcp_wmem = 4096 65536 134217728

Increase socket buffer sizes for high-throughput applications. tcp_tw_reuse allows reuse of connections in TIME_WAIT state, important for high-connection-rate servers.

Network Monitoring

Per-interface statistics

sar -n DEV 1

Socket statistics

ss -s

Connection tracking

netstat -s | grep -i "connections established"

Monitor for dropped packets in /proc/net/softnet_stat and TCP retransmits with netstat -s | grep retransmit.

File System Tuning

Mount Options

Optimize filesystem mount options for performance:

/etc/fstab

/dev/sda1 / ext4 noatime,nodiratime,data=ordered 0 0

The noatime option eliminates update timestamps on every file read, significantly reducing disk writes. Use nodiratime for directories.

I/O Limits with cgroups

Control per-process I/O with cgroups v2:

Limit read/write to 50 MB/s

echo "8:0 rbps=52428800 wbps=52428800" > /sys/fs/cgroup//io.max

Application-Level Tuning

Profile before tuning. Use perf for CPU profiling, flamegraphs for visualization, and strace for system call analysis:

Sample CPU stacks at 99Hz

perf record -F 99 -a -g -- sleep 30

perf script | ./stackcollapse-perf.pl | ./flamegraph.pl > profile.svg

Always measure before and after changes. A single benchmark run is unreliable -- run multiple iterations and report the median.

Summary

Linux performance tuning is a systematic process of identifying bottlenecks, applying targeted optimizations, and measuring the impact. Start with the USE method to find the bottleneck, use the right monitoring tools, and tune one parameter at a time. Common wins include setting the correct I/O scheduler, reducing swappiness for databases, disabling THP, and increasing network buffer sizes. Profile first, tune second, and always validate with benchmarks.