Linux deliberately fills almost all your RAM with cache. The "139 MB free" you're panicking about is the kernel doing exactly what it should. Swap is more nuanced in 2026 than "always off" — disk swap remains dangerous, but zram and AI training frameworks have changed the conversation. Here's the honest, updated picture.

The Alarm That Isn't One

You SSH into a production server. You run free -m. You see this:

               total        used        free      shared  buff/cache   available
Mem:           15727       15588         139           0       12261       12187
Swap:           3071         302        2769

Your heart skips. 15.5 GB used. Only 139 MB free.

You open a ticket. You ping the team. You consider rebooting.

Stop. The server is fine.

That output is one of the most consistently misread things in Linux operations. And in 2026 — with AI inference servers, vector databases, and GPU-backed workloads now running on Linux boxes that were never part of the traditional sysadmin curriculum — getting this wrong is more expensive than ever.

Let's break it apart, update the assumptions that have aged out, and build an accurate picture of Linux memory in the world we actually operate in today.

How Linux Actually Uses RAM

Linux never lets RAM sit idle.

Whenever free memory exists, the kernel uses it to speed up the system by filling it with two things:

• Buffers — filesystem metadata: directory entries, inode tables, permission bits. What the kernel needs to navigate the filesystem without touching the disk.
• Page Cache — the actual contents of files and block devices read from or written to disk recently. Every file opened, every binary loaded, every config file parsed at startup — kept in cache as long as RAM permits.

The kernel's reasoning: RAM is roughly 1,000 times faster than even the best NVMe SSD for the kind of small, random reads that swap and file access involve. If the kernel can serve a read from RAM instead of storage, it will.

This is not a bug. It is a fundamental design principle of the Linux VM subsystem, and it is why Linux outperforms other operating systems on I/O-heavy workloads.

Reading free -m Correctly in 2026

Modern Linux systems running procps-ng (current standard across Ubuntu, RHEL, Debian, Arch) show:

               total        used        free      shared  buff/cache   available
Mem:           15727        3291         121           0       12315       12187
Swap:           3071         302        2769

The Only Column That Matters: available

available is the only number worth reading. It accounts for reclaimable cache. 12.1 GB is available. The server is healthy.

Rule: Ignore free. Read available. Alert on available, not free.

The Old Format (Still Alive in Older Systems)

Older free versions showed this layout — and many tutorials still reference it:

              total   used   free  shared  buffers   cached
Mem:          15727  15588    139       0       33    12228
-/+ buffers/cache:   3326  12401
Swap:          3071    302   2769

In this format, used on Line 1 includes buffers and cache, making the server look critically full. The real number was always on the -/+ buffers/cache: line — 12401 MB free. Modern free collapsed this into the available column. If you're on an old system: ignore Line 1's free, read Line 2's free.

Why a Healthy Server Always Looks "Full"

Linux follows one principle: unused RAM is wasted performance.

The page cache grows aggressively over time. On a server running for weeks:

• Shared libraries loaded by every process are mapped once and shared across all of them
• Nginx static files serve from cache, bypassing disk entirely
• MySQL's hot table pages live in cache alongside InnoDB buffer pool
• PostgreSQL reads benefit from a second caching layer at the kernel level

When a new process needs memory, the kernel evicts least-recently-used cache pages and hands that RAM over immediately — transparently, without the application noticing.

A server with large genuinely free RAM is a server whose kernel is doing less than it should. High buff/cache is a sign of a well-utilized server, not a stressed one.

RAM in 2026: The Hardware Reality Has Shifted

This context matters for everything that follows about swap.

When swap was designed in the early Unix era, servers had 32–256 MB of RAM. RAM cost thousands of dollars per megabyte. Swap was a genuine engineering solution to a real hardware constraint — a way to run programs larger than physical memory allowed.

That world does not exist anymore.

RAM pricing today:

Cloud follows the same trend. AWS, GCP, and Azure offer memory-optimized instances at 256 GB, 512 GB, and 1.5 TB as standard catalogue options today.

The cost of adding 16 GB of RAM to a cloud instance is now less than one hour of an on-call engineer's time spent debugging a swap-induced incident.

Storage is fast — but the latency gap with RAM has not closed:

NVMe Gen5 can push 14 GB/s of sequential throughput. That is impressive. But swap does not use sequential throughput — it uses random small-block reads of cold pages scattered across the device. At that access pattern, even Gen5 NVMe is 1,000 times slower than DDR5 RAM. Cloud-attached block storage is 5,000–10,000 times slower.

That gap has not meaningfully closed in 30 years, and it will not close because it is a physical property of the two technologies, not an engineering limitation.

The Number That Should Worry You: Disk Swap in Use

Swap:   3071    302   2769

302 MB of disk swap is active. On a traditional web or database server, this is a warning sign.

Why Disk Swap Degrades Servers

When the kernel starts swapping to disk, it triggers a cascade:

1. Cold pages are written to the swap partition — disk I/O starts competing with application I/O.
2. When those pages are needed again, they must be read back — more disk I/O.
3. While processes wait for disk, they hold locks, connections, and queued requests.
4. Load average climbs. More processes queue. More swap I/O is triggered.
5. The disk I/O queue backs up. Everything slows — not just the swapping process.
6. MySQL queries start timing out. Nginx workers exhaust. API latency jumps from milliseconds to seconds.

The server does not crash cleanly. It degrades while appearing operational — which makes diagnosis slower and the incident longer.

Disk swap does not prevent a crash. It often converts a fast, clean OOM kill into a prolonged, ambiguous system-wide degradation.

Where the "Always Disable Swap" Assumption Needs Updating

Here is where the 2026 picture gets more nuanced than older advice.

zram: Swap That Lives in RAM

zram is a compressed RAM-based block device used as swap. Instead of writing cold pages to disk, zram compresses them and stores them in RAM. When those pages are needed, they are decompressed — still entirely in RAM, no disk I/O involved.

This changes the tradeoff significantly:

zram is now enabled by default on Ubuntu 22.04+, Fedora 33+, and Android (which has used it since Android 4.4). It is a legitimate solution for handling occasional memory pressure without the catastrophic disk I/O cascade.

For production servers, zram does not replace proper RAM sizing — but if you are constrained and cannot disable all swap, replacing disk swap with zram is a meaningful upgrade:

# Install and configure zram (Ubuntu/Debian)
apt install zram-tools

# Or manual setup
modprobe zram
echo lz4 > /sys/block/zram0/comp_algorithm   # lz4 = fast; zstd = better compression
echo 8G > /sys/block/zram0/disksize
mkswap /dev/zram0
swapon -p 100 /dev/zram0                      # Higher priority than disk swap

Bottom line on zram: It is not a substitute for RAM, but it is a far safer fallback than disk swap. If you cannot disable swap entirely, replace disk swap with zram.

AI Training: The Exception Where Offloading Makes Sense

For AI training workloads specifically, the conversation around swap has taken a different turn — but not because disk swap is good. Because purpose-built NVMe offloading frameworks have emerged.

DeepSpeed's ZeRO-Infinity and ZeRO-3 with CPU offload deliberately move optimizer states and parameters to CPU RAM and NVMe storage during training. PyTorch's OffloadOptimizer does the same. These frameworks:

• Know exactly when a tensor is needed and prefetch it with sufficient lead time
• Use high-bandwidth sequential I/O patterns, not random-access swap
• Are aware of the latency tradeoff and schedule around it
• Were designed specifically for the memory requirements of multi-hundred-billion parameter models

This is fundamentally different from OS-level swap. The OS swapper has no idea what data it is evicting or when it will be needed. ZeRO-Infinity knows. The I/O is predictable and pipelined.

If you are running AI training with DeepSpeed or FSDP with CPU/NVMe offload, that is a deliberate architectural choice — not swap as a safety net. The OS-level swap partition should still be off. These frameworks manage their own offloading independently.

Swap and AI Inference: Still Catastrophic

For AI inference — serving models to users — the original advice holds without qualification.

Running a large language model requires keeping model weights resident in memory. The numbers are not small:

When a request arrives, the inference engine reads through model weights layer by layer — in RAM, at RAM speed, sequentially. If any portion of those weights has been paged to disk:

• Each inference call triggers disk reads across potentially gigabytes of model data
• A request that normally takes 50–300 ms now takes 30–120 seconds
• Tokens stream in with multi-second gaps or time out entirely
• The GPU sits idle waiting for weights that are stuck behind disk I/O

vLLM, Ollama, llama.cpp, and TGI all load models at startup and expect those pages to stay resident indefinitely. None of them were designed to work around OS-level swap.

The KV cache parallel: The attention KV cache that LLM inference engines maintain per-request — storing previously computed key-value pairs so the model does not re-process the entire context on every new token — is conceptually identical to the Linux page cache. Both exist to keep recently computed or accessed data in fast memory rather than recomputing or re-reading it. If you understand why page cache eviction is bad, you understand why KV cache eviction to disk destroys inference performance.

Vector Databases

Qdrant, Weaviate, Milvus, and Chroma load their HNSW or IVF indexes into RAM at startup. These indexes scale with corpus size — hundreds of gigabytes for production AI systems. Any swap activity touching index data sends similarity search latency from single-digit milliseconds to multi-second responses, eliminating the performance advantage of vector search entirely.

On any server running AI inference or vector search: disk swap off, no exceptions.

Containers and Kubernetes: The Right Defaults

Kubernetes disables swap on nodes by default. This is intentional.

When a container exceeds its memory limit, the OOM killer terminates it — fast, clean, and immediately restartable by Kubernetes. This is predictable. Resource guarantees work as specified.

Swap breaks this model: a container that is swap-thrashing does not OOM-kill. It degrades — and it degrades the entire node, affecting all pods across all workloads, in a way that violates the isolation guarantees resource limits are supposed to provide.

Kubernetes 1.28 introduced alpha NodeSwap support with LimitedSwap mode — allowing swap only for memory exceeding a container's request (not its limit). Even this carefully constrained form is still alpha and not recommended for production nodes running latency-sensitive workloads.

The container ecosystem's default position: swap off on production nodes. Not because swap is universally evil, but because its failure mode is incompatible with the predictability that container orchestration depends on.

What To Do

1. Understand Your Actual Memory Footprint First

# The authoritative number: available memory for new processes
free -m

# Per-process memory sorted by RSS
ps aux --sort=-%mem | head -20

# Shared vs private memory per process (more accurate than ps)
smem -s rss -r | head -20

# Real-time view
htop    # Press M to sort by memory
btop    # Modern alternative, better visualization

If available is above 10–15% of total, your server is healthy regardless of what used shows.

2. Disable Disk Swap on Production Servers

# Check what swap is configured
swapon --show

# Disable immediately
swapoff -a

# Verify
free -m   # Swap line should show: 0  0  0

# Make permanent: edit /etc/fstab and comment out the swap line
# UUID=xxxx-xxxx   none   swap   sw   0   0
# becomes:
# #UUID=xxxx-xxxx  none   swap   sw   0   0

Do this only after confirming available memory is sufficient. If applications genuinely consume close to all physical RAM, disabling swap causes OOM kills — which is still a cleaner failure mode than swap thrashing, but requires either reducing application memory or adding RAM.

3. Replace Disk Swap with zram If You Need a Fallback

# Ubuntu 22.04+ (zram-tools package)
apt install zram-tools

# Check it is configured
swapon --show
# Should show /dev/zram0 with partition type zram

# Manual configuration for other distros
modprobe zram
echo lz4 > /sys/block/zram0/comp_algorithm
echo 8G > /sys/block/zram0/disksize
mkswap /dev/zram0
swapon -p 100 /dev/zram0

4. Tune vm.swappiness

# Minimize kernel's eagerness to swap (0 = last resort before OOM kill)
sysctl -w vm.swappiness=0

# Persist across reboots
echo "vm.swappiness=0" | tee /etc/sysctl.d/99-production.conf
sysctl -p /etc/sysctl.d/99-production.conf

For zram-only setups where you still want some swap pressure relief, vm.swappiness=10 or vm.swappiness=20 is a reasonable compromise — enough to let zram handle cold pages, not enough to start thrashing.

5. Size RAM Correctly for Your Workload

6. Monitor the Metrics That Matter

# Real-time swap activity
vmstat 2
# si = pages swapped IN/s (reading from swap)
# so = pages swapped OUT/s (writing to swap)
# Any sustained non-zero = active problem

# Memory pressure events
cat /proc/meminfo | grep -E "MemAvailable|SwapFree|SwapCached"

Monitoring alert thresholds (Prometheus node_exporter metrics):

Quick Reference

               total        used        free      shared  buff/cache   available
Mem:           15727        3291         121           0       12315       12187
Swap:           3071           0        3071

Updated Myth vs Reality (2026 Edition)

The Closing Point

Linux memory management is not broken. The kernel's aggressive use of every available byte for caching is one of the reasons Linux dominates production infrastructure globally — including the servers running the AI models you interact with daily.

The page cache and the KV cache in LLM inference follow the same principle: keep hot, recently accessed data in fast memory so you never pay the cost of recomputing or re-reading it. When you understand one, you understand the other.

For disk swap: the economics that justified it in the 1990s — RAM was scarce and expensive — are gone. RAM is cheap, abundant, and on the servers running modern AI workloads, absolutely non-negotiable. Disk swap on an LLM inference server is not a safety net. It is a guarantee that your server will appear operational while being completely unusable.

For zram: it is a real improvement over disk swap and a valid choice for development machines, constrained VMs, and container nodes that occasionally see memory pressure. It is not a substitute for provisioning enough RAM for your workload.

The available column in free -m tells you what is actually happening. Everything else is the kernel working as designed.

This article draws from real production experience managing Linux infrastructure for high-traffic applications, updated with current AI infrastructure operational patterns.

Tags: linux sysadmin devops performance memory-management ai llm infrastructure kubernetes backend mlops

Cover image suggestion: A terminal split-view — left showing free -m with the available column highlighted, right showing ollama or vllm loading a model — dark background, clean monospace font.

A network interface can be up. Servers can respond to ping. Storage drives can have plenty of available bandwidth.

Yet an AI training workload can come to a complete standstill.

We experienced this firsthand while deploying and operating Lustre, the open-source parallel file system widely used in HPC and AI infrastructure.

What began as a high-performance storage deployment eventually became an extended exercise in troubleshooting network anomalies, kernel-driver compatibility, metadata recovery, and filesystem availability.

The most difficult part wasn't just identifying the technical issues. It was dealing with repeated interruptions to customer workloads and the growing loss of confidence in the storage platform.

Eventually, we had to reconsider our approach and move toward an alternative storage solution.

Here are some of the lessons that experience taught me.

1. When Everything Looked Healthy, but Training Kept Failing

Our objective was to deliver high-performance shared storage for GPU-intensive AI training workloads.

The environment had enterprise NVMe storage, high-speed networking, dedicated object storage servers, and three Lustre metadata server nodes.

On paper, the infrastructure had significant performance headroom.

But once production workloads started running, we began experiencing serious instability:

• Compute nodes with Lustre mounts would stop responding to filesystem operations.
• AI training jobs would hang or fail.
• Metadata requests would time out, and clients would lose connections to metadata targets.
• Our engineering team repeatedly had to intervene on metadata services to restore normal operations.

What made this particularly frustrating was that storage drives were utilizing less than 10% of their available bandwidth during some of the observed failures.

The bottleneck wasn't an obvious shortage of raw disk throughput.

Something deeper in the infrastructure stack was preventing the distributed filesystem from operating reliably.

2. The Small Network Errors That Raised Big Questions

During troubleshooting, we began examining the network interfaces more closely.

One observation revealed approximately 2,500 Forward Error Correction (FEC) symbol errors per second.

Across different storage nodes, some interfaces had accumulated hundreds of millions or billions of symbol errors, along with smaller numbers of uncorrectable FEC blocks.

At the same time, Lustre reported RPC timeouts and lost connections to metadata targets.

One recurring message looked like this:

Lustre: Request sent has timed out for slow reply

Lustre: Connection to MDT0001 was lost;
in-progress operations using this service
will wait for recovery to complete.

These messages indicated that Lustre clients were experiencing metadata request timeouts and losing connectivity to a metadata target. Affected filesystem operations could remain blocked while the target recovered.

Our investigation also found that Lustre's own networking utility, lctl ping, failed from multiple clients with input/output errors.

lctl ping <MDS_NID> failed to ping <MDS_NID>: Input/output error

These observations made the network stack a major area of concern.

However, an important distinction is necessary.

FEC is designed to correct transmission errors. Large corrected-error counters alone don't prove packet loss or application failure. Uncorrectable errors, error-rate trends, and correlation with transport failures are more meaningful indicators.

We had evidence of network anomalies and Lustre communication failures, but establishing a single definitive cause for every incident would have required further investigation.

The lesson: An error that appears insignificant when measured at one infrastructure layer can have a much larger operational impact when it affects a distributed application.

Recovery Was Successful. The Customer Experience Wasn't.

In one incident, Lustre reported the following metadata target recovery statistics:

The recovery completed successfully.

But nearly six minutes of metadata recovery is significant when applications are waiting on filesystem operations.

With AI training workloads running across GPU nodes, intermittent storage stalls and repeated recovery events can translate into lost execution time, failed jobs, and operational disruption.

This highlighted a distinction that became central to our experience:

A storage system can recover successfully at the infrastructure level and still fail its customers at the application level.

Having multiple metadata servers also doesn't automatically guarantee uninterrupted operations. Lustre's recovery mechanisms are designed to preserve consistency and restore service, but clients may still experience delays while affected metadata targets recover.

3. The Kernel Compatibility Challenge We Encountered Earlier

Interestingly, the difficulties began before production.

During the initial deployment, we encountered compatibility issues between our network interface drivers and the Linux kernel environment required by the Lustre servers.

Unlike applications implemented entirely in user space, Lustre relies heavily on Linux kernel modules.

Traditional Lustre server deployments, particularly those using the ldiskfs backend, may use patched or specially qualified Linux kernel versions. ZFS-based Lustre server configurations and Lustre clients have different kernel requirements.

Our storage environment required a kernel and driver combination that wasn't directly compatible with the NIC drivers available to us.

Under tight delivery timelines, we compiled and adapted the drivers to proceed.

This allowed the deployment to move forward, but it exposed a broader architectural concern.

Compatibility Is a Property of the Complete Stack

Consider the dependencies involved:

Each layer has its own versions, compatibility requirements, and failure characteristics.

A driver upgrade that works on a standard Linux server may not be straightforward on a storage server running a specific kernel configuration.

Similarly, a network interface that works reliably for ordinary virtualization traffic hasn't necessarily been validated for sustained, latency-sensitive distributed storage operations.

In our case, the same class of hardware had passed preliminary tests in a virtualization environment.

But those tests didn't reproduce the actual Lustre workload.

We also observed RPC-level errors involving MDS_BATCH operations, illustrating another possible compatibility concern between Lustre client and server implementations.

Similar messages have been documented in upstream Lustre compatibility issues. That doesn't establish that we encountered the same software defect, but it demonstrates why version qualification matters.

The lesson was clear: hardware compatibility isn't just about whether a component works. It's about whether the entire stack works together under the intended production workload.

4. The Hidden Cost of Open-Source Infrastructure

I've spent much of my career working with open-source infrastructure, and I remain a strong advocate for it.

Linux, Kubernetes, Ceph, Lustre, and many other open-source technologies have demonstrated remarkable capabilities at scale.

But there's an important difference between having access to powerful technology and operating it as a reliable customer-facing service.

Open source provides flexibility, transparency, and control.

What it doesn't automatically provide is a fully validated combination of hardware, firmware, drivers, kernel versions, networking components, and production workloads.

Documentation and community knowledge exist, but troubleshooting a particular combination of modern hardware and software can still require significant engineering effort.

When a failure crosses multiple layers, the investigation becomes particularly challenging.

The network may pass basic connectivity tests. Hardware diagnostics may appear normal. Individual storage services may eventually recover.

Yet the overall system can remain unreliable.

Someone still has to own the complete integration problem.

The hidden cost of open-source infrastructure is not necessarily licensing. It's the engineering effort required to validate, integrate, troubleshoot, upgrade, and maintain the entire solution.

For organizations with strong HPC expertise and established hardware qualification processes, managing this complexity may be perfectly reasonable.

For customer-facing cloud infrastructure, however, that complexity must be considered alongside reliability expectations and operating costs.

5. Are User-Space Storage Architectures Changing the Equation?

This experience also influenced how I evaluate newer enterprise high-performance storage platforms.

Lustre is a kernel-integrated distributed filesystem. Several newer software-defined storage systems take a different approach by moving substantial portions of their performance-critical data planes into user space.

Technologies such as DPDK and SPDK allow storage software to optimize packet processing and device access through techniques such as polling, batching, and reduced context switching.

These approaches can provide greater control over I/O processing and reduce certain kernel dependencies.

However, user-space execution doesn't eliminate hardware compatibility challenges.

Network interfaces, firmware, drivers, operating systems, and storage clients still need appropriate qualification.

Some enterprise storage platforms also retain kernel-based client components even when their server-side data plane primarily operates in user space.

The more meaningful distinction is often the maturity of the overall product and its integration ecosystem.

Enterprise support alone is not a guarantee of reliability, and Lustre itself is available through commercially supported implementations.

But validated configurations, integrated tooling, and a clear support model can materially reduce the burden on infrastructure teams.

For me, this changed the way I evaluate storage platforms.

Peak throughput is only one part of the equation. Operational reliability and integration complexity deserve equal attention.

6. The Customer Experience Is the Ultimate Benchmark

Perhaps the hardest lesson from this experience wasn't technical.

As infrastructure engineers, we naturally want to isolate the failing component. Was the issue in the hardware, driver, kernel, network transport, metadata service, or an unexpected interaction between them?

These distinctions are essential for troubleshooting. But customers experience the infrastructure as a single service.

They don't care which component failed. They care that their AI training workloads stopped working.

Every interruption meant stalled training jobs, underutilized GPU resources, delayed work, and growing frustration.

Despite significant engineering effort, repeated filesystem disruptions gradually affected customer confidence. Eventually, we made the difficult decision to move toward an alternative storage solution.

That doesn't mean Lustre is fundamentally unsuitable for high-performance computing. It remains a proven technology used in some of the world's most demanding HPC environments.

But it reinforced an important lesson:

A technically capable solution isn't necessarily the right operational fit for every production environment.

Sometimes, maintaining customer trust requires reconsidering the architecture rather than continuing to troubleshoot an increasingly complex integration.

What I Would Do Differently

Looking back, five lessons stand out.

1. Validate hardware using real workloads.

Successful ping tests, basic throughput benchmarks, and generic virtualization testing are not sufficient qualification for distributed storage.

2. Treat kernel, drivers, firmware, and storage software as one compatibility unit.

The complete stack must be tested and maintained as a validated combination.

3. Monitor reliability at the application level.

A healthy interface or a successfully recovered metadata target doesn't necessarily mean applications were unaffected.

4. Evaluate operational maturity alongside performance.

Observability, recovery behavior, upgrade complexity, documentation, and support ownership matter as much as benchmark numbers.

5. Know when to reconsider the architecture.

Engineering persistence is important, but so are customer confidence, service reliability, and long-term operational sustainability.

Final Thoughts

Our experience reinforced a principle that applies to nearly every distributed infrastructure platform:

Reliability is not a property of an individual component. It's a property of the entire system.

A healthy SSD doesn't guarantee a healthy filesystem.

A reachable server doesn't guarantee reliable distributed communication.

Multiple metadata servers don't guarantee uninterrupted application access.

And a powerful open-source technology doesn't automatically guarantee seamless integration with every hardware and software combination.

As AI infrastructure evolves toward faster networks, denser NVMe storage, and more powerful GPU clusters, performance expectations will continue to increase.

But the most important question isn't simply how much throughput a system can deliver.

It's how consistently that system can support real customer workloads, under failures, upgrades, and everyday operational complexity.

Sometimes, the biggest improvement isn't another round of tuning. It's recognizing when the architecture needs to change.

Written from personal experience designing, deploying, and operating high-performance infrastructure for AI workloads. Technical observations have been generalized and identifying information omitted.

References and Further Reading

• Lustre: Getting Started
• Lustre Wiki: Compiling Lustre
• Lustre Recovery Overview
• DPDK Project
• Storage Performance Development Kit (SPDK)