SMR drives, ZFS and 92% full, and why compression won't save you

If a ZFS pool on spinning disks goes unresponsive under write load and iostat shows single-digit MB/s, stop looking at throughput. You aren't bandwidth-bound and nothing that reduces bytes written will help. That includes compression, which is the first thing everyone reaches for.

Ours took an evening to work out, and most of that evening was spent testing theories that couldn't have been true.

The machine is heavy, a workstation that also runs the CI agent that builds container images for a dozen or so applications. That combination is the whole story: a bulk-storage pool built for media and backups, quietly also serving as the working surface for a container build workload. Those two jobs want opposite things from a disk, and only one of them was considered when the pool was built.

The pool

The pool is slow, which was an accurate name before it became a funny one. Four 2TB drives in two mirrored pairs, with an NVMe partition as L2ARC:

$ zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH
slow  3.62T  3.35T   286G        -         -    55%    92%  1.00x  ONLINE

Two numbers matter more than the rest: 92% capacity and 55% fragmentation. Hold those.

The drives:

$ lsblk -d -o NAME,SIZE,MODEL,SERIAL,ROTA
NAME   SIZE MODEL                SERIAL     ROTA
sda    1.8T ST2000DM008-2FR102   ZFL62CFK      1
sdb    1.8T ST2000DM008-2FR102   ZFL62DT8      1
...

The ST2000DM008 is a Seagate BarraCuda, and it's SMR. That's not written on the box, it wasn't in the datasheet when these were bought, and Seagate didn't disclose it until it was found out in April 2020.

Why SMR and ZFS interact badly

Shingled recording overlaps the write tracks like roof tiles to get more density. Reads are fine. Writes are fine too, as long as they're sequential. But a small write into the middle of a shingled zone can't be done in place, because overwriting one track corrupts its neighbour. The drive has to read the zone, modify it and write it back.

The drive hides this with a CMR cache region and reorders work in the background, which is why an SMR disk benchmarks perfectly well for thirty seconds. Fill that cache faster than it drains and the drive falls back to doing read-modify-write inline, and service times go from milliseconds to seconds.

ZFS makes that worse in three specific ways, and all three of them get worse as the pool fills:

• Copy-on-write never overwrites in place. Every modified block is a new allocation somewhere else, so a workload that looks like "rewriting a file" looks to the disk like scattered writes plus metadata updates.
• A full, fragmented pool has nowhere tidy to put them. At 92% used and 55% fragmented, the allocator is fitting new blocks into small scattered free extents. On CMR that costs seeks. On SMR it costs whole-zone rewrites.
• Metadata amplifies it. Every write updates block pointers up the tree, and those are small scattered writes by their nature.

None of this is a bug. It's the intended behaviour of all three things meeting.

The measurement that settled it

The reason to trust "seek-bound, not bandwidth-bound" rather than assume it is that they look completely different in iostat, and only one of them is fixable by writing fewer bytes.

During the stall the pool was moving single-digit MB/s while being effectively unusable. Four 7,200rpm drives will do far better than that sequentially, so throughput was not the ceiling. Utilisation was pinned with tiny transfers and long service times, which is what seek starvation looks like.

You can see the same shape at idle. Here's the pool doing nothing much:

$ zpool iostat -v slow 2 2
                    capacity     operations     bandwidth
pool              alloc   free   read  write   read  write
slow              3.35T   286G      0    123      0  2.58M
  mirror-0        1.67T   142G      0     62      0  1.67M
  mirror-1        1.67T   144G      0     61      0   935K

123 write operations for 2.58 MB. That's about 21 KB per operation, against a 128K record size. The pool isn't writing big contiguous records, it's writing small scattered ones, and that's before anything is asking it to work.

The diagnostic, in one line: divide bandwidth by operations. If the answer is a small fraction of your recordsize, the pool is seek-bound and no amount of compression, cache or bigger disks in the same shape will fix it.

Why compression doesn't help

This is the bit worth internalising, because compression is genuinely excellent and it's genuinely irrelevant here.

$ zfs get -o name,value compressratio slow slow/docker-volumes
NAME                 VALUE
slow                 1.03x
slow/docker-volumes  2.43x

The container volumes compress well. So a compressed dataset writes fewer than half the bytes it otherwise would, and the pool still stalls, because compression reduces bytes and the constraint is operations.

A 128K record that compresses to 40K is still one allocation, still one set of metadata updates, still one seek. You've cut the bytes and left the operation count alone. Compression helps sequential bandwidth and it helps capacity. It does nothing whatsoever for seek count.

The same logic disposes of L2ARC. Our cache device holds 75GB and it's earning its keep on reads, but the stall is a write-path problem and a read cache cannot absorb writes.

What actually fixed it

Two changes, one small and one structural.

Turn off atime. With atime=on, every read becomes a metadata write. On a seek-bound pool that is a straightforward doubling of the thing that's hurting:

$ zfs set atime=off slow
$ zfs get -o name,value,source atime slow
NAME  VALUE  SOURCE
slow  off    local

Cheap, instant, and worth doing on any spinning pool that isn't serving a workload that needs access times. relatime is the middle ground if something does.

Move the write-heavy workload off the pool entirely. The real fix wasn't tuning ZFS, it was accepting that a container build workload has no business on shingled disks. Docker's data-root went to XFS on NVMe, and the buildkit state volume with it:

$ docker info | grep 'Root Dir'
 Docker Root Dir: /docker

$ findmnt -no SOURCE,FSTYPE,TARGET /docker
/dev/nvme0n1p5 xfs /docker

Builds are the pathological case for SMR: thousands of small files, heavy metadata churn, layers written and immediately discarded. Every property that makes a build fast is a property that makes a shingled drive slow. The pool went back to what it's good at, which is bulk data written once and read many times.

That's the lesson. There's a lot of ZFS tuning advice out there and some of it helps at the margin, but the pool was doing exactly what those drives are physically able to do. The problem was the workload's address, not the pool's configuration.

The red herring

Worth including because it cost us most of the evening.

There's a fifth drive in heavy, on the same SATA controller, which was failing and had been for a while, throwing command timeouts and ATA errors. It isn't in the pool and never was:

$ zpool status slow | grep -c ZK20BYWD
0

It looked like the obvious culprit and it wasn't. A failing device on a shared SATA controller absolutely can cause an outage, and this one did, because error recovery on one port stalls the controller and everything behind it goes away. But that's a different symptom with a different signature: everything freezes at once, including devices that were idle, and the kernel log fills with ATA resets.

The performance stall had none of that. All four pool members were clean:

$ zpool status slow
  scan: scrub repaired 0B in 03:49:04 with 0 errors on Mon Aug 10 03:53:55 2026
errors: No known data errors

Zero errors, zero repairs, a clean scrub. Two genuine faults in the same box, one of which explained a real outage and neither of which explained the thing we were chasing. The trap is that the moment you find a dying disk you stop looking, and a dying disk is such a satisfying answer that it takes discipline to notice it doesn't fit the symptom.

Buying, if you're replacing

The naming is deliberately unhelpful and it's the single most useful thing to write down:

"Red" and "Red Plus" being different technologies at nearly the same price, with nearly the same name, aimed at the same NAS buyers, is how most people end up here. Check the specific model number against the vendor's own SMR list rather than trusting the product line.

There's no reliable way to ask the drive itself over SATA. smartctl -i won't tell you, and neither will lsblk. Host-managed SMR reports itself through /sys/block/<dev>/queue/zoned, but these are drive-managed, so they lie by design and report none like any conventional disk. Model number, vendor list. That's it.

The limits of this

I haven't tested whether a much less full pool of the same drives behaves acceptably under the same load, and I suspect it would be better without being good. Capacity and fragmentation clearly matter, but I can't tell you where the cliff is, only that we're well past it.

If you're at 92% on shingled drives and things feel fine, they will continue to feel fine right up until something writes hard enough to exhaust the drives' cache. That's the property that makes this bite: it's a latent condition, not a gradual degradation, and the trigger is any unusually heavy write session.

The version of this that costs real money

Ours was one pool on one machine. The expensive version is a fleet purchase, and it goes wrong in three ways at once.

It is not on the spec sheet. SMR frequently is not stated in the model line you buy against, so whoever specifies the order cannot filter for it and whoever debugs it later does not know to ask.

It presents as a software fault. The symptom is stalling under write load, so it gets diagnosed as the filesystem, the application or the hypervisor. Weeks go into the wrong layer, by people who have no reason to suspect the drives.

And the fix is a re-buy. There is no configuration that rescues it. The cost is not engineering time, it is replacing a fleet of drives that were bought on price - by which point the saving that justified them is long gone.