Date: Sun, 27 Feb 2011 15:04:50 -0900 From: Royce Williams <royce.williams@gmail.com> To: freebsd-fs@freebsd.org Subject: Re: slow "zfs destroy snapshot" with predictable time pattern Message-ID: <AANLkTimfhBp2Y1kaPyOGtYCfHFn5bAFBCogDb2NCP6F4@mail.gmail.com> In-Reply-To: <AANLkTi=v9riXw85qoyrCBFA0s874jHGKc64ROabDVFgy@mail.gmail.com> References: <AANLkTi=v9riXw85qoyrCBFA0s874jHGKc64ROabDVFgy@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Feb 27, 2011 at 9:30 AM, Royce Williams <royce.williams@gmail.com> wrote: > I thought that 'zfs snapshot destroy' should be fast (on the order of > a few seconds), but mine are taking a predictably long time on a > pretty modest filesystem (details below). > > I discovered this when a typo caused many more snapshots than I > intended (every minute!); I had about 12,000 of them before I noticed. > Destroying the first snapshot took about 39 wallclock seconds on an > otherwise idle system. A few more destroys took almost exactly the > same amount of time. > > I know little about ZFS under the hood, but I wanted to investigate a > little bit. I scripted a loop of 'time zfs destroy snapshot' and let > it run overnight. Each destroy was consistently taking 37-40 seconds, > but then after hundreds of deletions in that time range, I saw a > jagged spike, followed by a consistent drop that has stayed in the > 23-25s range: > > [hours of 38-39s destroys snipped] > > real 0m38.205s > real 0m38.455s > real 0m38.580s > real 0m37.414s > real 0m35.330s <-- small drop here > real 0m35.347s > real 0m35.380s > real 0m35.355s > real 0m35.255s > real 0m35.514s > real 0m35.422s > real 0m35.464s > real 0m46.121s <-- small spike here > real 0m44.630s > real 0m46.021s > real 1m19.443s <-- big spike here > real 0m40.896s > real 0m22.848s <-- drop into the 20s range > real 0m29.039s > real 0m29.831s > real 0m26.348s > real 0m22.623s > real 0m29.314s > real 0m29.589s > real 0m26.573s > real 0m22.773s > > [hours of of 23-25s destroys snipped] > > I know very little about ZFS under the hood, but this model might fit the facts: > > * Normally, 'zfs destroy snapshot' is fast (on the order of a few seconds); > > * 'zfs destroy snapshot' has to briefly analyze all snapshots prior to > destruction; > > * A particular 'problem' snapshot can slow that full analysis by a > consistent amount of time; > > * Destroying that 'problem' snapshot drops the analysis time by that amount. > > > If my model is correct, I'm going to see one or more spikes, followed > by corresponding drops, until the destroys return to a reasonable > rate. > > This guy had a problem that might also fit that model -- that > particular snapshots can be very slow, and removing them removes the > time delay. That thread notes that it was due to a low-memory > condition, and OpenSolaris bug 6542681 was filed for it. I do not > think that my problem is because of low memory. > > http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg07647.html > > > I have stopped the destroys in case the remaining 'problem' snapshot is useful. > > The system is 8.1-SECURITY, amd64, 4GB RAM, no sysctl or loader > tweaks, ZFS v3, zpool v14, single 58GB ZFS pool. > > # zfs list > NAME USED AVAIL REFER MOUNTPOINT > atoz-backup 15.4G 58.0G 25.5K /atoz-backup > atoz-backup/usr 15.3G 58.0G 14.8G /atoz-backup/usr > > # df -ki | egrep 'atoz|Filesystem' > Filesystem 1024-blocks Used > Avail Capacity iused ifree %iused Mounted on > atoz-backup 60789979 25 > 60789953 0% 6 121579907 0% /atoz-backup > atoz-backup/usr 76281655 15491701 > 60789953 20% 714124 121579907 1% /atoz-backup/usr I left this message open-ended. I also left out the hardware info. What I meant to get at was: * Is ~35 seconds roughly normal for a "zfs destroy snapshot" on a filesystem with 723K files, single disk, under no other I/O or CPU load, with an Intel ICH9 SATA300 talking to a Seagate ST3160812AS (SATA 3)? Googling seems to indicate that it's not, even if I had 12,000 snapshots. * If this speed is not normal, does this smell like a bug? If so, I can help recreate or test, maybe on 8.2 or 9-CURRENT. * If this speed is normal, what search terms would have yielded that fact? (slow OR speed OR problem OR performance) plus "zfs destroy snapshot" didn't yield much that applied. * Is my model valid? I reasoned that if a snapshot is a separate set of references to all files in the filesystem, and the filesystem hasn't changed much over time, then destroying each snapshot should take a relatively constant amount of time. Even if my time expectation was wrong (that it would be quite a bit shorter, even for 723,000 files), I would not expect the destroy time to drop by 50% in the middle of the destroy series ... so I think that I'm missing something. * If my model isn't valid, what is a better model? For example, I had gotten down to just over 10,000 snapshots when I stopped the series of destroys; could there be a snapshot count threshold, under which each destroy now takes less time, in stair-step fashion? In other words: I'm trying to understand what I'm seeing. :-) Also, sorry for the bad subject line - "zfs destroy snapshot" is obviously the correct command; I've updated the subject line. Royce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTimfhBp2Y1kaPyOGtYCfHFn5bAFBCogDb2NCP6F4>