Date: Sun, 27 Feb 2011 15:04:50 -0900 From: Royce Williams <royce.williams@gmail.com> To: freebsd-fs@freebsd.org Subject: Re: slow "zfs destroy snapshot" with predictable time pattern Message-ID: <AANLkTimfhBp2Y1kaPyOGtYCfHFn5bAFBCogDb2NCP6F4@mail.gmail.com> In-Reply-To: <AANLkTi=v9riXw85qoyrCBFA0s874jHGKc64ROabDVFgy@mail.gmail.com> References: <AANLkTi=v9riXw85qoyrCBFA0s874jHGKc64ROabDVFgy@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Feb 27, 2011 at 9:30 AM, Royce Williams <royce.williams@gmail.com> wrote: > I thought that 'zfs snapshot destroy' should be fast (on the order of > a few seconds), but mine are taking a predictably long time on a > pretty modest filesystem (details below). > > I discovered this when a typo caused many more snapshots than I > intended (every minute!); I had about 12,000 of them before I noticed. > =A0Destroying the first snapshot took about 39 wallclock seconds on an > otherwise idle system. =A0A few more destroys took almost exactly the > same amount of time. > > I know little about ZFS under the hood, but I wanted to investigate a > little bit. I scripted a loop of 'time zfs destroy snapshot' and let > it run overnight. =A0Each destroy was consistently taking 37-40 seconds, > but then after hundreds of deletions in that time range, I saw a > jagged spike, followed by a consistent drop that has stayed in the > 23-25s range: > > [hours of 38-39s destroys snipped] > > real =A0 =A00m38.205s > real =A0 =A00m38.455s > real =A0 =A00m38.580s > real =A0 =A00m37.414s > real =A0 =A00m35.330s <-- small drop here > real =A0 =A00m35.347s > real =A0 =A00m35.380s > real =A0 =A00m35.355s > real =A0 =A00m35.255s > real =A0 =A00m35.514s > real =A0 =A00m35.422s > real =A0 =A00m35.464s > real =A0 =A00m46.121s =A0 <-- small spike here > real =A0 =A00m44.630s > real =A0 =A00m46.021s > real =A0 =A01m19.443s =A0 <-- big spike here > real =A0 =A00m40.896s > real =A0 =A00m22.848s =A0<-- drop into the 20s range > real =A0 =A00m29.039s > real =A0 =A00m29.831s > real =A0 =A00m26.348s > real =A0 =A00m22.623s > real =A0 =A00m29.314s > real =A0 =A00m29.589s > real =A0 =A00m26.573s > real =A0 =A00m22.773s > > [hours of of 23-25s destroys snipped] > > I know very little about ZFS under the hood, but this model might fit the= facts: > > * Normally, 'zfs destroy snapshot' is fast (on the order of a few seconds= ); > > * 'zfs destroy snapshot' has to briefly analyze all snapshots prior to > destruction; > > * A particular 'problem' snapshot can slow that full analysis by a > consistent amount of time; > > * Destroying that 'problem' snapshot drops the analysis time by that amou= nt. > > > If my model is correct, I'm going to see one or more spikes, followed > by corresponding drops, until the destroys return to a reasonable > rate. > > This guy had a problem that might also fit that model -- that > particular snapshots can be very slow, and removing them removes the > time delay. That thread notes that it was due to a low-memory > condition, and OpenSolaris bug 6542681 was filed for it. =A0I do not > think that my problem is because of low memory. > > =A0 =A0http://www.mail-archive.com/zfs-discuss@opensolaris.org/msg07647.h= tml > > > I have stopped the destroys in case the remaining 'problem' snapshot is u= seful. > > The system is 8.1-SECURITY, amd64, 4GB RAM, no sysctl or loader > tweaks, ZFS v3, zpool v14, single 58GB ZFS pool. > > # zfs list > NAME =A0 =A0 =A0 =A0 =A0 =A0 =A0USED =A0AVAIL =A0REFER =A0MOUNTPOINT > atoz-backup =A0 =A0 =A015.4G =A058.0G =A025.5K =A0/atoz-backup > atoz-backup/usr =A015.3G =A058.0G =A014.8G =A0/atoz-backup/usr > > # df -ki | egrep 'atoz|Filesystem' > Filesystem =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 10= 24-blocks =A0 =A0 Used > Avail Capacity iused =A0 =A0 ifree %iused =A0Mounted on > atoz-backup =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 60789979 =A0 =A0 =A0 25 > 60789953 =A0 =A0 0% =A0 =A0 =A0 6 121579907 =A0 =A00% =A0 /atoz-backup > atoz-backup/usr =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 7= 6281655 15491701 > 60789953 =A0 =A020% =A0714124 121579907 =A0 =A01% =A0 /atoz-backup/usr I left this message open-ended. I also left out the hardware info. What I meant to get at was: * Is ~35 seconds roughly normal for a "zfs destroy snapshot" on a filesystem with 723K files, single disk, under no other I/O or CPU load, with an Intel ICH9 SATA300 talking to a Seagate ST3160812AS (SATA 3)? Googling seems to indicate that it's not, even if I had 12,000 snapshots. * If this speed is not normal, does this smell like a bug? If so, I can help recreate or test, maybe on 8.2 or 9-CURRENT. * If this speed is normal, what search terms would have yielded that fact? (slow OR speed OR problem OR performance) plus "zfs destroy snapshot" didn't yield much that applied. * Is my model valid? I reasoned that if a snapshot is a separate set of references to all files in the filesystem, and the filesystem hasn't changed much over time, then destroying each snapshot should take a relatively constant amount of time. Even if my time expectation was wrong (that it would be quite a bit shorter, even for 723,000 files), I would not expect the destroy time to drop by 50% in the middle of the destroy series ... so I think that I'm missing something. * If my model isn't valid, what is a better model? For example, I had gotten down to just over 10,000 snapshots when I stopped the series of destroys; could there be a snapshot count threshold, under which each destroy now takes less time, in stair-step fashion? In other words: I'm trying to understand what I'm seeing. :-) Also, sorry for the bad subject line - "zfs destroy snapshot" is obviously the correct command; I've updated the subject line. Royce
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTimfhBp2Y1kaPyOGtYCfHFn5bAFBCogDb2NCP6F4>