Date: Tue, 15 Dec 2009 22:43:06 +0100 From: Ivan Voras <ivoras@freebsd.org> To: Wiktor Niesiobedzki <bsd@w.evip.pl> Cc: freebsd-fs <freebsd-fs@freebsd.org>, freebsd-hackers <freebsd-hackers@freebsd.org> Subject: Re: ZFS, compression, system load, pauses (livelocks?) Message-ID: <9bbcef730912151343p729b6d29w97e56bab96f264f9@mail.gmail.com> In-Reply-To: <2ae8edf30912151157t53267adek85af80b1e31fb4b@mail.gmail.com> References: <hg8015$999$1@ger.gmane.org> <2ae8edf30912151157t53267adek85af80b1e31fb4b@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
2009/12/15 Wiktor Niesiobedzki <bsd@w.evip.pl>: > 2009/12/15 Ivan Voras <ivoras@freebsd.org>: >> The context of this post is file servers running FreeBSD 8 and ZFS with >> compressed file systems on low-end hardware, or actually high-end hardwa= re >> on VMWare ESX 3.5 and 4, which kind of makes it low-end as far as storag= e is >> concerned. The servers are standby backup mirrors of production servers = - >> thus many writes, few reads. >> >> Running this setup I notice two things: >> >> 1) load averages get very high, though the only usage these systems get = is >> file system usage: >> 2) long pauses, in what looks like vfs.zfs.txg.timeout second intervals, >> which seemengly block everything, or at least the entire userland. These >> pauses are sometimes so long that file transfers fail, which must be >> avoided. >> >> Looking at the list of processes it looks like a large number of kernel = and >> userland processes are woken up at once. From the kernel side there are >> regularily all g_* threads, but also unrelated threads like bufdaemon, >> softdepflush, etc. and from the userland - top, syslog, cron, etc. It is >> like ZFS livelocks everything else. >> >> Any ideas on the "pauses" issue? >> > > Hi, > > I've a bit striped your post. It's kind of "me too" message (more > details here: http://lists.freebsd.org/pipermail/freebsd-geom/2009-Decemb= er/003810.html). > What I've figured out so far is, that lowering the kernel thread > priority (as pjd@ suggested) gives quite promising results (no > livelocks at all). Though my bottleneck were caused by GELI thread. > > The pattern there is like this: > > sched_prio(curthread, PRIBIO); > [...] > msleep(sc, &sc->sc_queue_mtx, PDROP | PRIBIO, =C2=A0"geli:w", 0); I have tried before reducing priority of ZFS taskqueues but only to PRIBIO, not below it - not much effect wrt "pauses".
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9bbcef730912151343p729b6d29w97e56bab96f264f9>