From owner-freebsd-fs@FreeBSD.ORG Thu Apr 25 17:44:28 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A4E068CE; Thu, 25 Apr 2013 17:44:28 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-oa0-f43.google.com (mail-oa0-f43.google.com [209.85.219.43]) by mx1.freebsd.org (Postfix) with ESMTP id 5DE49111B; Thu, 25 Apr 2013 17:44:28 +0000 (UTC) Received: by mail-oa0-f43.google.com with SMTP id k7so3102530oag.2 for ; Thu, 25 Apr 2013 10:44:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=CA+pLfK2K7mVs4zYCpxitEFWLS9UTqkwLdQ/yqbSSAI=; b=ckgW0k5+FEb6wzoPbzFmQ+9cxDbHcc2taEN4pQHG2j3OAFewKF3r0TCDTzjYgTljOC jirWzGJ5+GTL3B6+kpGs8zfFdygrj7ciyTst/xD6a6loB7wBUROWMIwdBmGfXkMnsK8N UVzxrbLVp0Fm52e94XaxfR26c6NJaAH/edAXte8ojyBp37ga75WLMzc1v+POhzcd198a 2Bl2iA6jbfjxLiO6l1jSX3l0Qf2tw3D23RP/cQEiNMv2iZlQG+CRQehbOLPmpDqwISDk Lft1a3k1tYS3dzwSZhVZLZNZVByMMK33mIHDnRgMeRrx5fFp91Fs4ntNAKoAghGzJ8dE LANg== MIME-Version: 1.0 X-Received: by 10.182.148.33 with SMTP id tp1mr15605056obb.83.1366911862588; Thu, 25 Apr 2013 10:44:22 -0700 (PDT) Received: by 10.76.141.15 with HTTP; Thu, 25 Apr 2013 10:44:22 -0700 (PDT) In-Reply-To: References: <51430744.6020004@FreeBSD.org> <51487CE1.5090703@FreeBSD.org> Date: Thu, 25 Apr 2013 10:44:22 -0700 Message-ID: Subject: Re: Strange slowdown when cache devices enabled in ZFS From: Freddie Cash To: Andriy Gapon Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Filesystems X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Apr 2013 17:44:28 -0000 On Thu, Apr 25, 2013 at 8:51 AM, Freddie Cash wrote: > I haven't had a chance to run any of the DTrace scripts on any of my ZFS > systems, but I have narrowed down the issue a bit. > > If I set primarycache=all and secondarycache=all, then adding an L2ARC > device to the pool will lead to zfskern{l2arc_feed_thread} taking up 100% > of one CPU core and stalling I/O to the pool. > > If I set primarycache=all and secondarycache=metadata, then adding an > L2ARC device to the pool speeds things up (zfs send/recv saturates a 1 Gbps > link; and the nightly rsync backups run finishes 4 hours earlier). > > I haven't tested the other two combinations (metadata/metadata; > metadata/all) as yet. > > This is consistent across two ZFS systems so far: > - 8-core Opteron 6100-series CPU with 48 GB of RAM; 44 GB ARC, 40 GB > metadata limit; 3x raidz2 > - 2x 8-core Opteron 6100-series CPU with 128 GB of RAM; 64 GB ARC, 60 GB > metadata limit; 5x raidz2 > > Still reading up on dtrace/hwpmc as time permits. Just wanted to pass > along the above to show I haven't forgotten about this yet. :) $JOB/$LIFE > slows things down sometimes. :) > And, I may have narrowed it down even further. It appears (still testing) that the following sysctl can be toggled to enable/disable the behaviour that leads to the 100% CPU usage and I/O stalls to the pool: vfs.zfs.l2arc_norw The default setting is 1, which (if I'm reading things right) means no reads from the L2ARC while writing data to the L2ARC. Setting this to 0 (which allows reads and writes to occur concurrently to L2ARC?) on a pool that is "stalled" makes things work again. Since these pools all have dedupe enabled, is it possible that the L2ARC feed thread searching the ARC for stuff to write to L2 is starving/blocking reads of the DDT from the L2ARC thus preventing the pool from writing any new data? I've managed to get the hotkernel dtrace script to work on one system. It's sampling with vfs.zfs.l2arc_norw=1 enabled and doing some zfs sends right now. Hopefully it will stall while I'm sampling. :) I'm also in the process of upgrading the biggest box to the latest 9-STABLE from this morning, and enabling DTrace on there. With run hotkernel with l2arc_norw enabled and disabled once the upgrade is complete. -- Freddie Cash fjwcash@gmail.com