From owner-freebsd-stable@FreeBSD.ORG Tue Dec 10 17:36:47 2013 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CF3CCB73 for ; Tue, 10 Dec 2013 17:36:47 +0000 (UTC) Received: from mail-oa0-x231.google.com (mail-oa0-x231.google.com [IPv6:2607:f8b0:4003:c02::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 9490A1FD6 for ; Tue, 10 Dec 2013 17:36:47 +0000 (UTC) Received: by mail-oa0-f49.google.com with SMTP id i4so5896965oah.36 for ; Tue, 10 Dec 2013 09:36:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ETJ/qHz4AeHn0NfOTbyGIZIKvMT13g4SD1/53rvsDQ8=; b=ERIktGjkIXJ1z/Of6fMZijUb9Rl5ZpY+APGe0BOAQg0Fa/n+EeXUfII1kCg58anUGW BBE4X7s8sa0dRMc3vjCzd//sBYXSDsebTzslou6+0ZRsALOft+/9xdpm9oH5c/ITpei0 T6smvonVX7+jBvKHd9DcR2uqFcK/JUHh3nP4XmMZJXbPCf884WiH75WleF6M1jGyKkPB IyJRq2HNJ6lCrgekUCfRkjVpKxdVa77dhQOsynhDKwvc9wVupkh7usXBAXBjYAziNRd+ 0GQ+YkaTe1nCXGBqtWCyZVYJHUQHWChdAgAlu5aQrQHYHUL4PtrKTj19/XpZCHkq/MWV 4qPA== MIME-Version: 1.0 X-Received: by 10.182.49.166 with SMTP id v6mr17644681obn.13.1386697006842; Tue, 10 Dec 2013 09:36:46 -0800 (PST) Received: by 10.183.24.136 with HTTP; Tue, 10 Dec 2013 09:36:46 -0800 (PST) In-Reply-To: References: Date: Tue, 10 Dec 2013 18:36:46 +0100 Message-ID: Subject: Re: Swap on ZFS - bad idea? (swap_pager: indefinite wait buffer) From: Oscar Prieto To: Benjamin Lutz Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Dec 2013 17:36:47 -0000 Personally I still have to read an announcement telling that swap on zfs is relliable. I'm still using a dedicated swap partition on every install due to that kind of behaviour. On Tue, Dec 10, 2013 at 11:41 AM, Benjamin Lutz wrote: > We have storage server that I've found unresponsive this morning. The > console was filled with these messages: > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 17344, size: 61440 > > The lines were all exactly the same, no variation in the numbers. > Ctrl-Alt-Delete had no effect. > > Now, the only thing related to swap that's special about this machine, is > that we use swap on ZFS, because we figured, silent disk corruption is > just as bad when it happens to swapped memory as it is when it happens to > stored data. However, might this be the problem? One thought I was having > was: ZFS hits its memory bounds, a process makes the kernel want to > allocate a swap buffer which causes more ZFS operations which never > complete because ZFS would need to do some housework to free memory, but > the kernel is waiting for swap to become available, some sort of deadlock? > I'm just speculating here, not knowing the internals of this subsystem. > > The machine is a quad core system with 12GiB RAM and a RAIDZ3 consisting > of 12x2TB disks. It's used as a storage system, with accesses mainly > through netatalk (an AFP server, what OS X uses) and a bit of iSCSI. I've > set the following sysctls: > > # > > http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Device_I.2FO_Queue_Size_.28I.2FO_Concurrency.29 > vfs.zfs.vdev.max_pending=20 > # limit kernel size to 10GB to leave some RAM for processes > vm.kmem_size_max="10737418240" > # increase number of mbuf clusters, see igb(4). The default is 25600 > kern.ipc.nmbclusters="102400" > > This sort of hang seems to happen every couple of months. I'd really like > some pointers on how to increase the stability of this system. Maybe there > are some more tunables I need to adjust? Or is swap on ZFS maybe just not > a great idea? > > Cheers, > Benjamin > -- > Benjamin Lutz | Software Engineer | BIOLAB Technology AG > Dufourstr. 80 | CH-8008 Zurich | www.biolab.ch | benjamin.lutz@biolab.ch > PHONE +41 44 295 97 13 | MOBILE +41 79 558 57 13 | FAX +41 44 295 97 19 > > > > This e-mail and the information it contains including attachments are > confidential and meant > only for use by the intended recipient(s); disclosure or copying is > strictly prohibited. If you > are not addressed, but in the possession of this e-mail, please notify the > sender immediately. > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >