From owner-freebsd-fs@FreeBSD.ORG Mon Apr 8 13:58:58 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2C420F7E for ; Mon, 8 Apr 2013 13:58:58 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from winston.madpilot.net (winston.madpilot.net [78.47.75.155]) by mx1.freebsd.org (Postfix) with ESMTP id AC302F6E for ; Mon, 8 Apr 2013 13:58:57 +0000 (UTC) Received: from winston.madpilot.net (localhost [127.0.0.1]) by winston.madpilot.net (Postfix) with ESMTP id 3ZktQN4VF9zFTXB; Mon, 8 Apr 2013 15:53:48 +0200 (CEST) X-Virus-Scanned: amavisd-new at madpilot.net Received: from winston.madpilot.net ([127.0.0.1]) by winston.madpilot.net (winston.madpilot.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g9n1kekShwbJ; Mon, 8 Apr 2013 15:53:43 +0200 (CEST) Received: from vwg82.hq.ignesti.it (unknown [77.246.14.1]) by winston.madpilot.net (Postfix) with ESMTPSA; Mon, 8 Apr 2013 15:53:43 +0200 (CEST) Message-ID: <5162CBE8.5050104@madpilot.net> Date: Mon, 08 Apr 2013 15:53:44 +0200 From: Guido Falsi User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130404 Thunderbird/17.0.5 MIME-Version: 1.0 To: Kai Gallasch Subject: Re: FreeBSD 9.1 and swap on zfs References: <9407C6ED-3B4C-4BA2-8B88-F8A998E0A847@free.de> In-Reply-To: <9407C6ED-3B4C-4BA2-8B88-F8A998E0A847@free.de> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2013 13:58:58 -0000 On 04/08/13 15:08, Kai Gallasch wrote: > Hi. > > When running a ZFS on root FreeBSD install.. > > is it for FreeBSD 9.1 (ZFS v28) still not advisable to use a vdev as swapspace? - like: > > # zfs create -V 8G \ > -o org.freebsd:swap=on \ > -o sync=disabled \ > -o primarycache=none \ > -o secondarycache=none rpool/swap > > # swapon /dev/zvol/rpool/swap > > Often voiced fears for swapping on zfs are, that at the moment the server starts to swap, ZFS will start to compete for memory with the short-on-memory server itself (reason for swapping) and the system will lock up shortly after. > > Seems a lot of ZFS-on-root single disk setups make use of an extra swap partition on the root-disk. > > People booting of a mirrored-zpool often seem to have a swap partition on both disks forming the zpool and use those as devices for a gmirror. They then use the gmirror device as swap. > > Which approach is the recommended one? > Swapping to ZFS *or* a swap partition / gmirror on top of two partitions? > I can share my experience, which is not definitive but I hope can help. I have various machines with ZFS on root. some with swap on ZVOL and some with swap on separate partitions (none are mirroring the swap using gmirror though). There is a race condition between ZFS' ARC and the VM system when very low memory conditions arise and this could happen and the machine just starves, I've seen this happen on machines when running buildworld -j without enough ram and also on machines running ports tinderbox or poudriere. This is not happening when using a separate swap partition. In such a case the machine swaps happily and just slows down as naturally expected when swapping a lot. I have noticed that setting the following properties on the ZFS ZVOL can help some: checksum off compression off (it's the default usually) primarycache metadata (maybe none would be even better) secondarycache none sync disabled (in case of a system reset swap data is not valuable anyway) (I'm also not really sure if setting secondarycache has any purpose when primarycache is metadata or none) but this will not completely solve the problem anyway. Also I'm quite sure that tuning ARC not to take all available memory can mitigate the problem too. But the basic race condition remains anyway. Also, this could be just an idea I have no data to corroborate this, it looks to me that ZVOL swap is somewhat slower than a separate partition at recovering swapped data back to ram. But again, I don't know how to properly test this. I have never used mirrored swap because I don't think swap data is anyway valuable, but I understand that having a machine die just because it lost half it's swap can be a problem, so if you need the machine to run rock solid through a disk failure having swap mirrored could be a really good idea. My suggestion is: if you want stability and don't have specific disk layout problems create a separate swap. If you can afford the need to hard reset the machine sometimes when high load sends it in a lockup or really can't make separate partitions, just go for ZVOLS. Just my 2 cents! -- Guido Falsi